ConsistentlyInconsistentYT--Pixeltovoxelprojector/tests/benchmarks/QUICKSTART.md

Benchmark Suite Quick Start Guide

Installation

1. Install Dependencies

cd /home/user/Pixeltovoxelprojector/tests/benchmarks
pip install -r requirements.txt

2. Verify Installation

python test_installation.py

This will check all dependencies and show what's available.

3. (Optional) Compile CUDA Benchmarks

If you have an NVIDIA GPU and CUDA toolkit installed:

make

This will compile voxel_benchmark for GPU performance testing.

Running Benchmarks

Quick Test (Fast)

For a quick performance check during development:

python quick_benchmark.py

Duration: ~30 seconds Tests: 2 core benchmarks with reduced iterations

Individual Suites

Run specific benchmark suites:

# Main benchmark suite
python benchmark_suite.py

# Camera pipeline benchmarks
python camera_benchmark.py

# Network benchmarks
python network_benchmark.py

# CUDA voxel benchmarks (if compiled)
./voxel_benchmark

Complete Suite (Recommended)

Run all benchmarks and generate comprehensive reports:

python run_all_benchmarks.py

Duration: 5-15 minutes Output: Complete performance analysis with graphs and reports

Understanding Results

Output Files

All results are saved to benchmark_results/:

benchmark_results/
├── results_YYYYMMDD_HHMMSS.json     # Raw benchmark data
├── results_YYYYMMDD_HHMMSS.csv      # CSV format
├── report_YYYYMMDD_HHMMSS.html      # HTML report with graphs
├── throughput_comparison.png         # Throughput bar chart
├── latency_distribution.png          # Latency percentiles
├── resource_utilization.png          # CPU/GPU/Memory usage
├── camera/                           # Camera benchmark results
│   └── camera_benchmark_*.json
├── network/                          # Network benchmark results
│   └── network_benchmark_*.json
├── combined_results_*.json           # All suites combined
└── summary_*.txt                     # Text summary

Key Metrics

Throughput (FPS)

• Higher is better
• Target: >30 FPS for real-time
• Indicates how many frames/operations per second

Latency (ms)

• Lower is better
• p50: Median latency
• p95: 95th percentile
• p99: 99th percentile (worst case)
• Target p99: <33ms for 30 FPS

GPU Utilization (%)

• 70-95% is optimal
• <50% may indicate CPU bottleneck

• 98% may indicate saturation

Memory Bandwidth (GB/s)

• Modern GPUs: 300-900 GB/s theoretical
• 60-80% of theoretical is good

Performance Baselines

Create Your Baseline

After first run:

python benchmark_suite.py
# When prompted: "Save these results as performance baseline? (y/n):"
# Type: y

This creates baselines.json for regression detection.

Check for Regressions

Future runs will automatically compare against baseline:

WARNING: Performance regressions detected:
  - Throughput regression: 28.50 < 35.00 FPS
  - Latency regression: 38.20 > 35.00 ms

Reset Baselines

To update baselines after optimization:

# Delete old baselines
rm benchmark_results/baselines.json

# Run benchmarks and save new baseline
python benchmark_suite.py
# Type: y when prompted

Common Workflows

Development Testing

Quick check after code changes:

python quick_benchmark.py

Pre-Commit Testing

Before committing performance-critical changes:

python benchmark_suite.py
# Check for regressions in output

Release Testing

Full performance validation before release:

python run_all_benchmarks.py
# Review HTML report
# Compare against previous releases

Hardware Comparison

Testing on different hardware:

# Machine A
python run_all_benchmarks.py
cp benchmark_results/combined_results_*.json results_machine_a.json

# Machine B
python run_all_benchmarks.py
cp benchmark_results/combined_results_*.json results_machine_b.json

# Compare results
python compare_results.py results_machine_a.json results_machine_b.json

Troubleshooting

ImportError: No module named 'X'

pip install -r requirements.txt

CUDA benchmarks fail

1. Check GPU is visible:

   nvidia-smi
   

2. Check CUDA toolkit:

   nvcc --version
   

3. Recompile:

   make clean
   make
   

Benchmarks are slow

This is normal! Benchmarks run many iterations for accuracy:

• Quick benchmark: ~30 seconds
• Full suite: 5-15 minutes

For faster testing, reduce iterations in the code:

suite.run_benchmark(..., iterations=10, warmup=2)  # Instead of 100/10

Memory errors with large grids

Reduce grid size in benchmark calls:

benchmark_voxel_ray_casting(grid_size=256)  # Instead of 500

Network benchmarks show errors

Network benchmarks use localhost (127.0.0.1) by default. For realistic results, test between separate machines:

benchmark.benchmark_tcp_throughput(host="192.168.1.100")

Advanced Usage

Custom Benchmarks

Add your own benchmark function:

def my_benchmark(param1, param2):
    # Your code to benchmark
    pass

# Run it
from benchmark_suite import BenchmarkSuite
suite = BenchmarkSuite()
suite.run_benchmark(
    "My Custom Test",
    my_benchmark,
    iterations=100,
    param1=value1,
    param2=value2
)

Adjust Test Parameters

Edit the benchmark scripts:

# In benchmark_suite.py
suite.run_benchmark(
    "Voxel Ray Casting",
    benchmark_voxel_ray_casting,
    iterations=200,      # More iterations = more accurate
    warmup=20,           # More warmup = more stable
    grid_size=1000,      # Larger grid = more realistic
    num_rays=5000        # More rays = stress test
)

Export for CI/CD

# Run and save results
python benchmark_suite.py

# Extract key metrics for CI
python -c "
import json
with open('benchmark_results/results_*.json') as f:
    data = json.load(f)
    for result in data:
        if result['throughput_fps'] < 30:
            exit(1)  # Fail CI
"

Best Practices

1. Consistent Environment

   • Close other applications
   • Ensure adequate cooling
   • Run on AC power (laptops)
   • Disable CPU throttling

2. Baseline Management

   • Create baseline on clean system
   • Update after major optimizations
   • Document hardware configuration
   • Version control baselines

3. Interpretation

   • Look at trends, not absolute values
   • Compare similar hardware only
   • Account for thermal throttling
   • Check multiple runs for consistency

4. Optimization Workflow

   • Baseline before changes
   • Make incremental changes
   • Benchmark after each change
   • Document what worked

Getting Help

• Check README.md for detailed documentation
• Run python test_installation.py to verify setup
• Review example output in documentation
• Check hardware compatibility

Next Steps

After running benchmarks:

1. Review HTML report for visual analysis
2. Identify bottlenecks (CPU, GPU, memory, network)
3. Optimize critical paths
4. Re-run and compare
5. Save new baseline if improvements validated

Happy benchmarking!