ConsistentlyInconsistentYT--Pixeltovoxelprojector/docs/README.md

8K Motion Tracking and Voxel Processing System

System Overview

A high-performance distributed system for real-time motion tracking, target detection, and 3D voxel reconstruction from multiple 8K camera pairs. The system combines thermal and monochrome imaging with GPU-accelerated processing to deliver sub-33ms latency at 30 FPS.

Key Capabilities

• Real-time 8K Processing: Processes 7680x4320 video streams at 30 FPS
• Multi-Modal Fusion: Combines thermal and monochrome imaging for enhanced detection
• Distributed Architecture: Scales across multiple GPU nodes with automatic load balancing
• CUDA Acceleration: GPU-optimized kernels for motion extraction and voxel processing
• Fault Tolerance: Automatic failover and recovery from node/camera failures
• Low Latency: Sub-33ms end-to-end pipeline latency

System Architecture

┌──────────────────────────────────────────────────────────────────┐
│                    Camera Layer (10 Pairs)                       │
│  ┌─────────────┐  ┌─────────────┐         ┌─────────────┐      │
│  │ Mono + Therm│  │ Mono + Therm│   ...   │ Mono + Therm│      │
│  │   Pair 0    │  │   Pair 1    │         │   Pair 9    │      │
│  └─────────────┘  └─────────────┘         └─────────────┘      │
└──────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌──────────────────────────────────────────────────────────────────┐
│                    Video Processing Layer                         │
│  • Hardware-accelerated decode (HEVC, H.264)                     │
│  • Motion extraction (C++ with OpenMP)                           │
│  • Frame synchronization (<10ms time diff)                       │
└──────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌──────────────────────────────────────────────────────────────────┐
│                    Fusion Layer                                   │
│  • Image registration and alignment                              │
│  • Multi-spectral detection                                      │
│  • False positive reduction                                      │
│  • Target tracking                                               │
└──────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌──────────────────────────────────────────────────────────────────┐
│                    Distributed Processing Layer                   │
│  • Task scheduling and load balancing                            │
│  • Multi-GPU coordination                                        │
│  • Fault tolerance and failover                                  │
└──────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌──────────────────────────────────────────────────────────────────┐
│                    Voxel Reconstruction Layer                     │
│  • Sparse voxel grid (CUDA accelerated)                          │
│  • 3D motion projection                                          │
│  • Spatial tracking                                              │
└──────────────────────────────────────────────────────────────────┘

---

Quick Start Guide

Prerequisites

• Hardware:

  • NVIDIA GPU with CUDA 11.0+ (RTX 3090/4090 recommended)
  • 32GB+ RAM
  • 10GbE network adapter (for distributed deployment)

• Software:

  • Ubuntu 20.04+ or compatible Linux distribution
  • CUDA Toolkit 11.0+
  • Python 3.8+
  • GCC 9.0+ with C++17 support

Installation

1. Clone the Repository

git clone <repository-url>
cd Pixeltovoxelprojector

2. Install System Dependencies

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install -y \
    build-essential \
    cmake \
    libopencv-dev \
    libprotobuf-dev \
    protobuf-compiler \
    python3-dev \
    python3-pip

# CUDA Toolkit (if not installed)
# Download from https://developer.nvidia.com/cuda-downloads

3. Install Python Dependencies

pip install -r requirements.txt

# Camera system dependencies
pip install -r requirements_camera.txt

4. Build C++ Extensions

python setup.py build_ext --inplace

This will compile:

• Motion extractor (C++ with OpenMP)
• Sparse voxel grid
• Stream manager
• Fusion engine
• CUDA extensions (if CUDA available)

5. Verify Installation

python verify_tracking_system.py

Expected output:

================================================================================
Camera Tracking System - Verification
================================================================================

1. Checking implementation files...
✓ Pose Tracker: /home/user/Pixeltovoxelprojector/src/camera/pose_tracker.py
✓ Orientation Manager: /home/user/Pixeltovoxelprojector/src/camera/orientation_manager.cpp
...
✓ ALL CHECKS PASSED!

Running Your First Example

Example 1: Single Stream Motion Extraction

cd src
python example_8k_pipeline.py --example 1

Example 2: Dual Camera Fusion

from src.fusion import FusionManager, FusionConfig

# Create fusion manager
config = FusionConfig(
    target_fps=30,
    enable_cuda=True,
    enable_false_positive_reduction=True
)
fusion_mgr = FusionManager(config)

# Add camera pair
pair_id = fusion_mgr.add_camera_pair(
    thermal_id=0,
    mono_id=1,
    baseline_m=0.5
)

# Start processing
fusion_mgr.start(num_workers=4)

# Process frame pairs
detections = fusion_mgr.process_frame_pair(
    pair_id, thermal_frame, mono_frame, timestamp
)

Example 3: Distributed Processing

# On master node
python examples/distributed_processing_example.py --master

# On worker nodes
python examples/distributed_processing_example.py --worker --master-ip 192.168.1.100

---

Installation Instructions

Detailed Installation

Option 1: Standard Installation

# Install package and dependencies
pip install -e .

# Run tests
python -m pytest tests/

Option 2: Development Installation

# Create virtual environment
python -m venv venv
source venv/bin/activate  # Linux/Mac
# or
venv\Scripts\activate  # Windows

# Install development dependencies
pip install -e ".[dev]"

# Install pre-commit hooks
pre-commit install

Option 3: Docker Installation

# Build Docker image
docker build -t 8k-motion-tracking .

# Run container
docker run --gpus all -it 8k-motion-tracking

Building C++ Extensions Manually

# Motion extractor
g++ -O3 -shared -std=c++17 -fopenmp -fPIC \
    $(python -m pybind11 --includes) \
    src/motion_extractor.cpp \
    -o motion_extractor_cpp$(python3-config --extension-suffix)

# CUDA extensions (requires CUDA toolkit)
nvcc -O3 --std=c++17 -Xcompiler -fPIC \
    -gencode arch=compute_86,code=sm_86 \
    cuda/voxel_cuda.cu -c -o voxel_cuda.o

g++ -shared voxel_cuda.o cuda/voxel_cuda_wrapper.cpp \
    -L/usr/local/cuda/lib64 -lcudart \
    -o voxel_cuda$(python3-config --extension-suffix)

---

Configuration Guide

Camera Configuration

Configure camera pairs in camera_config.json:

{
  "num_pairs": 10,
  "cameras": {
    "0": {
      "camera_id": 0,
      "pair_id": 0,
      "camera_type": "MONO",
      "connection": "GIGE_VISION",
      "ip_address": "192.168.1.10",
      "width": 7680,
      "height": 4320,
      "frame_rate": 30.0,
      "exposure_time": 10000.0,
      "gain": 1.0,
      "trigger_mode": "Hardware",
      "position": [0.0, 0.0, 0.0],
      "orientation": [[1,0,0], [0,1,0], [0,0,1]]
    },
    "1": {
      "camera_id": 1,
      "pair_id": 0,
      "camera_type": "THERMAL",
      "connection": "GIGE_VISION",
      "ip_address": "192.168.1.11",
      "width": 7680,
      "height": 4320,
      "frame_rate": 30.0
    }
  },
  "pairs": {
    "0": {
      "pair_id": 0,
      "mono_camera_id": 0,
      "thermal_camera_id": 1,
      "stereo_baseline": 0.5
    }
  }
}

Network Configuration

Configure cluster nodes in your application:

from src.network import ClusterConfig

cluster = ClusterConfig(
    node_id="node1",
    discovery_port=9999,
    data_port_range=(10000, 11000),
    heartbeat_interval=1.0,
    heartbeat_timeout=5.0,
    enable_rdma=True,
    rdma_device="mlx5_0"
)

# Start cluster
cluster.start(is_master=True)

Fusion Configuration

from src.fusion import FusionConfig

config = FusionConfig(
    target_fps=30,
    registration_update_interval_s=1.0,
    thermal_threshold=0.3,
    mono_threshold=0.2,
    confidence_threshold=0.6,
    max_range_km=5.0,
    enable_thermal_enhancement=True,
    enable_false_positive_reduction=True,
    enable_cuda=True,
    thermal_palette="iron"
)

Performance Tuning

Edit configuration parameters for your hardware:

# For RTX 4090
CUDA_CONFIG = {
    'compute_capability': '89',
    'max_threads_per_block': 1024,
    'shared_memory_per_block': 49152,
    'registers_per_thread': 128
}

# For high-throughput processing
PIPELINE_CONFIG = {
    'buffer_size': 60,  # frames
    'num_decoder_threads': 4,
    'num_processing_threads': 8,
    'enable_hardware_accel': True,
    'codec': 'hevc_cuvid'  # NVIDIA hardware decoder
}

---

API Reference

Core Classes

VideoProcessor

Main class for 8K video processing:

from src.video_processor import VideoProcessor, VideoStream

processor = VideoProcessor(
    use_hardware_accel=True,
    target_fps=30.0,
    enable_profiling=True
)

# Add video stream
processor.add_stream('camera1', VideoStream(
    path='/path/to/video.mp4',
    stream_type='monochrome',
    codec=VideoCodec.HEVC
))

# Register callback
def on_motion(motion_data):
    print(f"Detected {len(motion_data.coordinates)} objects")

processor.register_motion_callback(on_motion)

# Start processing
processor.start_processing()

FusionManager

Multi-modal sensor fusion:

from src.fusion import FusionManager, FusionConfig

fusion_mgr = FusionManager(FusionConfig())
fusion_mgr.start(num_workers=4)

# Process frames
detections = fusion_mgr.process_frame_pair(
    pair_id=0,
    thermal_image=thermal_frame,
    mono_image=mono_frame,
    timestamp=time.time()
)

# Get metrics
metrics = fusion_mgr.get_performance_metrics()
print(f"FPS: {metrics['avg_fps']:.2f}")
print(f"False positive reduction: {metrics['false_positive_reduction_rate']:.2%}")

DistributedProcessor

Cluster processing coordination:

from src.network import DistributedProcessor, ClusterConfig, DataPipeline

cluster = ClusterConfig()
pipeline = DataPipeline(num_cameras=10)
processor = DistributedProcessor(cluster, pipeline, num_cameras=10)

# Register task handler
def process_frame(task):
    # Process frame
    return result

processor.register_task_handler('process_frame', process_frame)

# Start distributed processing
processor.start()

# Submit tasks
task_id = processor.submit_camera_frame(camera_id, frame, metadata)
result = processor.wait_for_task(task_id, timeout=1.0)

CameraManager

Camera system management:

from src.camera import CameraManager, CameraConfiguration, CameraType

mgr = CameraManager(num_pairs=10)

# Add cameras
for i in range(20):
    config = CameraConfiguration(
        camera_id=i,
        pair_id=i // 2,
        camera_type=CameraType.MONO if i % 2 == 0 else CameraType.THERMAL,
        connection=ConnectionType.GIGE_VISION,
        ip_address=f"192.168.1.{10+i}"
    )
    mgr.add_camera(config)

# Initialize and start
mgr.initialize_all_cameras()
mgr.start_all_acquisition()
mgr.start_health_monitoring()

# Check health
health = mgr.get_system_health()
print(f"Cameras streaming: {health['streaming']}/{health['total_cameras']}")

---

Performance Benchmarks

See docs/PERFORMANCE.md for detailed benchmarks.

Quick Reference

Component	Throughput	Latency	GPU Usage
8K HEVC Decode (HW)	60+ FPS	5-8 ms	15%
Motion Extraction	35+ FPS	12-18 ms	40%
Fusion Processing	30+ FPS	8-12 ms	25%
Voxel Reconstruction	30+ FPS	5-8 ms	30%
End-to-End Pipeline	30 FPS	<33 ms	65%

Tested on: NVIDIA RTX 4090, Intel i9-13900K, 64GB RAM

---

Troubleshooting

Common Issues

1. CUDA Not Found

# Check CUDA installation
nvcc --version

# Set CUDA_HOME environment variable
export CUDA_HOME=/usr/local/cuda
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH

2. Low FPS / High Latency

• Enable hardware acceleration: use_hardware_accel=True
• Check GPU utilization: nvidia-smi
• Reduce buffer size to decrease latency
• Increase number of processing threads

3. Camera Connection Failures

# Check network connectivity
ping <camera-ip>

# Check GigE Vision devices
arv-tool-0.8 -l

# Increase network buffer size
sudo sysctl -w net.core.rmem_max=26214400
sudo sysctl -w net.core.rmem_default=26214400

4. Build Failures

# Install missing dependencies
pip install pybind11 numpy

# Clean and rebuild
python setup.py clean --all
python setup.py build_ext --inplace

Getting Help

• Check docs/DEPLOYMENT.md for deployment issues
• Check docs/API.md for API usage examples
• Review logs in /var/log/motion_tracking/
• Enable debug logging: logging.basicConfig(level=logging.DEBUG)

---

System Requirements

Minimum Requirements

• CPU: Intel Core i7 or AMD Ryzen 7
• RAM: 32GB
• GPU: NVIDIA RTX 3060 (12GB VRAM)
• Storage: 500GB SSD
• Network: 1GbE

Recommended Requirements

• CPU: Intel Core i9-13900K or AMD Ryzen 9 7950X
• RAM: 64GB DDR5
• GPU: NVIDIA RTX 4090 (24GB VRAM)
• Storage: 2TB NVMe SSD
• Network: 10GbE or InfiniBand

Multi-Node Cluster

• 4+ GPU nodes
• 10GbE or InfiniBand interconnect
• Sub-5ms inter-node latency
• Shared storage (NFS/Lustre) for calibration data

---

License

See LICENSE file for details.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Authors

See AUTHORS file for contributor list.

References

• Architecture Documentation
• API Reference
• Deployment Guide
• Performance Guide