ConsistentlyInconsistentYT--Pixeltovoxelprojector/BUILD.md

Build Instructions

Overview

This document provides comprehensive instructions for building and installing the Pixel to Voxel Projector system. The system supports multiple build methods and configurations.

Table of Contents

1. System Requirements
2. Quick Start
3. Detailed Build Methods
4. Docker Installation
5. Troubleshooting
6. Development Setup

---

System Requirements

Minimum Requirements

• OS: Ubuntu 20.04+ (Linux primary), Windows 10/11 with WSL2, macOS 11+
• CPU: Intel Core i7 or AMD Ryzen 7 (8 cores recommended)
• RAM: 16 GB (32 GB recommended for 8K video processing)
• Storage: 50 GB free space
• Python: 3.8 or higher

GPU Requirements (Recommended)

• GPU: NVIDIA RTX 3060 or better
  • RTX 3090/4090 recommended for optimal performance
  • Compute Capability 7.5+ required
• VRAM: 8 GB minimum (24 GB recommended)
• CUDA: 11.x or 12.x
• cuDNN: 8.x
• NVIDIA Driver: 470+ for CUDA 11, 525+ for CUDA 12

Software Dependencies

System Packages (Ubuntu/Debian)

sudo apt-get update
sudo apt-get install -y \
    build-essential \
    cmake \
    ninja-build \
    git \
    pkg-config \
    libomp-dev \
    ffmpeg \
    libavcodec-dev \
    libavformat-dev \
    libavutil-dev \
    libswscale-dev \
    libopencv-dev \
    libgl1-mesa-dev \
    libglu1-mesa-dev \
    protobuf-compiler \
    libprotobuf-dev \
    libzmq3-dev \
    liblz4-dev \
    libzstd-dev

---

Quick Start

Option 1: One-Command Install (Recommended)

# Clone the repository
git clone https://github.com/yourusername/Pixeltovoxelprojector.git
cd Pixeltovoxelprojector

# Install with all dependencies
pip install -e ".[full,dev]"

Option 2: Docker (Easiest)

# Build and run with GPU support
docker build -t pixeltovoxel:latest -f docker/Dockerfile .
docker run --gpus all -it --rm -v $(pwd):/app pixeltovoxel:latest

Option 3: Basic Install (CPU only)

pip install -e .

---

Detailed Build Methods

Method 1: Python setuptools (setup.py)

This is the primary build method using Python's setuptools with custom CUDA compilation.

Step 1: Install Python Dependencies

# Install basic dependencies
pip install -r requirements.txt

# For GPU support (choose based on your CUDA version)
pip install cupy-cuda11x  # For CUDA 11.x
# OR
pip install cupy-cuda12x  # For CUDA 12.x

Step 2: Build Extensions

# Development install (editable)
pip install -e .

# Production install
pip install .

# With optional dependencies
pip install -e ".[full,dev,cuda]"

Step 3: Verify Installation

# Test CUDA extensions
python -c "import voxel_cuda; print('CUDA extensions loaded successfully')"

# Run quick test
python tests/benchmarks/test_installation.py

Build Configuration

The setup.py build supports several environment variables:

# Specify CUDA path
export CUDA_HOME=/usr/local/cuda-12.0

# Set compute architectures
export TORCH_CUDA_ARCH_LIST="8.6;8.9"  # RTX 3090/4090

# Parallel build
export MAX_JOBS=8

# Build with custom flags
CFLAGS="-march=native" pip install -e .

Method 2: CMake Build System

For more control over the build process, use CMake.

Step 1: Configure Build

mkdir build
cd build

# Configure with CMake
cmake .. \
    -GNinja \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_CUDA=ON \
    -DBUILD_PYTHON_BINDINGS=ON \
    -DUSE_OPENMP=ON \
    -DENABLE_FAST_MATH=ON \
    -DCMAKE_CUDA_ARCHITECTURES="86;89"

Step 2: Build

# Build with Ninja (fast)
ninja

# Or with make
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

Step 3: Install

# System-wide install (requires sudo)
sudo ninja install

# Or install to custom prefix
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/.local
ninja install

CMake Build Options

Option	Default	Description
BUILD_CUDA	ON	Build CUDA extensions
BUILD_TESTS	ON	Build test suite
BUILD_BENCHMARKS	ON	Build benchmarks
BUILD_PYTHON_BINDINGS	ON	Build Python bindings
USE_OPENMP	ON	Enable OpenMP
ENABLE_FAST_MATH	ON	Enable fast math
CMAKE_CUDA_ARCHITECTURES	"86;89"	Target GPU architectures

Example configurations:

# Debug build
cmake .. -DCMAKE_BUILD_TYPE=Debug -DBUILD_CUDA=ON

# CPU-only build
cmake .. -DBUILD_CUDA=OFF -DUSE_OPENMP=ON

# Minimal build
cmake .. -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF

# Custom CUDA architectures
cmake .. -DCMAKE_CUDA_ARCHITECTURES="75;80;86;89"

Method 3: Protocol Buffer Compilation

Protocol buffers are automatically compiled during setup, but you can compile them manually:

cd src/protocols

# Compile all .proto files
for proto in *.proto; do
    protoc --python_out=. --grpc_python_out=. -I. "$proto"
done

# Or use the Python grpcio-tools
python -m grpc_tools.protoc \
    -I. \
    --python_out=. \
    --grpc_python_out=. \
    motion_protocol.proto

---

Docker Installation

Prerequisites

• Docker 20.10+
• Docker Compose 2.0+
• NVIDIA Container Toolkit

Install NVIDIA Container Toolkit

# Add NVIDIA package repositories
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
    sudo tee /etc/apt/sources.list.d/nvidia-docker.list

# Install nvidia-container-toolkit
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# Restart Docker
sudo systemctl restart docker

Build Docker Image

# Build image
docker build -t pixeltovoxel:latest -f docker/Dockerfile .

# Build with specific CUDA version
docker build --build-arg CUDA_VERSION=11.8.0 \
    -t pixeltovoxel:cuda11 \
    -f docker/Dockerfile .

Run Container

# Basic run with GPU
docker run --gpus all -it --rm \
    -v $(pwd):/app \
    pixeltovoxel:latest

# With GUI support (X11)
xhost +local:docker
docker run --gpus all -it --rm \
    -e DISPLAY=$DISPLAY \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -v $(pwd):/app \
    pixeltovoxel:latest

# With Jupyter Lab
docker run --gpus all -p 8888:8888 -it --rm \
    -v $(pwd):/app \
    pixeltovoxel:latest \
    jupyter lab --ip=0.0.0.0 --allow-root --no-browser

# Specific GPU selection
docker run --gpus '"device=0"' -it --rm \
    -v $(pwd):/app \
    pixeltovoxel:latest

Docker Compose

# Start all services
docker-compose -f docker/docker-compose.yml up -d

# Start specific service
docker-compose -f docker/docker-compose.yml up pixeltovoxel

# With custom directories
DATA_DIR=/path/to/data OUTPUT_DIR=/path/to/output \
    docker-compose -f docker/docker-compose.yml up

# View logs
docker-compose -f docker/docker-compose.yml logs -f

# Stop all services
docker-compose -f docker/docker-compose.yml down

---

Troubleshooting

CUDA Not Found

Problem: CUDA not found during build

Solutions:

# Set CUDA_HOME environment variable
export CUDA_HOME=/usr/local/cuda-12.0
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH

# Verify CUDA installation
nvcc --version
nvidia-smi

GPU Not Detected

Problem: GPU not accessible in Docker

Solutions:

# Install nvidia-container-toolkit
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

# Check Docker GPU support
docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi

Compilation Errors

Problem: C++ compilation fails

Solutions:

# Update compiler
sudo apt-get install -y build-essential g++-10

# Use specific compiler
export CC=gcc-10
export CXX=g++-10

# Clean and rebuild
rm -rf build/
pip install -e . --force-reinstall --no-cache-dir

Memory Errors

Problem: Out of memory during compilation

Solutions:

# Limit parallel jobs
export MAX_JOBS=4
pip install -e .

# Use swap space
sudo fallocate -l 16G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Protocol Buffer Errors

Problem: protoc command not found or protobuf import errors

Solutions:

# Install protocol buffers
sudo apt-get install -y protobuf-compiler libprotobuf-dev
pip install protobuf grpcio grpcio-tools

# Recompile protocol buffers
cd src/protocols
protoc --python_out=. *.proto

Python Import Errors

Problem: Cannot import compiled extensions

Solutions:

# Check installation
pip show pixeltovoxelprojector

# Set PYTHONPATH
export PYTHONPATH=/path/to/Pixeltovoxelprojector:$PYTHONPATH

# Reinstall in development mode
pip install -e . --force-reinstall

---

Development Setup

Setting Up Development Environment

# Clone repository
git clone https://github.com/yourusername/Pixeltovoxelprojector.git
cd Pixeltovoxelprojector

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install development dependencies
pip install -e ".[dev,full]"

# Install pre-commit hooks (optional)
pip install pre-commit
pre-commit install

Running Tests

# Run all tests
pytest tests/

# Run specific test file
pytest tests/benchmarks/test_installation.py

# Run with coverage
pytest --cov=src tests/

# Run benchmarks
python tests/benchmarks/run_all_benchmarks.py

Code Quality

# Format code
black src/ tests/

# Check style
flake8 src/ tests/

# Type checking
mypy src/

# Run all checks
./scripts/check_code_quality.sh  # If available

Building Documentation

# Install documentation dependencies
pip install sphinx sphinx-rtd-theme

# Build HTML documentation
cd docs
make html

# View documentation
python -m http.server -d _build/html

---

Performance Optimization

CPU Optimization

# Build with native architecture optimization
CFLAGS="-march=native -O3" pip install -e .

# Enable OpenMP
export OMP_NUM_THREADS=16  # Set to your CPU core count

GPU Optimization

# Set compute architecture for your GPU
export TORCH_CUDA_ARCH_LIST="8.6"  # RTX 3090
# or
export TORCH_CUDA_ARCH_LIST="8.9"  # RTX 4090

# Enable TensorFloat-32
export NVIDIA_TF32_OVERRIDE=1

# Optimize CUDA compilation
export CUDA_LAUNCH_BLOCKING=0

Memory Optimization

# Reduce memory usage during build
export MAX_JOBS=4

# Use shared memory for data transfer
export USE_SHM=1

# Set CUDA memory fraction
export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512

---

Next Steps

After successful installation:

1. Verify Installation: Run python tests/benchmarks/test_installation.py
2. Run Examples: Try examples in the examples/ directory
3. Read Documentation: Check src/README.md for usage instructions
4. Run Benchmarks: Execute python tests/benchmarks/run_all_benchmarks.py

---

Additional Resources

• Project README
• API Documentation
• Examples
• Contributing Guidelines
• Issue Tracker

---

Support

For build issues:

1. Check this document first
2. Search existing issues on GitHub
3. Create a new issue with:
   • Your system configuration
   • Complete error messages
   • Build commands used
   • Output of nvidia-smi, nvcc --version, python --version

---

Last Updated: 2025-01-13 Version: 1.0.0