ConsistentlyInconsistentYT--Pixeltovoxelprojector/CI_CD_QUICKSTART.md

CI/CD Pipeline - Quick Start Guide

Overview

This project includes a comprehensive CI/CD pipeline for automated testing, building, and deployment of the PixelToVoxel 8K motion tracking system.

Quick Setup

1. Enable GitHub Actions

The workflows are already configured in .github/workflows/. They will automatically run on:

• Push to main, develop, or claude/** branches
• Pull requests to main or develop
• Manual workflow dispatch

2. Configure Self-Hosted GPU Runners (Optional)

For GPU tests and benchmarks, you'll need self-hosted runners:

# On your GPU machine
mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-x64-2.311.0.tar.gz -L \
  https://github.com/actions/runner/releases/download/v2.311.0/actions-runner-linux-x64-2.311.0.tar.gz
tar xzf actions-runner-linux-x64-2.311.0.tar.gz

# Configure runner (get token from GitHub repo settings)
./config.sh --url https://github.com/YOUR_ORG/YOUR_REPO \
  --token YOUR_TOKEN \
  --labels self-hosted,linux,gpu

# Run as service
sudo ./svc.sh install
sudo ./svc.sh start

3. Configure Secrets (Optional)

For Docker Hub publishing, add these secrets in GitHub repo settings:

• DOCKERHUB_USERNAME: Your Docker Hub username
• DOCKERHUB_TOKEN: Docker Hub access token

Local Development

Build Project

# Clean build with dependencies
./scripts/build.sh --clean --deps --dev

# With CUDA support
./scripts/build.sh --cuda

# Release build
./scripts/build.sh --release --install

Run Tests

# Quick test suite
./scripts/run_tests.sh --quick

# All unit tests
./scripts/run_tests.sh --unit --coverage

# Integration tests
./scripts/run_tests.sh --integration

# Performance benchmarks
./scripts/run_tests.sh --benchmark

# With GPU tests
./scripts/run_tests.sh --all --gpu

# CPU only
./scripts/run_tests.sh --all --cpu-only

Docker Build

# CPU image
docker build --target cpu-runtime -t pixeltovoxel:cpu .

# GPU image
docker build --target gpu-runtime -t pixeltovoxel:gpu \
  --build-arg CUDA_VERSION=12.0.0 .

# Development image
docker build --target development -t pixeltovoxel:dev .

# Run with GPU
docker run --rm -it --gpus all pixeltovoxel:gpu

CI/CD Workflows

Main CI Pipeline (.github/workflows/ci.yml)

Jobs:

1. lint - Code quality checks (Black, Flake8, Pylint)
2. test-cpu - Unit tests on multiple Python versions
3. test-gpu - GPU-accelerated tests with CUDA
4. integration-tests - Full pipeline validation
5. benchmarks - Performance testing with regression detection
6. build - Build verification and package creation
7. security - Vulnerability scanning (Trivy, Bandit)

Coverage Threshold: 80% (configurable in workflow)

Docker Build Pipeline (.github/workflows/docker.yml)

Jobs:

1. build-cpu - CPU-only production image
2. build-gpu - GPU-enabled images (CUDA 12.0, 11.8)
3. build-dev - Development environment
4. test-images - Container functionality tests
5. scan-images - Security vulnerability scanning
6. publish-release - Automated release publishing

Triggered On:

• Push to main/develop
• Version tags (v*.*.*)
• Manual dispatch

Common Tasks

Run Tests Before Committing

# Quick validation
./scripts/run_tests.sh --quick --cpu-only

# Full test suite (if you have GPU)
./scripts/run_tests.sh --all --coverage

Check Code Quality

# Format code
black src/ tests/

# Sort imports
isort src/ tests/

# Check for issues
flake8 src/ tests/
pylint src/

Update Performance Baseline

After intentional performance changes:

# Run benchmarks
./scripts/run_tests.sh --benchmark

# Update baseline
cp tests/benchmarks/benchmark_results/latest.json \
   tests/benchmarks/benchmark_results/baseline.json

# Commit updated baseline
git add tests/benchmarks/benchmark_results/baseline.json
git commit -m "Update performance baseline"

Create a Release

# Tag version
git tag -a v1.0.0 -m "Release version 1.0.0"
git push origin v1.0.0

# CI automatically builds and publishes Docker images

Monitoring CI/CD

Check Workflow Status

# Using GitHub CLI
gh run list
gh run view <run-id>
gh run watch

# Or visit GitHub Actions tab in repository

View Test Results

Test results and artifacts are available in the GitHub Actions UI:

• Code coverage reports
• Benchmark results
• Build artifacts
• Security scan results

Docker Images

Published images are available at:

• GitHub Container Registry: ghcr.io/YOUR_ORG/YOUR_REPO
• Docker Hub: YOUR_ORG/pixeltovoxelprojector (if configured)

Pull images:

# Latest CPU
docker pull ghcr.io/YOUR_ORG/YOUR_REPO:latest-cpu

# Latest GPU
docker pull ghcr.io/YOUR_ORG/YOUR_REPO:latest-gpu

# Specific version
docker pull ghcr.io/YOUR_ORG/YOUR_REPO:v1.0.0-gpu-cuda12.0

Troubleshooting

Tests Failing Locally

# Clean environment
./scripts/build.sh --clean

# Reinstall dependencies
pip install -r requirements.txt
pip install -e .[dev,cuda,full]

# Rebuild extensions
./scripts/build.sh --cuda

# Run tests with verbose output
./scripts/run_tests.sh --unit --verbose

Docker Build Issues

# Clean Docker cache
docker system prune -a -f

# Build with no cache
docker build --no-cache --target cpu-runtime -t test .

# Check build logs
docker build --progress=plain --target cpu-runtime -t test .

CI Failing on GPU Tests

If you don't have GPU runners:

1. Comment out or skip GPU tests in .github/workflows/ci.yml
2. Or set up self-hosted GPU runners (see setup guide above)
3. Or use GitHub-hosted runners with GPU support (if available)

Performance Requirements

Minimum Hardware for GPU Tests

• GPU: NVIDIA GPU with Compute Capability ≥ 7.0
• RAM: 16GB minimum, 32GB recommended
• Storage: 100GB+ for Docker images and test data
• CUDA: Version 11.8 or 12.0

Expected CI Times

• Lint: ~2 minutes
• CPU Tests: ~5-10 minutes per Python version
• GPU Tests: ~10-15 minutes
• Integration Tests: ~15-20 minutes
• Benchmarks: ~20-30 minutes
• Docker Builds: ~10-15 minutes per image

Advanced Configuration

Customize Coverage Threshold

Edit .github/workflows/ci.yml:

env:
  MIN_COVERAGE: '80'  # Change to desired percentage

Adjust Benchmark Regression Threshold

Edit test execution or workflow:

python3 tests/benchmarks/compare_benchmarks.py \
  --threshold 15.0  # Change from default 10.0

Add More Python Versions

Edit .github/workflows/ci.yml:

strategy:
  matrix:
    python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']

Resources

• Full Documentation: See docs/CI_CD_GUIDE.md
• Test Scripts: See scripts/run_tests.sh
• Build Scripts: See scripts/build.sh
• Dockerfile: See Dockerfile
• Workflows: See .github/workflows/

Support

For issues:

1. Check workflow logs in GitHub Actions
2. Review docs/CI_CD_GUIDE.md for detailed troubleshooting
3. Check runner logs for self-hosted runners
4. Open an issue on GitHub

---

Quick Commands Reference:

# Development
./scripts/build.sh --clean --dev --cuda
./scripts/run_tests.sh --quick

# Before commit
black src/ tests/
./scripts/run_tests.sh --unit --coverage

# Docker
docker build --target development -t pixeltovoxel:dev .
docker run --rm -it --gpus all -v $(pwd):/workspace pixeltovoxel:dev

# Release
git tag -a v1.0.0 -m "Release 1.0.0"
git push origin v1.0.0

---

Last Updated: 2025-11-13