ConsistentlyInconsistentYT--Pixeltovoxelprojector/NETWORK_QUICKSTART.md

Network Infrastructure Quick Start Guide

Installation

1. Install Dependencies

# Navigate to project directory
cd /home/user/Pixeltovoxelprojector

# Install core dependencies
pip install -r src/network/requirements.txt

# Optional: Install RDMA support (for InfiniBand)
# pip install pyverbs

# Optional: Install advanced shared memory
# pip install posix_ipc

2. Verify Installation

# Run simple test
python3 -c "from src.network import ClusterConfig, DataPipeline, DistributedProcessor; print('OK')"

---

Quick Start: Single Node

Basic Example

from src.network import ClusterConfig, DataPipeline, DistributedProcessor
import numpy as np
import time

# 1. Initialize cluster (single node)
cluster = ClusterConfig()
cluster.start(is_master=True)
time.sleep(1)

# 2. Create data pipeline
pipeline = DataPipeline(
    buffer_capacity=32,
    frame_shape=(1080, 1920, 3),  # HD resolution
    enable_rdma=False,
    enable_shared_memory=True
)

# 3. Initialize processor
processor = DistributedProcessor(
    cluster_config=cluster,
    data_pipeline=pipeline,
    num_cameras=2
)

# 4. Register task handler
def my_task_handler(task):
    frame = task.input_data['frame']
    # Process frame here
    result = np.mean(frame)
    return {'average': result}

processor.register_task_handler('process_frame', my_task_handler)

# 5. Start processing
processor.start()
time.sleep(1)

# 6. Submit a frame
frame = np.random.rand(1080, 1920, 3).astype(np.float32)
from src.network import FrameMetadata

metadata = FrameMetadata(
    frame_id=0,
    camera_id=0,
    timestamp=time.time(),
    width=1920,
    height=1080,
    channels=3,
    dtype='float32',
    compressed=False,
    checksum='',
    sequence_number=0
)

task_id = processor.submit_camera_frame(0, frame, metadata)

# 7. Wait for result
result = processor.wait_for_task(task_id, timeout=5.0)
print(f"Result: {result}")

# 8. Cleanup
processor.stop()
cluster.stop()
pipeline.cleanup()

---

Quick Start: Multi-Node Cluster

On Each Node

Master Node (run first):

from src.network import ClusterConfig
import time

cluster = ClusterConfig(
    discovery_port=9999,
    enable_rdma=True  # Set False if no InfiniBand
)

cluster.start(is_master=True)

# Keep running
try:
    while True:
        time.sleep(1)
        status = cluster.get_cluster_status()
        print(f"Nodes: {status['online_nodes']}, GPUs: {status['total_gpus']}")
except KeyboardInterrupt:
    cluster.stop()

Worker Nodes (run on other machines):

from src.network import ClusterConfig
import time

cluster = ClusterConfig(
    discovery_port=9999,
    enable_rdma=True
)

cluster.start(is_master=False)

# Keep running
try:
    while True:
        time.sleep(10)
except KeyboardInterrupt:
    cluster.stop()

Run Distributed Processing

On master node:

from src.network import ClusterConfig, DataPipeline, DistributedProcessor
import time

# Initialize (master node)
cluster = ClusterConfig(enable_rdma=True)
cluster.start(is_master=True)
time.sleep(3)  # Wait for node discovery

# Create pipeline
pipeline = DataPipeline(
    buffer_capacity=64,
    frame_shape=(2160, 3840, 3),  # 8K
    enable_rdma=True,
    enable_shared_memory=True,
    shm_size_mb=2048
)

# Create processor
processor = DistributedProcessor(
    cluster_config=cluster,
    data_pipeline=pipeline,
    num_cameras=10,
    enable_fault_tolerance=True
)

# Register handler and start
def process_voxel_frame(task):
    # Your processing logic here
    return {'status': 'ok'}

processor.register_task_handler('process_frame', process_voxel_frame)
processor.start()
time.sleep(2)

# Allocate cameras
allocation = cluster.allocate_cameras(10)
print(f"Camera allocation: {allocation}")

# Get system health
health = processor.get_system_health()
print(f"System health: {health['status']}")
print(f"Active workers: {health['active_workers']}")

# Submit frames...
# (see full example in examples/distributed_processing_example.py)

---

Running Examples

Full Distributed Processing Demo

python3 examples/distributed_processing_example.py

Output:

• Cluster initialization
• Node discovery
• Camera allocation
• Task processing
• Performance statistics

Network Benchmark

python3 examples/benchmark_network.py

Tests:

• Ring buffer latency
• Data pipeline throughput
• Task scheduling overhead
• End-to-end latency

---

Configuration Options

ClusterConfig

Parameter	Default	Description
discovery_port	9999	UDP port for node discovery
heartbeat_interval	1.0	Seconds between heartbeats
heartbeat_timeout	5.0	Timeout before node offline
enable_rdma	True	Enable InfiniBand RDMA

DataPipeline

Parameter	Default	Description
buffer_capacity	64	Frames per ring buffer
frame_shape	(1080,1920,3)	Frame dimensions
enable_rdma	True	Use RDMA for transfers
enable_shared_memory	True	Use shared memory IPC
shm_size_mb	1024	Shared memory size (MB)

DistributedProcessor

Parameter	Default	Description
num_cameras	10	Number of camera pairs
enable_fault_tolerance	True	Auto failover on failure

---

Monitoring

Get Real-Time Statistics

# Cluster status
cluster_status = cluster.get_cluster_status()
print(f"Online nodes: {cluster_status['online_nodes']}")
print(f"Total GPUs: {cluster_status['total_gpus']}")

# Processing statistics
stats = processor.get_statistics()
print(f"Tasks completed: {stats['tasks_completed']}")
print(f"Success rate: {stats['success_rate']*100:.1f}%")
print(f"Avg execution time: {stats['avg_execution_time']*1000:.2f}ms")

# Pipeline statistics
pipeline_stats = stats['pipeline']
print(f"Frames processed: {pipeline_stats['frames_processed']}")
print(f"Throughput: {pipeline_stats['bytes_transferred']/1e9:.2f} GB")

# System health
health = processor.get_system_health()
print(f"Status: {health['status']}")
print(f"Avg latency: {health['avg_latency_ms']:.2f}ms")

---

Network Configuration

InfiniBand Setup

1. Verify InfiniBand devices:

ibstat
ibv_devices

2. Check connectivity:

# On node 1
ib_send_lat

# On node 2
ib_send_lat <node1_ip>

3. Expected latency: <1 μs

10GbE Setup

1. Enable jumbo frames:

sudo ip link set eth0 mtu 9000

2. Verify:

ip link show eth0 | grep mtu

3. Test bandwidth:

# On receiver
iperf3 -s

# On sender
iperf3 -c <receiver_ip> -t 10

4. Expected throughput: 9+ Gbps

---

Troubleshooting

Issue: Nodes not discovering each other

Solution:

# Check firewall
sudo ufw allow 9999/udp

# Check network connectivity
ping <other_node_ip>

# Verify broadcast is enabled
sudo sysctl net.ipv4.icmp_echo_ignore_broadcasts=0

Issue: RDMA not available

Solution:

# Disable RDMA
cluster = ClusterConfig(enable_rdma=False)
pipeline = DataPipeline(enable_rdma=False)

Issue: GPU not detected

Solution:

# Check NVIDIA driver
nvidia-smi

# Install pynvml
pip install pynvml

# Verify CUDA
python3 -c "import pynvml; pynvml.nvmlInit(); print('OK')"

Issue: High latency (>5ms)

Solutions:

• Enable jumbo frames (MTU 9000)
• Check network utilization: iftop -i eth0
• Optimize topology: cluster.optimize_network_topology()
• Reduce CPU usage on nodes

Issue: Tasks failing

Solutions:

# Check error logs
stats = processor.get_statistics()
print(f"Failed tasks: {stats['tasks_failed']}")

# Review specific task
task = processor.task_registry.get(task_id)
if task:
    print(f"Error: {task.error}")

# Increase timeout
result = processor.wait_for_task(task_id, timeout=60.0)

---

Performance Tuning

For Maximum Throughput

# Larger buffers
pipeline = DataPipeline(
    buffer_capacity=128,  # Increased from 64
    frame_shape=(2160, 3840, 3)
)

# More workers per GPU
# (automatically scales with available GPUs)

For Minimum Latency

# Smaller buffers (reduces queueing delay)
pipeline = DataPipeline(
    buffer_capacity=16,
    frame_shape=(2160, 3840, 3)
)

# Enable RDMA
cluster = ClusterConfig(enable_rdma=True)
pipeline = DataPipeline(enable_rdma=True)

# High priority tasks
task.priority = 10  # Higher = processed first

For Reliability

# Enable all fault tolerance features
processor = DistributedProcessor(
    cluster_config=cluster,
    data_pipeline=pipeline,
    num_cameras=10,
    enable_fault_tolerance=True  # Must be True
)

# Increase retries
task.max_retries = 5  # Default is 3

# Shorter heartbeat interval
cluster = ClusterConfig(
    heartbeat_interval=0.5,  # More frequent checks
    heartbeat_timeout=3.0     # Faster failure detection
)

---

Best Practices

1. Always start master node first, wait 2-3 seconds before starting workers
2. Enable RDMA for 10+ cameras to achieve target latency
3. Monitor system health using get_system_health() every few seconds
4. Set appropriate timeouts based on expected task duration
5. Test failover before production deployment
6. Log all events for debugging and analysis
7. Profile regularly using built-in statistics
8. Reserve compute headroom (20-30%) for load spikes

---

Next Steps

1. Read full architecture documentation: DISTRIBUTED_ARCHITECTURE.md
2. Review example code: examples/distributed_processing_example.py
3. Run benchmarks: examples/benchmark_network.py
4. Customize task handlers for your workload
5. Deploy to production cluster
6. Set up monitoring and alerting

---

Additional Resources

• Architecture Details: /home/user/Pixeltovoxelprojector/DISTRIBUTED_ARCHITECTURE.md
• Example Code: /home/user/Pixeltovoxelprojector/examples/
• API Documentation: Inline code comments in /home/user/Pixeltovoxelprojector/src/network/

---

Need Help?

• Check inline code documentation
• Review examples directory
• See troubleshooting section above
• Examine debug logs (set logging.level=DEBUG)