PM2 Deployment Guide

This guide covers PM2-based deployment for Hyperscape, including secrets management, database mode detection, and streaming configuration.

Overview

Hyperscape uses PM2 for production process management on Vast.ai and other GPU servers. The deployment system includes:

Automatic secrets loading from /tmp/hyperscape-secrets.env
Database mode auto-detection (local vs remote)
Xvfb virtual display for WebGPU streaming
Chrome Beta for stable streaming capture
Multi-platform RTMP streaming with auto-detection

Quick Start

Deploy to Vast.ai

# On Vast.ai instance
cd /root/hyperscape
bash scripts/deploy-vast.sh

This script:

Pulls latest code from origin/main
Installs system dependencies (Chrome Beta, FFmpeg, Xvfb)
Loads secrets from /tmp/hyperscape-secrets.env
Auto-detects database mode from DATABASE_URL
Builds core packages
Applies database migrations
Starts Xvfb virtual display
Launches duel stack via PM2

PM2 Commands

# View process status
bunx pm2 status

# View logs
bunx pm2 logs hyperscape-duel

# Restart stack
bunx pm2 restart hyperscape-duel

# Stop stack
bunx pm2 stop hyperscape-duel

# Delete from PM2
bunx pm2 delete hyperscape-duel

Secrets Management

Secrets File Format

PM2 loads secrets from /tmp/hyperscape-secrets.env at config load time:

# /tmp/hyperscape-secrets.env
DATABASE_URL=postgresql://user:pass@host:5432/db
ELIZAOS_CLOUD_API_KEY=your-elizacloud-key
TWITCH_STREAM_KEY=live_123456789_abcdefghij
KICK_STREAM_KEY=your-kick-key
DUEL_ARENA_ORACLE_EVM_PRIVATE_KEY=0x...
DUEL_ARENA_ORACLE_SOLANA_AUTHORITY_SECRET=base64:...

Why Direct File Loading?

Problem: bunx pm2 doesn’t reliably inherit exported environment variables from the deploy shell script. Solution: ecosystem.config.cjs reads the secrets file directly at config load time:

const fs = require("fs");
const SECRETS_FILES = [
  "/tmp/hyperscape-secrets.env",
  require("path").join(__dirname, ".env.production"),
];

for (const secretsPath of SECRETS_FILES) {
  if (fs.existsSync(secretsPath)) {
    const lines = fs.readFileSync(secretsPath, "utf-8").split("\n");
    for (const line of lines) {
      // Parse and load into process.env
    }
  }
}

Creating Secrets File

GitHub Actions (.github/workflows/deploy-vast.yml):

- name: Create secrets file
  run: |
    cat > /tmp/hyperscape-secrets.env << 'EOF'
    DATABASE_URL=${{ secrets.DATABASE_URL }}
    ELIZAOS_CLOUD_API_KEY=${{ secrets.ELIZAOS_CLOUD_API_KEY }}
    TWITCH_STREAM_KEY=${{ secrets.TWITCH_STREAM_KEY }}
    TWITCH_RTMP_STREAM_KEY=${{ secrets.TWITCH_STREAM_KEY }}
    KICK_STREAM_KEY=${{ secrets.KICK_STREAM_KEY }}
    EOF

Manual Deployment:

# SSH into server
ssh root@your-vast-instance

# Create secrets file
cat > /tmp/hyperscape-secrets.env << 'EOF'
DATABASE_URL=postgresql://...
ELIZAOS_CLOUD_API_KEY=...
TWITCH_STREAM_KEY=...
EOF

# Deploy
cd /root/hyperscape
bash scripts/deploy-vast.sh

Database Mode Auto-Detection

How It Works

The deployment system automatically detects whether to use local or remote PostgreSQL:

// ecosystem.config.cjs
if (!process.env.DUEL_DATABASE_MODE && process.env.DATABASE_URL) {
  const dbHost = new URL(process.env.DATABASE_URL).hostname;
  const isLocal = ["localhost", "127.0.0.1", "0.0.0.0", "::1"].includes(dbHost);
  process.env.DUEL_DATABASE_MODE = isLocal ? "local" : "remote";
}

Local Mode

Triggers:

DATABASE_URL contains localhost, 127.0.0.1, 0.0.0.0, or ::1
OR DUEL_DATABASE_MODE=local explicitly set

Behavior:

Starts local PostgreSQL via pg_ctlcluster
Creates database and user if needed
Sets USE_LOCAL_POSTGRES=true

Remote Mode

Triggers:

DATABASE_URL contains any other hostname
OR DUEL_DATABASE_MODE=remote explicitly set

Behavior:

Uses external PostgreSQL (Neon, Railway, etc.)
Skips local PostgreSQL setup
Sets USE_LOCAL_POSTGRES=false

Manual Override

# Force remote mode even with localhost URL
export DUEL_DATABASE_MODE=remote

# Force local mode even with remote URL
export DUEL_DATABASE_MODE=local

PostgreSQL Connection Pool

Configuration

// ecosystem.config.cjs
env: {
  POSTGRES_POOL_MAX: "20",  // Increased from 10 (March 2026)
  POSTGRES_POOL_MIN: "2",
}

Why 20 Connections?

Concurrent Agents: Up to 10 AI agents querying database simultaneously
Bank Queries: Each agent can make 5 concurrent bank queries
Server Queries: Game server needs connections for player data
Headroom: Extra capacity for spikes and migrations

Tuning

If you see “timeout exceeded when trying to connect” errors:

# Increase pool size
export POSTGRES_POOL_MAX=30

# Increase timeout
export POSTGRES_POOL_TIMEOUT_MS=90000

Xvfb Virtual Display

Why Xvfb?

WebGPU requires a window context, even on headless servers. Xvfb provides a virtual X11 display.

Startup Order

Critical: Xvfb must start BEFORE PM2:

# deploy-vast.sh
echo "[deploy] Starting Xvfb virtual display..."
pkill -f "Xvfb :99" || true
sleep 1
Xvfb :99 -screen 0 1280x720x24 &
export DISPLAY=:99
echo "[deploy] Xvfb started on DISPLAY=$DISPLAY"

# Then start PM2
bunx pm2 start ecosystem.config.cjs --update-env

PM2 Environment

ecosystem.config.cjs explicitly forwards DISPLAY:

env: {
  DISPLAY: process.env.DISPLAY || ":99",
  // ... other vars
}

Troubleshooting

Error: cannot open display Solution:

# Check if Xvfb is running
ps aux | grep Xvfb

# Restart Xvfb
pkill -f "Xvfb :99"
Xvfb :99 -screen 0 1280x720x24 &

# Verify DISPLAY
echo $DISPLAY  # Should output :99

# Restart PM2
bunx pm2 restart hyperscape-duel

Chrome Beta Streaming

Why Chrome Beta?

Stability: More stable than Dev/Canary channels
WebGPU Support: Full WebGPU support with ANGLE backend
Compatibility: Better driver compatibility than native Vulkan

Installation

# deploy-vast.sh
wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | apt-key add -
echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list
apt-get update && apt-get install -y google-chrome-beta

Configuration

// ecosystem.config.cjs
env: {
  STREAM_CAPTURE_CHANNEL: "chrome-beta",
  STREAM_CAPTURE_ANGLE: "default",  // Auto-select best backend
  STREAM_CAPTURE_WIDTH: "1280",
  STREAM_CAPTURE_HEIGHT: "720",
}

ANGLE Backend Selection

Default (Recommended):

Automatically selects best backend for the system
Tries Vulkan → OpenGL → D3D11 in order
Most compatible across different GPUs

Vulkan (Legacy):

Native Vulkan backend
Can crash on incompatible drivers
Not recommended for production

RTMP Streaming

Auto-Detection

Stream destinations are auto-detected from available keys:

# deploy-vast.sh
DESTS=""
if [ -n "${TWITCH_STREAM_KEY:-${TWITCH_RTMP_STREAM_KEY:-}}" ]; then
    DESTS="twitch"
fi
if [ -n "${KICK_STREAM_KEY:-}" ]; then
    DESTS="${DESTS:+${DESTS},}kick"
fi
export STREAM_ENABLED_DESTINATIONS="$DESTS"

Supported Platforms

Platform	Key Variable	URL Variable
Twitch	`TWITCH_STREAM_KEY` or `TWITCH_RTMP_STREAM_KEY`	`TWITCH_STREAM_URL` (default: `rtmp://live.twitch.tv/app`)
Kick	`KICK_STREAM_KEY`	`KICK_RTMP_URL` (default: `rtmps://fa723fc1b171.global-contribute.live-video.net/app`)
YouTube	`YOUTUBE_STREAM_KEY` or `YOUTUBE_RTMP_STREAM_KEY`	`YOUTUBE_STREAM_URL` (default: `rtmp://a.rtmp.youtube.com/live2`)

Manual Configuration

# Override auto-detection
export STREAM_ENABLED_DESTINATIONS=twitch,kick,youtube

# Or disable streaming
export STREAM_ENABLED_DESTINATIONS=

Health Checks

Local Services

The deploy script waits for services to become healthy:

# Server health
curl -fsS http://127.0.0.1:5555/health

# Streaming state
curl -fsS http://127.0.0.1:5555/api/streaming/state

# CDN health (if using local CDN)
curl -fsS http://127.0.0.1:8080/health

PM2 Status

# Check process status
bunx pm2 status

# Expected output:
# ┌─────┬──────────────────┬─────────┬─────────┬─────────┬──────────┐
# │ id  │ name             │ mode    │ ↺       │ status  │ cpu      │
# ├─────┼──────────────────┼─────────┼─────────┼─────────┼──────────┤
# │ 0   │ hyperscape-duel  │ fork    │ 0       │ online  │ 45%      │
# └─────┴──────────────────┴─────────┴─────────┴─────────┴──────────┘

Port Proxying

Vast.ai requires port proxying for external access:

# deploy-vast.sh
# Game server: internal 5555 -> external 35143
nohup socat TCP-LISTEN:35143,reuseaddr,fork TCP:127.0.0.1:5555 > /dev/null 2>&1 &

# WebSocket: internal 5555 -> external 35079
nohup socat TCP-LISTEN:35079,reuseaddr,fork TCP:127.0.0.1:5555 > /dev/null 2>&1 &

# CDN: internal 8080 -> external 35144
nohup socat TCP-LISTEN:35144,reuseaddr,fork TCP:127.0.0.1:8080 > /dev/null 2>&1 &

Logs

PM2 Logs

# Tail all logs
bunx pm2 logs hyperscape-duel

# Tail error logs only
bunx pm2 logs hyperscape-duel --err

# Tail output logs only
bunx pm2 logs hyperscape-duel --out

# View last 200 lines
bunx pm2 logs hyperscape-duel --lines 200

Log Files

# Error log
tail -f logs/duel-error.log

# Output log
tail -f logs/duel-out.log

Troubleshooting

Server Crashes on Startup

Check logs:

bunx pm2 logs hyperscape-duel --lines 500
tail -n 200 logs/duel-error.log

Common causes:

Missing DATABASE_URL → Check /tmp/hyperscape-secrets.env
Database connection timeout → Increase POSTGRES_POOL_MAX
WebGPU initialization failed → Check GPU driver and Xvfb

Streaming Not Starting

Check streaming state:

curl http://127.0.0.1:5555/api/streaming/state

Common causes:

Missing stream keys → Check TWITCH_STREAM_KEY, KICK_STREAM_KEY
Xvfb not running → ps aux | grep Xvfb
Chrome Beta not installed → google-chrome-beta --version
DISPLAY not set → echo $DISPLAY (should be :99)

Database Connection Errors

Error: timeout exceeded when trying to connect Solutions:

Increase connection pool:
```
export POSTGRES_POOL_MAX=30
```
Check database is accessible:
```
psql "$DATABASE_URL" -c "SELECT 1"
```
Verify connection string:
```
echo $DATABASE_URL
```

PM2 Not Forwarding Environment Variables

Problem: Environment variables set in shell are not available to PM2 processes. Solution: Use /tmp/hyperscape-secrets.env instead of shell exports:

# ❌ Don't rely on shell exports
export DATABASE_URL=postgresql://...
bunx pm2 start ecosystem.config.cjs

# ✅ Use secrets file
cat > /tmp/hyperscape-secrets.env << 'EOF'
DATABASE_URL=postgresql://...
EOF
bunx pm2 start ecosystem.config.cjs

Advanced Configuration

Custom Database Mode

# Force remote mode
export DUEL_DATABASE_MODE=remote

# Force local mode
export DUEL_DATABASE_MODE=local

Custom Streaming Configuration

# Use different Chrome channel
export STREAM_CAPTURE_CHANNEL=chrome-dev

# Use specific ANGLE backend
export STREAM_CAPTURE_ANGLE=vulkan  # or opengl, d3d11

# Custom resolution
export STREAM_CAPTURE_WIDTH=1920
export STREAM_CAPTURE_HEIGHT=1080

Custom Xvfb Display

# Use different display number
export DISPLAY=:100

# Start Xvfb on custom display
Xvfb :100 -screen 0 1920x1080x24 &

Monitoring

Process Health

# Check if process is running
bunx pm2 status | grep hyperscape-duel

# Check uptime
bunx pm2 show hyperscape-duel | grep uptime

# Check memory usage
bunx pm2 show hyperscape-duel | grep memory

Service Health

# Server health
curl http://127.0.0.1:5555/health

# Streaming state
curl http://127.0.0.1:5555/api/streaming/state

# Duel context
curl http://127.0.0.1:5555/api/streaming/duel-context

Auto-Restart Configuration

// ecosystem.config.cjs
{
  autorestart: true,
  max_restarts: 999999,
  min_uptime: "10s",
  restart_delay: 10000,
  exp_backoff_restart_delay: 2000,
  max_memory_restart: "4G",
}

Deployment Checklist

ecosystem.config.cjs - PM2 configuration
scripts/deploy-vast.sh - Deployment script
.github/workflows/deploy-vast.yml - CI/CD workflow
packages/server/.env.example - Environment variable documentation
docs/duel-stack.md - Duel stack documentation

Configuration

Operations

​PM2 Deployment Guide

​Overview

​Quick Start

​Deploy to Vast.ai

​PM2 Commands

​Secrets Management

​Secrets File Format

​Why Direct File Loading?

​Creating Secrets File

​Database Mode Auto-Detection

​How It Works

​Local Mode

​Remote Mode

​Manual Override

​PostgreSQL Connection Pool

​Configuration

​Why 20 Connections?

​Tuning

​Xvfb Virtual Display

​Why Xvfb?

​Startup Order

​PM2 Environment

​Troubleshooting

​Chrome Beta Streaming

​Why Chrome Beta?

​Installation

​Configuration

​ANGLE Backend Selection

​RTMP Streaming

​Auto-Detection

​Supported Platforms

​Manual Configuration

​Health Checks

​Local Services

​PM2 Status

​Port Proxying

​Logs

​PM2 Logs

​Log Files

​Troubleshooting

​Server Crashes on Startup

​Streaming Not Starting

​Database Connection Errors

​PM2 Not Forwarding Environment Variables

​Advanced Configuration

​Custom Database Mode

​Custom Streaming Configuration

​Custom Xvfb Display

​Monitoring

​Process Health

​Service Health

​Auto-Restart Configuration

​Deployment Checklist

​Related Files

PM2 Deployment Guide

Overview

Quick Start

Deploy to Vast.ai

PM2 Commands

Secrets Management

Secrets File Format

Why Direct File Loading?

Creating Secrets File

Database Mode Auto-Detection

How It Works

Local Mode

Remote Mode

Manual Override

PostgreSQL Connection Pool

Configuration

Why 20 Connections?

Tuning

Xvfb Virtual Display

Why Xvfb?

Startup Order

PM2 Environment

Troubleshooting

Chrome Beta Streaming

Why Chrome Beta?

Installation

Configuration

ANGLE Backend Selection

RTMP Streaming

Auto-Detection

Supported Platforms

Manual Configuration

Health Checks

Local Services

PM2 Status

Port Proxying

Logs

PM2 Logs

Log Files

Troubleshooting

Server Crashes on Startup

Streaming Not Starting

Database Connection Errors

PM2 Not Forwarding Environment Variables

Advanced Configuration

Custom Database Mode

Custom Streaming Configuration

Custom Xvfb Display

Monitoring

Process Health

Service Health

Auto-Restart Configuration

Deployment Checklist

Related Files