Deployment - Hyperscape

Deployment Overview

Hyperscape uses a split deployment model:

Server: Railway, Fly.io, or Docker host
Client: Vercel, Netlify, or static hosting
Database: PostgreSQL (Neon recommended)
Assets: CDN or object storage

Environment Variables

Server Production

# Required
DATABASE_URL=postgresql://...
JWT_SECRET=your-secret-key
PRIVY_APP_ID=your-app-id
PRIVY_APP_SECRET=your-app-secret
ADMIN_CODE=your-admin-code  # Required for admin API endpoints (maintenance mode, logs, restart)

# Optional
PORT=5555
PUBLIC_CDN_URL=https://cdn.example.com
LIVEKIT_API_KEY=...
LIVEKIT_API_SECRET=...
ORACLE_SETTLEMENT_DELAY_MS=7000  # Delay oracle publish to sync with stream (default: 7000ms)

Client Production

PUBLIC_PRIVY_APP_ID=your-app-id
PUBLIC_API_URL=https://api.hyperscape.lol
PUBLIC_WS_URL=wss://api.hyperscape.lol
PUBLIC_CDN_URL=https://cdn.hyperscape.lol

Production Domains

Hyperscape supports multiple production domains with CORS configuration (added in commits bb292c1, 7ff88d1): Game Domains:

hyperscape.gg - Primary game domain (added Feb 2026)
play.hyperscape.club - Alternative game domain

Betting Domains:

hyperscape.bet - Betting platform (added Feb 2026)
hyperbet.win - Additional betting domain (added Feb 2026)

CORS Configuration: The server and betting keeper automatically allow these domains:

// From packages/server/src/startup/http-server.ts
const ALLOWED_ORIGINS = [
  'https://hyperscape.gg',
  'https://hyperscape.bet',
  'https://hyperbet.win',
  'https://play.hyperscape.club'
];

// CORS middleware configuration
fastify.register(cors, {
  origin: (origin, callback) => {
    if (!origin || ALLOWED_ORIGINS.includes(origin)) {
      callback(null, true);
    } else {
      callback(new Error('Not allowed by CORS'));
    }
  },
  credentials: true,
  maxAge: 86400  // 24 hours
});

Subdomain Pattern Support: The betting keeper supports subdomain patterns for flexible deployment:

// From packages/gold-betting-demo/keeper/src/service.ts
const ALLOWED_ORIGIN_PATTERNS = [
  /^https:\/\/.*\.hyperscape\.bet$/,
  /^https:\/\/.*\.hyperbet\.win$/
];

Tauri Mobile Deep Links: Mobile apps support deep linking from production domains:

// packages/app/src-tauri/tauri.conf.json
{
  "identifier": "com.hyperscape.app",
  "deepLinkProtocols": ["hyperscape"],
  "associatedDomains": [
    "hyperscape.gg",
    "play.hyperscape.club"
  ]
}

Website Game Link: The marketing website now links to the primary game domain:

// From packages/website/src/lib/links.ts
export const GAME_URL = 'https://hyperscape.gg';

PUBLIC_PRIVY_APP_ID must match between client and server.

Railway Deployment

Create Railway project

Connect your GitHub repository to Railway.

Configure build

Set build command: bun run build Set start command: bun start

Add PostgreSQL

Add PostgreSQL service from Railway marketplace.

Set environment variables

Add all required server environment variables.

Cloudflare Pages Deployment

The client automatically deploys to Cloudflare Pages on push to main via GitHub Actions (added commit 37c3629, Feb 26 2026).

Automated Deployment

The .github/workflows/deploy-pages.yml workflow triggers on:

Pushes to main branch
Changes to packages/client/** or packages/shared/** (shared contains packet definitions)
Changes to package.json or bun.lockb
Manual workflow dispatch

The client depends on packages/shared for packet definitions. When packets change on the server, the client must rebuild to stay in sync.

Deployment Process:

# Build client with production environment variables
- name: Build client (includes shared + physx dependencies via turbo)
  run: bun run build:client
  env:
    NODE_OPTIONS: '--max-old-space-size=4096'
    PUBLIC_PRIVY_APP_ID: ${{ secrets.PUBLIC_PRIVY_APP_ID }}
    PUBLIC_API_URL: https://hyperscape-production.up.railway.app
    PUBLIC_WS_URL: wss://hyperscape-production.up.railway.app/ws
    PUBLIC_CDN_URL: https://assets.hyperscape.club
    PUBLIC_APP_URL: https://hyperscape.gg

# Deploy to Cloudflare Pages
- name: Deploy to Cloudflare Pages
  run: |
    # Extract first line of commit message (avoid multi-line issues)
    COMMIT_MSG=$(echo "${{ github.event.head_commit.message }}" | head -1 | tr -d '"' | cut -c1-100)
    npx wrangler pages deploy dist \
      --project-name=hyperscape \
      --branch=${{ github.ref_name }} \
      --commit-hash=${{ github.sha }} \
      --commit-message="$COMMIT_MSG" \
      --commit-dirty=true
  env:
    CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}

Multi-Line Commit Message Fix (commit 3e4bb48): Wrangler fails on multi-line commit messages. The workflow now extracts only the first line:

COMMIT_MSG=$(echo "${{ github.event.head_commit.message }}" | head -1 | tr -d '"' | cut -c1-100)

Production URLs:

Primary: https://hyperscape.gg
Alternative: https://hyperscape.club
Preview: https://<commit-sha>.hyperscape.pages.dev

Required GitHub Secrets

Secret	Purpose
`CLOUDFLARE_API_TOKEN`	Cloudflare API token with Pages write access
`PUBLIC_PRIVY_APP_ID`	Privy app ID (must match server)

Cloudflare R2 CORS Configuration

Assets are served from Cloudflare R2 with CORS enabled for cross-origin loading:

# Configure R2 CORS (run once)
bash scripts/configure-r2-cors.sh

CORS Configuration:

{
  "allowed": {
    "origins": ["*"],
    "methods": ["GET", "HEAD"],
    "headers": ["*"]
  },
  "exposed": ["ETag"],
  "maxAge": 3600
}

Why This Format: The wrangler API requires nested allowed.origins/methods/headers structure (not flat allowedOrigins). The old format caused wrangler r2 bucket cors set to fail (commit 055779a). Benefits:

Allows assets.hyperscape.club to serve to all domains
Supports hyperscape.gg, hyperscape.club, and preview URLs
Enables cross-origin asset loading for Cloudflare Pages → R2

Manual Deployment

Deploy manually using wrangler:

cd packages/client
bun run build
npx wrangler pages deploy dist --project-name=hyperscape

Vercel Client Deployment

Import project

Import packages/client directory to Vercel.

Configure build

Root directory: packages/client Build command: bun run build Output directory: dist

Set environment variables

Add PUBLIC_* environment variables.

Database Setup

Neon PostgreSQL

Create database at neon.tech
Copy connection string
Set as DATABASE_URL in server environment

Migrations

cd packages/server
bunx drizzle-kit push      # Apply schema
bunx drizzle-kit migrate   # Run migrations

CDN Setup

Option 1: Self-Hosted

Use the included Docker CDN:

bun run cdn:up

Configure PUBLIC_CDN_URL to point to your CDN host.

Option 2: Cloud Storage

Upload assets to S3, R2, or similar:

Build assets: bun run assets:optimize
Upload packages/server/world/assets/
Set PUBLIC_CDN_URL to bucket URL

Vast.ai GPU Deployment

Hyperscape deploys to Vast.ai for GPU-accelerated streaming with automated CI/CD via GitHub Actions.

Automated Instance Provisioning (NEW)

The scripts/vast-provision.sh script automatically finds and rents GPU instances with display driver support:

Install Vast.ai CLI

pip install vastai
vastai set api-key YOUR_API_KEY

Run Provisioner

./scripts/vast-provision.sh

The script will:

Search for instances with gpu_display_active=true (REQUIRED for WebGPU)
Filter by reliability (≥95%), GPU RAM (≥20GB), price (≤$2/hr)
Show top 5 available instances
Automatically rent the best instance
Wait for instance to be ready
Output SSH connection details

Update GitHub Secrets

gh secret set VAST_HOST --body '<ssh-host>'
gh secret set VAST_PORT --body '<ssh-port>'

Trigger Deployment

gh workflow run deploy-vast.yml

CRITICAL: Only rent instances with gpu_display_active=true. Compute-only GPUs cannot run WebGPU streaming.

Configuration Options: Edit scripts/vast-provision.sh to customize search criteria:

MIN_GPU_RAM=20          # GB - RTX 4090 has 24GB
MIN_RELIABILITY=0.95    # 95% uptime
MAX_PRICE_PER_HOUR=2.0  # USD per hour
PREFERRED_GPUS="RTX_4090,RTX_3090,RTX_A6000,A100"
DISK_SPACE=100          # GB minimum

Output:

═══════════════════════════════════════════════════════════════════
Instance provisioned successfully!
═══════════════════════════════════════════════════════════════════

Instance Details:
  Instance ID: 12345678
  GPU: RTX 4090 (24 GB)
  Display Driver: ENABLED ✓
  SSH Host: ssh.vast.ai
  SSH Port: 35022
  Public IP: 1.2.3.4

SSH Connection:
  ssh -p 35022 root@ssh.vast.ai

Update GitHub Secrets:
  VAST_HOST=ssh.vast.ai
  VAST_PORT=35022

Automated Deployment

The .github/workflows/deploy-vast.yml workflow automatically deploys to Vast.ai on push to main:

# Triggers on successful CI completion or manual dispatch
on:
  workflow_run:
    workflows: ["CI"]
    types: [completed]
    branches: [main]
  workflow_dispatch:  # NEW: Manual deployment trigger (commit b1f41d5)

Manual Deployment (commit b1f41d5): You can now trigger Vast.ai deployments manually from GitHub Actions UI:

Go to Actions tab in GitHub
Select “Deploy to Vast.ai” workflow
Click “Run workflow”
Select branch (usually main)
Click “Run workflow”

This is useful for:

Deploying hotfixes without waiting for CI
Re-deploying after Vast.ai instance restart
Testing deployment process

Deployment Process:

Write Secrets to /tmp - Saves secrets to /tmp/hyperscape-secrets.env before git operations (commit 684b203)
Enter Maintenance Mode - Pauses new duel cycles, waits for active markets to resolve
SSH Deploy - Connects to Vast.ai instance, pulls latest code, builds, and restarts
Auto-Detect Configuration - Database mode, stream destinations, GPU rendering mode
Start Xvfb - Virtual display started before PM2 (commit 294a36c)
PM2 Restart - Reads secrets from /tmp, auto-detects database mode (commits 684b203, 3df4370)
Exit Maintenance Mode - Resumes duel cycles after health check passes

Graceful Restart API (Zero-Downtime Deployments)

The server provides a graceful restart API for zero-downtime deployments during active duels: Request Graceful Restart:

POST /admin/graceful-restart
Headers:
  x-admin-code: <ADMIN_CODE>

Response:
  {
    "success": true,
    "message": "Graceful restart requested",
    "duelActive": true,
    "willRestartAfterDuel": true
  }

Check Restart Status:

GET /admin/restart-status
Headers:
  x-admin-code: <ADMIN_CODE>

Response:
  {
    "restartPending": true,
    "duelActive": true,
    "currentPhase": "FIGHTING"
  }

Behavior:

If no duel active: restarts immediately via SIGTERM
If duel in progress: waits until RESOLUTION phase completes
PM2 automatically restarts the server with new code
No interruption to active duels or streams

Use Cases:

Deploy hotfixes during active streaming
Update server code without stopping duels
Restart after configuration changes

Maintenance Mode API (March 2026)

The server provides a comprehensive maintenance mode system for zero-downtime deployments: Enter Maintenance Mode:

POST /admin/maintenance/enter
Headers:
  x-admin-code: <ADMIN_CODE>
  Content-Type: application/json
Body:
  {
    "reason": "deployment",
    "timeoutMs": 300000  # 5 minutes
  }

Response:
  {
    "success": true,
    "status": {
      "active": true,
      "safeToDeploy": true,
      "currentPhase": "IDLE",
      "marketStatus": "resolved",
      "pendingMarkets": 0
    }
  }

Exit Maintenance Mode:

POST /admin/maintenance/exit
Headers:
  x-admin-code: <ADMIN_CODE>

Response:
  {
    "success": true,
    "status": {
      "active": false,
      "safeToDeploy": true
    }
  }

Check Status:

GET /admin/maintenance/status
Headers:
  x-admin-code: <ADMIN_CODE>

Response:
  {
    "active": false,
    "enteredAt": null,
    "reason": null,
    "safeToDeploy": true,
    "currentPhase": "FIGHTING",
    "marketStatus": "betting",
    "pendingMarkets": 1
  }

Get Live Logs:

GET /admin/logs
Headers:
  x-admin-code: <ADMIN_CODE>

Response:
  {
    "logs": [
      {
        "timestamp": 1710187234567,
        "level": "INFO",
        "system": "DuelScheduler",
        "message": "Duel started",
        "data": { "duelId": "duel-123" }
      }
    ]
  }

Restart Server:

POST /admin/restart
Headers:
  x-admin-code: <ADMIN_CODE>

Response:
  {
    "success": true,
    "message": "Restarting server in 2 seconds..."
  }

The restart endpoint calls process.exit(0) after a 2-second delay. Ensure you have a process manager (PM2) configured to automatically restart the server.

Manual Maintenance Mode: Helper scripts are available for manual control:

# Enter maintenance mode
bash scripts/pre-deploy-maintenance.sh

# Exit maintenance mode
bash scripts/post-deploy-resume.sh

Client-Side Maintenance Banner: The client automatically displays a maintenance banner when the server enters maintenance mode:

Polls /health endpoint every 5 seconds
Displays red warning banner when maintenanceMode: true
Banner appears across all screens (game, admin, leaderboard, streaming)
Message: “SERVER MAINTENANCE IMMINENT - GAME WILL PAUSE AFTER CURRENT DUEL”

Maintenance Mode Behavior:

Prevents new duel cycles from starting
Waits for active duels to complete
Pauses betting markets
Sets safeToDeploy: true when safe to restart
Resumes automatically on exit or timeout

Required GitHub Secrets

Configure these in repository settings → Secrets → Actions:

Secret	Purpose
`VAST_HOST`	Vast.ai instance IP address
`VAST_PORT`	SSH port (usually 35022)
`VAST_SSH_KEY`	Private SSH key for instance access
`DATABASE_URL`	PostgreSQL connection string
`SOLANA_DEPLOYER_PRIVATE_KEY`	Base58 Solana keypair for market operations
`TWITCH_STREAM_KEY`	Twitch stream key
`X_STREAM_KEY`	X/Twitter stream key
`X_RTMP_URL`	X/Twitter RTMP URL
`KICK_STREAM_KEY`	Kick stream key
`KICK_RTMP_URL`	Kick RTMP URL
`ADMIN_CODE`	Admin code for maintenance mode API
`VAST_SERVER_URL`	Public server URL (e.g., https://hyperscape.gg)

Deployment Script Improvements (March 2026)

The scripts/deploy-vast.sh script has been significantly enhanced with recent improvements: MediaRecorder Streaming Mode (Commits 72c667a, 7284882):

Switched from CDP screencast to MediaRecorder mode for streaming capture
Uses canvas.captureStream() → WebSocket → FFmpeg pipeline
More reliable under Xvfb + WebGPU on Vast instances
Requires internalCapture=1 URL parameter for canvas capture bridge
Eliminates stream freezing and stalling issues

PM2 Secrets Loading (Commits 684b203, 3df4370):

Writes secrets to /tmp/hyperscape-secrets.env before git operations
ecosystem.config.cjs reads secrets file directly at config load time
Auto-detects DUEL_DATABASE_MODE from DATABASE_URL hostname
Prevents sanitizeRuntimeEnv() from stripping DATABASE_URL in remote mode
Ensures secrets persist through git reset operations

Chrome Beta for Linux WebGPU Support (Commit 154f0b6, March 13, 2026):

Reverted from Chrome Canary back to Chrome Beta (google-chrome-beta) for Linux NVIDIA
Chrome Canary was experiencing instability issues on Linux NVIDIA GPUs
Chrome Beta provides better stability for production streaming
Uses Vulkan ANGLE backend (--use-angle=vulkan) for optimal performance

Xvfb Display Setup (Commits 704b955, 294a36c):

Starts Xvfb before PM2 to ensure virtual display is available
Exports DISPLAY=:99 to environment
ecosystem.config.cjs explicitly sets DISPLAY=:99 in PM2 environment
Prevents “cannot open display” errors during RTMP streaming

Remote Database Auto-Detection (Commit dd51c7f):

Auto-detects remote database mode from DATABASE_URL environment variable
Sets USE_LOCAL_POSTGRES=false when remote database detected
Prevents Docker PostgreSQL conflicts on Vast.ai instances

APT Fix-Broken (Commit dd51c7f):

Added apt --fix-broken install -y before package installation
Resolves dependency conflicts on fresh Vast.ai instances
Prevents deployment failures from broken package states

Streaming Destination Auto-Detection (Commit 41dc606):

STREAM_ENABLED_DESTINATIONS now uses || logic for fallback
Auto-detects enabled destinations from configured stream keys
Explicitly forwards stream keys through PM2 environment
Added TWITCH_RTMP_STREAM_KEY alias to secrets file

First-Time Setup Support (Commit 6302fa4):

Auto-clones repository if it doesn’t exist on fresh Vast.ai instances
Eliminates manual repository setup step

Bun Installation Check (Commit abfe0ce):

Always checks and installs bun if missing
Ensures bun is available before running build commands

The scripts/deploy-vast.sh script handles the full deployment with these improvements: Key Steps:

Load secrets from /tmp/hyperscape-secrets.env (commit 684b203)
Auto-detect database mode from DATABASE_URL (commit 3df4370)
Auto-detect stream destinations from available keys (commit 41dc606)
Configure DNS resolution (some Vast containers use internal-only DNS)
Pull latest code from main branch
Restore environment variables after git reset (commits eec04b0, dda4396, 4a6aaaf)
Install system dependencies (build-essential, ffmpeg, Vulkan drivers, Chrome Beta, PulseAudio)
GPU rendering detection and configuration (commits dd649da, e51a332, 30bdaf0, 725e934, 012450c):
- Check for NVIDIA GPU and DRI devices
- Try Xorg mode first (if DRI available)
- Detect Xorg swrast fallback and switch to headless EGL if needed
- Fall back to Xvfb mode if Xorg fails
- Fall back to headless EGL mode if X11 not available
- Install NVIDIA Xorg drivers and configure headless X server
- Force NVIDIA-only Vulkan ICD to avoid Mesa conflicts
Install Chrome Beta channel for WebGPU support (commit 547714e)
Setup PulseAudio for audio capture (commits 3b6f1ee, aab66b0, b9d2e41):

Create virtual sink (chrome_audio) for Chrome audio output
Configure user-mode PulseAudio with proper permissions
Export PULSE_SERVER environment variable

Install Playwright and dependencies
Build core packages (physx, decimation, impostors, procgen, asset-forge, shared)
Setup Solana keypair from SOLANA_DEPLOYER_PRIVATE_KEY (commit 8a677dc)
Push database schema with drizzle-kit and warmup connection pool
Tear down existing processes (commit b466233):
- Use pm2 kill instead of pm2 delete to restart daemon with fresh env
- Clean up legacy watchdog processes
Start port proxies (socat) for external access
Start Xvfb virtual display (commits 704b955, 294a36c):
- Start Xvfb before PM2 to ensure DISPLAY is available
- Export DISPLAY=:99 to environment
Export GPU environment variables for PM2
Start duel stack via PM2 (commits 684b203, 3df4370):
- PM2 reads secrets from /tmp/hyperscape-secrets.env
- Auto-detects database mode from DATABASE_URL
- Explicitly forwards DISPLAY, DATABASE_URL, and stream keys
Wait for health check to pass (up to 120 seconds)
Run streaming diagnostics (commit cf53ad4)

Environment Variable Persistence (commits eec04b0, dda4396, 4a6aaaf): Problem: git reset operations in deploy script would overwrite the .env file, losing DATABASE_URL and stream keys. Solution: Write secrets to /tmp before git reset, then restore after:

# Save secrets to /tmp (survives git reset)
echo "DATABASE_URL=$DATABASE_URL" > /tmp/hyperscape-secrets.env
echo "TWITCH_STREAM_KEY=$TWITCH_STREAM_KEY" >> /tmp/hyperscape-secrets.env
# ... other secrets ...

# Pull latest code (git reset happens here)
git fetch origin main
git reset --hard origin/main

# Restore secrets from /tmp
cat /tmp/hyperscape-secrets.env >> packages/server/.env
rm /tmp/hyperscape-secrets.env

Why This Matters:

Prevents database connection loss during deployment
Ensures stream keys persist across deployments
Required for zero-downtime deployments

Stream Key Export (commits 7ee730d, a71d4ba, 50f8bec): Stream keys must be explicitly unset and re-exported before PM2 start:

# Unset stale environment variables
unset TWITCH_STREAM_KEY X_STREAM_KEY X_RTMP_URL KICK_STREAM_KEY KICK_RTMP_URL

# Re-source .env file to get correct values
source /root/hyperscape/packages/server/.env

# Log which keys are configured (masked)
echo "Stream keys configured:"
[ -n "$TWITCH_STREAM_KEY" ] && echo "  - Twitch: ***${TWITCH_STREAM_KEY: -4}"
[ -n "$KICK_STREAM_KEY" ] && echo "  - Kick: ***${KICK_STREAM_KEY: -4}"
[ -n "$X_STREAM_KEY" ] && echo "  - X: ***${X_STREAM_KEY: -4}"

# Start PM2 with clean environment
bunx pm2 start ecosystem.config.cjs

Why This Matters:

Vast.ai servers can have stale stream keys from previous deployments
Stale values override .env file values
Explicitly unsetting ensures PM2 picks up correct keys
Prevents streams from going to wrong Twitch/X/Kick accounts

Port Mappings:

Internal	External	Service
5555	35143	HTTP API
5555	35079	WebSocket
8080	35144	CDN

Solana Keypair Setup

The deployment automatically configures Solana keypairs from environment variables:

# SOLANA_DEPLOYER_PRIVATE_KEY is decoded and written to:
# - ~/.config/solana/id.json (Solana CLI default)
# - deployer-keypair.json (legacy location)

# Script: scripts/decode-key.ts
# Converts base58 private key to JSON byte array format

Environment Variable Fallbacks:

// From ecosystem.config.cjs
SOLANA_ARENA_AUTHORITY_SECRET: process.env.SOLANA_DEPLOYER_PRIVATE_KEY || "",
SOLANA_ARENA_REPORTER_SECRET: process.env.SOLANA_DEPLOYER_PRIVATE_KEY || "",
SOLANA_ARENA_KEEPER_SECRET: process.env.SOLANA_DEPLOYER_PRIVATE_KEY || "",

All three roles (authority, reporter, keeper) default to the same deployer keypair for simplified configuration.

System Requirements

Vast.ai Instance Specs:

GPU: NVIDIA with Vulkan support (RTX 3060 Ti or better)
RAM: 16GB minimum
Storage: 50GB minimum
OS: Ubuntu 22.04 or Debian 12

Installed Dependencies:

Bun (latest)
FFmpeg (system package, not static build)
Chrome Beta channel (google-chrome-beta) - Updated March 13, 2026 for better production stability
Playwright Chromium
Vulkan drivers (mesa-vulkan-drivers, vulkan-tools)
Xorg or EGL support (for GPU rendering)
PulseAudio (for audio capture)
socat (for port proxying)

GPU Rendering Requirements (commits e51a332, 30bdaf0, 012450c, Feb 27-28 2026): The system requires hardware GPU rendering for WebGPU. Three modes are supported (tried in order):

Xorg Mode (preferred if DRI/DRM available):
- Requires /dev/dri/card0 or similar DRM device
- Full hardware GPU acceleration
- Best performance
Xvfb Mode (fallback when Xorg fails):
- Virtual framebuffer + GPU rendering via ANGLE/Vulkan
- Works when DRI/DRM not available
- Requires X11 protocol support
Headless EGL Mode (fallback for containers without X11):
- Works without X server or DRM/DRI access
- Uses Chrome’s --headless=new with direct EGL rendering
- Hardware GPU acceleration via NVIDIA EGL
- Ideal for Vast.ai containers where NVIDIA kernel module fails to initialize for Xorg
- Uses --use-gl=egl --ozone-platform=headless flags

Swrast Detection (commit 725e934):

Deployment script detects when Xorg falls back to swrast (software rendering)
Automatically switches to headless EGL mode when swrast detected
Prevents unusable software rendering for WebGPU streaming

Software rendering (SwiftShader, Lavapipe) is NOT supported - too slow for streaming. See Also: GPU Rendering Guide for complete GPU configuration

Streaming Configuration

The deployment uses these streaming settings (updated March 2026):

# From ecosystem.config.cjs (commits 72c667a, 547714e, 684b203, 3df4370)
STREAM_CAPTURE_MODE=mediarecorder          # Changed from cdp (commit 72c667a)
                                           # mediarecorder: canvas.captureStream() → WebSocket → FFmpeg
                                           # cdp: Chrome DevTools Protocol screencast (may stall under Xvfb)
STREAM_CAPTURE_HEADLESS=false              # Xorg mode (or "new" for headless EGL)
STREAM_CAPTURE_USE_EGL=false               # true for headless EGL mode
STREAM_CAPTURE_CHANNEL=chrome-beta         # Changed from chrome-unstable (commit 547714e)
STREAM_CAPTURE_ANGLE=default               # Changed from vulkan (commit 547714e)
                                           # default: auto-selects best backend for system
STREAM_CAPTURE_DISABLE_WEBGPU=false        # WebGPU enabled
STREAMING_CANONICAL_PLATFORM=twitch        # Lower latency than YouTube
STREAMING_PUBLIC_DELAY_MS=0                # No delay (real-time)

# GPU Rendering (auto-detected)
GPU_RENDERING_MODE=xorg                    # xorg | xvfb-vulkan | headless-egl
DISPLAY=:99                                # Explicitly set in PM2 env (commit 704b955)
DUEL_CAPTURE_USE_XVFB=true                # true for Xvfb mode (commit 294a36c)
                                          # Xvfb started before PM2 in deploy-vast.sh
STREAM_CAPTURE_USE_EGL=false              # true for headless EGL mode
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json

# Database Configuration (auto-detected - commit 3df4370)
DUEL_DATABASE_MODE=remote                  # Auto-detected from DATABASE_URL hostname
DATABASE_URL=postgresql://...              # Explicitly forwarded through PM2 (commit 5d415fc)

# CDN Configuration (commit 2173086)
PUBLIC_CDN_URL=https://assets.hyperscape.club  # Unified CDN URL (replaced DUEL_PUBLIC_CDN_URL)

# Audio Capture (commits 3b6f1ee, aab66b0, b9d2e41)
STREAM_AUDIO_ENABLED=true
PULSE_AUDIO_DEVICE=chrome_audio.monitor
PULSE_SERVER=unix:/tmp/pulse-runtime/pulse/native
XDG_RUNTIME_DIR=/tmp/pulse-runtime

Frame Pacing (Commits 522fe37, e2c9fbf): The streaming pipeline enforces 30fps frame pacing to eliminate buffering:

Frame Pacing Guard: Skips frames arriving faster than 85% of 33.3ms target interval
Xvfb Compositor: Runs at 30fps without vsync (game is capped at 30fps)
everyNthFrame: Set to 1 (Xvfb delivers at 30fps, no frame skipping needed)
Resolution: 1280x720 matches capture viewport, eliminates upscaling overhead

Impact: Eliminates stream buffering, smoother playback for viewers, reduced bandwidth usage. Multi-Platform Streaming: Streams simultaneously to:

Twitch (rtmp://live.twitch.tv/app)
Kick (rtmps://fa723fc1b171.global-contribute.live-video.net/app) - Fixed in commit 5dbd239
X/Twitter (rtmp://sg.pscp.tv:80/x)
YouTube explicitly disabled (commit b466233)

BREAKING CHANGE - WebGPU Required (commit 47782ed, Feb 27 2026):

All WebGL fallback code removed
STREAM_CAPTURE_DISABLE_WEBGPU and DUEL_FORCE_WEBGL_FALLBACK flags ignored
Deployment FAILS if WebGPU cannot initialize (no soft fallbacks)
Headless mode NOT supported (WebGPU requires display server: Xorg or Xvfb)
DUEL_USE_PRODUCTION_CLIENT=true recommended for faster page loads (180s timeout fix)
STREAM_GOP_SIZE now configurable via environment variable (default: 60 frames)

Audio Streaming (commits 3b6f1ee, aab66b0, b9d2e41):

Game music and sound effects captured via PulseAudio
Virtual sink (chrome_audio) routes Chrome audio to FFmpeg
Graceful fallback to silent audio if PulseAudio unavailable

See Also:

GPU Rendering Guide - GPU configuration
Audio Streaming Guide - PulseAudio setup

Health Monitoring

The deployment includes comprehensive health checks:

# Health endpoint
GET /health

Response:
{
  "status": "healthy",
  "uptime": 12345,
  "maintenance": false
}

Post-Deploy Diagnostics: The deploy script automatically runs streaming diagnostics:

Checks streaming API state
Verifies game client is running
Checks RTMP status file
Lists FFmpeg processes
Shows recent PM2 logs filtered for streaming keywords

Troubleshooting

Stream not appearing on platforms:

Check stream keys are configured:

# SSH into Vast instance
ssh -p 35022 root@<vast-ip>

# Check environment variables
cd /root/hyperscape
cat packages/server/.env | grep STREAM_KEY

Check FFmpeg processes:

ps aux | grep ffmpeg

Check RTMP status:

cat packages/server/public/live/rtmp-status.json

Check PM2 logs:

bunx pm2 logs hyperscape-duel --lines 100 | grep -i "rtmp\|stream\|ffmpeg"

Database connection issues: The deployment writes DATABASE_URL to packages/server/.env after git reset to prevent it from being overwritten. GPU rendering issues: Check Vulkan support:

vulkaninfo --summary
nvidia-smi

If Vulkan fails, the system falls back to GL ANGLE backend. WebGPU diagnostics (NEW - commit d5c6884): Check WebGPU initialization logs:

bunx pm2 logs hyperscape-duel --lines 500 | grep -A 20 "GPU Diagnostics"
bunx pm2 logs hyperscape-duel --lines 500 | grep -i "webgpu\\|preflight"

Browser timeout issues (NEW - commit 4be263a): If page load times out (180s limit), enable production client build:

# In packages/server/.env
DUEL_USE_PRODUCTION_CLIENT=true

This serves pre-built client via vite preview instead of dev server, eliminating JIT compilation delays.

CI/CD Configuration

GitHub Actions

The repository includes several CI/CD workflows with recent reliability improvements (Feb 2026):

Build and Test (`.github/workflows/ci.yml`)

Runs on every push to main:

Installs Foundry for MUD contracts tests
Runs all package tests with increased timeouts for CI
Validates manifest JSON files
Checks TypeScript compilation

Key Features:

Caches dependencies for faster builds
Runs tests in parallel across packages
Fails fast on first error
Uses --frozen-lockfile to prevent npm rate-limiting (commit 08aa151, Feb 25, 2026)

Frozen Lockfile Fix (commit 08aa151): All CI workflows now use bun install --frozen-lockfile to prevent npm 403 rate-limiting errors:

# .github/workflows/ci.yml
- name: Install dependencies
  run: bun install --frozen-lockfile

Why This Matters:

bun install without --frozen-lockfile tries to resolve packages fresh from npm even when lockfile exists
Under CI load this triggers npm rate-limiting (403 Forbidden)
--frozen-lockfile ensures bun uses only the committed lockfile for resolution
Eliminates npm registry calls entirely in CI
Applied to all workflows: ci.yml, integration.yml, typecheck.yml, deploy-*.yml

Impact:

CI workflows now run reliably without npm rate-limiting failures
Faster builds (no network calls to npm registry)
Deterministic builds (exact versions from lockfile)

Integration Tests (`.github/workflows/integration.yml`)

Runs integration tests with database setup: Database Schema Creation (commit eb8652a): The integration workflow uses drizzle-kit push for declarative schema creation instead of server migrations:

# .github/workflows/integration.yml
- name: Create database schema
  run: |
    cd packages/server
    bunx drizzle-kit push
  env:
    DATABASE_URL: postgresql://postgres:postgres@localhost:5432/hyperscape_test

- name: Run integration tests
  run: bun test:integration
  env:
    SKIP_MIGRATIONS: true  # Skip server migrations (schema already created)
    DATABASE_URL: postgresql://postgres:postgres@localhost:5432/hyperscape_test

Why This Approach:

Server’s built-in migrations have FK ordering issues (migration 0050 references arena_rounds from older migrations)
drizzle-kit push creates schema declaratively without these problems
Prevents “relation already exists” errors on fresh test databases
SKIP_MIGRATIONS=true tells server to skip migration system (schema already created)
Fixed in commits: eb8652a (CI integration), 6a5f4ee (table validation skip)

Migration 0050 Fix (commit e4b6489): Migration 0050 was also fixed to add IF NOT EXISTS guards for idempotency:

-- Before (caused errors on fresh databases)
CREATE TABLE agent_duel_stats (...);
CREATE INDEX idx_agent_duel_stats_character_id ON agent_duel_stats(character_id);

-- After (fixed in commit e4b6489)
CREATE TABLE IF NOT EXISTS agent_duel_stats (...);
CREATE INDEX IF NOT EXISTS idx_agent_duel_stats_character_id ON agent_duel_stats(character_id);

This allows the server’s migration system to work correctly on fresh databases when SKIP_MIGRATIONS is not set.

Deployment Workflows

Railway: .github/workflows/deploy-railway.yml
Cloudflare: .github/workflows/deploy-cloudflare.yml
Vast.ai: .github/workflows/deploy-vast.yml

Environment Variables Required:

RAILWAY_TOKEN - Railway API token
CLOUDFLARE_API_TOKEN - Cloudflare API token
VAST_API_KEY - Vast.ai API key

Docker Build Configuration

Server Dockerfile:

FROM node:20-bookworm-slim

# Install build dependencies
RUN apt-get update && apt-get install -y \
    build-essential \
    git-lfs \
    python3 \
    python3-pip

# Install Bun
RUN curl -fsSL https://bun.sh/install | bash

# Set CI=true to skip asset download in production
ENV CI=true

# Build application
RUN bun install
RUN bun run build

Key Points:

Uses bookworm-slim for Python 3.11+ support
Includes build-essential for native module compilation
Sets CI=true to skip asset download (assets served from CDN)
Installs git-lfs for asset checks

Streaming Infrastructure

Browser Capture Configuration

The streaming system uses Playwright with Chrome for game capture: Chrome Flags for WebGPU:

// Stable configuration for RTX 5060 Ti and similar GPUs
const chromeFlags = [
  '--use-gl=angle',              // Use ANGLE backend (Vulkan ICD broken on some GPUs)
  '--use-angle=gl',              // Force OpenGL backend
  '--disable-gpu-sandbox',       // Required for headless GPU access
  '--enable-unsafe-webgpu',      // Enable WebGPU in headless mode
  '--no-sandbox',                // Required for Docker
  '--disable-setuid-sandbox'
];

FFmpeg Configuration:

# Use system FFmpeg (static builds cause SIGSEGV on some systems)
apt-get install -y ffmpeg

# Verify installation
ffmpeg -version

Playwright Dependencies:

# Install Playwright browsers and system dependencies
npx playwright install --with-deps chromium

# Or install manually
apt-get install -y \
  libnss3 \
  libnspr4 \
  libatk1.0-0 \
  libatk-bridge2.0-0 \
  libcups2 \
  libdrm2 \
  libxkbcommon0 \
  libxcomposite1 \
  libxdamage1 \
  libxfixes3 \
  libxrandr2 \
  libgbm1 \
  libasound2

GPU Compatibility

Tested Configurations:

✅ RTX 3060 Ti (Vulkan)
✅ RTX 4090 (Vulkan)
⚠️ RTX 5060 Ti (GL ANGLE only, Vulkan ICD broken)

Fallback Modes:

WebGPU + Vulkan (preferred, best performance)
WebGPU + GL ANGLE (RTX 5060 Ti, stable)
WebGL + Swiftshader (CPU fallback, lowest performance)

Environment Variables:

# Force specific rendering backend
CHROME_BACKEND=angle        # Use GL ANGLE
CHROME_BACKEND=vulkan       # Use Vulkan (default)
CHROME_BACKEND=swiftshader  # Use CPU fallback

Xvfb Configuration

For headful mode with GPU compositing:

# Start Xvfb
Xvfb :99 -screen 0 1920x1080x24 &
export DISPLAY=:99

# Run game server with GPU acceleration
bun run dev:streaming

Docker Configuration:

# Install Xvfb for headful GPU compositing
RUN apt-get install -y xvfb

# Start Xvfb in entrypoint
CMD Xvfb :99 -screen 0 1920x1080x24 & \
    export DISPLAY=:99 && \
    bun run start

Chrome Dev Channel

For latest WebGPU features on Vast.ai:

# Install Chrome Dev channel
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list
apt-get update
apt-get install -y google-chrome-unstable

# Use in Playwright
const browser = await chromium.launch({
  executablePath: '/usr/bin/google-chrome-unstable'
});

Solana Betting Infrastructure

CLOB Market Mainnet Migration

The betting system migrated to CLOB (Central Limit Order Book) market program on Solana mainnet in February 2026 (commits dba3e03, 35c14f9): Program Address Updates:

// From packages/gold-betting-demo/anchor/programs/fight_oracle/src/lib.rs
declare_id!("FightOracle111111111111111111111111111111111");  // Mainnet address

// From packages/gold-betting-demo/anchor/programs/gold_clob_market/src/lib.rs
declare_id!("GoldCLOB1111111111111111111111111111111111111");  // Mainnet address

IDL Updates: All IDL files updated with mainnet program addresses:

// From packages/gold-betting-demo/keeper/src/idl/fight_oracle.json
{
  "address": "FightOracle111111111111111111111111111111111",
  "metadata": {
    "name": "fight_oracle",
    "version": "0.1.0"
  }
}

Bot Rewrite for CLOB Instructions: The betting bot was rewritten to use CLOB market instructions instead of binary market:

// From packages/gold-betting-demo/keeper/src/bot.ts

// CLOB market instructions
await program.methods.initializeConfig(/* ... */).rpc();
await program.methods.initializeMatch(/* ... */).rpc();
await program.methods.initializeOrderBook(/* ... */).rpc();
await program.methods.resolveMatch(/* ... */).rpc();

// Removed binary market instructions:
// - seedMarket
// - createVault
// - placeBet

Server Configuration: Arena config fallback updated to mainnet fight oracle:

// From packages/server/src/arena/config.ts
const DEFAULT_FIGHT_ORACLE = "FightOracle111111111111111111111111111111111";

Frontend Configuration: Updated .env.mainnet with all VITE_ environment variables:

# packages/gold-betting-demo/app/.env.mainnet
VITE_FIGHT_ORACLE_PROGRAM_ID=FightOracle111111111111111111111111111111111
VITE_GOLD_CLOB_MARKET_PROGRAM_ID=GoldCLOB1111111111111111111111111111111111111
VITE_SOLANA_RPC_URL=https://api.mainnet-beta.solana.com

Migration Checklist:

Update program addresses in Rust code
Regenerate IDL files with anchor build
Update keeper bot logic for CLOB instructions
Update server arena config with mainnet program IDs
Update frontend .env.mainnet with all VITE_ vars
Test on devnet before mainnet deployment
Verify program deployment on Solana Explorer

Native App Releases

Hyperscape automatically builds native desktop and mobile apps for tagged releases.

Creating a Release

# Tag a new version
git tag v1.0.0
git push origin v1.0.0

This triggers .github/workflows/build-app.yml which builds:

Windows: .msi installer (x64)
macOS: .dmg installer (universal binary: Intel + Apple Silicon)
Linux: .AppImage (portable) and .deb (Debian/Ubuntu)
iOS: .ipa bundle
Android: .apk bundle

Download Portal

Built apps are published to:

GitHub Releases: https://github.com/HyperscapeAI/hyperscape/releases
Public Portal: https://hyperscapeai.github.io/hyperscape/

Required GitHub Secrets

Configure these in repository settings for automated builds:

Secret	Purpose	Required For
`APPLE_CERTIFICATE`	Code signing certificate (base64)	macOS, iOS
`APPLE_CERTIFICATE_PASSWORD`	Certificate password	macOS, iOS
`APPLE_SIGNING_IDENTITY`	Developer ID	macOS, iOS
`APPLE_ID`	Apple ID email	macOS, iOS
`APPLE_PASSWORD`	App-specific password	macOS, iOS
`APPLE_TEAM_ID`	Developer team ID	macOS, iOS
`TAURI_PRIVATE_KEY`	Updater signing key	All platforms
`TAURI_KEY_PASSWORD`	Key password	All platforms

The build workflow is enabled as of commit cb57325 (Feb 25, 2026). See docs/native-release.md in the repository for complete setup instructions.

Security & Browser Requirements

WebGPU Requirement

As of February 2026, Hyperscape requires WebGPU for rendering. All shaders use Three.js Shading Language (TSL) which only works with WebGPU. Browser Support:

Chrome 113+ (WebGPU enabled by default)
Edge 113+
Safari 18+ (macOS Sonoma+)
Firefox Nightly (experimental)

Unsupported Browsers: Users on browsers without WebGPU support see a user-friendly error screen:

// From packages/shared/src/systems/client/ClientGraphics.ts
if (!navigator.gpu) {
  // Show error screen with browser upgrade instructions
  throw new Error('WebGPU not supported. Please upgrade your browser.');
}

Why WebGPU Only:

All procedural shaders (grass, terrain, particles) use TSL
TSL compiles to WGSL (WebGPU Shading Language)
No WebGL fallback possible without rewriting all shaders
Commit: 3bc59db (February 26, 2026)

CSRF Protection Updates

The CSRF middleware was updated to support cross-origin clients (commit cd29a76): Problem:

CSRF uses SameSite=Strict cookies which cannot be sent in cross-origin requests
Cloudflare Pages (hyperscape.gg) → Railway backend caused “Missing CSRF token” errors
Cross-origin requests already protected by Origin validation + JWT auth

Solution:

// From packages/server/src/middleware/csrf.ts
const KNOWN_CROSS_ORIGIN_CLIENTS = [
  /^https:\/\/.*\.hyperscape\.gg$/,
  /^https:\/\/hyperscape\.gg$/,        // Apex domain support
  /^https:\/\/.*\.hyperbet\.win$/,
  /^https:\/\/hyperbet\.win$/,         // Apex domain support
  /^https:\/\/.*\.hyperscape\.bet$/,
  /^https:\/\/hyperscape\.bet$/,       // Apex domain support
];

// Skip CSRF validation for known cross-origin clients
if (origin && KNOWN_CROSS_ORIGIN_CLIENTS.some(pattern => pattern.test(origin))) {
  return; // Skip CSRF check
}

Security Layers:

Origin header validation (http-server.ts preHandler hook)
JWT bearer token authentication (Authorization header)
CSRF cookie validation (same-origin requests only)

JWT Secret Enforcement

JWT secret is now required in production and staging environments (commit 3bc59db):

// From packages/server/src/startup/config.ts
if (NODE_ENV === 'production' || NODE_ENV === 'staging') {
  if (!JWT_SECRET) {
    throw new Error('JWT_SECRET is required in production/staging');
  }
}

if (NODE_ENV === 'development' && !JWT_SECRET) {
  console.warn('WARNING: JWT_SECRET not set in development');
}

Generate Secure Secret:

openssl rand -base64 32

​Deployment Overview

​Environment Variables

​Server Production

​Client Production

​Production Domains

​Railway Deployment

​Cloudflare Pages Deployment

​Automated Deployment

​Required GitHub Secrets

​Cloudflare R2 CORS Configuration

​Manual Deployment

​Vercel Client Deployment

​Database Setup

​Neon PostgreSQL

​Migrations

​CDN Setup

​Option 1: Self-Hosted

​Option 2: Cloud Storage

​Vast.ai GPU Deployment

​Automated Instance Provisioning (NEW)

​Automated Deployment

​Automated Deployment

​Graceful Restart API (Zero-Downtime Deployments)

​Maintenance Mode API (March 2026)

​Required GitHub Secrets

​Deployment Script Improvements (March 2026)

​Solana Keypair Setup

​System Requirements

​Streaming Configuration

​Health Monitoring

​Troubleshooting

​CI/CD Configuration

​GitHub Actions

​Build and Test (.github/workflows/ci.yml)

​Integration Tests (.github/workflows/integration.yml)

​Deployment Workflows

​Docker Build Configuration

​Streaming Infrastructure

​Browser Capture Configuration

​GPU Compatibility

​Xvfb Configuration

​Chrome Dev Channel

​Solana Betting Infrastructure

​CLOB Market Mainnet Migration

​Native App Releases

​Creating a Release

​Download Portal

​Required GitHub Secrets

​Security & Browser Requirements

​WebGPU Requirement

​CSRF Protection Updates

​JWT Secret Enforcement

​Production Checklist

Deployment Overview

Environment Variables

Server Production

Client Production

Production Domains

Railway Deployment

Cloudflare Pages Deployment

Automated Deployment

Required GitHub Secrets

Cloudflare R2 CORS Configuration

Manual Deployment

Vercel Client Deployment

Database Setup

Neon PostgreSQL

Migrations

CDN Setup

Option 1: Self-Hosted

Option 2: Cloud Storage

Vast.ai GPU Deployment

Automated Instance Provisioning (NEW)

Automated Deployment

Automated Deployment

Graceful Restart API (Zero-Downtime Deployments)

Maintenance Mode API (March 2026)

Required GitHub Secrets

Deployment Script Improvements (March 2026)

Solana Keypair Setup

System Requirements

Streaming Configuration

Health Monitoring

Troubleshooting

CI/CD Configuration

GitHub Actions

Build and Test (`.github/workflows/ci.yml`)

Integration Tests (`.github/workflows/integration.yml`)

Deployment Workflows

Docker Build Configuration

Streaming Infrastructure

Browser Capture Configuration

GPU Compatibility

Xvfb Configuration

Chrome Dev Channel

Solana Betting Infrastructure

CLOB Market Mainnet Migration

Native App Releases

Creating a Release

Download Portal

Required GitHub Secrets

Security & Browser Requirements

WebGPU Requirement

CSRF Protection Updates

JWT Secret Enforcement

Production Checklist