> ## Documentation Index
> Fetch the complete documentation index at: https://hyperscape-ai-mintlify-docs-update.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Deployment

> Deploy Hyperscape to production environments

## Deployment Overview

Hyperscape uses a split deployment model:

* **Server**: Railway, Fly.io, or Docker host
* **Client**: Vercel, Netlify, or static hosting
* **Database**: PostgreSQL (Neon recommended)
* **Assets**: CDN or object storage

## Environment Variables

### Server Production

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Required
DATABASE_URL=postgresql://...
JWT_SECRET=your-secret-key
PRIVY_APP_ID=your-app-id
PRIVY_APP_SECRET=your-app-secret
ADMIN_CODE=your-admin-code  # Required for admin API endpoints (maintenance mode, logs, restart)

# Optional
PORT=5555
PUBLIC_CDN_URL=https://cdn.example.com
LIVEKIT_API_KEY=...
LIVEKIT_API_SECRET=...
ORACLE_SETTLEMENT_DELAY_MS=7000  # Delay oracle publish to sync with stream (default: 7000ms)
```

### Client Production

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
PUBLIC_PRIVY_APP_ID=your-app-id
PUBLIC_API_URL=https://api.hyperscape.lol
PUBLIC_WS_URL=wss://api.hyperscape.lol
PUBLIC_CDN_URL=https://cdn.hyperscape.lol
```

### Production Domains

Hyperscape supports multiple production domains with CORS configuration (added in commits bb292c1, 7ff88d1):

**Game Domains:**

* `hyperscape.gg` - Primary game domain (added Feb 2026)
* `play.hyperscape.club` - Alternative game domain

**Betting Domains:**

* `hyperscape.bet` - Betting platform (added Feb 2026)
* `hyperbet.win` - Additional betting domain (added Feb 2026)

**CORS Configuration:**

The server and betting keeper automatically allow these domains:

```typescript theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// From packages/server/src/startup/http-server.ts
const ALLOWED_ORIGINS = [
  'https://hyperscape.gg',
  'https://hyperscape.bet',
  'https://hyperbet.win',
  'https://play.hyperscape.club'
];

// CORS middleware configuration
fastify.register(cors, {
  origin: (origin, callback) => {
    if (!origin || ALLOWED_ORIGINS.includes(origin)) {
      callback(null, true);
    } else {
      callback(new Error('Not allowed by CORS'));
    }
  },
  credentials: true,
  maxAge: 86400  // 24 hours
});
```

**Subdomain Pattern Support:**

The betting keeper supports subdomain patterns for flexible deployment:

```typescript theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// From packages/gold-betting-demo/keeper/src/service.ts
const ALLOWED_ORIGIN_PATTERNS = [
  /^https:\/\/.*\.hyperscape\.bet$/,
  /^https:\/\/.*\.hyperbet\.win$/
];
```

**Tauri Mobile Deep Links:**

Mobile apps support deep linking from production domains:

```json theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// packages/app/src-tauri/tauri.conf.json
{
  "identifier": "com.hyperscape.app",
  "deepLinkProtocols": ["hyperscape"],
  "associatedDomains": [
    "hyperscape.gg",
    "play.hyperscape.club"
  ]
}
```

**Website Game Link:**

The marketing website now links to the primary game domain:

```typescript theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// From packages/website/src/lib/links.ts
export const GAME_URL = 'https://hyperscape.gg';
```

<Warning>
  `PUBLIC_PRIVY_APP_ID` must match between client and server.
</Warning>

## Railway Deployment

<Steps>
  <Step title="Create Railway project">
    Connect your GitHub repository to Railway.
  </Step>

  <Step title="Configure build">
    Set build command: `bun run build`
    Set start command: `bun start`
  </Step>

  <Step title="Add PostgreSQL">
    Add PostgreSQL service from Railway marketplace.
  </Step>

  <Step title="Set environment variables">
    Add all required server environment variables.
  </Step>
</Steps>

## Cloudflare Pages Deployment

The client automatically deploys to Cloudflare Pages on push to main via GitHub Actions (added commit 37c3629, Feb 26 2026).

### Automated Deployment

The `.github/workflows/deploy-pages.yml` workflow triggers on:

* Pushes to main branch
* Changes to `packages/client/**` or `packages/shared/**` (shared contains packet definitions)
* Changes to `package.json` or `bun.lockb`
* Manual workflow dispatch

<Info>
  The client depends on `packages/shared` for packet definitions. When packets change on the server, the client must rebuild to stay in sync.
</Info>

**Deployment Process:**

```yaml theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Build client with production environment variables
- name: Build client (includes shared + physx dependencies via turbo)
  run: bun run build:client
  env:
    NODE_OPTIONS: '--max-old-space-size=4096'
    PUBLIC_PRIVY_APP_ID: ${{ secrets.PUBLIC_PRIVY_APP_ID }}
    PUBLIC_API_URL: https://hyperscape-production.up.railway.app
    PUBLIC_WS_URL: wss://hyperscape-production.up.railway.app/ws
    PUBLIC_CDN_URL: https://assets.hyperscape.club
    PUBLIC_APP_URL: https://hyperscape.gg

# Deploy to Cloudflare Pages
- name: Deploy to Cloudflare Pages
  run: |
    # Extract first line of commit message (avoid multi-line issues)
    COMMIT_MSG=$(echo "${{ github.event.head_commit.message }}" | head -1 | tr -d '"' | cut -c1-100)
    npx wrangler pages deploy dist \
      --project-name=hyperscape \
      --branch=${{ github.ref_name }} \
      --commit-hash=${{ github.sha }} \
      --commit-message="$COMMIT_MSG" \
      --commit-dirty=true
  env:
    CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
```

**Multi-Line Commit Message Fix (commit 3e4bb48):**

Wrangler fails on multi-line commit messages. The workflow now extracts only the first line:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
COMMIT_MSG=$(echo "${{ github.event.head_commit.message }}" | head -1 | tr -d '"' | cut -c1-100)
```

**Production URLs:**

* Primary: `https://hyperscape.gg`
* Alternative: `https://hyperscape.club`
* Preview: `https://<commit-sha>.hyperscape.pages.dev`

### Required GitHub Secrets

| Secret                 | Purpose                                      |
| ---------------------- | -------------------------------------------- |
| `CLOUDFLARE_API_TOKEN` | Cloudflare API token with Pages write access |
| `PUBLIC_PRIVY_APP_ID`  | Privy app ID (must match server)             |

### Cloudflare R2 CORS Configuration

Assets are served from Cloudflare R2 with CORS enabled for cross-origin loading:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Configure R2 CORS (run once)
bash scripts/configure-r2-cors.sh
```

**CORS Configuration:**

```json theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
{
  "allowed": {
    "origins": ["*"],
    "methods": ["GET", "HEAD"],
    "headers": ["*"]
  },
  "exposed": ["ETag"],
  "maxAge": 3600
}
```

**Why This Format:**

The wrangler API requires nested `allowed.origins/methods/headers` structure (not flat `allowedOrigins`). The old format caused `wrangler r2 bucket cors set` to fail (commit 055779a).

**Benefits:**

* Allows `assets.hyperscape.club` to serve to all domains
* Supports hyperscape.gg, hyperscape.club, and preview URLs
* Enables cross-origin asset loading for Cloudflare Pages → R2

### Manual Deployment

Deploy manually using wrangler:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
cd packages/client
bun run build
npx wrangler pages deploy dist --project-name=hyperscape
```

## Vercel Client Deployment

<Steps>
  <Step title="Import project">
    Import `packages/client` directory to Vercel.
  </Step>

  <Step title="Configure build">
    Root directory: `packages/client`
    Build command: `bun run build`
    Output directory: `dist`
  </Step>

  <Step title="Set environment variables">
    Add `PUBLIC_*` environment variables.
  </Step>
</Steps>

## Database Setup

### Neon PostgreSQL

1. Create database at [neon.tech](https://neon.tech)
2. Copy connection string
3. Set as `DATABASE_URL` in server environment

### Migrations

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
cd packages/server
bunx drizzle-kit push      # Apply schema
bunx drizzle-kit migrate   # Run migrations
```

## CDN Setup

### Option 1: Self-Hosted

Use the included Docker CDN:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
bun run cdn:up
```

Configure `PUBLIC_CDN_URL` to point to your CDN host.

### Option 2: Cloud Storage

Upload assets to S3, R2, or similar:

1. Build assets: `bun run assets:optimize`
2. Upload `packages/server/world/assets/`
3. Set `PUBLIC_CDN_URL` to bucket URL

## Vast.ai GPU Deployment

Hyperscape deploys to Vast.ai for GPU-accelerated streaming with automated CI/CD via GitHub Actions.

### Automated Instance Provisioning (NEW)

The `scripts/vast-provision.sh` script automatically finds and rents GPU instances with display driver support:

<Steps>
  <Step title="Install Vast.ai CLI">
    ```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
    pip install vastai
    vastai set api-key YOUR_API_KEY
    ```
  </Step>

  <Step title="Run Provisioner">
    ```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
    ./scripts/vast-provision.sh
    ```

    The script will:

    * Search for instances with `gpu_display_active=true` (REQUIRED for WebGPU)
    * Filter by reliability (≥95%), GPU RAM (≥20GB), price (≤\$2/hr)
    * Show top 5 available instances
    * Automatically rent the best instance
    * Wait for instance to be ready
    * Output SSH connection details
  </Step>

  <Step title="Update GitHub Secrets">
    ```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
    gh secret set VAST_HOST --body '<ssh-host>'
    gh secret set VAST_PORT --body '<ssh-port>'
    ```
  </Step>

  <Step title="Trigger Deployment">
    ```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
    gh workflow run deploy-vast.yml
    ```
  </Step>
</Steps>

<Warning>
  **CRITICAL**: Only rent instances with `gpu_display_active=true`. Compute-only GPUs cannot run WebGPU streaming.
</Warning>

**Configuration Options:**

Edit `scripts/vast-provision.sh` to customize search criteria:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
MIN_GPU_RAM=20          # GB - RTX 4090 has 24GB
MIN_RELIABILITY=0.95    # 95% uptime
MAX_PRICE_PER_HOUR=2.0  # USD per hour
PREFERRED_GPUS="RTX_4090,RTX_3090,RTX_A6000,A100"
DISK_SPACE=100          # GB minimum
```

**Output:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
═══════════════════════════════════════════════════════════════════
Instance provisioned successfully!
═══════════════════════════════════════════════════════════════════

Instance Details:
  Instance ID: 12345678
  GPU: RTX 4090 (24 GB)
  Display Driver: ENABLED ✓
  SSH Host: ssh.vast.ai
  SSH Port: 35022
  Public IP: 1.2.3.4

SSH Connection:
  ssh -p 35022 root@ssh.vast.ai

Update GitHub Secrets:
  VAST_HOST=ssh.vast.ai
  VAST_PORT=35022
```

### Automated Deployment

### Automated Deployment

The `.github/workflows/deploy-vast.yml` workflow automatically deploys to Vast.ai on push to main:

```yaml theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Triggers on successful CI completion or manual dispatch
on:
  workflow_run:
    workflows: ["CI"]
    types: [completed]
    branches: [main]
  workflow_dispatch:  # NEW: Manual deployment trigger (commit b1f41d5)
```

**Manual Deployment (commit b1f41d5):**

You can now trigger Vast.ai deployments manually from GitHub Actions UI:

1. Go to Actions tab in GitHub
2. Select "Deploy to Vast.ai" workflow
3. Click "Run workflow"
4. Select branch (usually main)
5. Click "Run workflow"

This is useful for:

* Deploying hotfixes without waiting for CI
* Re-deploying after Vast.ai instance restart
* Testing deployment process

**Deployment Process:**

1. **Write Secrets to /tmp** - Saves secrets to `/tmp/hyperscape-secrets.env` before git operations (commit 684b203)
2. **Enter Maintenance Mode** - Pauses new duel cycles, waits for active markets to resolve
3. **SSH Deploy** - Connects to Vast.ai instance, pulls latest code, builds, and restarts
4. **Auto-Detect Configuration** - Database mode, stream destinations, GPU rendering mode
5. **Start Xvfb** - Virtual display started before PM2 (commit 294a36c)
6. **PM2 Restart** - Reads secrets from `/tmp`, auto-detects database mode (commits 684b203, 3df4370)
7. **Exit Maintenance Mode** - Resumes duel cycles after health check passes

### Graceful Restart API (Zero-Downtime Deployments)

The server provides a graceful restart API for zero-downtime deployments during active duels:

**Request Graceful Restart:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
POST /admin/graceful-restart
Headers:
  x-admin-code: <ADMIN_CODE>

Response:
  {
    "success": true,
    "message": "Graceful restart requested",
    "duelActive": true,
    "willRestartAfterDuel": true
  }
```

**Check Restart Status:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
GET /admin/restart-status
Headers:
  x-admin-code: <ADMIN_CODE>

Response:
  {
    "restartPending": true,
    "duelActive": true,
    "currentPhase": "FIGHTING"
  }
```

**Behavior:**

* If no duel active: restarts immediately via SIGTERM
* If duel in progress: waits until RESOLUTION phase completes
* PM2 automatically restarts the server with new code
* No interruption to active duels or streams

**Use Cases:**

* Deploy hotfixes during active streaming
* Update server code without stopping duels
* Restart after configuration changes

### Maintenance Mode API (March 2026)

The server provides a comprehensive maintenance mode system for zero-downtime deployments:

**Enter Maintenance Mode:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
POST /admin/maintenance/enter
Headers:
  x-admin-code: <ADMIN_CODE>
  Content-Type: application/json
Body:
  {
    "reason": "deployment",
    "timeoutMs": 300000  # 5 minutes
  }

Response:
  {
    "success": true,
    "status": {
      "active": true,
      "safeToDeploy": true,
      "currentPhase": "IDLE",
      "marketStatus": "resolved",
      "pendingMarkets": 0
    }
  }
```

**Exit Maintenance Mode:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
POST /admin/maintenance/exit
Headers:
  x-admin-code: <ADMIN_CODE>

Response:
  {
    "success": true,
    "status": {
      "active": false,
      "safeToDeploy": true
    }
  }
```

**Check Status:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
GET /admin/maintenance/status
Headers:
  x-admin-code: <ADMIN_CODE>

Response:
  {
    "active": false,
    "enteredAt": null,
    "reason": null,
    "safeToDeploy": true,
    "currentPhase": "FIGHTING",
    "marketStatus": "betting",
    "pendingMarkets": 1
  }
```

**Get Live Logs:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
GET /admin/logs
Headers:
  x-admin-code: <ADMIN_CODE>

Response:
  {
    "logs": [
      {
        "timestamp": 1710187234567,
        "level": "INFO",
        "system": "DuelScheduler",
        "message": "Duel started",
        "data": { "duelId": "duel-123" }
      }
    ]
  }
```

**Restart Server:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
POST /admin/restart
Headers:
  x-admin-code: <ADMIN_CODE>

Response:
  {
    "success": true,
    "message": "Restarting server in 2 seconds..."
  }
```

<Warning>
  The restart endpoint calls `process.exit(0)` after a 2-second delay. Ensure you have a process manager (PM2) configured to automatically restart the server.
</Warning>

**Manual Maintenance Mode:**

Helper scripts are available for manual control:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Enter maintenance mode
bash scripts/pre-deploy-maintenance.sh

# Exit maintenance mode
bash scripts/post-deploy-resume.sh
```

**Client-Side Maintenance Banner:**

The client automatically displays a maintenance banner when the server enters maintenance mode:

* Polls `/health` endpoint every 5 seconds
* Displays red warning banner when `maintenanceMode: true`
* Banner appears across all screens (game, admin, leaderboard, streaming)
* Message: "SERVER MAINTENANCE IMMINENT - GAME WILL PAUSE AFTER CURRENT DUEL"

**Maintenance Mode Behavior:**

* Prevents new duel cycles from starting
* Waits for active duels to complete
* Pauses betting markets
* Sets `safeToDeploy: true` when safe to restart
* Resumes automatically on exit or timeout

### Required GitHub Secrets

Configure these in repository settings → Secrets → Actions:

| Secret                        | Purpose                                                                  |
| ----------------------------- | ------------------------------------------------------------------------ |
| `VAST_HOST`                   | Vast.ai instance IP address                                              |
| `VAST_PORT`                   | SSH port (usually 35022)                                                 |
| `VAST_SSH_KEY`                | Private SSH key for instance access                                      |
| `DATABASE_URL`                | PostgreSQL connection string                                             |
| `SOLANA_DEPLOYER_PRIVATE_KEY` | Base58 Solana keypair for market operations                              |
| `TWITCH_STREAM_KEY`           | Twitch stream key                                                        |
| `X_STREAM_KEY`                | X/Twitter stream key                                                     |
| `X_RTMP_URL`                  | X/Twitter RTMP URL                                                       |
| `KICK_STREAM_KEY`             | Kick stream key                                                          |
| `KICK_RTMP_URL`               | Kick RTMP URL                                                            |
| `ADMIN_CODE`                  | Admin code for maintenance mode API                                      |
| `VAST_SERVER_URL`             | Public server URL (e.g., [https://hyperscape.gg](https://hyperscape.gg)) |

### Deployment Script Improvements (March 2026)

The `scripts/deploy-vast.sh` script has been significantly enhanced with recent improvements:

**MediaRecorder Streaming Mode** (Commits 72c667a, 7284882):

* Switched from CDP screencast to MediaRecorder mode for streaming capture
* Uses `canvas.captureStream()` → WebSocket → FFmpeg pipeline
* More reliable under Xvfb + WebGPU on Vast instances
* Requires `internalCapture=1` URL parameter for canvas capture bridge
* Eliminates stream freezing and stalling issues

**PM2 Secrets Loading** (Commits 684b203, 3df4370):

* Writes secrets to `/tmp/hyperscape-secrets.env` before git operations
* `ecosystem.config.cjs` reads secrets file directly at config load time
* Auto-detects `DUEL_DATABASE_MODE` from `DATABASE_URL` hostname
* Prevents `sanitizeRuntimeEnv()` from stripping `DATABASE_URL` in remote mode
* Ensures secrets persist through git reset operations

**Chrome Beta for Linux WebGPU Support** (Commit 154f0b6, March 13, 2026):

* Reverted from Chrome Canary back to Chrome Beta (`google-chrome-beta`) for Linux NVIDIA
* Chrome Canary was experiencing instability issues on Linux NVIDIA GPUs
* Chrome Beta provides better stability for production streaming
* Uses Vulkan ANGLE backend (`--use-angle=vulkan`) for optimal performance

**Xvfb Display Setup** (Commits 704b955, 294a36c):

* Starts Xvfb before PM2 to ensure virtual display is available
* Exports `DISPLAY=:99` to environment
* `ecosystem.config.cjs` explicitly sets `DISPLAY=:99` in PM2 environment
* Prevents "cannot open display" errors during RTMP streaming

**Remote Database Auto-Detection (Commit dd51c7f):**

* Auto-detects remote database mode from `DATABASE_URL` environment variable
* Sets `USE_LOCAL_POSTGRES=false` when remote database detected
* Prevents Docker PostgreSQL conflicts on Vast.ai instances

**APT Fix-Broken (Commit dd51c7f):**

* Added `apt --fix-broken install -y` before package installation
* Resolves dependency conflicts on fresh Vast.ai instances
* Prevents deployment failures from broken package states

**Streaming Destination Auto-Detection (Commit 41dc606):**

* `STREAM_ENABLED_DESTINATIONS` now uses `||` logic for fallback
* Auto-detects enabled destinations from configured stream keys
* Explicitly forwards stream keys through PM2 environment
* Added `TWITCH_RTMP_STREAM_KEY` alias to secrets file

**First-Time Setup Support (Commit 6302fa4):**

* Auto-clones repository if it doesn't exist on fresh Vast.ai instances
* Eliminates manual repository setup step

**Bun Installation Check (Commit abfe0ce):**

* Always checks and installs bun if missing
* Ensures bun is available before running build commands

The `scripts/deploy-vast.sh` script handles the full deployment with these improvements:

**Key Steps:**

1. **Load secrets from `/tmp/hyperscape-secrets.env`** (commit 684b203)
2. **Auto-detect database mode from DATABASE\_URL** (commit 3df4370)
3. **Auto-detect stream destinations from available keys** (commit 41dc606)
4. Configure DNS resolution (some Vast containers use internal-only DNS)
5. Pull latest code from main branch
6. Restore environment variables after git reset (commits eec04b0, dda4396, 4a6aaaf)
7. Install system dependencies (build-essential, ffmpeg, Vulkan drivers, Chrome Beta, PulseAudio)
8. **GPU rendering detection and configuration** (commits dd649da, e51a332, 30bdaf0, 725e934, 012450c):
   * Check for NVIDIA GPU and DRI devices
   * Try Xorg mode first (if DRI available)
   * Detect Xorg swrast fallback and switch to headless EGL if needed
   * Fall back to Xvfb mode if Xorg fails
   * Fall back to headless EGL mode if X11 not available
   * Install NVIDIA Xorg drivers and configure headless X server
   * Force NVIDIA-only Vulkan ICD to avoid Mesa conflicts
9. Install Chrome Beta channel for WebGPU support (commit 547714e)
10. **Setup PulseAudio for audio capture** (commits 3b6f1ee, aab66b0, b9d2e41):

* Create virtual sink (`chrome_audio`) for Chrome audio output
* Configure user-mode PulseAudio with proper permissions
* Export PULSE\_SERVER environment variable

11. Install Playwright and dependencies
12. Build core packages (physx, decimation, impostors, procgen, asset-forge, shared)
13. Setup Solana keypair from `SOLANA_DEPLOYER_PRIVATE_KEY` (commit 8a677dc)
14. Push database schema with drizzle-kit and warmup connection pool
15. **Tear down existing processes** (commit b466233):
    * Use `pm2 kill` instead of `pm2 delete` to restart daemon with fresh env
    * Clean up legacy watchdog processes
16. Start port proxies (socat) for external access
17. **Start Xvfb virtual display** (commits 704b955, 294a36c):
    * Start Xvfb before PM2 to ensure DISPLAY is available
    * Export `DISPLAY=:99` to environment
18. Export GPU environment variables for PM2
19. **Start duel stack via PM2** (commits 684b203, 3df4370):
    * PM2 reads secrets from `/tmp/hyperscape-secrets.env`
    * Auto-detects database mode from DATABASE\_URL
    * Explicitly forwards DISPLAY, DATABASE\_URL, and stream keys
20. Wait for health check to pass (up to 120 seconds)
21. Run streaming diagnostics (commit cf53ad4)

**Environment Variable Persistence (commits eec04b0, dda4396, 4a6aaaf):**

**Problem:** `git reset` operations in deploy script would overwrite the .env file, losing DATABASE\_URL and stream keys.

**Solution:** Write secrets to `/tmp` before git reset, then restore after:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Save secrets to /tmp (survives git reset)
echo "DATABASE_URL=$DATABASE_URL" > /tmp/hyperscape-secrets.env
echo "TWITCH_STREAM_KEY=$TWITCH_STREAM_KEY" >> /tmp/hyperscape-secrets.env
# ... other secrets ...

# Pull latest code (git reset happens here)
git fetch origin main
git reset --hard origin/main

# Restore secrets from /tmp
cat /tmp/hyperscape-secrets.env >> packages/server/.env
rm /tmp/hyperscape-secrets.env
```

**Why This Matters:**

* Prevents database connection loss during deployment
* Ensures stream keys persist across deployments
* Required for zero-downtime deployments

**Stream Key Export (commits 7ee730d, a71d4ba, 50f8bec):**

Stream keys must be explicitly unset and re-exported before PM2 start:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Unset stale environment variables
unset TWITCH_STREAM_KEY X_STREAM_KEY X_RTMP_URL KICK_STREAM_KEY KICK_RTMP_URL

# Re-source .env file to get correct values
source /root/hyperscape/packages/server/.env

# Log which keys are configured (masked)
echo "Stream keys configured:"
[ -n "$TWITCH_STREAM_KEY" ] && echo "  - Twitch: ***${TWITCH_STREAM_KEY: -4}"
[ -n "$KICK_STREAM_KEY" ] && echo "  - Kick: ***${KICK_STREAM_KEY: -4}"
[ -n "$X_STREAM_KEY" ] && echo "  - X: ***${X_STREAM_KEY: -4}"

# Start PM2 with clean environment
bunx pm2 start ecosystem.config.cjs
```

**Why This Matters:**

* Vast.ai servers can have stale stream keys from previous deployments
* Stale values override .env file values
* Explicitly unsetting ensures PM2 picks up correct keys
* Prevents streams from going to wrong Twitch/X/Kick accounts

**Port Mappings:**

| Internal | External | Service   |
| -------- | -------- | --------- |
| 5555     | 35143    | HTTP API  |
| 5555     | 35079    | WebSocket |
| 8080     | 35144    | CDN       |

### Solana Keypair Setup

The deployment automatically configures Solana keypairs from environment variables:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# SOLANA_DEPLOYER_PRIVATE_KEY is decoded and written to:
# - ~/.config/solana/id.json (Solana CLI default)
# - deployer-keypair.json (legacy location)

# Script: scripts/decode-key.ts
# Converts base58 private key to JSON byte array format
```

**Environment Variable Fallbacks:**

```javascript theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// From ecosystem.config.cjs
SOLANA_ARENA_AUTHORITY_SECRET: process.env.SOLANA_DEPLOYER_PRIVATE_KEY || "",
SOLANA_ARENA_REPORTER_SECRET: process.env.SOLANA_DEPLOYER_PRIVATE_KEY || "",
SOLANA_ARENA_KEEPER_SECRET: process.env.SOLANA_DEPLOYER_PRIVATE_KEY || "",
```

All three roles (authority, reporter, keeper) default to the same deployer keypair for simplified configuration.

### System Requirements

**Vast.ai Instance Specs:**

* GPU: NVIDIA with Vulkan support (RTX 3060 Ti or better)
* RAM: 16GB minimum
* Storage: 50GB minimum
* OS: Ubuntu 22.04 or Debian 12

**Installed Dependencies:**

* Bun (latest)
* FFmpeg (system package, not static build)
* Chrome Beta channel (google-chrome-beta) - **Updated March 13, 2026 for better production stability**
* Playwright Chromium
* Vulkan drivers (mesa-vulkan-drivers, vulkan-tools)
* Xorg or EGL support (for GPU rendering)
* PulseAudio (for audio capture)
* socat (for port proxying)

**GPU Rendering Requirements (commits e51a332, 30bdaf0, 012450c, Feb 27-28 2026):**

The system requires hardware GPU rendering for WebGPU. Three modes are supported (tried in order):

1. **Xorg Mode** (preferred if DRI/DRM available):
   * Requires `/dev/dri/card0` or similar DRM device
   * Full hardware GPU acceleration
   * Best performance

2. **Xvfb Mode** (fallback when Xorg fails):
   * Virtual framebuffer + GPU rendering via ANGLE/Vulkan
   * Works when DRI/DRM not available
   * Requires X11 protocol support

3. **Headless EGL Mode** (fallback for containers without X11):
   * Works without X server or DRM/DRI access
   * Uses Chrome's `--headless=new` with direct EGL rendering
   * Hardware GPU acceleration via NVIDIA EGL
   * Ideal for Vast.ai containers where NVIDIA kernel module fails to initialize for Xorg
   * Uses `--use-gl=egl --ozone-platform=headless` flags

**Swrast Detection (commit 725e934):**

* Deployment script detects when Xorg falls back to swrast (software rendering)
* Automatically switches to headless EGL mode when swrast detected
* Prevents unusable software rendering for WebGPU streaming

**Software rendering (SwiftShader, Lavapipe) is NOT supported** - too slow for streaming.

**See Also:** [GPU Rendering Guide](/devops/gpu-rendering) for complete GPU configuration

### Streaming Configuration

The deployment uses these streaming settings (updated March 2026):

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# From ecosystem.config.cjs (commits 72c667a, 547714e, 684b203, 3df4370)
STREAM_CAPTURE_MODE=mediarecorder          # Changed from cdp (commit 72c667a)
                                           # mediarecorder: canvas.captureStream() → WebSocket → FFmpeg
                                           # cdp: Chrome DevTools Protocol screencast (may stall under Xvfb)
STREAM_CAPTURE_HEADLESS=false              # Xorg mode (or "new" for headless EGL)
STREAM_CAPTURE_USE_EGL=false               # true for headless EGL mode
STREAM_CAPTURE_CHANNEL=chrome-beta         # Changed from chrome-unstable (commit 547714e)
STREAM_CAPTURE_ANGLE=default               # Changed from vulkan (commit 547714e)
                                           # default: auto-selects best backend for system
STREAM_CAPTURE_DISABLE_WEBGPU=false        # WebGPU enabled
STREAMING_CANONICAL_PLATFORM=twitch        # Lower latency than YouTube
STREAMING_PUBLIC_DELAY_MS=0                # No delay (real-time)

# GPU Rendering (auto-detected)
GPU_RENDERING_MODE=xorg                    # xorg | xvfb-vulkan | headless-egl
DISPLAY=:99                                # Explicitly set in PM2 env (commit 704b955)
DUEL_CAPTURE_USE_XVFB=true                # true for Xvfb mode (commit 294a36c)
                                          # Xvfb started before PM2 in deploy-vast.sh
STREAM_CAPTURE_USE_EGL=false              # true for headless EGL mode
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json

# Database Configuration (auto-detected - commit 3df4370)
DUEL_DATABASE_MODE=remote                  # Auto-detected from DATABASE_URL hostname
DATABASE_URL=postgresql://...              # Explicitly forwarded through PM2 (commit 5d415fc)

# CDN Configuration (commit 2173086)
PUBLIC_CDN_URL=https://assets.hyperscape.club  # Unified CDN URL (replaced DUEL_PUBLIC_CDN_URL)

# Audio Capture (commits 3b6f1ee, aab66b0, b9d2e41)
STREAM_AUDIO_ENABLED=true
PULSE_AUDIO_DEVICE=chrome_audio.monitor
PULSE_SERVER=unix:/tmp/pulse-runtime/pulse/native
XDG_RUNTIME_DIR=/tmp/pulse-runtime
```

**Frame Pacing (Commits 522fe37, e2c9fbf):**

The streaming pipeline enforces 30fps frame pacing to eliminate buffering:

* **Frame Pacing Guard**: Skips frames arriving faster than 85% of 33.3ms target interval
* **Xvfb Compositor**: Runs at 30fps without vsync (game is capped at 30fps)
* **everyNthFrame**: Set to 1 (Xvfb delivers at 30fps, no frame skipping needed)
* **Resolution**: 1280x720 matches capture viewport, eliminates upscaling overhead

**Impact**: Eliminates stream buffering, smoother playback for viewers, reduced bandwidth usage.

**Multi-Platform Streaming:**

Streams simultaneously to:

* Twitch (rtmp\://live.twitch.tv/app)
* Kick (rtmps\://fa723fc1b171.global-contribute.live-video.net/app) - Fixed in commit 5dbd239
* X/Twitter (rtmp\://sg.pscp.tv:80/x)
* YouTube explicitly disabled (commit b466233)

**BREAKING CHANGE - WebGPU Required (commit 47782ed, Feb 27 2026):**

* All WebGL fallback code removed
* `STREAM_CAPTURE_DISABLE_WEBGPU` and `DUEL_FORCE_WEBGL_FALLBACK` flags ignored
* Deployment FAILS if WebGPU cannot initialize (no soft fallbacks)
* Headless mode NOT supported (WebGPU requires display server: Xorg or Xvfb)
* `DUEL_USE_PRODUCTION_CLIENT=true` recommended for faster page loads (180s timeout fix)
* `STREAM_GOP_SIZE` now configurable via environment variable (default: 60 frames)

**Audio Streaming (commits 3b6f1ee, aab66b0, b9d2e41):**

* Game music and sound effects captured via PulseAudio
* Virtual sink (`chrome_audio`) routes Chrome audio to FFmpeg
* Graceful fallback to silent audio if PulseAudio unavailable

**See Also:**

* [GPU Rendering Guide](/devops/gpu-rendering) - GPU configuration
* [Audio Streaming Guide](/devops/audio-streaming) - PulseAudio setup

### Health Monitoring

The deployment includes comprehensive health checks:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Health endpoint
GET /health

Response:
{
  "status": "healthy",
  "uptime": 12345,
  "maintenance": false
}
```

**Post-Deploy Diagnostics:**

The deploy script automatically runs streaming diagnostics:

* Checks streaming API state
* Verifies game client is running
* Checks RTMP status file
* Lists FFmpeg processes
* Shows recent PM2 logs filtered for streaming keywords

### Troubleshooting

**Stream not appearing on platforms:**

1. Check stream keys are configured:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# SSH into Vast instance
ssh -p 35022 root@<vast-ip>

# Check environment variables
cd /root/hyperscape
cat packages/server/.env | grep STREAM_KEY
```

2. Check FFmpeg processes:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
ps aux | grep ffmpeg
```

3. Check RTMP status:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
cat packages/server/public/live/rtmp-status.json
```

4. Check PM2 logs:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
bunx pm2 logs hyperscape-duel --lines 100 | grep -i "rtmp\|stream\|ffmpeg"
```

**Database connection issues:**

The deployment writes `DATABASE_URL` to `packages/server/.env` after git reset to prevent it from being overwritten.

**GPU rendering issues:**

Check Vulkan support:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
vulkaninfo --summary
nvidia-smi
```

If Vulkan fails, the system falls back to GL ANGLE backend.

**WebGPU diagnostics (NEW - commit d5c6884):**

Check WebGPU initialization logs:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
bunx pm2 logs hyperscape-duel --lines 500 | grep -A 20 "GPU Diagnostics"
bunx pm2 logs hyperscape-duel --lines 500 | grep -i "webgpu\\|preflight"
```

**Browser timeout issues (NEW - commit 4be263a):**

If page load times out (180s limit), enable production client build:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# In packages/server/.env
DUEL_USE_PRODUCTION_CLIENT=true
```

This serves pre-built client via `vite preview` instead of dev server, eliminating JIT compilation delays.

## CI/CD Configuration

### GitHub Actions

The repository includes several CI/CD workflows with recent reliability improvements (Feb 2026):

#### Build and Test (`.github/workflows/ci.yml`)

Runs on every push to main:

* Installs Foundry for MUD contracts tests
* Runs all package tests with increased timeouts for CI
* Validates manifest JSON files
* Checks TypeScript compilation

**Key Features:**

* Caches dependencies for faster builds
* Runs tests in parallel across packages
* Fails fast on first error
* Uses `--frozen-lockfile` to prevent npm rate-limiting (commit 08aa151, Feb 25, 2026)

**Frozen Lockfile Fix (commit 08aa151):**

All CI workflows now use `bun install --frozen-lockfile` to prevent npm 403 rate-limiting errors:

```yaml theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# .github/workflows/ci.yml
- name: Install dependencies
  run: bun install --frozen-lockfile
```

**Why This Matters:**

* `bun install` without `--frozen-lockfile` tries to resolve packages fresh from npm even when lockfile exists
* Under CI load this triggers npm rate-limiting (403 Forbidden)
* `--frozen-lockfile` ensures bun uses only the committed lockfile for resolution
* Eliminates npm registry calls entirely in CI
* Applied to all workflows: ci.yml, integration.yml, typecheck.yml, deploy-\*.yml

**Impact:**

* CI workflows now run reliably without npm rate-limiting failures
* Faster builds (no network calls to npm registry)
* Deterministic builds (exact versions from lockfile)

#### Integration Tests (`.github/workflows/integration.yml`)

Runs integration tests with database setup:

**Database Schema Creation (commit eb8652a):**

The integration workflow uses `drizzle-kit push` for declarative schema creation instead of server migrations:

```yaml theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# .github/workflows/integration.yml
- name: Create database schema
  run: |
    cd packages/server
    bunx drizzle-kit push
  env:
    DATABASE_URL: postgresql://postgres:postgres@localhost:5432/hyperscape_test

- name: Run integration tests
  run: bun test:integration
  env:
    SKIP_MIGRATIONS: true  # Skip server migrations (schema already created)
    DATABASE_URL: postgresql://postgres:postgres@localhost:5432/hyperscape_test
```

**Why This Approach:**

* Server's built-in migrations have FK ordering issues (migration 0050 references `arena_rounds` from older migrations)
* `drizzle-kit push` creates schema declaratively without these problems
* Prevents "relation already exists" errors on fresh test databases
* `SKIP_MIGRATIONS=true` tells server to skip migration system (schema already created)
* Fixed in commits: eb8652a (CI integration), 6a5f4ee (table validation skip)

**Migration 0050 Fix (commit e4b6489):**

Migration 0050 was also fixed to add `IF NOT EXISTS` guards for idempotency:

```sql theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
-- Before (caused errors on fresh databases)
CREATE TABLE agent_duel_stats (...);
CREATE INDEX idx_agent_duel_stats_character_id ON agent_duel_stats(character_id);

-- After (fixed in commit e4b6489)
CREATE TABLE IF NOT EXISTS agent_duel_stats (...);
CREATE INDEX IF NOT EXISTS idx_agent_duel_stats_character_id ON agent_duel_stats(character_id);
```

This allows the server's migration system to work correctly on fresh databases when `SKIP_MIGRATIONS` is not set.

#### Deployment Workflows

* **Railway**: `.github/workflows/deploy-railway.yml`
* **Cloudflare**: `.github/workflows/deploy-cloudflare.yml`
* **Vast.ai**: `.github/workflows/deploy-vast.yml`

**Environment Variables Required:**

* `RAILWAY_TOKEN` - Railway API token
* `CLOUDFLARE_API_TOKEN` - Cloudflare API token
* `VAST_API_KEY` - Vast.ai API key

### Docker Build Configuration

**Server Dockerfile:**

```dockerfile theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
FROM node:20-bookworm-slim

# Install build dependencies
RUN apt-get update && apt-get install -y \
    build-essential \
    git-lfs \
    python3 \
    python3-pip

# Install Bun
RUN curl -fsSL https://bun.sh/install | bash

# Set CI=true to skip asset download in production
ENV CI=true

# Build application
RUN bun install
RUN bun run build
```

**Key Points:**

* Uses bookworm-slim for Python 3.11+ support
* Includes build-essential for native module compilation
* Sets CI=true to skip asset download (assets served from CDN)
* Installs git-lfs for asset checks

## Streaming Infrastructure

### Browser Capture Configuration

The streaming system uses Playwright with Chrome for game capture:

**Chrome Flags for WebGPU:**

```typescript theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// Stable configuration for RTX 5060 Ti and similar GPUs
const chromeFlags = [
  '--use-gl=angle',              // Use ANGLE backend (Vulkan ICD broken on some GPUs)
  '--use-angle=gl',              // Force OpenGL backend
  '--disable-gpu-sandbox',       // Required for headless GPU access
  '--enable-unsafe-webgpu',      // Enable WebGPU in headless mode
  '--no-sandbox',                // Required for Docker
  '--disable-setuid-sandbox'
];
```

**FFmpeg Configuration:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Use system FFmpeg (static builds cause SIGSEGV on some systems)
apt-get install -y ffmpeg

# Verify installation
ffmpeg -version
```

**Playwright Dependencies:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Install Playwright browsers and system dependencies
npx playwright install --with-deps chromium

# Or install manually
apt-get install -y \
  libnss3 \
  libnspr4 \
  libatk1.0-0 \
  libatk-bridge2.0-0 \
  libcups2 \
  libdrm2 \
  libxkbcommon0 \
  libxcomposite1 \
  libxdamage1 \
  libxfixes3 \
  libxrandr2 \
  libgbm1 \
  libasound2
```

### GPU Compatibility

**Tested Configurations:**

* ✅ RTX 3060 Ti (Vulkan)
* ✅ RTX 4090 (Vulkan)
* ⚠️ RTX 5060 Ti (GL ANGLE only, Vulkan ICD broken)

**Fallback Modes:**

1. **WebGPU + Vulkan** (preferred, best performance)
2. **WebGPU + GL ANGLE** (RTX 5060 Ti, stable)
3. **WebGL + Swiftshader** (CPU fallback, lowest performance)

**Environment Variables:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Force specific rendering backend
CHROME_BACKEND=angle        # Use GL ANGLE
CHROME_BACKEND=vulkan       # Use Vulkan (default)
CHROME_BACKEND=swiftshader  # Use CPU fallback
```

### Xvfb Configuration

For headful mode with GPU compositing:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Start Xvfb
Xvfb :99 -screen 0 1920x1080x24 &
export DISPLAY=:99

# Run game server with GPU acceleration
bun run dev:streaming
```

**Docker Configuration:**

```dockerfile theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Install Xvfb for headful GPU compositing
RUN apt-get install -y xvfb

# Start Xvfb in entrypoint
CMD Xvfb :99 -screen 0 1920x1080x24 & \
    export DISPLAY=:99 && \
    bun run start
```

### Chrome Dev Channel

For latest WebGPU features on Vast.ai:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Install Chrome Dev channel
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list
apt-get update
apt-get install -y google-chrome-unstable

# Use in Playwright
const browser = await chromium.launch({
  executablePath: '/usr/bin/google-chrome-unstable'
});
```

## Solana Betting Infrastructure

### CLOB Market Mainnet Migration

The betting system migrated to CLOB (Central Limit Order Book) market program on Solana mainnet in February 2026 (commits dba3e03, 35c14f9):

**Program Address Updates:**

```rust theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// From packages/gold-betting-demo/anchor/programs/fight_oracle/src/lib.rs
declare_id!("FightOracle111111111111111111111111111111111");  // Mainnet address

// From packages/gold-betting-demo/anchor/programs/gold_clob_market/src/lib.rs
declare_id!("GoldCLOB1111111111111111111111111111111111111");  // Mainnet address
```

**IDL Updates:**

All IDL files updated with mainnet program addresses:

```json theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// From packages/gold-betting-demo/keeper/src/idl/fight_oracle.json
{
  "address": "FightOracle111111111111111111111111111111111",
  "metadata": {
    "name": "fight_oracle",
    "version": "0.1.0"
  }
}
```

**Bot Rewrite for CLOB Instructions:**

The betting bot was rewritten to use CLOB market instructions instead of binary market:

```typescript theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// From packages/gold-betting-demo/keeper/src/bot.ts

// CLOB market instructions
await program.methods.initializeConfig(/* ... */).rpc();
await program.methods.initializeMatch(/* ... */).rpc();
await program.methods.initializeOrderBook(/* ... */).rpc();
await program.methods.resolveMatch(/* ... */).rpc();

// Removed binary market instructions:
// - seedMarket
// - createVault
// - placeBet
```

**Server Configuration:**

Arena config fallback updated to mainnet fight oracle:

```typescript theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// From packages/server/src/arena/config.ts
const DEFAULT_FIGHT_ORACLE = "FightOracle111111111111111111111111111111111";
```

**Frontend Configuration:**

Updated `.env.mainnet` with all VITE\_ environment variables:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# packages/gold-betting-demo/app/.env.mainnet
VITE_FIGHT_ORACLE_PROGRAM_ID=FightOracle111111111111111111111111111111111
VITE_GOLD_CLOB_MARKET_PROGRAM_ID=GoldCLOB1111111111111111111111111111111111111
VITE_SOLANA_RPC_URL=https://api.mainnet-beta.solana.com
```

**Migration Checklist:**

* [ ] Update program addresses in Rust code
* [ ] Regenerate IDL files with `anchor build`
* [ ] Update keeper bot logic for CLOB instructions
* [ ] Update server arena config with mainnet program IDs
* [ ] Update frontend .env.mainnet with all VITE\_ vars
* [ ] Test on devnet before mainnet deployment
* [ ] Verify program deployment on Solana Explorer

## Native App Releases

Hyperscape automatically builds native desktop and mobile apps for tagged releases.

### Creating a Release

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Tag a new version
git tag v1.0.0
git push origin v1.0.0
```

This triggers `.github/workflows/build-app.yml` which builds:

* **Windows**: `.msi` installer (x64)
* **macOS**: `.dmg` installer (universal binary: Intel + Apple Silicon)
* **Linux**: `.AppImage` (portable) and `.deb` (Debian/Ubuntu)
* **iOS**: `.ipa` bundle
* **Android**: `.apk` bundle

### Download Portal

Built apps are published to:

* **GitHub Releases**: [https://github.com/HyperscapeAI/hyperscape/releases](https://github.com/HyperscapeAI/hyperscape/releases)
* **Public Portal**: [https://hyperscapeai.github.io/hyperscape/](https://hyperscapeai.github.io/hyperscape/)

### Required GitHub Secrets

Configure these in repository settings for automated builds:

| Secret                       | Purpose                           | Required For  |
| ---------------------------- | --------------------------------- | ------------- |
| `APPLE_CERTIFICATE`          | Code signing certificate (base64) | macOS, iOS    |
| `APPLE_CERTIFICATE_PASSWORD` | Certificate password              | macOS, iOS    |
| `APPLE_SIGNING_IDENTITY`     | Developer ID                      | macOS, iOS    |
| `APPLE_ID`                   | Apple ID email                    | macOS, iOS    |
| `APPLE_PASSWORD`             | App-specific password             | macOS, iOS    |
| `APPLE_TEAM_ID`              | Developer team ID                 | macOS, iOS    |
| `TAURI_PRIVATE_KEY`          | Updater signing key               | All platforms |
| `TAURI_KEY_PASSWORD`         | Key password                      | All platforms |

<Info>
  The build workflow is enabled as of commit cb57325 (Feb 25, 2026). See `docs/native-release.md` in the repository for complete setup instructions.
</Info>

## Security & Browser Requirements

### WebGPU Requirement

As of February 2026, Hyperscape **requires WebGPU** for rendering. All shaders use Three.js Shading Language (TSL) which only works with WebGPU.

**Browser Support:**

* Chrome 113+ (WebGPU enabled by default)
* Edge 113+
* Safari 18+ (macOS Sonoma+)
* Firefox Nightly (experimental)

**Unsupported Browsers:**

Users on browsers without WebGPU support see a user-friendly error screen:

```typescript theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// From packages/shared/src/systems/client/ClientGraphics.ts
if (!navigator.gpu) {
  // Show error screen with browser upgrade instructions
  throw new Error('WebGPU not supported. Please upgrade your browser.');
}
```

**Why WebGPU Only:**

* All procedural shaders (grass, terrain, particles) use TSL
* TSL compiles to WGSL (WebGPU Shading Language)
* No WebGL fallback possible without rewriting all shaders
* Commit: 3bc59db (February 26, 2026)

### CSRF Protection Updates

The CSRF middleware was updated to support cross-origin clients (commit cd29a76):

**Problem:**

* CSRF uses `SameSite=Strict` cookies which cannot be sent in cross-origin requests
* Cloudflare Pages (hyperscape.gg) → Railway backend caused "Missing CSRF token" errors
* Cross-origin requests already protected by Origin validation + JWT auth

**Solution:**

```typescript theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// From packages/server/src/middleware/csrf.ts
const KNOWN_CROSS_ORIGIN_CLIENTS = [
  /^https:\/\/.*\.hyperscape\.gg$/,
  /^https:\/\/hyperscape\.gg$/,        // Apex domain support
  /^https:\/\/.*\.hyperbet\.win$/,
  /^https:\/\/hyperbet\.win$/,         // Apex domain support
  /^https:\/\/.*\.hyperscape\.bet$/,
  /^https:\/\/hyperscape\.bet$/,       // Apex domain support
];

// Skip CSRF validation for known cross-origin clients
if (origin && KNOWN_CROSS_ORIGIN_CLIENTS.some(pattern => pattern.test(origin))) {
  return; // Skip CSRF check
}
```

**Security Layers:**

1. Origin header validation (http-server.ts preHandler hook)
2. JWT bearer token authentication (Authorization header)
3. CSRF cookie validation (same-origin requests only)

### JWT Secret Enforcement

JWT secret is now **required** in production and staging environments (commit 3bc59db):

```typescript theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// From packages/server/src/startup/config.ts
if (NODE_ENV === 'production' || NODE_ENV === 'staging') {
  if (!JWT_SECRET) {
    throw new Error('JWT_SECRET is required in production/staging');
  }
}

if (NODE_ENV === 'development' && !JWT_SECRET) {
  console.warn('WARNING: JWT_SECRET not set in development');
}
```

**Generate Secure Secret:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
openssl rand -base64 32
```

## Production Checklist

* [ ] PostgreSQL database provisioned
* [ ] Environment variables configured
* [ ] JWT\_SECRET generated and set (REQUIRED in production)
* [ ] ADMIN\_CODE set (REQUIRED for production security)
* [ ] Privy credentials set (both client and server)
* [ ] CDN serving assets with CORS configured
* [ ] WebSocket URL configured
* [ ] SSL/TLS enabled
* [ ] Vast.ai API key configured (if using GPU deployment)
* [ ] CI/CD workflows configured with required secrets
* [ ] DNS configured (Google DNS for Vast.ai instances)
* [ ] Solana program addresses updated for mainnet (if using betting)
* [ ] CORS domains configured for production domains
* [ ] GitHub secrets configured for native app builds (if releasing)
* [ ] WebGPU-compatible browsers verified for users
* [ ] Maintenance mode API tested with ADMIN\_CODE