> ## Documentation Index
> Fetch the complete documentation index at: https://hyperscape-ai-mintlify-docs-update.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Audio streaming

> Configure PulseAudio for game audio capture

## Overview

Hyperscape captures game music and sound effects for RTMP streams using PulseAudio virtual sinks. This allows viewers to hear the game audio alongside video.

## Architecture

```
Chrome (Game Audio) → PulseAudio Virtual Sink → FFmpeg Monitor → RTMP Stream
```

**Key Components:**

1. **PulseAudio** - Audio server with virtual sink
2. **chrome\_audio sink** - Virtual audio device for Chrome output
3. **chrome\_audio.monitor** - Monitor device for FFmpeg capture
4. **FFmpeg** - Captures from monitor and encodes to AAC

## PulseAudio Setup

### Installation

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Install PulseAudio
apt-get install -y pulseaudio pulseaudio-utils
```

### User Mode Configuration

**Why user mode:**

* More reliable than system mode
* Better permissions handling
* Easier debugging

**Setup:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Create runtime directory
export XDG_RUNTIME_DIR=/tmp/pulse-runtime
mkdir -p "$XDG_RUNTIME_DIR"
chmod 700 "$XDG_RUNTIME_DIR"

# Create PulseAudio config directory
mkdir -p /root/.config/pulse

# Create minimal config
cat > /root/.config/pulse/default.pa << 'EOF'
.fail
load-module module-null-sink sink_name=chrome_audio sink_properties=device.description="ChromeAudio"
set-default-sink chrome_audio
load-module module-native-protocol-unix auth-anonymous=1
EOF

# Start PulseAudio
pulseaudio --start --exit-idle-time=-1 --daemonize=yes
```

### Virtual Sink Creation

**Create chrome\_audio sink:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
pactl load-module module-null-sink \
  sink_name=chrome_audio \
  sink_properties=device.description="ChromeAudio"

# Set as default sink
pactl set-default-sink chrome_audio
```

**Verify sink exists:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
pactl list short sinks
# Should show: chrome_audio
```

### Permissions

**Add user to pulse-access group:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
usermod -aG pulse-access root
```

**Create /run/pulse directory:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
mkdir -p /run/pulse
chmod 777 /run/pulse
```

**Export PULSE\_SERVER:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
export PULSE_SERVER=unix:/tmp/pulse-runtime/pulse/native
```

## Chrome Configuration

### Audio Output

Chrome automatically uses the default PulseAudio sink when launched with `PULSE_SERVER` set:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Environment variables for Chrome
export PULSE_SERVER=unix:/tmp/pulse-runtime/pulse/native
export XDG_RUNTIME_DIR=/tmp/pulse-runtime

# Chrome will output audio to chrome_audio sink
```

**No additional Chrome flags needed** - PulseAudio integration is automatic.

## FFmpeg Configuration

### Audio Input

**Capture from PulseAudio monitor:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# FFmpeg args for audio capture
-thread_queue_size 1024 \
-use_wallclock_as_timestamps 1 \
-f pulse \
-ac 2 \
-ar 44100 \
-i chrome_audio.monitor
```

**Parameters:**

* `thread_queue_size 1024` - Prevent buffer underruns
* `use_wallclock_as_timestamps 1` - Real-time timing
* `-f pulse` - PulseAudio input format
* `-ac 2` - Stereo audio (2 channels)
* `-ar 44100` - Sample rate (44.1 kHz)
* `chrome_audio.monitor` - Monitor device (captures sink output)

### Audio Processing

**Async resampling for drift recovery:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Audio filter
-af aresample=async=1000:first_pts=0
```

**Why:**

* Recovers from audio drift when video/audio desync
* `async=1000` - Resample if drift exceeds 1000 samples (22ms at 44.1kHz)
* `first_pts=0` - Reset PTS to prevent timestamp issues

### Audio Encoding

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# AAC encoding
-c:a aac \
-b:a 128k \
-ar 44100 \
-flags +global_header
```

**Parameters:**

* `-c:a aac` - AAC codec (required for RTMP)
* `-b:a 128k` - Audio bitrate (128 kbps default)
* `-ar 44100` - Output sample rate
* `-flags +global_header` - Required for RTMP/FLV

## Environment Variables

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Enable audio capture
STREAM_AUDIO_ENABLED=true

# PulseAudio device (monitor of virtual sink)
PULSE_AUDIO_DEVICE=chrome_audio.monitor

# PulseAudio server socket
PULSE_SERVER=unix:/tmp/pulse-runtime/pulse/native

# Runtime directory for PulseAudio
XDG_RUNTIME_DIR=/tmp/pulse-runtime

# Audio bitrate (kbps)
STREAM_AUDIO_BITRATE_KBPS=128
```

## Graceful Fallback

If PulseAudio is not available, FFmpeg automatically falls back to silent audio:

```typescript theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
// From packages/server/src/streaming/rtmp-bridge.ts
let usePulseAudio = audioEnabled && process.platform === 'linux';

// Check if PulseAudio is accessible
if (usePulseAudio) {
  try {
    execSync('pactl info', { timeout: 2000, stdio: 'pipe' });
    const sinks = execSync('pactl list short sinks', { timeout: 2000, stdio: 'pipe' }).toString();
    if (!sinks.includes('chrome_audio')) {
      usePulseAudio = false;
    }
  } catch {
    usePulseAudio = false;
  }
}

if (usePulseAudio) {
  // Use PulseAudio
  args.push('-f', 'pulse', '-i', 'chrome_audio.monitor');
} else {
  // Fallback to silent audio
  args.push('-f', 'lavfi', '-i', 'anullsrc=r=44100:cl=stereo');
}
```

**Why silent fallback:**

* Many RTMP servers require an audio track
* Silent audio prevents stream rejection
* Better than failing the entire stream

## Troubleshooting

### PulseAudio Not Running

**Check status:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
pulseaudio --check
echo $?  # 0 = running, 1 = not running
```

**Start PulseAudio:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
pulseaudio --start --exit-idle-time=-1 --daemonize=yes
```

**Check logs:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
journalctl -u pulseaudio --no-pager -n 50
```

### chrome\_audio Sink Missing

**List sinks:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
pactl list short sinks
```

**Create sink manually:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
pactl load-module module-null-sink \
  sink_name=chrome_audio \
  sink_properties=device.description="ChromeAudio"
```

**Set as default:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
pactl set-default-sink chrome_audio
```

### No Audio in Stream

**Check FFmpeg is capturing:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Look for pulse input in FFmpeg logs
bunx pm2 logs hyperscape-duel | grep -i pulse
```

**Test audio capture:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Record 5 seconds of audio
ffmpeg -f pulse -i chrome_audio.monitor -t 5 test.wav

# Play back
ffplay test.wav
```

**Check Chrome is outputting audio:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Monitor PulseAudio activity
pactl subscribe

# Should show events when Chrome plays audio
```

### Audio Dropouts

**Symptoms:**

* Intermittent audio cutting out
* Audio desync from video
* Crackling or stuttering

**Fixes:**

**1. Increase buffer size:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
STREAM_AUDIO_BUFFER_SIZE=2048  # Default: 1024
```

**2. Check for buffer underruns:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# FFmpeg logs will show:
# [pulse @ 0x...] Thread message queue blocking; consider raising the thread_queue_size option
```

**3. Increase thread queue size:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# In rtmp-bridge.ts
-thread_queue_size 2048  # Increased from 1024
```

**4. Enable async resampling:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Already enabled by default
-af aresample=async=1000:first_pts=0
```

### Audio/Video Desync

**Symptoms:**

* Audio ahead or behind video
* Lip sync issues
* Audio drift over time

**Fixes:**

**1. Use wall clock timestamps:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Already enabled by default
-use_wallclock_as_timestamps 1
```

**2. Enable async resampling:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Resamples when drift exceeds 22ms
-af aresample=async=1000:first_pts=0
```

**3. Match video buffer size:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Video and audio should have same thread_queue_size
-thread_queue_size 1024  # Both video and audio
```

**4. Remove -shortest flag:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Don't use -shortest - causes audio dropouts during video buffering
# Removed in commit b9d2e41
```

### Permission Errors

**Error:**

```
Connection failure: Connection refused
pa_context_connect() failed: Connection refused
```

**Fix:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Check PULSE_SERVER is set
echo $PULSE_SERVER

# Check socket exists
ls -la /tmp/pulse-runtime/pulse/native

# Check permissions
chmod 700 /tmp/pulse-runtime
```

**Error:**

```
Access denied
```

**Fix:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Add user to pulse-access group
usermod -aG pulse-access root

# Create /run/pulse with proper permissions
mkdir -p /run/pulse
chmod 777 /run/pulse
```

## Audio Quality Settings

### Bitrate

**Default:** 128 kbps (good quality, low bandwidth)

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Low quality (save bandwidth)
STREAM_AUDIO_BITRATE_KBPS=64

# Medium quality (default)
STREAM_AUDIO_BITRATE_KBPS=128

# High quality (music-focused streams)
STREAM_AUDIO_BITRATE_KBPS=192
```

### Sample Rate

**Default:** 44.1 kHz (CD quality)

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Lower quality (save bandwidth)
STREAM_AUDIO_SAMPLE_RATE=22050

# Standard quality (default)
STREAM_AUDIO_SAMPLE_RATE=44100

# High quality (unnecessary for game audio)
STREAM_AUDIO_SAMPLE_RATE=48000
```

### Channels

**Default:** Stereo (2 channels)

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Mono (save bandwidth)
STREAM_AUDIO_CHANNELS=1

# Stereo (default)
STREAM_AUDIO_CHANNELS=2
```

## Monitoring

### Check PulseAudio Status

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Check if running
pulseaudio --check && echo "Running" || echo "Not running"

# Get info
pactl info

# List sinks
pactl list short sinks

# Monitor activity
pactl subscribe
```

### Check Audio Levels

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Monitor sink volume
pactl list sinks | grep -A 10 chrome_audio

# Check if audio is flowing
pactl list sink-inputs
```

### FFmpeg Audio Stats

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# FFmpeg logs show audio stats
bunx pm2 logs hyperscape-duel | grep -i "audio\|aac\|pulse"

# Look for:
# - Audio: aac (LC) 44100 Hz, stereo, fltp, 128 kb/s
# - Stream #0:1(und): Audio: aac
```

## Performance Impact

### CPU Usage

**PulseAudio:** \~1-2% CPU
**FFmpeg AAC encoding:** \~5-10% CPU per stream

**Total audio overhead:** \~10-15% CPU

### Memory Usage

**PulseAudio:** \~50-100 MB RAM
**FFmpeg audio buffers:** \~10-20 MB RAM

**Total audio overhead:** \~100-150 MB RAM

### Bandwidth

**Audio bitrate:** 128 kbps = 16 KB/s = \~1 MB/minute

**Impact on total stream:**

* Video: 4500 kbps (default)
* Audio: 128 kbps
* **Total:** 4628 kbps (\~3% overhead)

## Disabling Audio

To disable audio capture and use silent audio:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
STREAM_AUDIO_ENABLED=false
```

FFmpeg will use silent audio source:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
-f lavfi -i anullsrc=r=44100:cl=stereo
```

**When to disable:**

* Debugging audio issues
* Reducing CPU usage
* Testing video-only streams
* Copyright concerns (music in game)

## Advanced Configuration

### Custom PulseAudio Config

Create custom `~/.config/pulse/default.pa`:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Load virtual sink
load-module module-null-sink \
  sink_name=chrome_audio \
  sink_properties=device.description="ChromeAudio"

# Set as default
set-default-sink chrome_audio

# Enable network protocol (optional)
load-module module-native-protocol-unix auth-anonymous=1

# Load other modules as needed
# load-module module-echo-cancel
# load-module module-equalizer-sink
```

### Audio Filters

**Normalize audio levels:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
-af "loudnorm=I=-16:TP=-1.5:LRA=11,aresample=async=1000:first_pts=0"
```

**Reduce noise:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
-af "highpass=f=200,lowpass=f=3000,aresample=async=1000:first_pts=0"
```

**Compress dynamic range:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
-af "acompressor=threshold=-20dB:ratio=4:attack=5:release=50,aresample=async=1000:first_pts=0"
```

### Multiple Audio Sources

**Mix game audio + microphone:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Game audio from PulseAudio
-f pulse -i chrome_audio.monitor

# Microphone input
-f pulse -i default

# Mix both sources
-filter_complex "[0:a][1:a]amix=inputs=2:duration=longest"
```

## Deployment Integration

### Vast.ai Deployment

The `deploy-vast.sh` script automatically configures PulseAudio:

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# From scripts/deploy-vast.sh (lines 150-200)

# Kill any existing PulseAudio
pulseaudio --kill 2>/dev/null || true
pkill -9 pulseaudio 2>/dev/null || true

# Setup runtime directory
export XDG_RUNTIME_DIR=/tmp/pulse-runtime
mkdir -p "$XDG_RUNTIME_DIR"
chmod 700 "$XDG_RUNTIME_DIR"

# Create config
mkdir -p /root/.config/pulse
cat > /root/.config/pulse/default.pa << 'EOF'
.fail
load-module module-null-sink sink_name=chrome_audio sink_properties=device.description="ChromeAudio"
set-default-sink chrome_audio
load-module module-native-protocol-unix auth-anonymous=1
EOF

# Start PulseAudio
pulseaudio --start --exit-idle-time=-1 --daemonize=yes

# Verify sink exists
pactl list short sinks | grep chrome_audio || \
  pactl load-module module-null-sink sink_name=chrome_audio
```

### PM2 Environment

**Export for PM2:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# From ecosystem.config.cjs
env: {
  PULSE_SERVER: 'unix:/tmp/pulse-runtime/pulse/native',
  XDG_RUNTIME_DIR: '/tmp/pulse-runtime',
  STREAM_AUDIO_ENABLED: 'true',
  PULSE_AUDIO_DEVICE: 'chrome_audio.monitor'
}
```

## Audio Stability Improvements

### Buffer Configuration (Commit b9d2e41)

**Three key changes to prevent audio dropouts:**

**1. Buffer both audio and video adequately:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Audio input buffering
-thread_queue_size 1024  # Prevents buffer underruns

# Video input buffering (increased for a/v sync)
-thread_queue_size 1024  # Increased from 512
```

**2. Use wall clock timestamps:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Maintains real-time timing for PulseAudio
-use_wallclock_as_timestamps 1
```

**3. Async resampling for drift recovery:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Resyncs audio when drift exceeds 22ms
-af aresample=async=1000:first_pts=0
```

**4. Remove -shortest flag:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Don't use -shortest - causes audio dropouts during video buffering
# Removed in commit b9d2e41
```

### Timing Synchronization

**Problem:** Audio and video can drift over time, causing desync.

**Solution:** Three-layer sync strategy:

**1. Wall clock timestamps:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
-use_wallclock_as_timestamps 1
```

Ensures PulseAudio uses real-time clock for timestamps.

**2. Async resampling:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
-af aresample=async=1000:first_pts=0
```

Resamples audio when drift exceeds threshold.

**3. Matched buffer sizes:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Both video and audio use same buffer size
-thread_queue_size 1024
```

Prevents one stream from getting ahead of the other.

## Testing

### Test Audio Capture

**Record 5 seconds:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
ffmpeg -f pulse -i chrome_audio.monitor -t 5 test.wav
```

**Play back:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
ffplay test.wav
```

**Check for audio:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Should show audio waveform
ffmpeg -i test.wav -filter_complex showwavespic=s=1280x720 -frames:v 1 waveform.png
```

### Test Full Pipeline

**1. Start PulseAudio:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
pulseaudio --start
```

**2. Play test audio to sink:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
paplay --device=chrome_audio /usr/share/sounds/alsa/Front_Center.wav
```

**3. Capture from monitor:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
ffmpeg -f pulse -i chrome_audio.monitor -t 5 test.wav
```

**4. Verify audio was captured:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
ffplay test.wav
```

### Monitor Live Stream

**Check FFmpeg audio stats:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
bunx pm2 logs hyperscape-duel | grep -E "Audio:|aac|pulse"
```

**Expected output:**

```
Stream #0:1(und): Audio: aac (LC), 44100 Hz, stereo, fltp, 128 kb/s
```

## Common Issues

### No Audio in Stream

**Checklist:**

1. ✅ PulseAudio running: `pulseaudio --check`
2. ✅ chrome\_audio sink exists: `pactl list short sinks | grep chrome_audio`
3. ✅ PULSE\_SERVER set: `echo $PULSE_SERVER`
4. ✅ FFmpeg using pulse input: `pm2 logs | grep pulse`
5. ✅ Chrome outputting audio: `pactl list sink-inputs`

**Debug:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Check PulseAudio logs
journalctl -u pulseaudio -n 50

# Check FFmpeg logs
bunx pm2 logs hyperscape-duel --lines 100 | grep -i audio

# Test capture manually
ffmpeg -f pulse -i chrome_audio.monitor -t 5 test.wav
```

### Audio Crackling

**Causes:**

* Buffer underruns
* CPU overload
* Sample rate mismatch

**Fixes:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Increase buffer size
-thread_queue_size 2048

# Lower audio quality
STREAM_AUDIO_BITRATE_KBPS=96

# Check CPU usage
top -p $(pgrep ffmpeg)
```

### Audio Ahead of Video

**Cause:** Audio processing faster than video encoding

**Fix:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Increase video buffer to match audio
-thread_queue_size 1024  # Both video and audio

# Use async resampling
-af aresample=async=1000:first_pts=0
```

### Permission Denied

**Error:**

```
pa_context_connect() failed: Access denied
```

**Fix:**

```bash theme={"theme":{"light":"github-light","dark":"tokyo-night"}}
# Check user is in pulse-access group
groups root | grep pulse-access

# Add if missing
usermod -aG pulse-access root

# Restart PulseAudio
pulseaudio --kill
pulseaudio --start
```

## Related Documentation

* [GPU Rendering](/devops/gpu-rendering) - GPU configuration
* [Deployment](/guides/deployment) - Full deployment guide
* [Configuration](/devops/configuration) - Environment variables