Audio streaming - Hyperscape

Overview

Hyperscape captures game music and sound effects for RTMP streams using PulseAudio virtual sinks. This allows viewers to hear the game audio alongside video.

Architecture

Chrome (Game Audio) → PulseAudio Virtual Sink → FFmpeg Monitor → RTMP Stream

Key Components:

PulseAudio - Audio server with virtual sink
chrome_audio sink - Virtual audio device for Chrome output
chrome_audio.monitor - Monitor device for FFmpeg capture
FFmpeg - Captures from monitor and encodes to AAC

PulseAudio Setup

Installation

# Install PulseAudio
apt-get install -y pulseaudio pulseaudio-utils

User Mode Configuration

Why user mode:

More reliable than system mode
Better permissions handling
Easier debugging

Setup:

# Create runtime directory
export XDG_RUNTIME_DIR=/tmp/pulse-runtime
mkdir -p "$XDG_RUNTIME_DIR"
chmod 700 "$XDG_RUNTIME_DIR"

# Create PulseAudio config directory
mkdir -p /root/.config/pulse

# Create minimal config
cat > /root/.config/pulse/default.pa << 'EOF'
.fail
load-module module-null-sink sink_name=chrome_audio sink_properties=device.description="ChromeAudio"
set-default-sink chrome_audio
load-module module-native-protocol-unix auth-anonymous=1
EOF

# Start PulseAudio
pulseaudio --start --exit-idle-time=-1 --daemonize=yes

Virtual Sink Creation

Create chrome_audio sink:

pactl load-module module-null-sink \
  sink_name=chrome_audio \
  sink_properties=device.description="ChromeAudio"

# Set as default sink
pactl set-default-sink chrome_audio

Verify sink exists:

pactl list short sinks
# Should show: chrome_audio

Permissions

Add user to pulse-access group:

usermod -aG pulse-access root

Create /run/pulse directory:

mkdir -p /run/pulse
chmod 777 /run/pulse

Export PULSE_SERVER:

export PULSE_SERVER=unix:/tmp/pulse-runtime/pulse/native

Chrome Configuration

Audio Output

Chrome automatically uses the default PulseAudio sink when launched with PULSE_SERVER set:

# Environment variables for Chrome
export PULSE_SERVER=unix:/tmp/pulse-runtime/pulse/native
export XDG_RUNTIME_DIR=/tmp/pulse-runtime

# Chrome will output audio to chrome_audio sink

No additional Chrome flags needed - PulseAudio integration is automatic.

FFmpeg Configuration

Audio Input

Capture from PulseAudio monitor:

# FFmpeg args for audio capture
-thread_queue_size 1024 \
-use_wallclock_as_timestamps 1 \
-f pulse \
-ac 2 \
-ar 44100 \
-i chrome_audio.monitor

Parameters:

thread_queue_size 1024 - Prevent buffer underruns
use_wallclock_as_timestamps 1 - Real-time timing
-f pulse - PulseAudio input format
-ac 2 - Stereo audio (2 channels)
-ar 44100 - Sample rate (44.1 kHz)
chrome_audio.monitor - Monitor device (captures sink output)

Audio Processing

Async resampling for drift recovery:

# Audio filter
-af aresample=async=1000:first_pts=0

Why:

Recovers from audio drift when video/audio desync
async=1000 - Resample if drift exceeds 1000 samples (22ms at 44.1kHz)
first_pts=0 - Reset PTS to prevent timestamp issues

Audio Encoding

# AAC encoding
-c:a aac \
-b:a 128k \
-ar 44100 \
-flags +global_header

Parameters:

-c:a aac - AAC codec (required for RTMP)
-b:a 128k - Audio bitrate (128 kbps default)
-ar 44100 - Output sample rate
-flags +global_header - Required for RTMP/FLV

Environment Variables

# Enable audio capture
STREAM_AUDIO_ENABLED=true

# PulseAudio device (monitor of virtual sink)
PULSE_AUDIO_DEVICE=chrome_audio.monitor

# PulseAudio server socket
PULSE_SERVER=unix:/tmp/pulse-runtime/pulse/native

# Runtime directory for PulseAudio
XDG_RUNTIME_DIR=/tmp/pulse-runtime

# Audio bitrate (kbps)
STREAM_AUDIO_BITRATE_KBPS=128

Graceful Fallback

If PulseAudio is not available, FFmpeg automatically falls back to silent audio:

// From packages/server/src/streaming/rtmp-bridge.ts
let usePulseAudio = audioEnabled && process.platform === 'linux';

// Check if PulseAudio is accessible
if (usePulseAudio) {
  try {
    execSync('pactl info', { timeout: 2000, stdio: 'pipe' });
    const sinks = execSync('pactl list short sinks', { timeout: 2000, stdio: 'pipe' }).toString();
    if (!sinks.includes('chrome_audio')) {
      usePulseAudio = false;
    }
  } catch {
    usePulseAudio = false;
  }
}

if (usePulseAudio) {
  // Use PulseAudio
  args.push('-f', 'pulse', '-i', 'chrome_audio.monitor');
} else {
  // Fallback to silent audio
  args.push('-f', 'lavfi', '-i', 'anullsrc=r=44100:cl=stereo');
}

Why silent fallback:

Many RTMP servers require an audio track
Silent audio prevents stream rejection
Better than failing the entire stream

Troubleshooting

PulseAudio Not Running

Check status:

pulseaudio --check
echo $?  # 0 = running, 1 = not running

Start PulseAudio:

pulseaudio --start --exit-idle-time=-1 --daemonize=yes

Check logs:

journalctl -u pulseaudio --no-pager -n 50

chrome_audio Sink Missing

List sinks:

pactl list short sinks

Create sink manually:

pactl load-module module-null-sink \
  sink_name=chrome_audio \
  sink_properties=device.description="ChromeAudio"

Set as default:

pactl set-default-sink chrome_audio

No Audio in Stream

Check FFmpeg is capturing:

# Look for pulse input in FFmpeg logs
bunx pm2 logs hyperscape-duel | grep -i pulse

Test audio capture:

# Record 5 seconds of audio
ffmpeg -f pulse -i chrome_audio.monitor -t 5 test.wav

# Play back
ffplay test.wav

Check Chrome is outputting audio:

# Monitor PulseAudio activity
pactl subscribe

# Should show events when Chrome plays audio

Audio Dropouts

Symptoms:

Intermittent audio cutting out
Audio desync from video
Crackling or stuttering

Fixes: 1. Increase buffer size:

STREAM_AUDIO_BUFFER_SIZE=2048  # Default: 1024

2. Check for buffer underruns:

# FFmpeg logs will show:
# [pulse @ 0x...] Thread message queue blocking; consider raising the thread_queue_size option

3. Increase thread queue size:

# In rtmp-bridge.ts
-thread_queue_size 2048  # Increased from 1024

4. Enable async resampling:

# Already enabled by default
-af aresample=async=1000:first_pts=0

Audio/Video Desync

Symptoms:

Audio ahead or behind video
Lip sync issues
Audio drift over time

Fixes: 1. Use wall clock timestamps:

# Already enabled by default
-use_wallclock_as_timestamps 1

2. Enable async resampling:

# Resamples when drift exceeds 22ms
-af aresample=async=1000:first_pts=0

3. Match video buffer size:

# Video and audio should have same thread_queue_size
-thread_queue_size 1024  # Both video and audio

4. Remove -shortest flag:

# Don't use -shortest - causes audio dropouts during video buffering
# Removed in commit b9d2e41

Permission Errors

Error:

Connection failure: Connection refused
pa_context_connect() failed: Connection refused

Fix:

# Check PULSE_SERVER is set
echo $PULSE_SERVER

# Check socket exists
ls -la /tmp/pulse-runtime/pulse/native

# Check permissions
chmod 700 /tmp/pulse-runtime

Error:

Access denied

Fix:

# Add user to pulse-access group
usermod -aG pulse-access root

# Create /run/pulse with proper permissions
mkdir -p /run/pulse
chmod 777 /run/pulse

Audio Quality Settings

Bitrate

Default: 128 kbps (good quality, low bandwidth)

# Low quality (save bandwidth)
STREAM_AUDIO_BITRATE_KBPS=64

# Medium quality (default)
STREAM_AUDIO_BITRATE_KBPS=128

# High quality (music-focused streams)
STREAM_AUDIO_BITRATE_KBPS=192

Sample Rate

Default: 44.1 kHz (CD quality)

# Lower quality (save bandwidth)
STREAM_AUDIO_SAMPLE_RATE=22050

# Standard quality (default)
STREAM_AUDIO_SAMPLE_RATE=44100

# High quality (unnecessary for game audio)
STREAM_AUDIO_SAMPLE_RATE=48000

Channels

Default: Stereo (2 channels)

# Mono (save bandwidth)
STREAM_AUDIO_CHANNELS=1

# Stereo (default)
STREAM_AUDIO_CHANNELS=2

Monitoring

Check PulseAudio Status

# Check if running
pulseaudio --check && echo "Running" || echo "Not running"

# Get info
pactl info

# List sinks
pactl list short sinks

# Monitor activity
pactl subscribe

Check Audio Levels

# Monitor sink volume
pactl list sinks | grep -A 10 chrome_audio

# Check if audio is flowing
pactl list sink-inputs

FFmpeg Audio Stats

# FFmpeg logs show audio stats
bunx pm2 logs hyperscape-duel | grep -i "audio\|aac\|pulse"

# Look for:
# - Audio: aac (LC) 44100 Hz, stereo, fltp, 128 kb/s
# - Stream #0:1(und): Audio: aac

Performance Impact

CPU Usage

PulseAudio: ~1-2% CPU FFmpeg AAC encoding: ~5-10% CPU per stream Total audio overhead: ~10-15% CPU

Memory Usage

PulseAudio: ~50-100 MB RAM FFmpeg audio buffers: ~10-20 MB RAM Total audio overhead: ~100-150 MB RAM

Bandwidth

Audio bitrate: 128 kbps = 16 KB/s = ~1 MB/minute Impact on total stream:

Video: 4500 kbps (default)
Audio: 128 kbps
Total: 4628 kbps (~3% overhead)

Disabling Audio

To disable audio capture and use silent audio:

STREAM_AUDIO_ENABLED=false

FFmpeg will use silent audio source:

-f lavfi -i anullsrc=r=44100:cl=stereo

When to disable:

Debugging audio issues
Reducing CPU usage
Testing video-only streams
Copyright concerns (music in game)

Advanced Configuration

Custom PulseAudio Config

Create custom ~/.config/pulse/default.pa:

# Load virtual sink
load-module module-null-sink \
  sink_name=chrome_audio \
  sink_properties=device.description="ChromeAudio"

# Set as default
set-default-sink chrome_audio

# Enable network protocol (optional)
load-module module-native-protocol-unix auth-anonymous=1

# Load other modules as needed
# load-module module-echo-cancel
# load-module module-equalizer-sink

Audio Filters

Normalize audio levels:

-af "loudnorm=I=-16:TP=-1.5:LRA=11,aresample=async=1000:first_pts=0"

Reduce noise:

-af "highpass=f=200,lowpass=f=3000,aresample=async=1000:first_pts=0"

Compress dynamic range:

-af "acompressor=threshold=-20dB:ratio=4:attack=5:release=50,aresample=async=1000:first_pts=0"

Multiple Audio Sources

Mix game audio + microphone:

# Game audio from PulseAudio
-f pulse -i chrome_audio.monitor

# Microphone input
-f pulse -i default

# Mix both sources
-filter_complex "[0:a][1:a]amix=inputs=2:duration=longest"

Deployment Integration

Vast.ai Deployment

The deploy-vast.sh script automatically configures PulseAudio:

# From scripts/deploy-vast.sh (lines 150-200)

# Kill any existing PulseAudio
pulseaudio --kill 2>/dev/null || true
pkill -9 pulseaudio 2>/dev/null || true

# Setup runtime directory
export XDG_RUNTIME_DIR=/tmp/pulse-runtime
mkdir -p "$XDG_RUNTIME_DIR"
chmod 700 "$XDG_RUNTIME_DIR"

# Create config
mkdir -p /root/.config/pulse
cat > /root/.config/pulse/default.pa << 'EOF'
.fail
load-module module-null-sink sink_name=chrome_audio sink_properties=device.description="ChromeAudio"
set-default-sink chrome_audio
load-module module-native-protocol-unix auth-anonymous=1
EOF

# Start PulseAudio
pulseaudio --start --exit-idle-time=-1 --daemonize=yes

# Verify sink exists
pactl list short sinks | grep chrome_audio || \
  pactl load-module module-null-sink sink_name=chrome_audio

PM2 Environment

Export for PM2:

# From ecosystem.config.cjs
env: {
  PULSE_SERVER: 'unix:/tmp/pulse-runtime/pulse/native',
  XDG_RUNTIME_DIR: '/tmp/pulse-runtime',
  STREAM_AUDIO_ENABLED: 'true',
  PULSE_AUDIO_DEVICE: 'chrome_audio.monitor'
}

Audio Stability Improvements

Buffer Configuration (Commit b9d2e41)

Three key changes to prevent audio dropouts: 1. Buffer both audio and video adequately:

# Audio input buffering
-thread_queue_size 1024  # Prevents buffer underruns

# Video input buffering (increased for a/v sync)
-thread_queue_size 1024  # Increased from 512

2. Use wall clock timestamps:

# Maintains real-time timing for PulseAudio
-use_wallclock_as_timestamps 1

3. Async resampling for drift recovery:

# Resyncs audio when drift exceeds 22ms
-af aresample=async=1000:first_pts=0

4. Remove -shortest flag:

# Don't use -shortest - causes audio dropouts during video buffering
# Removed in commit b9d2e41

Timing Synchronization

Problem: Audio and video can drift over time, causing desync. Solution: Three-layer sync strategy: 1. Wall clock timestamps:

-use_wallclock_as_timestamps 1

Ensures PulseAudio uses real-time clock for timestamps. 2. Async resampling:

-af aresample=async=1000:first_pts=0

Resamples audio when drift exceeds threshold. 3. Matched buffer sizes:

# Both video and audio use same buffer size
-thread_queue_size 1024

Prevents one stream from getting ahead of the other.

Testing

Test Audio Capture

Record 5 seconds:

ffmpeg -f pulse -i chrome_audio.monitor -t 5 test.wav

Play back:

ffplay test.wav

Check for audio:

# Should show audio waveform
ffmpeg -i test.wav -filter_complex showwavespic=s=1280x720 -frames:v 1 waveform.png

Test Full Pipeline

1. Start PulseAudio:

pulseaudio --start

2. Play test audio to sink:

paplay --device=chrome_audio /usr/share/sounds/alsa/Front_Center.wav

3. Capture from monitor:

ffmpeg -f pulse -i chrome_audio.monitor -t 5 test.wav

4. Verify audio was captured:

ffplay test.wav

Monitor Live Stream

Check FFmpeg audio stats:

bunx pm2 logs hyperscape-duel | grep -E "Audio:|aac|pulse"

Expected output:

Stream #0:1(und): Audio: aac (LC), 44100 Hz, stereo, fltp, 128 kb/s

Common Issues

No Audio in Stream

Checklist:

✅ PulseAudio running: pulseaudio --check
✅ chrome_audio sink exists: pactl list short sinks | grep chrome_audio
✅ PULSE_SERVER set: echo $PULSE_SERVER
✅ FFmpeg using pulse input: pm2 logs | grep pulse
✅ Chrome outputting audio: pactl list sink-inputs

Debug:

# Check PulseAudio logs
journalctl -u pulseaudio -n 50

# Check FFmpeg logs
bunx pm2 logs hyperscape-duel --lines 100 | grep -i audio

# Test capture manually
ffmpeg -f pulse -i chrome_audio.monitor -t 5 test.wav

Audio Crackling

Causes:

Buffer underruns
CPU overload
Sample rate mismatch

Fixes:

# Increase buffer size
-thread_queue_size 2048

# Lower audio quality
STREAM_AUDIO_BITRATE_KBPS=96

# Check CPU usage
top -p $(pgrep ffmpeg)

Audio Ahead of Video

Cause: Audio processing faster than video encoding Fix:

# Increase video buffer to match audio
-thread_queue_size 1024  # Both video and audio

# Use async resampling
-af aresample=async=1000:first_pts=0

Permission Denied

Error:

pa_context_connect() failed: Access denied

Fix:

# Check user is in pulse-access group
groups root | grep pulse-access

# Add if missing
usermod -aG pulse-access root

# Restart PulseAudio
pulseaudio --kill
pulseaudio --start

GPU Rendering - GPU configuration
Deployment - Full deployment guide
Configuration - Environment variables

​Overview

​Architecture

​PulseAudio Setup

​Installation

​User Mode Configuration

​Virtual Sink Creation

​Permissions

​Chrome Configuration

​Audio Output

​FFmpeg Configuration

​Audio Input

​Audio Processing

​Audio Encoding

​Environment Variables

​Graceful Fallback

​Troubleshooting

​PulseAudio Not Running

​chrome_audio Sink Missing

​No Audio in Stream

​Audio Dropouts

​Audio/Video Desync

​Permission Errors

​Audio Quality Settings

​Bitrate

​Sample Rate

​Channels

​Monitoring

​Check PulseAudio Status

​Check Audio Levels

​FFmpeg Audio Stats

​Performance Impact

​CPU Usage

​Memory Usage

​Bandwidth

​Disabling Audio

​Advanced Configuration

​Custom PulseAudio Config

​Audio Filters

​Multiple Audio Sources

​Deployment Integration

​Vast.ai Deployment

​PM2 Environment

​Audio Stability Improvements

​Buffer Configuration (Commit b9d2e41)

​Timing Synchronization

​Testing

​Test Audio Capture

​Test Full Pipeline

​Monitor Live Stream

​Common Issues

​No Audio in Stream

​Audio Crackling

​Audio Ahead of Video

​Permission Denied

​Related Documentation

Overview

Architecture

PulseAudio Setup

Installation

User Mode Configuration

Virtual Sink Creation

Permissions

Chrome Configuration

Audio Output

FFmpeg Configuration

Audio Input

Audio Processing

Audio Encoding

Environment Variables

Graceful Fallback

Troubleshooting

PulseAudio Not Running

chrome_audio Sink Missing

No Audio in Stream

Audio Dropouts

Audio/Video Desync

Permission Errors

Audio Quality Settings

Bitrate

Sample Rate

Channels

Monitoring

Check PulseAudio Status

Check Audio Levels

FFmpeg Audio Stats

Performance Impact

CPU Usage

Memory Usage

Bandwidth

Disabling Audio

Advanced Configuration

Custom PulseAudio Config

Audio Filters

Multiple Audio Sources

Deployment Integration

Vast.ai Deployment

PM2 Environment

Audio Stability Improvements

Buffer Configuration (Commit b9d2e41)

Timing Synchronization

Testing

Test Audio Capture

Test Full Pipeline

Monitor Live Stream

Common Issues

No Audio in Stream

Audio Crackling

Audio Ahead of Video

Permission Denied

Related Documentation