Monitoring & Diagnostics

Health Checking

The lyrebird-diagnostics.sh script provides three diagnostic modes with increasing depth:

Diagnostic Exit Codes

Orchestrator Integration

Diagnostics are accessible from the Orchestrator menu without remembering commands:

• Main Menu → 5 → 1: Quick health check
• Main Menu → 5 → 2: Full diagnostic
• Main Menu → 5 → 3: Debug diagnostic

What Gets Checked

Monitoring Best Practices

Recommended Schedule

# Add to /etc/cron.d/lyrebird-monitoring

# Quick check every 5 minutes (via stream manager's install command)
*/5 * * * * root /path/to/lyrebird-stream-manager.sh monitor

# Full diagnostic daily at 6 AM
0 6 * * * root /path/to/lyrebird-diagnostics.sh full >> /var/log/lyrebird-daily-check.log 2>&1

# Storage cleanup daily at 3 AM
0 3 * * * root /path/to/lyrebird-storage.sh cleanup --quiet

Log Rotation

Install the provided logrotate configuration to prevent log files from filling the disk:

# Install log rotation
sudo cp config/lyrebird-logrotate.conf /etc/logrotate.d/lyrebird

# Test configuration
sudo logrotate -d /etc/logrotate.d/lyrebird

# Force rotation (for testing)
sudo logrotate -f /etc/logrotate.d/lyrebird

Prometheus Metrics

Export metrics for Grafana dashboards and alerting:

# Start HTTP metrics server
./lyrebird-metrics.sh serve --port 9100

# Or use as a textfile collector for node_exporter
./lyrebird-metrics.sh > /var/lib/node_exporter/textfile_collector/lyrebird.prom

Grafana Dashboard Setup

1. Start the metrics server on port 9100
2. Add a Prometheus scrape job pointing to your server
3. Import a dashboard in Grafana using the lyrebird_* metrics

Key metrics to dashboard:

• lyrebird_stream_active — Alert when 0
• lyrebird_mediamtx_up — Alert when 0
• lyrebird_disk_usage_percent — Alert when > 80%
• lyrebird_cpu_usage_percent — Alert when > 80%
• lyrebird_usb_device_connected — Alert when 0

Webhook Alerting

Setup

# Interactive webhook setup
./lyrebird-alerts.sh setup

# Test your webhook
./lyrebird-alerts.sh test

What Events Trigger Alerts

• Stream goes down or comes back up
• USB device disconnects or reconnects
• Disk usage exceeds warning/critical thresholds
• MediaMTX service goes down or recovers
• Network connectivity lost or restored

Rate Limiting

Alerts are rate-limited by default to prevent spam. Duplicate alerts within the configured window (default: 300 seconds) are suppressed. Adjust in /etc/lyrebird/alerts.conf:

LYREBIRD_ALERT_RATE_LIMIT=300  # Seconds between duplicate alerts

Performance & Optimization

Codec Selection

Sample Rate Selection

Bitrate Tuning

Resource Usage Estimates

Thread Queue Tuning

For high stream counts (4+), increase the thread queue size to reduce buffer underruns:

# In /etc/mediamtx/audio-devices.conf
DEFAULT_THREAD_QUEUE=16384  # Default is 8192

System Tuning

File Descriptor Limits

# Check current limits
ulimit -n

# Increase limits (add to /etc/security/limits.conf)
echo "* soft nofile 8192" | sudo tee -a /etc/security/limits.conf
echo "* hard nofile 16384" | sudo tee -a /etc/security/limits.conf
echo "mediamtx soft nofile 4096" | sudo tee -a /etc/security/limits.conf
echo "mediamtx hard nofile 8192" | sudo tee -a /etc/security/limits.conf

# Apply (requires reboot)
sudo reboot

Network Buffer Sizes

# Increase UDP buffer sizes for streaming
sudo sysctl -w net.core.rmem_max=26214400
sudo sysctl -w net.core.wmem_max=26214400

# Make permanent
echo "net.core.rmem_max=26214400" | sudo tee -a /etc/sysctl.conf
echo "net.core.wmem_max=26214400" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

USB Power Management

# Disable USB auto-suspend (prevents devices going to sleep)
echo -1 | sudo tee /sys/bus/usb/devices/*/power/autosuspend

# Make permanent via udev rule
echo 'ACTION=="add", SUBSYSTEM=="usb", ATTR{idVendor}=="YOUR_VID", ATTR{power/autosuspend}="-1"' \
  | sudo tee /etc/udev/rules.d/50-usb-power.rules

Raspberry Pi Optimization

# Use low-quality settings for Raspberry Pi
sudo ./lyrebird-mic-check.sh -g --quality=low

# Consider mono recording to halve bandwidth
DEFAULT_CHANNELS=1

# Reduce stream count to 1-2 maximum

Raspberry Pi Recommended Settings

# /etc/mediamtx/audio-devices.conf (Raspberry Pi)
DEFAULT_SAMPLE_RATE=16000    # or 44100 at most
DEFAULT_CHANNELS=1           # Mono to reduce CPU
DEFAULT_BITRATE=64k          # Low bitrate
DEFAULT_CODEC=opus
DEFAULT_THREAD_QUEUE=4096    # Smaller queue to reduce memory