Dymas/jarri-spooty
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Document deterministic yt-dlp hardening and Docker persistence

Browse Source
This commit is contained in:
Tor Wingalen
2026-05-15 21:07:01 +02:00
parent 48517a5a85
commit 5f4d7c9c4e
2 changed files with 221 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+54 -35
View File
@@ -10,37 +10,60 @@ This project follows a practical chronological changelog rather than autogenerat
## Added ## Added
### Spotify OAuth Integration ### Spotify
- Added Spotify OAuth login flow - Added Spotify OAuth login flow
- Added `/api/spotify/login` - Added `/api/spotify/login`
- Added `/api/spotify/callback` - Added `/api/spotify/callback`
- Added `/api/spotify/status` - Added `/api/spotify/status`
- Added persistent Spotify user access token storage - Added persistent Spotify user token storage
- Added authenticated Spotify playlist retrieval - Added authenticated Spotify playlist retrieval
- Added OAuth frontend bootstrap flow - Added OAuth frontend bootstrap flow
- Added support for playlists exceeding 300 tracks
- Added deterministic playlist pagination handling
### Large Playlist Support ### YouTube
- Fixed Spotify playlist truncation at 100 tracks - Added deterministic YouTube candidate scoring
- Added full Spotify pagination support - Added duration-aware YouTube matching
- Verified support for playlists exceeding 300 tracks - Added failed-candidate rejection memory
- Improved playlist queue handling stability - Added automatic retry/fallback handling
- Added explicit YouTube candidate exclusion support
- Added detailed YouTube candidate debug logging
- Added explicit yt-dlp CLI execution
- Added deterministic yt-dlp stderr capture
- Added age-gated video detection
- Added invalid-format detection
- Added Docker-compatible cookie handling
- Added temporary cookie-copy isolation
- Added retry-aware queue coordination
### Queue Hardening ### Queue Hardening
- Added configurable YouTube search pacing - Added configurable YouTube search pacing
- Added configurable YouTube download pacing - Added configurable YouTube download pacing
- Added sequential search queue behavior - Added sequential queue coordination
- Added retry collision prevention
- Added deterministic queue replacement handling
- Added explicit retry-state transitions
- Reduced aggressive request bursts toward YouTube - Reduced aggressive request bursts toward YouTube
- Improved queue logging visibility
### Docker
- Added persistent SQLite bind-mount recommendations
- Added persistent runtime config recommendations
- Added yt-dlp installation to runtime container
- Improved Docker runtime observability
- Improved hardened Docker startup flow
- Improved deployment documentation
### Security ### Security
- Added token-based frontend access mode - Added token-based frontend access mode
- Added external secret handling recommendations - Added external secret handling recommendations
- Added isolated Spotify credential storage - Added isolated Spotify credential storage
- Improved Docker credential handling documentation - Added safer cookie handling flow
- Improved Docker secret isolation documentation
### Frontend ### Frontend
@@ -49,37 +72,33 @@ This project follows a practical chronological changelog rather than autogenerat
- Improved playlist rendering behavior - Improved playlist rendering behavior
- Improved large playlist progress tracking - Improved large playlist progress tracking
### Docker ## Improved
- Improved hardened Docker runtime flow - Improved YouTube throttling resistance
- Added explicit environment variable examples - Improved age-gated video recovery behavior
- Added safer default pacing recommendations - Improved operational observability
- Improved deployment documentation - Improved cover-art embedding resilience
- Improved queue stability during large playlist imports
- Improved deterministic runtime behavior
- Improved Docker persistence handling
## Unreleased ## Fixed
### Added - Fixed Spotify playlist truncation at 100 tracks
- Spotify OAuth login flow - Fixed incomplete playlist retrieval behavior
- Persistent Spotify user token handling - Fixed repeated retry loops against failed YouTube videos
- Large playlist pagination support (>100 tracks) - Fixed queue deadlocks during retries
- Deterministic YouTube candidate scoring - Fixed retry-button queue collisions
- Duration-aware YouTube matching - Fixed container database loss on restart
- Detailed YouTube candidate debug logging - Fixed yt-dlp wrapper opacity issues
- Improved Docker deployment guidance - Fixed hidden yt-dlp stderr failures
- Fixed multiple age-gated fallback failure paths
### Improved - Fixed invalid YouTube candidate reuse
- YouTube pacing and throttling resistance - Fixed several Docker persistence edge cases
- Cover art embedding resilience
- Queue stability during large playlist imports
### Fixed
- Spotify playlist truncation at 100 tracks
- Incorrect live/acoustic YouTube selections
- Playlist retrieval failures on large public playlists
--- ---
# 2.4.2 # 2.4.2 (Original Spooty Endpoint)
## Fixed ## Fixed
README.md
+167 -34
View File
@@ -14,6 +14,13 @@ This hardened branch focuses on:
- Safer credential handling - Safer credential handling
- More resilient cover-art embedding - More resilient cover-art embedding
- Better queue stability - Better queue stability
- Deterministic YouTube fallback handling
- Explicit yt-dlp CLI execution
- Automatic failed-candidate rejection
- Improved age-gated video handling
- Persistent SQLite state across container restarts
- Deterministic Docker config persistence
- Improved operational observability
--- ---
@@ -30,6 +37,9 @@ This hardened branch focuses on:
- Large playlist pagination support - Large playlist pagination support
- Download pacing controls - Download pacing controls
- YouTube cookie support - YouTube cookie support
- Automatic YouTube retry/fallback handling
- Failed YouTube candidate rejection memory
- Deterministic yt-dlp error classification
--- ---
@@ -80,7 +90,7 @@ Copy:
Create a secure env file: Create a secure env file:
~~~bash ```bash
sudo mkdir -p /etc/tokens sudo mkdir -p /etc/tokens
sudo tee /etc/tokens/spotify.env > /dev/null <<'EOF' sudo tee /etc/tokens/spotify.env > /dev/null <<'EOF'
@@ -90,25 +100,44 @@ EOF
sudo chown root:$USER /etc/tokens/spotify.env sudo chown root:$USER /etc/tokens/spotify.env
sudo chmod 640 /etc/tokens/spotify.env sudo chmod 640 /etc/tokens/spotify.env
~~~ ```
Never commit this file. Never commit this file.
--- ---
## 3. Export YouTube Cookies (Recommended)
YouTube increasingly rate-limits or age-gates anonymous downloads.
Export a Netscape-format `cookies.txt` from a logged-in browser session.
Recommended storage:
```bash
sudo cp cookies.txt /etc/tokens/youtube.cookies.txt
sudo chown root:$USER /etc/tokens/youtube.cookies.txt
sudo chmod 640 /etc/tokens/youtube.cookies.txt
```
---
# Docker Run # Docker Run
~~~bash ```bash
docker run --rm -p 3000:3000 \ docker run --rm -p 3000:3000 \
--env-file /etc/tokens/spotify.env \ --env-file /etc/tokens/spotify.env \
-e SPOTIFY_REDIRECT_URI='http://127.0.0.1:3000/api/spotify/callback' \ -e SPOTIFY_REDIRECT_URI='http://127.0.0.1:3000/api/spotify/callback' \
-e AUTH_ENABLED=true \ -e AUTH_ENABLED=true \
-e SPOOTY_AUTH_TOKEN=change_this_token \ -e SPOOTY_AUTH_TOKEN=change_this_token \
-e YT_SEARCH_DELAY_MS=7000 \ -e YT_SEARCH_DELAY_MS=7000 \
-e YT_DOWNLOADS_PER_MINUTE=3 \ -e YT_DOWNLOADS_PER_MINUTE=6 \
-e YT_COOKIES_FILE=/spooty/config/youtube.cookies.txt \
-v "$PWD/downloads:/spooty/backend/downloads" \ -v "$PWD/downloads:/spooty/backend/downloads" \
-v "$PWD/spooty-config:/spooty/backend/config" \
-v "/etc/tokens/youtube.cookies.txt:/spooty/config/youtube.cookies.txt:ro" \
spootyfy-hardened:local spootyfy-hardened:local
~~~ ```
Open: Open:
@@ -126,7 +155,7 @@ Then:
# Docker Compose # Docker Compose
~~~yaml ```yaml
services: services:
spooty: spooty:
image: spootyfy-hardened:local image: spootyfy-hardened:local
@@ -145,11 +174,15 @@ services:
SPOOTY_AUTH_TOKEN: "change_this_token" SPOOTY_AUTH_TOKEN: "change_this_token"
YT_SEARCH_DELAY_MS: "7000" YT_SEARCH_DELAY_MS: "7000"
YT_DOWNLOADS_PER_MINUTE: "3" YT_DOWNLOADS_PER_MINUTE: "6"
YT_COOKIES_FILE: "/spooty/config/youtube.cookies.txt"
volumes: volumes:
- ./downloads:/spooty/backend/downloads - ./downloads:/spooty/backend/downloads
~~~ - ./spooty-config:/spooty/backend/config
- /etc/tokens/youtube.cookies.txt:/spooty/config/youtube.cookies.txt:ro
```
--- ---
@@ -162,23 +195,25 @@ services:
- ffmpeg - ffmpeg
- Python3 - Python3
- Redis - Redis
- yt-dlp
--- ---
## Build ## Build
~~~bash ```bash
npm install npm install
npm run build npm run build
~~~ docker build -t spootyfy-hardened:local .
```
--- ---
## Run ## Run
~~~bash ```bash
npm run start npm run start
~~~ ```
--- ---
@@ -196,8 +231,8 @@ npm run start
| SPOTIFY_REDIRECT_URI | unset | OAuth callback URI | | SPOTIFY_REDIRECT_URI | unset | OAuth callback URI |
| AUTH_ENABLED | `false` | Enable frontend token auth | | AUTH_ENABLED | `false` | Enable frontend token auth |
| SPOOTY_AUTH_TOKEN | unset | Frontend auth token | | SPOOTY_AUTH_TOKEN | unset | Frontend auth token |
| YT_SEARCH_DELAY_MS | `5000` | Delay between YouTube searches | | YT_SEARCH_DELAY_MS | `7000` | Delay between YouTube searches |
| YT_DOWNLOADS_PER_MINUTE | `3` | Download pacing | | YT_DOWNLOADS_PER_MINUTE | `6` | Download pacing |
| YT_COOKIES | unset | Browser cookie extraction | | YT_COOKIES | unset | Browser cookie extraction |
| YT_COOKIES_FILE | unset | Netscape cookies.txt path | | YT_COOKIES_FILE | unset | Netscape cookies.txt path |
@@ -213,9 +248,9 @@ Two supported methods exist.
## Browser Cookies (Native Install Only) ## Browser Cookies (Native Install Only)
~~~bash ```bash
YT_COOKIES=firefox YT_COOKIES=firefox
~~~ ```
Supported: Supported:
@@ -236,15 +271,63 @@ Export a Netscape-format `cookies.txt`.
Mount into container: Mount into container:
~~~bash ```bash
-v /path/to/cookies.txt:/spooty/config/cookies.txt -v /path/to/cookies.txt:/spooty/config/youtube.cookies.txt:ro
~~~ ```
Then: Then:
~~~bash ```bash
-e YT_COOKIES_FILE=/spooty/config/cookies.txt -e YT_COOKIES_FILE=/spooty/config/youtube.cookies.txt
~~~ ```
---
# Deterministic YouTube Fallback Handling
The hardened branch now uses direct `yt-dlp` CLI execution rather than relying entirely on wrapper abstractions.
This provides:
- Explicit stderr visibility
- Better Docker compatibility
- Deterministic retry handling
- Automatic failed-candidate rejection
- Better age-gated video handling
- Improved operational observability
If a YouTube candidate fails:
1. The failed URL is recorded
2. The candidate is rejected
3. A new YouTube search is performed
4. The next-best valid candidate is attempted automatically
This prevents infinite retry loops against dead or restricted videos.
---
# Persistent Docker State
The hardened Docker flow strongly recommends persistent bind mounts for:
- downloads
- SQLite database
- runtime config
Recommended:
```bash
-v "$PWD/downloads:/spooty/backend/downloads" \
-v "$PWD/spooty-config:/spooty/backend/config"
```
This prevents:
- database resets on container recreation
- queue state loss
- retry-state loss
- playlist metadata loss
--- ---
@@ -272,10 +355,45 @@ Aggressive YouTube access can trigger:
Recommended safe pacing: Recommended safe pacing:
~~~bash ```bash
-e YT_SEARCH_DELAY_MS=7000 -e YT_SEARCH_DELAY_MS=7000
-e YT_DOWNLOADS_PER_MINUTE=3 -e YT_DOWNLOADS_PER_MINUTE=6
~~~ ```
The hardened branch also includes additional internal pacing and retry coordination to reduce:
- repeated failed candidate loops
- aggressive retry bursts
- queue collisions
- YouTube anti-bot triggers
---
# yt-dlp Notes
Modern YouTube behavior changes frequently.
The hardened branch now:
- Uses explicit yt-dlp CLI execution
- Captures stderr deterministically
- Detects age-gated failures
- Detects missing downloadable formats
- Handles retry/fallback selection explicitly
Some videos may still fail due to:
- regional restrictions
- removed videos
- YouTube anti-bot measures
- invalid cookies
- unavailable formats
Recommended:
- fresh cookies.txt exports
- authenticated YouTube sessions
- moderate pacing settings
--- ---
@@ -288,17 +406,19 @@ Never commit:
- cookies.txt - cookies.txt
- downloaded music - downloaded music
- local databases - local databases
- spooty-config/
Recommended `.gitignore` additions: Recommended `.gitignore` additions:
~~~gitignore ```gitignore
downloads/ downloads/
config/ config/
spooty-config/
*.sqlite *.sqlite
cookies.txt cookies.txt
.env .env
.env.local .env.local
~~~ ```
--- ---
@@ -318,10 +438,19 @@ cookies.txt
### YouTube ### YouTube
- Added configurable search pacing - Added deterministic YouTube candidate scoring
- Added configurable download pacing - Added duration-aware YouTube matching
- Reduced aggressive YouTube request bursts - Added failed-candidate rejection memory
- Improved queue throttling behavior - Added automatic retry/fallback handling
- Added explicit YouTube candidate exclusion support
- Added detailed YouTube candidate debug logging
- Added explicit yt-dlp CLI execution
- Added deterministic yt-dlp stderr capture
- Added age-gated video detection
- Added invalid-format detection
- Added Docker-compatible cookie handling
- Added temporary cookie-copy isolation
- Added retry-aware queue coordination
### Backend ### Backend
@@ -330,6 +459,7 @@ cookies.txt
- Improved Docker runtime stability - Improved Docker runtime stability
- Improved logging around Spotify retrieval - Improved logging around Spotify retrieval
- Added auth bootstrap flow support - Added auth bootstrap flow support
- Added deterministic retry-state handling
### Frontend ### Frontend
@@ -351,17 +481,20 @@ cookies.txt
Recommended local test run: Recommended local test run:
~~~bash ```bash
docker run --rm -p 3000:3000 \ docker run --rm -p 3000:3000 \
--env-file /etc/tokens/spotify.env \ --env-file /etc/tokens/spotify.env \
-e SPOTIFY_REDIRECT_URI='http://127.0.0.1:3000/api/spotify/callback' \ -e SPOTIFY_REDIRECT_URI='http://127.0.0.1:3000/api/spotify/callback' \
-e AUTH_ENABLED=true \ -e AUTH_ENABLED=true \
-e SPOOTY_AUTH_TOKEN=test-token \ -e SPOOTY_AUTH_TOKEN=test-token \
-e YT_SEARCH_DELAY_MS=7000 \ -e YT_SEARCH_DELAY_MS=7000 \
-e YT_DOWNLOADS_PER_MINUTE=3 \ -e YT_DOWNLOADS_PER_MINUTE=6 \
-e YT_COOKIES_FILE=/spooty/config/youtube.cookies.txt \
-v "$PWD/downloads:/spooty/backend/downloads" \ -v "$PWD/downloads:/spooty/backend/downloads" \
-v "$PWD/spooty-config:/spooty/backend/config" \
-v "/etc/tokens/youtube.cookies.txt:/spooty/config/youtube.cookies.txt:ro" \
spootyfy-hardened:local spootyfy-hardened:local
~~~ ```
--- ---