Dymas/jarri-spooty
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Document deterministic yt-dlp hardening and Docker persistence

Browse Source
This commit is contained in:
Tor Wingalen
2026-05-15 21:07:01 +02:00
parent 48517a5a85
commit 9ea3c6a271
2 changed files with 221 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+54 -35
View File
@@ -10,37 +10,60 @@ This project follows a practical chronological changelog rather than autogenerat
## Added
### Spotify OAuth Integration
### Spotify
- Added Spotify OAuth login flow
- Added `/api/spotify/login`
- Added `/api/spotify/callback`
- Added `/api/spotify/status`
- Added persistent Spotify user access token storage
- Added persistent Spotify user token storage
- Added authenticated Spotify playlist retrieval
- Added OAuth frontend bootstrap flow
- Added support for playlists exceeding 300 tracks
- Added deterministic playlist pagination handling
### Large Playlist Support
### YouTube
- Fixed Spotify playlist truncation at 100 tracks
- Added full Spotify pagination support
- Verified support for playlists exceeding 300 tracks
- Improved playlist queue handling stability
- Added deterministic YouTube candidate scoring
- Added duration-aware YouTube matching
- Added failed-candidate rejection memory
- Added automatic retry/fallback handling
- Added explicit YouTube candidate exclusion support
- Added detailed YouTube candidate debug logging
- Added explicit yt-dlp CLI execution
- Added deterministic yt-dlp stderr capture
- Added age-gated video detection
- Added invalid-format detection
- Added Docker-compatible cookie handling
- Added temporary cookie-copy isolation
- Added retry-aware queue coordination
### Queue Hardening
- Added configurable YouTube search pacing
- Added configurable YouTube download pacing
- Added sequential search queue behavior
- Added sequential queue coordination
- Added retry collision prevention
- Added deterministic queue replacement handling
- Added explicit retry-state transitions
- Reduced aggressive request bursts toward YouTube
- Improved queue logging visibility
### Docker
- Added persistent SQLite bind-mount recommendations
- Added persistent runtime config recommendations
- Added yt-dlp installation to runtime container
- Improved Docker runtime observability
- Improved hardened Docker startup flow
- Improved deployment documentation
### Security
- Added token-based frontend access mode
- Added external secret handling recommendations
- Added isolated Spotify credential storage
- Improved Docker credential handling documentation
- Added safer cookie handling flow
- Improved Docker secret isolation documentation
### Frontend
@@ -49,37 +72,33 @@ This project follows a practical chronological changelog rather than autogenerat
- Improved playlist rendering behavior
- Improved large playlist progress tracking
### Docker
## Improved
- Improved hardened Docker runtime flow
- Added explicit environment variable examples
- Added safer default pacing recommendations
- Improved deployment documentation
- Improved YouTube throttling resistance
- Improved age-gated video recovery behavior
- Improved operational observability
- Improved cover-art embedding resilience
- Improved queue stability during large playlist imports
- Improved deterministic runtime behavior
- Improved Docker persistence handling
## Unreleased
## Fixed
### Added
- Spotify OAuth login flow
- Persistent Spotify user token handling
- Large playlist pagination support (>100 tracks)
- Deterministic YouTube candidate scoring
- Duration-aware YouTube matching
- Detailed YouTube candidate debug logging
- Improved Docker deployment guidance
### Improved
- YouTube pacing and throttling resistance
- Cover art embedding resilience
- Queue stability during large playlist imports
### Fixed
- Spotify playlist truncation at 100 tracks
- Incorrect live/acoustic YouTube selections
- Playlist retrieval failures on large public playlists
- Fixed Spotify playlist truncation at 100 tracks
- Fixed incomplete playlist retrieval behavior
- Fixed repeated retry loops against failed YouTube videos
- Fixed queue deadlocks during retries
- Fixed retry-button queue collisions
- Fixed container database loss on restart
- Fixed yt-dlp wrapper opacity issues
- Fixed hidden yt-dlp stderr failures
- Fixed multiple age-gated fallback failure paths
- Fixed invalid YouTube candidate reuse
- Fixed several Docker persistence edge cases
---
# 2.4.2
# 2.4.2 (Original Spooty Endpoint)
## Fixed
README.md
+167 -34
View File
@@ -14,6 +14,13 @@ This hardened branch focuses on:
- Safer credential handling
- More resilient cover-art embedding
- Better queue stability
- Deterministic YouTube fallback handling
- Explicit yt-dlp CLI execution
- Automatic failed-candidate rejection
- Improved age-gated video handling
- Persistent SQLite state across container restarts
- Deterministic Docker config persistence
- Improved operational observability
---
@@ -30,6 +37,9 @@ This hardened branch focuses on:
- Large playlist pagination support
- Download pacing controls
- YouTube cookie support
- Automatic YouTube retry/fallback handling
- Failed YouTube candidate rejection memory
- Deterministic yt-dlp error classification
---
@@ -80,7 +90,7 @@ Copy:
Create a secure env file:
~~~bash
```bash
sudo mkdir -p /etc/tokens
sudo tee /etc/tokens/spotify.env > /dev/null <<'EOF'
@@ -90,25 +100,44 @@ EOF
sudo chown root:$USER /etc/tokens/spotify.env
sudo chmod 640 /etc/tokens/spotify.env
~~~
```
Never commit this file.
---
## 3. Export YouTube Cookies (Recommended)
YouTube increasingly rate-limits or age-gates anonymous downloads.
Export a Netscape-format `cookies.txt` from a logged-in browser session.
Recommended storage:
```bash
sudo cp cookies.txt /etc/tokens/youtube.cookies.txt
sudo chown root:$USER /etc/tokens/youtube.cookies.txt
sudo chmod 640 /etc/tokens/youtube.cookies.txt
```
---
# Docker Run
~~~bash
```bash
docker run --rm -p 3000:3000 \
--env-file /etc/tokens/spotify.env \
-e SPOTIFY_REDIRECT_URI='http://127.0.0.1:3000/api/spotify/callback' \
-e AUTH_ENABLED=true \
-e SPOOTY_AUTH_TOKEN=change_this_token \
-e YT_SEARCH_DELAY_MS=7000 \
-e YT_DOWNLOADS_PER_MINUTE=3 \
-e YT_DOWNLOADS_PER_MINUTE=6 \
-e YT_COOKIES_FILE=/spooty/config/youtube.cookies.txt \
-v "$PWD/downloads:/spooty/backend/downloads" \
-v "$PWD/spooty-config:/spooty/backend/config" \
-v "/etc/tokens/youtube.cookies.txt:/spooty/config/youtube.cookies.txt:ro" \
spootyfy-hardened:local
~~~
```
Open:
@@ -126,7 +155,7 @@ Then:
# Docker Compose
~~~yaml
```yaml
services:
spooty:
image: spootyfy-hardened:local
@@ -145,11 +174,15 @@ services:
SPOOTY_AUTH_TOKEN: "change_this_token"
YT_SEARCH_DELAY_MS: "7000"
YT_DOWNLOADS_PER_MINUTE: "3"
YT_DOWNLOADS_PER_MINUTE: "6"
YT_COOKIES_FILE: "/spooty/config/youtube.cookies.txt"
volumes:
- ./downloads:/spooty/backend/downloads
~~~
- ./spooty-config:/spooty/backend/config
- /etc/tokens/youtube.cookies.txt:/spooty/config/youtube.cookies.txt:ro
```
---
@@ -162,23 +195,25 @@ services:
- ffmpeg
- Python3
- Redis
- yt-dlp
---
## Build
~~~bash
```bash
npm install
npm run build
~~~
docker build -t spootyfy-hardened:local .
```
---
## Run
~~~bash
```bash
npm run start
~~~
```
---
@@ -196,8 +231,8 @@ npm run start
| SPOTIFY_REDIRECT_URI | unset | OAuth callback URI |
| AUTH_ENABLED | `false` | Enable frontend token auth |
| SPOOTY_AUTH_TOKEN | unset | Frontend auth token |
| YT_SEARCH_DELAY_MS | `5000` | Delay between YouTube searches |
| YT_DOWNLOADS_PER_MINUTE | `3` | Download pacing |
| YT_SEARCH_DELAY_MS | `7000` | Delay between YouTube searches |
| YT_DOWNLOADS_PER_MINUTE | `6` | Download pacing |
| YT_COOKIES | unset | Browser cookie extraction |
| YT_COOKIES_FILE | unset | Netscape cookies.txt path |
@@ -213,9 +248,9 @@ Two supported methods exist.
## Browser Cookies (Native Install Only)
~~~bash
```bash
YT_COOKIES=firefox
~~~
```
Supported:
@@ -236,15 +271,63 @@ Export a Netscape-format `cookies.txt`.
Mount into container:
~~~bash
-v /path/to/cookies.txt:/spooty/config/cookies.txt
~~~
```bash
-v /path/to/cookies.txt:/spooty/config/youtube.cookies.txt:ro
```
Then:
~~~bash
-e YT_COOKIES_FILE=/spooty/config/cookies.txt
~~~
```bash
-e YT_COOKIES_FILE=/spooty/config/youtube.cookies.txt
```
---
# Deterministic YouTube Fallback Handling
The hardened branch now uses direct `yt-dlp` CLI execution rather than relying entirely on wrapper abstractions.
This provides:
- Explicit stderr visibility
- Better Docker compatibility
- Deterministic retry handling
- Automatic failed-candidate rejection
- Better age-gated video handling
- Improved operational observability
If a YouTube candidate fails:
1. The failed URL is recorded
2. The candidate is rejected
3. A new YouTube search is performed
4. The next-best valid candidate is attempted automatically
This prevents infinite retry loops against dead or restricted videos.
---
# Persistent Docker State
The hardened Docker flow strongly recommends persistent bind mounts for:
- downloads
- SQLite database
- runtime config
Recommended:
```bash
-v "$PWD/downloads:/spooty/backend/downloads" \
-v "$PWD/spooty-config:/spooty/backend/config"
```
This prevents:
- database resets on container recreation
- queue state loss
- retry-state loss
- playlist metadata loss
---
@@ -272,10 +355,45 @@ Aggressive YouTube access can trigger:
Recommended safe pacing:
~~~bash
```bash
-e YT_SEARCH_DELAY_MS=7000
-e YT_DOWNLOADS_PER_MINUTE=3
~~~
-e YT_DOWNLOADS_PER_MINUTE=6
```
The hardened branch also includes additional internal pacing and retry coordination to reduce:
- repeated failed candidate loops
- aggressive retry bursts
- queue collisions
- YouTube anti-bot triggers
---
# yt-dlp Notes
Modern YouTube behavior changes frequently.
The hardened branch now:
- Uses explicit yt-dlp CLI execution
- Captures stderr deterministically
- Detects age-gated failures
- Detects missing downloadable formats
- Handles retry/fallback selection explicitly
Some videos may still fail due to:
- regional restrictions
- removed videos
- YouTube anti-bot measures
- invalid cookies
- unavailable formats
Recommended:
- fresh cookies.txt exports
- authenticated YouTube sessions
- moderate pacing settings
---
@@ -288,17 +406,19 @@ Never commit:
- cookies.txt
- downloaded music
- local databases
- spooty-config/
Recommended `.gitignore` additions:
~~~gitignore
```gitignore
downloads/
config/
spooty-config/
*.sqlite
cookies.txt
.env
.env.local
~~~
```
---
@@ -318,10 +438,19 @@ cookies.txt
### YouTube
- Added configurable search pacing
- Added configurable download pacing
- Reduced aggressive YouTube request bursts
- Improved queue throttling behavior
- Added deterministic YouTube candidate scoring
- Added duration-aware YouTube matching
- Added failed-candidate rejection memory
- Added automatic retry/fallback handling
- Added explicit YouTube candidate exclusion support
- Added detailed YouTube candidate debug logging
- Added explicit yt-dlp CLI execution
- Added deterministic yt-dlp stderr capture
- Added age-gated video detection
- Added invalid-format detection
- Added Docker-compatible cookie handling
- Added temporary cookie-copy isolation
- Added retry-aware queue coordination
### Backend
@@ -330,6 +459,7 @@ cookies.txt
- Improved Docker runtime stability
- Improved logging around Spotify retrieval
- Added auth bootstrap flow support
- Added deterministic retry-state handling
### Frontend
@@ -351,17 +481,20 @@ cookies.txt
Recommended local test run:
~~~bash
```bash
docker run --rm -p 3000:3000 \
--env-file /etc/tokens/spotify.env \
-e SPOTIFY_REDIRECT_URI='http://127.0.0.1:3000/api/spotify/callback' \
-e AUTH_ENABLED=true \
-e SPOOTY_AUTH_TOKEN=test-token \
-e YT_SEARCH_DELAY_MS=7000 \
-e YT_DOWNLOADS_PER_MINUTE=3 \
-e YT_DOWNLOADS_PER_MINUTE=6 \
-e YT_COOKIES_FILE=/spooty/config/youtube.cookies.txt \
-v "$PWD/downloads:/spooty/backend/downloads" \
-v "$PWD/spooty-config:/spooty/backend/config" \
-v "/etc/tokens/youtube.cookies.txt:/spooty/config/youtube.cookies.txt:ro" \
spootyfy-hardened:local
~~~
```
---