diff --git a/CHANGELOG.md b/CHANGELOG.md index 0cbd812..8097af8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,37 +10,60 @@ This project follows a practical chronological changelog rather than autogenerat ## Added -### Spotify OAuth Integration +### Spotify - Added Spotify OAuth login flow - Added `/api/spotify/login` - Added `/api/spotify/callback` - Added `/api/spotify/status` -- Added persistent Spotify user access token storage +- Added persistent Spotify user token storage - Added authenticated Spotify playlist retrieval - Added OAuth frontend bootstrap flow +- Added support for playlists exceeding 300 tracks +- Added deterministic playlist pagination handling -### Large Playlist Support +### YouTube -- Fixed Spotify playlist truncation at 100 tracks -- Added full Spotify pagination support -- Verified support for playlists exceeding 300 tracks -- Improved playlist queue handling stability +- Added deterministic YouTube candidate scoring +- Added duration-aware YouTube matching +- Added failed-candidate rejection memory +- Added automatic retry/fallback handling +- Added explicit YouTube candidate exclusion support +- Added detailed YouTube candidate debug logging +- Added explicit yt-dlp CLI execution +- Added deterministic yt-dlp stderr capture +- Added age-gated video detection +- Added invalid-format detection +- Added Docker-compatible cookie handling +- Added temporary cookie-copy isolation +- Added retry-aware queue coordination ### Queue Hardening - Added configurable YouTube search pacing - Added configurable YouTube download pacing -- Added sequential search queue behavior +- Added sequential queue coordination +- Added retry collision prevention +- Added deterministic queue replacement handling +- Added explicit retry-state transitions - Reduced aggressive request bursts toward YouTube -- Improved queue logging visibility + +### Docker + +- Added persistent SQLite bind-mount recommendations +- Added persistent runtime config recommendations +- Added yt-dlp installation to runtime container +- Improved Docker runtime observability +- Improved hardened Docker startup flow +- Improved deployment documentation ### Security - Added token-based frontend access mode - Added external secret handling recommendations - Added isolated Spotify credential storage -- Improved Docker credential handling documentation +- Added safer cookie handling flow +- Improved Docker secret isolation documentation ### Frontend @@ -49,37 +72,33 @@ This project follows a practical chronological changelog rather than autogenerat - Improved playlist rendering behavior - Improved large playlist progress tracking -### Docker +## Improved -- Improved hardened Docker runtime flow -- Added explicit environment variable examples -- Added safer default pacing recommendations -- Improved deployment documentation +- Improved YouTube throttling resistance +- Improved age-gated video recovery behavior +- Improved operational observability +- Improved cover-art embedding resilience +- Improved queue stability during large playlist imports +- Improved deterministic runtime behavior +- Improved Docker persistence handling -## Unreleased +## Fixed -### Added -- Spotify OAuth login flow -- Persistent Spotify user token handling -- Large playlist pagination support (>100 tracks) -- Deterministic YouTube candidate scoring -- Duration-aware YouTube matching -- Detailed YouTube candidate debug logging -- Improved Docker deployment guidance - -### Improved -- YouTube pacing and throttling resistance -- Cover art embedding resilience -- Queue stability during large playlist imports - -### Fixed -- Spotify playlist truncation at 100 tracks -- Incorrect live/acoustic YouTube selections -- Playlist retrieval failures on large public playlists +- Fixed Spotify playlist truncation at 100 tracks +- Fixed incomplete playlist retrieval behavior +- Fixed repeated retry loops against failed YouTube videos +- Fixed queue deadlocks during retries +- Fixed retry-button queue collisions +- Fixed container database loss on restart +- Fixed yt-dlp wrapper opacity issues +- Fixed hidden yt-dlp stderr failures +- Fixed multiple age-gated fallback failure paths +- Fixed invalid YouTube candidate reuse +- Fixed several Docker persistence edge cases --- -# 2.4.2 +# 2.4.2 (Original Spooty Endpoint) ## Fixed diff --git a/README.md b/README.md index 92bb5bb..4379add 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,13 @@ This hardened branch focuses on: - Safer credential handling - More resilient cover-art embedding - Better queue stability +- Deterministic YouTube fallback handling +- Explicit yt-dlp CLI execution +- Automatic failed-candidate rejection +- Improved age-gated video handling +- Persistent SQLite state across container restarts +- Deterministic Docker config persistence +- Improved operational observability --- @@ -30,6 +37,9 @@ This hardened branch focuses on: - Large playlist pagination support - Download pacing controls - YouTube cookie support +- Automatic YouTube retry/fallback handling +- Failed YouTube candidate rejection memory +- Deterministic yt-dlp error classification --- @@ -80,7 +90,7 @@ Copy: Create a secure env file: -~~~bash +```bash sudo mkdir -p /etc/tokens sudo tee /etc/tokens/spotify.env > /dev/null <<'EOF' @@ -90,25 +100,44 @@ EOF sudo chown root:$USER /etc/tokens/spotify.env sudo chmod 640 /etc/tokens/spotify.env -~~~ +``` Never commit this file. --- +## 3. Export YouTube Cookies (Recommended) + +YouTube increasingly rate-limits or age-gates anonymous downloads. + +Export a Netscape-format `cookies.txt` from a logged-in browser session. + +Recommended storage: + +```bash +sudo cp cookies.txt /etc/tokens/youtube.cookies.txt +sudo chown root:$USER /etc/tokens/youtube.cookies.txt +sudo chmod 640 /etc/tokens/youtube.cookies.txt +``` + +--- + # Docker Run -~~~bash +```bash docker run --rm -p 3000:3000 \ --env-file /etc/tokens/spotify.env \ -e SPOTIFY_REDIRECT_URI='http://127.0.0.1:3000/api/spotify/callback' \ -e AUTH_ENABLED=true \ -e SPOOTY_AUTH_TOKEN=change_this_token \ -e YT_SEARCH_DELAY_MS=7000 \ - -e YT_DOWNLOADS_PER_MINUTE=3 \ + -e YT_DOWNLOADS_PER_MINUTE=6 \ + -e YT_COOKIES_FILE=/spooty/config/youtube.cookies.txt \ -v "$PWD/downloads:/spooty/backend/downloads" \ + -v "$PWD/spooty-config:/spooty/backend/config" \ + -v "/etc/tokens/youtube.cookies.txt:/spooty/config/youtube.cookies.txt:ro" \ spootyfy-hardened:local -~~~ +``` Open: @@ -126,7 +155,7 @@ Then: # Docker Compose -~~~yaml +```yaml services: spooty: image: spootyfy-hardened:local @@ -145,11 +174,15 @@ services: SPOOTY_AUTH_TOKEN: "change_this_token" YT_SEARCH_DELAY_MS: "7000" - YT_DOWNLOADS_PER_MINUTE: "3" + YT_DOWNLOADS_PER_MINUTE: "6" + + YT_COOKIES_FILE: "/spooty/config/youtube.cookies.txt" volumes: - ./downloads:/spooty/backend/downloads -~~~ + - ./spooty-config:/spooty/backend/config + - /etc/tokens/youtube.cookies.txt:/spooty/config/youtube.cookies.txt:ro +``` --- @@ -162,23 +195,25 @@ services: - ffmpeg - Python3 - Redis +- yt-dlp --- ## Build -~~~bash +```bash npm install npm run build -~~~ +docker build -t spootyfy-hardened:local . +``` --- ## Run -~~~bash +```bash npm run start -~~~ +``` --- @@ -196,8 +231,8 @@ npm run start | SPOTIFY_REDIRECT_URI | unset | OAuth callback URI | | AUTH_ENABLED | `false` | Enable frontend token auth | | SPOOTY_AUTH_TOKEN | unset | Frontend auth token | -| YT_SEARCH_DELAY_MS | `5000` | Delay between YouTube searches | -| YT_DOWNLOADS_PER_MINUTE | `3` | Download pacing | +| YT_SEARCH_DELAY_MS | `7000` | Delay between YouTube searches | +| YT_DOWNLOADS_PER_MINUTE | `6` | Download pacing | | YT_COOKIES | unset | Browser cookie extraction | | YT_COOKIES_FILE | unset | Netscape cookies.txt path | @@ -213,9 +248,9 @@ Two supported methods exist. ## Browser Cookies (Native Install Only) -~~~bash +```bash YT_COOKIES=firefox -~~~ +``` Supported: @@ -236,15 +271,63 @@ Export a Netscape-format `cookies.txt`. Mount into container: -~~~bash --v /path/to/cookies.txt:/spooty/config/cookies.txt -~~~ +```bash +-v /path/to/cookies.txt:/spooty/config/youtube.cookies.txt:ro +``` Then: -~~~bash --e YT_COOKIES_FILE=/spooty/config/cookies.txt -~~~ +```bash +-e YT_COOKIES_FILE=/spooty/config/youtube.cookies.txt +``` + +--- + +# Deterministic YouTube Fallback Handling + +The hardened branch now uses direct `yt-dlp` CLI execution rather than relying entirely on wrapper abstractions. + +This provides: + +- Explicit stderr visibility +- Better Docker compatibility +- Deterministic retry handling +- Automatic failed-candidate rejection +- Better age-gated video handling +- Improved operational observability + +If a YouTube candidate fails: + +1. The failed URL is recorded +2. The candidate is rejected +3. A new YouTube search is performed +4. The next-best valid candidate is attempted automatically + +This prevents infinite retry loops against dead or restricted videos. + +--- + +# Persistent Docker State + +The hardened Docker flow strongly recommends persistent bind mounts for: + +- downloads +- SQLite database +- runtime config + +Recommended: + +```bash +-v "$PWD/downloads:/spooty/backend/downloads" \ +-v "$PWD/spooty-config:/spooty/backend/config" +``` + +This prevents: + +- database resets on container recreation +- queue state loss +- retry-state loss +- playlist metadata loss --- @@ -272,10 +355,45 @@ Aggressive YouTube access can trigger: Recommended safe pacing: -~~~bash +```bash -e YT_SEARCH_DELAY_MS=7000 --e YT_DOWNLOADS_PER_MINUTE=3 -~~~ +-e YT_DOWNLOADS_PER_MINUTE=6 +``` + +The hardened branch also includes additional internal pacing and retry coordination to reduce: + +- repeated failed candidate loops +- aggressive retry bursts +- queue collisions +- YouTube anti-bot triggers + +--- + +# yt-dlp Notes + +Modern YouTube behavior changes frequently. + +The hardened branch now: + +- Uses explicit yt-dlp CLI execution +- Captures stderr deterministically +- Detects age-gated failures +- Detects missing downloadable formats +- Handles retry/fallback selection explicitly + +Some videos may still fail due to: + +- regional restrictions +- removed videos +- YouTube anti-bot measures +- invalid cookies +- unavailable formats + +Recommended: + +- fresh cookies.txt exports +- authenticated YouTube sessions +- moderate pacing settings --- @@ -288,17 +406,19 @@ Never commit: - cookies.txt - downloaded music - local databases +- spooty-config/ Recommended `.gitignore` additions: -~~~gitignore +```gitignore downloads/ config/ +spooty-config/ *.sqlite cookies.txt .env .env.local -~~~ +``` --- @@ -318,10 +438,19 @@ cookies.txt ### YouTube -- Added configurable search pacing -- Added configurable download pacing -- Reduced aggressive YouTube request bursts -- Improved queue throttling behavior +- Added deterministic YouTube candidate scoring +- Added duration-aware YouTube matching +- Added failed-candidate rejection memory +- Added automatic retry/fallback handling +- Added explicit YouTube candidate exclusion support +- Added detailed YouTube candidate debug logging +- Added explicit yt-dlp CLI execution +- Added deterministic yt-dlp stderr capture +- Added age-gated video detection +- Added invalid-format detection +- Added Docker-compatible cookie handling +- Added temporary cookie-copy isolation +- Added retry-aware queue coordination ### Backend @@ -330,6 +459,7 @@ cookies.txt - Improved Docker runtime stability - Improved logging around Spotify retrieval - Added auth bootstrap flow support +- Added deterministic retry-state handling ### Frontend @@ -351,17 +481,20 @@ cookies.txt Recommended local test run: -~~~bash +```bash docker run --rm -p 3000:3000 \ --env-file /etc/tokens/spotify.env \ -e SPOTIFY_REDIRECT_URI='http://127.0.0.1:3000/api/spotify/callback' \ -e AUTH_ENABLED=true \ -e SPOOTY_AUTH_TOKEN=test-token \ -e YT_SEARCH_DELAY_MS=7000 \ - -e YT_DOWNLOADS_PER_MINUTE=3 \ + -e YT_DOWNLOADS_PER_MINUTE=6 \ + -e YT_COOKIES_FILE=/spooty/config/youtube.cookies.txt \ -v "$PWD/downloads:/spooty/backend/downloads" \ + -v "$PWD/spooty-config:/spooty/backend/config" \ + -v "/etc/tokens/youtube.cookies.txt:/spooty/config/youtube.cookies.txt:ro" \ spootyfy-hardened:local -~~~ +``` ---