504 lines
10 KiB
Markdown
504 lines
10 KiB
Markdown
# Spooty Hardened
|
|
|
|
Self-hosted Spotify playlist and track downloader built with NestJS + Angular.
|
|
|
|
Spooty does not download audio from Spotify itself.
|
|
It retrieves metadata from Spotify and locates matching audio on YouTube.
|
|
|
|
This hardened branch focuses on:
|
|
|
|
- Large playlist support (>100 tracks)
|
|
- Spotify OAuth login flow
|
|
- Better YouTube pacing and throttling resistance
|
|
- Improved Docker deployment
|
|
- Safer credential handling
|
|
- More resilient cover-art embedding
|
|
- Better queue stability
|
|
- Deterministic YouTube fallback handling
|
|
- Explicit yt-dlp CLI execution
|
|
- Automatic failed-candidate rejection
|
|
- Improved age-gated video handling
|
|
- Persistent SQLite state across container restarts
|
|
- Deterministic Docker config persistence
|
|
- Improved operational observability
|
|
|
|
---
|
|
|
|
# Features
|
|
|
|
- Download Spotify playlists
|
|
- Download individual Spotify tracks
|
|
- Playlist auto-subscription support
|
|
- Automatic YouTube matching
|
|
- MP3 tagging and embedded cover art
|
|
- Docker deployment
|
|
- Queue-based download system
|
|
- Spotify OAuth integration
|
|
- Large playlist pagination support
|
|
- Download pacing controls
|
|
- YouTube cookie support
|
|
- Automatic YouTube retry/fallback handling
|
|
- Failed YouTube candidate rejection memory
|
|
- Deterministic yt-dlp error classification
|
|
|
|
---
|
|
|
|
# Important Notice
|
|
|
|
Use this software responsibly.
|
|
|
|
Only download music you legally own or are permitted to access.
|
|
|
|
The maintainers are not responsible for misuse.
|
|
|
|
---
|
|
|
|
# Supported URLs
|
|
|
|
- Spotify playlists
|
|
- Spotify tracks
|
|
|
|
Example:
|
|
|
|
https://open.spotify.com/playlist/...
|
|
https://open.spotify.com/track/...
|
|
|
|
---
|
|
|
|
# Quick Start (Recommended)
|
|
|
|
## 1. Create Spotify Developer App
|
|
|
|
Go to:
|
|
|
|
https://developer.spotify.com/dashboard
|
|
|
|
Create an application.
|
|
|
|
Add this Redirect URI:
|
|
|
|
http://127.0.0.1:3000/api/spotify/callback
|
|
|
|
Copy:
|
|
|
|
- Client ID
|
|
- Client Secret
|
|
|
|
---
|
|
|
|
## 2. Store Credentials Outside Repository
|
|
|
|
Create a secure env file:
|
|
|
|
```bash
|
|
sudo mkdir -p /etc/tokens
|
|
|
|
sudo tee /etc/tokens/spotify.env > /dev/null <<'EOF'
|
|
SPOTIFY_CLIENT_ID=your_client_id
|
|
SPOTIFY_CLIENT_SECRET=your_client_secret
|
|
EOF
|
|
|
|
sudo chown root:$USER /etc/tokens/spotify.env
|
|
sudo chmod 640 /etc/tokens/spotify.env
|
|
```
|
|
|
|
Never commit this file.
|
|
|
|
---
|
|
|
|
## 3. Export YouTube Cookies (Recommended)
|
|
|
|
YouTube increasingly rate-limits or age-gates anonymous downloads.
|
|
|
|
Export a Netscape-format `cookies.txt` from a logged-in browser session.
|
|
|
|
Recommended storage:
|
|
|
|
```bash
|
|
sudo cp cookies.txt /etc/tokens/youtube.cookies.txt
|
|
sudo chown root:$USER /etc/tokens/youtube.cookies.txt
|
|
sudo chmod 640 /etc/tokens/youtube.cookies.txt
|
|
```
|
|
|
|
---
|
|
|
|
# Docker Run
|
|
|
|
```bash
|
|
docker run --rm -p 3000:3000 \
|
|
--env-file /etc/tokens/spotify.env \
|
|
-e SPOTIFY_REDIRECT_URI='http://127.0.0.1:3000/api/spotify/callback' \
|
|
-e AUTH_ENABLED=true \
|
|
-e SPOOTY_AUTH_TOKEN=change_this_token \
|
|
-e YT_SEARCH_DELAY_MS=7000 \
|
|
-e YT_DOWNLOADS_PER_MINUTE=6 \
|
|
-e YT_COOKIES_FILE=/spooty/config/youtube.cookies.txt \
|
|
-v "$PWD/downloads:/spooty/backend/downloads" \
|
|
-v "$PWD/spooty-config:/spooty/backend/config" \
|
|
-v "/etc/tokens/youtube.cookies.txt:/spooty/config/youtube.cookies.txt:ro" \
|
|
spootyfy-hardened:local
|
|
```
|
|
|
|
Open:
|
|
|
|
http://127.0.0.1:3000/?token=change_this_token
|
|
|
|
Then:
|
|
|
|
1. Click "Connect Spotify"
|
|
2. Login to Spotify
|
|
3. Approve access
|
|
4. Paste playlist URL
|
|
5. Download
|
|
|
|
---
|
|
|
|
# Docker Compose
|
|
|
|
```yaml
|
|
services:
|
|
spooty:
|
|
image: spootyfy-hardened:local
|
|
container_name: spooty
|
|
restart: unless-stopped
|
|
|
|
ports:
|
|
- "3000:3000"
|
|
|
|
env_file:
|
|
- /etc/tokens/spotify.env
|
|
|
|
environment:
|
|
SPOTIFY_REDIRECT_URI: "http://127.0.0.1:3000/api/spotify/callback"
|
|
AUTH_ENABLED: "true"
|
|
SPOOTY_AUTH_TOKEN: "change_this_token"
|
|
|
|
YT_SEARCH_DELAY_MS: "7000"
|
|
YT_DOWNLOADS_PER_MINUTE: "6"
|
|
|
|
YT_COOKIES_FILE: "/spooty/config/youtube.cookies.txt"
|
|
|
|
volumes:
|
|
- ./downloads:/spooty/backend/downloads
|
|
- ./spooty-config:/spooty/backend/config
|
|
- /etc/tokens/youtube.cookies.txt:/spooty/config/youtube.cookies.txt:ro
|
|
```
|
|
|
|
---
|
|
|
|
# Build From Source
|
|
|
|
## Requirements
|
|
|
|
- Node.js 20.20.0
|
|
- Docker
|
|
- ffmpeg
|
|
- Python3
|
|
- Redis
|
|
- yt-dlp
|
|
|
|
---
|
|
|
|
## Build
|
|
|
|
```bash
|
|
npm install
|
|
npm run build
|
|
docker build -t spootyfy-hardened:local .
|
|
```
|
|
|
|
---
|
|
|
|
## Run
|
|
|
|
```bash
|
|
npm run start
|
|
```
|
|
|
|
---
|
|
|
|
# Environment Variables
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| DB_PATH | `./config/db.sqlite` | SQLite database path |
|
|
| DOWNLOADS_PATH | `./downloads` | Download directory |
|
|
| FORMAT | `mp3` | Output format |
|
|
| REDIS_HOST | `localhost` | Redis host |
|
|
| REDIS_PORT | `6379` | Redis port |
|
|
| SPOTIFY_CLIENT_ID | unset | Spotify client ID |
|
|
| SPOTIFY_CLIENT_SECRET | unset | Spotify client secret |
|
|
| SPOTIFY_REDIRECT_URI | unset | OAuth callback URI |
|
|
| AUTH_ENABLED | `false` | Enable frontend token auth |
|
|
| SPOOTY_AUTH_TOKEN | unset | Frontend auth token |
|
|
| YT_SEARCH_DELAY_MS | `7000` | Delay between YouTube searches |
|
|
| YT_DOWNLOADS_PER_MINUTE | `6` | Download pacing |
|
|
| YT_COOKIES | unset | Browser cookie extraction |
|
|
| YT_COOKIES_FILE | unset | Netscape cookies.txt path |
|
|
|
|
---
|
|
|
|
# YouTube Cookies
|
|
|
|
YouTube may throttle or block requests without cookies.
|
|
|
|
Two supported methods exist.
|
|
|
|
---
|
|
|
|
## Browser Cookies (Native Install Only)
|
|
|
|
```bash
|
|
YT_COOKIES=firefox
|
|
```
|
|
|
|
Supported:
|
|
|
|
- firefox
|
|
- chrome
|
|
- chromium
|
|
- edge
|
|
- brave
|
|
- opera
|
|
|
|
This does NOT work reliably in Docker.
|
|
|
|
---
|
|
|
|
## Cookies File (Recommended)
|
|
|
|
Export a Netscape-format `cookies.txt`.
|
|
|
|
Mount into container:
|
|
|
|
```bash
|
|
-v /path/to/cookies.txt:/spooty/config/youtube.cookies.txt:ro
|
|
```
|
|
|
|
Then:
|
|
|
|
```bash
|
|
-e YT_COOKIES_FILE=/spooty/config/youtube.cookies.txt
|
|
```
|
|
|
|
---
|
|
|
|
# Deterministic YouTube Fallback Handling
|
|
|
|
The hardened branch now uses direct `yt-dlp` CLI execution rather than relying entirely on wrapper abstractions.
|
|
|
|
This provides:
|
|
|
|
- Explicit stderr visibility
|
|
- Better Docker compatibility
|
|
- Deterministic retry handling
|
|
- Automatic failed-candidate rejection
|
|
- Better age-gated video handling
|
|
- Improved operational observability
|
|
|
|
If a YouTube candidate fails:
|
|
|
|
1. The failed URL is recorded
|
|
2. The candidate is rejected
|
|
3. A new YouTube search is performed
|
|
4. The next-best valid candidate is attempted automatically
|
|
|
|
This prevents infinite retry loops against dead or restricted videos.
|
|
|
|
---
|
|
|
|
# Persistent Docker State
|
|
|
|
The hardened Docker flow strongly recommends persistent bind mounts for:
|
|
|
|
- downloads
|
|
- SQLite database
|
|
- runtime config
|
|
|
|
Recommended:
|
|
|
|
```bash
|
|
-v "$PWD/downloads:/spooty/backend/downloads" \
|
|
-v "$PWD/spooty-config:/spooty/backend/config"
|
|
```
|
|
|
|
This prevents:
|
|
|
|
- database resets on container recreation
|
|
- queue state loss
|
|
- retry-state loss
|
|
- playlist metadata loss
|
|
|
|
---
|
|
|
|
# Large Playlist Support
|
|
|
|
The hardened branch fixes several Spotify API limitations:
|
|
|
|
- Proper playlist pagination
|
|
- >100 track playlists
|
|
- OAuth-authenticated playlist access
|
|
- Queue stability improvements
|
|
|
|
Verified working with playlists containing 300+ tracks.
|
|
|
|
---
|
|
|
|
# Queue Pacing
|
|
|
|
Aggressive YouTube access can trigger:
|
|
|
|
- HTTP 302 loops
|
|
- CAPTCHA
|
|
- temporary throttling
|
|
- incomplete downloads
|
|
|
|
Recommended safe pacing:
|
|
|
|
```bash
|
|
-e YT_SEARCH_DELAY_MS=7000
|
|
-e YT_DOWNLOADS_PER_MINUTE=6
|
|
```
|
|
|
|
The hardened branch also includes additional internal pacing and retry coordination to reduce:
|
|
|
|
- repeated failed candidate loops
|
|
- aggressive retry bursts
|
|
- queue collisions
|
|
- YouTube anti-bot triggers
|
|
|
|
---
|
|
|
|
# yt-dlp Notes
|
|
|
|
Modern YouTube behavior changes frequently.
|
|
|
|
The hardened branch now:
|
|
|
|
- Uses explicit yt-dlp CLI execution
|
|
- Captures stderr deterministically
|
|
- Detects age-gated failures
|
|
- Detects missing downloadable formats
|
|
- Handles retry/fallback selection explicitly
|
|
|
|
Some videos may still fail due to:
|
|
|
|
- regional restrictions
|
|
- removed videos
|
|
- YouTube anti-bot measures
|
|
- invalid cookies
|
|
- unavailable formats
|
|
|
|
Recommended:
|
|
|
|
- fresh cookies.txt exports
|
|
- authenticated YouTube sessions
|
|
- moderate pacing settings
|
|
|
|
---
|
|
|
|
# Security Notes
|
|
|
|
Never commit:
|
|
|
|
- Spotify secrets
|
|
- OAuth tokens
|
|
- cookies.txt
|
|
- downloaded music
|
|
- local databases
|
|
- spooty-config/
|
|
|
|
Recommended `.gitignore` additions:
|
|
|
|
```gitignore
|
|
downloads/
|
|
config/
|
|
spooty-config/
|
|
*.sqlite
|
|
cookies.txt
|
|
.env
|
|
.env.local
|
|
```
|
|
|
|
---
|
|
|
|
# Hardened Branch Changelog
|
|
|
|
## Unreleased Hardened Changes
|
|
|
|
### Spotify
|
|
|
|
- Added Spotify OAuth login flow
|
|
- Added persistent Spotify user token storage
|
|
- Added authenticated playlist retrieval
|
|
- Fixed large playlist pagination
|
|
- Fixed playlist truncation at 100 tracks
|
|
- Added playlist retrieval debugging
|
|
- Added OAuth status endpoint
|
|
|
|
### YouTube
|
|
|
|
- Added deterministic YouTube candidate scoring
|
|
- Added duration-aware YouTube matching
|
|
- Added failed-candidate rejection memory
|
|
- Added automatic retry/fallback handling
|
|
- Added explicit YouTube candidate exclusion support
|
|
- Added detailed YouTube candidate debug logging
|
|
- Added explicit yt-dlp CLI execution
|
|
- Added deterministic yt-dlp stderr capture
|
|
- Added age-gated video detection
|
|
- Added invalid-format detection
|
|
- Added Docker-compatible cookie handling
|
|
- Added temporary cookie-copy isolation
|
|
- Added retry-aware queue coordination
|
|
|
|
### Backend
|
|
|
|
- Added safer queue pacing logic
|
|
- Added explicit search processor throttling
|
|
- Improved Docker runtime stability
|
|
- Improved logging around Spotify retrieval
|
|
- Added auth bootstrap flow support
|
|
- Added deterministic retry-state handling
|
|
|
|
### Frontend
|
|
|
|
- Added Spotify login integration
|
|
- Added frontend auth token bootstrap
|
|
- Fixed playlist progress handling for large playlists
|
|
- Improved playlist rendering stability
|
|
|
|
### Security
|
|
|
|
- Moved secrets to external env files
|
|
- Improved Docker secret handling recommendations
|
|
- Added security documentation
|
|
- Prevented accidental secret inclusion in repository
|
|
|
|
---
|
|
|
|
# Development Notes
|
|
|
|
Recommended local test run:
|
|
|
|
```bash
|
|
docker run --rm -p 3000:3000 \
|
|
--env-file /etc/tokens/spotify.env \
|
|
-e SPOTIFY_REDIRECT_URI='http://127.0.0.1:3000/api/spotify/callback' \
|
|
-e AUTH_ENABLED=true \
|
|
-e SPOOTY_AUTH_TOKEN=test-token \
|
|
-e YT_SEARCH_DELAY_MS=7000 \
|
|
-e YT_DOWNLOADS_PER_MINUTE=6 \
|
|
-e YT_COOKIES_FILE=/spooty/config/youtube.cookies.txt \
|
|
-v "$PWD/downloads:/spooty/backend/downloads" \
|
|
-v "$PWD/spooty-config:/spooty/backend/config" \
|
|
-v "/etc/tokens/youtube.cookies.txt:/spooty/config/youtube.cookies.txt:ro" \
|
|
spootyfy-hardened:local
|
|
```
|
|
|
|
---
|
|
|
|
# License
|
|
|
|
MIT
|