Add curl fallback for protected HLS segments

CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## 0.52.25 - 2026-09-07
+
+- Added a curl-backed media fetch fallback for native HLS segment requests that receive `HTTP 403` from Python's HTTP client.
+- Installed `curl` in the Docker image and documented `KAIZOKU_MEDIA_HTTP_CLIENT` and `KAIZOKU_CURL_BIN` controls for protected provider CDNs.
+
 ## 0.52.24 - 2026-09-07
 
 - Changed native HLS media requests to keep the provider referer without adding a synthetic `Origin` header, improving compatibility with CDNs that reject segment fetches with `HTTP 403`.

Dockerfile

@@ -13,6 +13,7 @@ RUN apt-get update \
     && apt-get install -y --no-install-recommends \
         bash \
         ca-certificates \
+        curl \
         ffmpeg \
         nodejs \
         npm \

README.md

@@ -68,6 +68,7 @@ Useful environment variables:
 - `KAIZOKU_REMOTE_PATH_ROOTS=/downloads,/jellyfin/tv,/jellyfin/movies` to limit remote filesystem browsing; comma-separated and platform path separators are accepted.
 - `KAIZOKU_MODE=sub`, `KAIZOKU_QUALITY=best`, and `KAIZOKU_DEBUG=1` for runtime defaults and diagnostics.
 - `KAIZOKU_SEGMENT_RETRIES=8`, `KAIZOKU_SEGMENT_RETRY_DELAY=1.25`, `KAIZOKU_SEGMENT_RETRY_MAX_DELAY=15`, and `KAIZOKU_SEGMENT_DOWNLOAD_DELAY=0` to tune retry/backoff and optional pacing behavior for native HLS segment downloads.
+- `KAIZOKU_MEDIA_HTTP_CLIENT=auto` to let protected HLS segment fetches fall back from Python HTTP to `curl` after `HTTP 403`; use `curl` or `urllib` to force one client, and `KAIZOKU_CURL_BIN=/path/to/curl` to prefer a specific curl-compatible binary.
 
 ## Docker
 
@@ -84,7 +85,7 @@ The compose file builds the local checkout and runs Kaizoku on port `8421` by de
 
 Use `/jellyfin/tv` for the Jellyfin TV directory and `/jellyfin/movies` for the Jellyfin movie directory inside the Config page. Override `JELLYFIN_TV_DIR` and `JELLYFIN_MOVIE_DIR` to point those container paths at your real host Jellyfin library folders.
 
-The Docker image installs Python, Node.js, npm, `ffmpeg`, `openssl`, and `util-linux` for optional UID/GID switching, then runs `npm ci --omit=dev` for the provider bridge. Downloads go through Kaizoku's provider bridge and `provider_downloader.py`.
+The Docker image installs Python, Node.js, npm, `curl`, `ffmpeg`, `openssl`, and `util-linux` for optional UID/GID switching, then runs `npm ci --omit=dev` for the provider bridge. Downloads go through Kaizoku's provider bridge and `provider_downloader.py`.
 
 Useful Compose overrides:
 
@@ -100,7 +101,7 @@ Keep `./.kaizoku` mounted for production instances. That directory contains the
 
 ## Download Flow
 
-Kaizoku stores provider-backed show IDs as `provider:id`, for example `anikoto:some-show-slug`. Queue jobs resolve the episode source through `providers/bridge.js`, then `provider_downloader.py` downloads the media with `ffmpeg` into a staging directory. If a provider returns multiple servers for the requested sub or dub mode, Kaizoku tries the same-mode server sources in quality order before falling back to another provider. It does not use sub sources for dub downloads, or dub sources for sub downloads. If a provider returns a master HLS playlist, Kaizoku selects the highest-bandwidth variant before starting `ffmpeg`. When a media playlist uses extensionless, SnapCDN, or disguised CDN segments, Kaizoku skips direct ffmpeg and uses a StrawVerse-style segment downloader; if HLS preflight fails, it stays on that native path instead of falling through to ffmpeg. The native downloader fetches the media playlist, downloads and concatenates segments itself with browser-like media headers and the provider referer, strips short PNG wrappers when present, retries temporary HTTP failures such as `429 Too Many Requests`, optionally paces requests with `KAIZOKU_SEGMENT_DOWNLOAD_DELAY`, then remuxes the local transport stream to MP4. Direct ffmpeg attempts also have a timeout guard so stalled HLS inputs can fall back cleanly. Each episode is written as a temporary `.mp4.part` file and moved into the downloads library after that episode succeeds, so already-finished episodes from a larger batch survive if a later episode fails. Retrying that failed queue job requests only the remaining episodes while keeping the original episode range for display and watchlist sync. Finalization preserves episode numbers from staged `SxxEyy` or `Episode yy` filenames before applying configured season/episode offsets.
+Kaizoku stores provider-backed show IDs as `provider:id`, for example `anikoto:some-show-slug`. Queue jobs resolve the episode source through `providers/bridge.js`, then `provider_downloader.py` downloads the media with `ffmpeg` into a staging directory. If a provider returns multiple servers for the requested sub or dub mode, Kaizoku tries the same-mode server sources in quality order before falling back to another provider. It does not use sub sources for dub downloads, or dub sources for sub downloads. If a provider returns a master HLS playlist, Kaizoku selects the highest-bandwidth variant before starting `ffmpeg`. When a media playlist uses extensionless, SnapCDN, or disguised CDN segments, Kaizoku skips direct ffmpeg and uses a StrawVerse-style segment downloader; if HLS preflight fails, it stays on that native path instead of falling through to ffmpeg. The native downloader fetches the media playlist, downloads and concatenates segments itself with browser-like media headers and the provider referer, can fall back to `curl` after protected CDN `HTTP 403` responses, strips short PNG wrappers when present, retries temporary HTTP failures such as `429 Too Many Requests`, optionally paces requests with `KAIZOKU_SEGMENT_DOWNLOAD_DELAY`, then remuxes the local transport stream to MP4. Direct ffmpeg attempts also have a timeout guard so stalled HLS inputs can fall back cleanly. Each episode is written as a temporary `.mp4.part` file and moved into the downloads library after that episode succeeds, so already-finished episodes from a larger batch survive if a later episode fails. Retrying that failed queue job requests only the remaining episodes while keeping the original episode range for display and watchlist sync. Finalization preserves episode numbers from staged `SxxEyy` or `Episode yy` filenames before applying configured season/episode offsets.
 
 If the primary provider cannot list, resolve, or download a requested episode, Kaizoku searches the same title on the remaining providers and tries the matching episode there. Existing finalization code moves staged files into the configured library layout, preserving data already present in production download folders.
 

VERSION

@@ -1 +1 @@
-0.52.24
+0.52.25

provider_downloader.py

@@ -10,6 +10,7 @@ import shutil
 import subprocess
 import sys
 import time
+import urllib.error
 from urllib.parse import urljoin, urlparse
 import urllib.request
 from pathlib import Path
@@ -19,6 +20,7 @@ PROJECT_ROOT = Path(__file__).resolve().parent
 BRIDGE = PROJECT_ROOT / "providers" / "bridge.js"
 PROVIDERS = ("anikoto", "anineko", "pahe")
 RETRYABLE_HTTP_STATUS = {408, 425, 429, 500, 502, 503, 504}
+CURL_STATUS_MARKER = b"\nKAIZOKU_HTTP_STATUS:"
 
 
 def clean_component(value, default="Anime"):
@@ -212,10 +214,75 @@ def request_headers(headers=None, referer=None):
     return merged
 
 
+def curl_candidates():
+    configured = str(os.environ.get("KAIZOKU_CURL_BIN") or "").strip()
+    names = [
+        configured,
+        "curl_chrome120",
+        "curl_chrome116",
+        "curl_chrome110",
+        "curl-impersonate",
+        "curl",
+    ]
+    candidates = []
+    for name in names:
+        if not name or name in candidates:
+            continue
+        resolved = shutil.which(name)
+        if resolved:
+            candidates.append(resolved)
+    return candidates
+
+
+def curl_fetch_bytes(url, headers=None, timeout=30):
+    candidates = curl_candidates()
+    if not candidates:
+        raise RuntimeError("curl is not available for media fetch fallback.")
+    clean_headers = request_headers(headers)
+    last_error = None
+    for curl_bin in candidates:
+        cmd = [
+            curl_bin,
+            "--location",
+            "--silent",
+            "--show-error",
+            "--max-time",
+            str(max(1, int(timeout or 30))),
+        ]
+        for key, value in clean_headers.items():
+            if value:
+                cmd.extend(["--header", f"{key}: {value}"])
+        cmd.extend(["--write-out", f"{CURL_STATUS_MARKER.decode('ascii')}%{{http_code}}", "--output", "-", str(url)])
+        proc = subprocess.run(cmd, capture_output=True, check=False)
+        payload, marker, status_bytes = proc.stdout.rpartition(CURL_STATUS_MARKER)
+        if marker:
+            try:
+                status = int(status_bytes.strip() or b"0")
+            except ValueError:
+                status = 0
+        else:
+            payload = proc.stdout
+            status = 0
+        if 200 <= status < 300 and proc.returncode == 0:
+            return payload
+        message = (proc.stderr or b"curl media fetch failed").decode("utf-8", errors="replace").strip()
+        last_error = urllib.error.HTTPError(str(url), status or 0, message, hdrs={}, fp=None)
+    raise last_error or RuntimeError("curl media fetch failed.")
+
+
 def fetch_bytes(url, headers=None, timeout=30):
+    client = str(os.environ.get("KAIZOKU_MEDIA_HTTP_CLIENT") or "auto").strip().lower()
+    if client == "curl":
+        return curl_fetch_bytes(url, headers=headers, timeout=timeout)
     request = urllib.request.Request(url, headers=request_headers(headers))
-    with urllib.request.urlopen(request, timeout=timeout) as response:
+    try:
-        return response.read()
+        with urllib.request.urlopen(request, timeout=timeout) as response:
+            return response.read()
+    except urllib.error.HTTPError as exc:
+        if client != "urllib" and exc.code == 403 and curl_candidates():
+            print(f"urllib media fetch got HTTP 403; retrying with curl fallback.", file=sys.stderr, flush=True)
+            return curl_fetch_bytes(url, headers=headers, timeout=timeout)
+        raise
 
 
 def int_env(name, default, minimum=0):

test_app.py

@@ -4529,6 +4529,38 @@ seg-3.ts
         self.assertNotIn("Origin", headers)
         self.assertEqual(headers["Sec-Fetch-Dest"], "video")
 
+    def test_curl_fetch_bytes_returns_body_before_status_marker(self):
+        proc = mock.Mock(returncode=0, stdout=b"video-bytes\nKAIZOKU_HTTP_STATUS:200", stderr=b"")
+
+        with mock.patch.object(provider_downloader, "curl_candidates", return_value=["/usr/bin/curl"]), mock.patch.object(
+            provider_downloader.subprocess, "run", return_value=proc
+        ) as run:
+            data = provider_downloader.curl_fetch_bytes("https://cdn.example/segment.ts", headers={"Referer": "https://player.example/"})
+
+        self.assertEqual(data, b"video-bytes")
+        self.assertIn("--header", run.call_args.args[0])
+        self.assertIn("Referer: https://player.example/", run.call_args.args[0])
+
+    def test_fetch_bytes_retries_forbidden_media_with_curl_fallback(self):
+        error = urllib.error.HTTPError(
+            "https://cdn.example/segment.ts",
+            403,
+            "Forbidden",
+            hdrs={},
+            fp=None,
+        )
+
+        try:
+            with mock.patch.object(provider_downloader.urllib.request, "urlopen", side_effect=error), mock.patch.object(
+                provider_downloader, "curl_candidates", return_value=["/usr/bin/curl"]
+            ), mock.patch.object(provider_downloader, "curl_fetch_bytes", return_value=b"video") as curl_fetch:
+                data = provider_downloader.fetch_bytes("https://cdn.example/segment.ts")
+        finally:
+            error.close()
+
+        self.assertEqual(data, b"video")
+        curl_fetch.assert_called_once()
+
     def test_hls_segments_need_native_download_for_extensionless_provider_urls(self):
         self.assertTrue(
             provider_downloader.hls_segments_need_native_download(