Preserve HLS media session cookies

2026-09-07 21:52:08 +02:00
parent baf5298631
commit d39d783738
5 changed files with 141 additions and 6 deletions
@@ -1,5 +1,10 @@
 # Changelog

+## 0.52.27 - 2026-09-07
+
+- Preserved provider CDN session cookies from HLS playlist requests across segment and encryption-key downloads, including curl fallback requests.
+- Improved curl media errors to identify the attempted browser wrappers and their HTTP status or exit code.
+
 ## 0.52.26 - 2026-09-07

 - Added curl-impersonate browser wrapper binaries to the Docker image so protected HLS segment requests can use browser-like TLS fingerprints.
@@ -68,7 +68,7 @@ Useful environment variables:
 - `KAIZOKU_REMOTE_PATH_ROOTS=/downloads,/jellyfin/tv,/jellyfin/movies` to limit remote filesystem browsing; comma-separated and platform path separators are accepted.
 - `KAIZOKU_MODE=sub`, `KAIZOKU_QUALITY=best`, and `KAIZOKU_DEBUG=1` for runtime defaults and diagnostics.
 - `KAIZOKU_SEGMENT_RETRIES=8`, `KAIZOKU_SEGMENT_RETRY_DELAY=1.25`, `KAIZOKU_SEGMENT_RETRY_MAX_DELAY=15`, and `KAIZOKU_SEGMENT_DOWNLOAD_DELAY=0` to tune retry/backoff and optional pacing behavior for native HLS segment downloads.
- `KAIZOKU_MEDIA_HTTP_CLIENT=auto` to let protected HLS segment fetches fall back from Python HTTP to `curl` or curl-impersonate after `HTTP 403`; use `curl` or `urllib` to force one client, and `KAIZOKU_CURL_BIN=/path/to/curl_chrome142` to prefer a specific curl-compatible binary.
+- `KAIZOKU_MEDIA_HTTP_CLIENT=auto` to let protected HLS segment fetches fall back from Python HTTP to `curl` or curl-impersonate after `HTTP 403`; use `curl` or `urllib` to force one client, and `KAIZOKU_CURL_BIN=/path/to/curl_chrome142` to prefer a specific curl-compatible binary. Playlist, segment, and encryption-key requests preserve provider CDN session cookies for the duration of the download process.

 ## Docker

@@ -101,7 +101,7 @@ Keep `./.kaizoku` mounted for production instances. That directory contains the

 ## Download Flow

-Kaizoku stores provider-backed show IDs as `provider:id`, for example `anikoto:some-show-slug`. Queue jobs resolve the episode source through `providers/bridge.js`, then `provider_downloader.py` downloads the media with `ffmpeg` into a staging directory. If a provider returns multiple servers for the requested sub or dub mode, Kaizoku tries the same-mode server sources in quality order before falling back to another provider. It does not use sub sources for dub downloads, or dub sources for sub downloads. If a provider returns a master HLS playlist, Kaizoku selects the highest-bandwidth variant before starting `ffmpeg`. When a media playlist uses extensionless, SnapCDN, or disguised CDN segments, Kaizoku skips direct ffmpeg and uses a StrawVerse-style segment downloader; if HLS preflight fails, it stays on that native path instead of falling through to ffmpeg. The native downloader fetches the media playlist, downloads and concatenates segments itself with browser-like media headers and the provider referer, can fall back to curl-impersonate browser wrappers after protected CDN `HTTP 403` responses, strips short PNG wrappers when present, retries temporary HTTP failures such as `429 Too Many Requests`, optionally paces requests with `KAIZOKU_SEGMENT_DOWNLOAD_DELAY`, then remuxes the local transport stream to MP4. Direct ffmpeg attempts also have a timeout guard so stalled HLS inputs can fall back cleanly. Each episode is written as a temporary `.mp4.part` file and moved into the downloads library after that episode succeeds, so already-finished episodes from a larger batch survive if a later episode fails. Retrying that failed queue job requests only the remaining episodes while keeping the original episode range for display and watchlist sync. Finalization preserves episode numbers from staged `SxxEyy` or `Episode yy` filenames before applying configured season/episode offsets.
+Kaizoku stores provider-backed show IDs as `provider:id`, for example `anikoto:some-show-slug`. Queue jobs resolve the episode source through `providers/bridge.js`, then `provider_downloader.py` downloads the media with `ffmpeg` into a staging directory. If a provider returns multiple servers for the requested sub or dub mode, Kaizoku tries the same-mode server sources in quality order before falling back to another provider. It does not use sub sources for dub downloads, or dub sources for sub downloads. If a provider returns a master HLS playlist, Kaizoku selects the highest-bandwidth variant before starting `ffmpeg`. When a media playlist uses extensionless, SnapCDN, or disguised CDN segments, Kaizoku skips direct ffmpeg and uses a StrawVerse-style segment downloader; if HLS preflight fails, it stays on that native path instead of falling through to ffmpeg. The native downloader fetches the media playlist, preserves provider CDN session cookies across playlist, segment, and encryption-key requests, downloads and concatenates segments itself with browser-like media headers and the provider referer, can fall back to curl-impersonate browser wrappers after protected CDN `HTTP 403` responses, strips short PNG wrappers when present, retries temporary HTTP failures such as `429 Too Many Requests`, optionally paces requests with `KAIZOKU_SEGMENT_DOWNLOAD_DELAY`, then remuxes the local transport stream to MP4. Direct ffmpeg attempts also have a timeout guard so stalled HLS inputs can fall back cleanly. Each episode is written as a temporary `.mp4.part` file and moved into the downloads library after that episode succeeds, so already-finished episodes from a larger batch survive if a later episode fails. Retrying that failed queue job requests only the remaining episodes while keeping the original episode range for display and watchlist sync. Finalization preserves episode numbers from staged `SxxEyy` or `Episode yy` filenames before applying configured season/episode offsets.

 If the primary provider cannot list, resolve, or download a requested episode, Kaizoku searches the same title on the remaining providers and tries the matching episode there. Existing finalization code moves staged files into the configured library layout, preserving data already present in production download folders.

@@ -1 +1 @@
-0.52.26
+0.52.27
@@ -3,12 +3,15 @@
 """Download episodes from Kaizoku's StrawVerse-compatible provider bridge."""

 import argparse
+import atexit
+import http.cookiejar
 import json
 import os
 import re
 import shutil
 import subprocess
 import sys
+import tempfile
 import time
 import urllib.error
 from urllib.parse import urljoin, urlparse
@@ -21,6 +24,8 @@ BRIDGE = PROJECT_ROOT / "providers" / "bridge.js"
 PROVIDERS = ("anikoto", "anineko", "pahe")
 RETRYABLE_HTTP_STATUS = {408, 425, 429, 500, 502, 503, 504}
 CURL_STATUS_MARKER = b"\nKAIZOKU_HTTP_STATUS:"
+MEDIA_COOKIE_JAR = http.cookiejar.CookieJar()
+MEDIA_CURL_COOKIE_DIR = None


 def clean_component(value, default="Anime"):
@@ -214,6 +219,28 @@ def request_headers(headers=None, referer=None):
    return merged


+def media_request(url, headers=None):
+    request = urllib.request.Request(url, headers=request_headers(headers))
+    MEDIA_COOKIE_JAR.add_cookie_header(request)
+    return request
+
+
+def media_cookie_header(url):
+    request = urllib.request.Request(url)
+    MEDIA_COOKIE_JAR.add_cookie_header(request)
+    return request.get_header("Cookie") or ""
+
+
+def curl_cookie_file():
+    global MEDIA_CURL_COOKIE_DIR
+    if MEDIA_CURL_COOKIE_DIR is None:
+        MEDIA_CURL_COOKIE_DIR = tempfile.TemporaryDirectory(prefix="kaizoku-media-cookies-")
+        atexit.register(MEDIA_CURL_COOKIE_DIR.cleanup)
+        cookie_file = Path(MEDIA_CURL_COOKIE_DIR.name) / "cookies.txt"
+        cookie_file.touch(mode=0o600)
+    return Path(MEDIA_CURL_COOKIE_DIR.name) / "cookies.txt"
+
+
 def curl_candidates():
    def candidate_score(name):
        text = Path(str(name or "")).name
@@ -279,8 +306,13 @@ def curl_fetch_bytes(url, headers=None, timeout=30):
    if not candidates:
        raise RuntimeError("curl is not available for media fetch fallback.")
    last_error = None
+    failures = []
+    cookie_file = curl_cookie_file()
    for curl_bin in candidates:
        clean_headers = curl_request_headers(curl_bin, headers)
+        cookie_header = media_cookie_header(url)
+        if cookie_header and not any(str(key).lower() == "cookie" for key in clean_headers):
+            clean_headers["Cookie"] = cookie_header
        cmd = [
            curl_bin,
            "--location",
@@ -288,6 +320,10 @@ def curl_fetch_bytes(url, headers=None, timeout=30):
            "--show-error",
            "--max-time",
            str(max(1, int(timeout or 30))),
+            "--cookie",
+            str(cookie_file),
+            "--cookie-jar",
+            str(cookie_file),
        ]
        for key, value in clean_headers.items():
            if value:
@@ -305,7 +341,12 @@ def curl_fetch_bytes(url, headers=None, timeout=30):
            status = 0
        if 200 <= status < 300 and proc.returncode == 0:
            return payload
-        message = (proc.stderr or b"curl media fetch failed").decode("utf-8", errors="replace").strip()
+        client_name = Path(curl_bin).name
+        detail = (proc.stderr or b"").decode("utf-8", errors="replace").strip()
+        failures.append(f"{client_name}: HTTP {status or 0}" if status else f"{client_name}: exit {proc.returncode}")
+        message = "; ".join(failures)
+        if detail:
+            message = f"{message} ({detail})"
        last_error = urllib.error.HTTPError(str(url), status or 0, message, hdrs={}, fp=None)
    raise last_error or RuntimeError("curl media fetch failed.")

@@ -314,9 +355,10 @@ def fetch_bytes(url, headers=None, timeout=30):
    client = str(os.environ.get("KAIZOKU_MEDIA_HTTP_CLIENT") or "auto").strip().lower()
    if client == "curl":
        return curl_fetch_bytes(url, headers=headers, timeout=timeout)
-    request = urllib.request.Request(url, headers=request_headers(headers))
+    request = media_request(url, headers=headers)
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
+            MEDIA_COOKIE_JAR.extract_cookies(response, request)
            return response.read()
    except urllib.error.HTTPError as exc:
        if client != "urllib" and exc.code == 403 and curl_candidates():
@@ -1,5 +1,7 @@
 #!/usr/bin/env python3
 import base64
+from email.message import Message
+import http.cookiejar
 import importlib.util
 import io
 import json
@@ -4535,7 +4537,9 @@ seg-3.ts
    def test_curl_fetch_bytes_returns_body_before_status_marker(self):
        proc = mock.Mock(returncode=0, stdout=b"video-bytes\nKAIZOKU_HTTP_STATUS:200", stderr=b"")

-        with mock.patch.object(provider_downloader, "curl_candidates", return_value=["/usr/bin/curl"]), mock.patch.object(
+        with tempfile.TemporaryDirectory() as temp_root, mock.patch.object(
+            provider_downloader, "curl_cookie_file", return_value=Path(temp_root) / "cookies.txt"
+        ), mock.patch.object(provider_downloader, "curl_candidates", return_value=["/usr/bin/curl"]), mock.patch.object(
            provider_downloader.subprocess, "run", return_value=proc
        ) as run:
            data = provider_downloader.curl_fetch_bytes("https://cdn.example/segment.ts", headers={"Referer": "https://player.example/"})
@@ -4543,6 +4547,90 @@ seg-3.ts
        self.assertEqual(data, b"video-bytes")
        self.assertIn("--header", run.call_args.args[0])
        self.assertIn("Referer: https://player.example/", run.call_args.args[0])
+        self.assertIn("--cookie-jar", run.call_args.args[0])
+
+    def test_media_request_preserves_playlist_cookie_for_segments(self):
+        jar = http.cookiejar.CookieJar()
+        cookie = http.cookiejar.Cookie(
+            version=0,
+            name="cdn_session",
+            value="allowed",
+            port=None,
+            port_specified=False,
+            domain="cdn.example",
+            domain_specified=True,
+            domain_initial_dot=False,
+            path="/",
+            path_specified=True,
+            secure=True,
+            expires=None,
+            discard=True,
+            comment=None,
+            comment_url=None,
+            rest={},
+            rfc2109=False,
+        )
+        jar.set_cookie(cookie)
+
+        with mock.patch.object(provider_downloader, "MEDIA_COOKIE_JAR", jar):
+            request = provider_downloader.media_request("https://cdn.example/video/segment.ts")
+
+        self.assertEqual(request.get_header("Cookie"), "cdn_session=allowed")
+
+    def test_fetch_bytes_keeps_set_cookie_for_next_media_request(self):
+        class Response:
+            def __init__(self, url, body, cookie=""):
+                self.url = url
+                self.body = body
+                self.headers = Message()
+                if cookie:
+                    self.headers.add_header("Set-Cookie", cookie)
+
+            def __enter__(self):
+                return self
+
+            def __exit__(self, *_args):
+                return False
+
+            def info(self):
+                return self.headers
+
+            def geturl(self):
+                return self.url
+
+            def read(self):
+                return self.body
+
+        requests = []
+        responses = iter(
+            [
+                Response("https://cdn.example/index.m3u8", b"playlist", "cdn_session=allowed; Path=/; Secure"),
+                Response("https://cdn.example/video/segment.ts", b"video"),
+            ]
+        )
+
+        def fake_urlopen(request, timeout):
+            requests.append(request)
+            return next(responses)
+
+        with mock.patch.object(provider_downloader, "MEDIA_COOKIE_JAR", http.cookiejar.CookieJar()), mock.patch.object(
+            provider_downloader.urllib.request, "urlopen", side_effect=fake_urlopen
+        ):
+            provider_downloader.fetch_bytes("https://cdn.example/index.m3u8")
+            provider_downloader.fetch_bytes("https://cdn.example/video/segment.ts")
+
+        self.assertEqual(requests[1].get_header("Cookie"), "cdn_session=allowed")
+
+    def test_curl_fetch_uses_python_media_session_cookie(self):
+        proc = mock.Mock(returncode=0, stdout=b"video\nKAIZOKU_HTTP_STATUS:200", stderr=b"")
+        with tempfile.TemporaryDirectory() as temp_root, mock.patch.object(
+            provider_downloader, "curl_cookie_file", return_value=Path(temp_root) / "cookies.txt"
+        ), mock.patch.object(provider_downloader, "curl_candidates", return_value=["/usr/bin/curl"]), mock.patch.object(
+            provider_downloader, "media_cookie_header", return_value="cdn_session=allowed"
+        ), mock.patch.object(provider_downloader.subprocess, "run", return_value=proc) as run:
+            provider_downloader.curl_fetch_bytes("https://cdn.example/segment.ts")
+
+        self.assertIn("Cookie: cdn_session=allowed", run.call_args.args[0])

    def test_curl_request_headers_preserve_browser_wrapper_defaults(self):
        headers = provider_downloader.curl_request_headers(
@@ -1 +1 @@
 .52.26
 .52.27