Fix native HLS segment request headers

CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## 0.52.24 - 2026-09-07
+
+- Changed native HLS media requests to keep the provider referer without adding a synthetic `Origin` header, improving compatibility with CDNs that reject segment fetches with `HTTP 403`.
+- Fixed native HLS segment failures so a failed source is not retried from preflight before the downloader moves on to the next source.
+
 ## 0.52.23 - 2026-09-05
 
 - Changed HLS preflight failures to stay on the native segment path instead of falling through to direct ffmpeg, preventing SnapCDN playlists from being hammered after a guard failure.

README.md

@@ -100,7 +100,7 @@ Keep `./.kaizoku` mounted for production instances. That directory contains the
 
 ## Download Flow
 
-Kaizoku stores provider-backed show IDs as `provider:id`, for example `anikoto:some-show-slug`. Queue jobs resolve the episode source through `providers/bridge.js`, then `provider_downloader.py` downloads the media with `ffmpeg` into a staging directory. If a provider returns multiple servers for the requested sub or dub mode, Kaizoku tries the same-mode server sources in quality order before falling back to another provider. It does not use sub sources for dub downloads, or dub sources for sub downloads. If a provider returns a master HLS playlist, Kaizoku selects the highest-bandwidth variant before starting `ffmpeg`. When a media playlist uses extensionless, SnapCDN, or disguised CDN segments, Kaizoku skips direct ffmpeg and uses a StrawVerse-style segment downloader; if HLS preflight fails, it stays on that native path instead of falling through to ffmpeg. The native downloader fetches the media playlist, downloads and concatenates segments itself, strips short PNG wrappers when present, retries temporary HTTP failures such as `429 Too Many Requests`, optionally paces requests with `KAIZOKU_SEGMENT_DOWNLOAD_DELAY`, then remuxes the local transport stream to MP4. Direct ffmpeg attempts also have a timeout guard so stalled HLS inputs can fall back cleanly. Each episode is written as a temporary `.mp4.part` file and moved into the downloads library after that episode succeeds, so already-finished episodes from a larger batch survive if a later episode fails. Retrying that failed queue job requests only the remaining episodes while keeping the original episode range for display and watchlist sync. Finalization preserves episode numbers from staged `SxxEyy` or `Episode yy` filenames before applying configured season/episode offsets.
+Kaizoku stores provider-backed show IDs as `provider:id`, for example `anikoto:some-show-slug`. Queue jobs resolve the episode source through `providers/bridge.js`, then `provider_downloader.py` downloads the media with `ffmpeg` into a staging directory. If a provider returns multiple servers for the requested sub or dub mode, Kaizoku tries the same-mode server sources in quality order before falling back to another provider. It does not use sub sources for dub downloads, or dub sources for sub downloads. If a provider returns a master HLS playlist, Kaizoku selects the highest-bandwidth variant before starting `ffmpeg`. When a media playlist uses extensionless, SnapCDN, or disguised CDN segments, Kaizoku skips direct ffmpeg and uses a StrawVerse-style segment downloader; if HLS preflight fails, it stays on that native path instead of falling through to ffmpeg. The native downloader fetches the media playlist, downloads and concatenates segments itself with browser-like media headers and the provider referer, strips short PNG wrappers when present, retries temporary HTTP failures such as `429 Too Many Requests`, optionally paces requests with `KAIZOKU_SEGMENT_DOWNLOAD_DELAY`, then remuxes the local transport stream to MP4. Direct ffmpeg attempts also have a timeout guard so stalled HLS inputs can fall back cleanly. Each episode is written as a temporary `.mp4.part` file and moved into the downloads library after that episode succeeds, so already-finished episodes from a larger batch survive if a later episode fails. Retrying that failed queue job requests only the remaining episodes while keeping the original episode range for display and watchlist sync. Finalization preserves episode numbers from staged `SxxEyy` or `Episode yy` filenames before applying configured season/episode offsets.
 
 If the primary provider cannot list, resolve, or download a requested episode, Kaizoku searches the same title on the remaining providers and tries the matching episode there. Existing finalization code moves staged files into the configured library layout, preserving data already present in production download folders.
 

VERSION

@@ -1 +1 @@
-0.52.23
+0.52.24

provider_downloader.py

@@ -194,23 +194,21 @@ def ffmpeg_headers(headers):
 
 def request_headers(headers=None, referer=None):
     merged = {
-        "User-Agent": "Mozilla/5.0",
+        "User-Agent": (
+            "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) "
+            "Gecko/20100101 Firefox/121.0"
+        ),
         "Accept": "*/*",
         "Accept-Language": "en-US,en;q=0.9",
         "Cache-Control": "no-cache",
         "Pragma": "no-cache",
+        "Sec-Fetch-Dest": "video",
+        "Sec-Fetch-Mode": "no-cors",
+        "Sec-Fetch-Site": "cross-site",
     }
     merged.update(headers or {})
     if referer and not (merged.get("Referer") or merged.get("referer")):
         merged["Referer"] = referer
-    final_referer = merged.get("Referer") or merged.get("referer")
-    if final_referer and not (merged.get("Origin") or merged.get("origin")):
-        try:
-            parsed = urlparse(final_referer)
-            if parsed.scheme and parsed.netloc:
-                merged["Origin"] = f"{parsed.scheme}://{parsed.netloc}"
-        except Exception:
-            pass
     return merged
 
 
@@ -591,30 +589,6 @@ def download_episode(stream, target, episode_number=None, episode_index=None, ep
         try:
             media_url, playlist = hls_media_playlist(stream, input_url)
             segments = parse_hls_segments(media_url, playlist)
-            if hls_segments_need_native_download(segments):
-                message = "HLS playlist uses provider segments that need native download; skipping direct ffmpeg."
-                print(message, flush=True)
-                emit_progress(
-                    phase="preflight",
-                    message=message,
-                    episode=episode_number,
-                    episode_index=episode_index,
-                    episode_total=episode_total,
-                    percent=0,
-                )
-                code = download_hls_segments(
-                    stream,
-                    media_url,
-                    target,
-                    partial,
-                    episode_number=episode_number,
-                    episode_index=episode_index,
-                    episode_total=episode_total,
-                    playlist_text=playlist,
-                )
-                if code == 0:
-                    return 0
-                return code
         except Exception as exc:
             print(f"Native HLS preflight failed: {exc}", file=sys.stderr)
             try:
@@ -633,6 +607,34 @@ def download_episode(stream, target, episode_number=None, episode_index=None, ep
             except Exception as segment_exc:
                 print(f"Segment downloader failed after HLS preflight error: {segment_exc}", file=sys.stderr)
                 return 1
+        if hls_segments_need_native_download(segments):
+            message = "HLS playlist uses provider segments that need native download; skipping direct ffmpeg."
+            print(message, flush=True)
+            emit_progress(
+                phase="preflight",
+                message=message,
+                episode=episode_number,
+                episode_index=episode_index,
+                episode_total=episode_total,
+                percent=0,
+            )
+            try:
+                code = download_hls_segments(
+                    stream,
+                    media_url,
+                    target,
+                    partial,
+                    episode_number=episode_number,
+                    episode_index=episode_index,
+                    episode_total=episode_total,
+                    playlist_text=playlist,
+                )
+                if code == 0:
+                    return 0
+                return code
+            except Exception as segment_exc:
+                print(f"Segment downloader failed: {segment_exc}", file=sys.stderr)
+                return 1
     attempts = [
         [
             *ffmpeg_base_command(stream, input_url),

test_app.py

@@ -4522,6 +4522,13 @@ seg-3.ts
 
         self.assertEqual(provider_downloader.strip_png_header(payload), b"VIDEO")
 
+    def test_request_headers_keep_referer_without_synthetic_origin(self):
+        headers = provider_downloader.request_headers({"Referer": "https://megaplay.example/"})
+
+        self.assertEqual(headers["Referer"], "https://megaplay.example/")
+        self.assertNotIn("Origin", headers)
+        self.assertEqual(headers["Sec-Fetch-Dest"], "video")
+
     def test_hls_segments_need_native_download_for_extensionless_provider_urls(self):
         self.assertTrue(
             provider_downloader.hls_segments_need_native_download(
@@ -4561,6 +4568,25 @@ seg-3.ts
         download_segments.assert_called_once()
         run_ffmpeg.assert_not_called()
 
+    def test_native_segment_failure_does_not_retry_same_source_from_preflight(self):
+        playlist = """#EXTM3U
+#EXTINF:4.0,
+https://shard-102.snapcdn.top/anime/show/episode/seg-f1-00083.css
+"""
+        with tempfile.TemporaryDirectory() as temp_root:
+            target = Path(temp_root) / "Episode.mp4"
+            stream = {"url": "https://cdn.example/master.m3u8", "isM3U8": True, "headers": {}}
+            with mock.patch.object(provider_downloader, "resolve_hls_input_url", return_value=stream["url"]), mock.patch.object(
+                provider_downloader, "hls_media_playlist", return_value=(stream["url"], playlist)
+            ), mock.patch.object(
+                provider_downloader, "download_hls_segments", side_effect=RuntimeError("segment forbidden")
+            ) as download_segments, mock.patch.object(provider_downloader, "run_ffmpeg") as run_ffmpeg:
+                code = provider_downloader.download_episode(stream, target, episode_number=1)
+
+        self.assertEqual(code, 1)
+        download_segments.assert_called_once()
+        run_ffmpeg.assert_not_called()
+
     def test_fetch_bytes_with_retries_recovers_from_segment_rate_limit(self):
         error = urllib.error.HTTPError(
             "https://cdn.example/video/raw-segment",