Update providers and deduplicate streams

2026-09-18 21:45:28 +02:00
parent 9c4c6437f4
commit c71f3db6a5
7 changed files with 387 additions and 65 deletions
@@ -1,5 +1,11 @@
 # Changelog

+## 0.52.33 - 2026-09-18
+
+- Updated AniNeko to `3.0.4`, using its current recently-added catalog route.
+- Updated AnimePahe to `5.0.1`, adding upstream request pacing, rate-limit backoff, response validation, and stale UUID recovery while preserving Kaizoku's newer Anikoto patches.
+- Deduplicated provider streams that resolve to the same URL and request headers so server aliases no longer repeat a failing CDN request and trigger avoidable `HTTP 429` responses.
+
 ## 0.52.32 - 2026-09-10

 - Preferred reachable alternate Megaplay CDN source variants for Anikoto streams, fixing real episode downloads where the default decoded CDN playlist returned `HTTP 403`.
@@ -103,6 +103,8 @@ Keep `./.kaizoku` mounted for production instances. That directory contains the

 Kaizoku stores provider-backed show IDs as `provider:id`, for example `anikoto:some-show-slug`. Queue jobs resolve the episode source through `providers/bridge.js`, then `provider_downloader.py` downloads the media with `ffmpeg` into a staging directory. If a provider returns multiple servers for the requested sub or dub mode, Kaizoku tries every same-mode server source in quality order before falling back to another provider, including when resolving one server's playlist raises an HTTP error. It does not use sub sources for dub downloads, or dub sources for sub downloads. If a provider returns a master HLS playlist, Kaizoku selects the highest-bandwidth variant before starting `ffmpeg`. When a media playlist uses extensionless, SnapCDN, or disguised CDN segments, Kaizoku skips direct ffmpeg and uses a StrawVerse-style segment downloader; if HLS preflight fails, it stays on that native path instead of falling through to ffmpeg. The native downloader fetches the media playlist, preserves provider CDN session cookies across playlist, segment, and encryption-key requests, downloads and concatenates segments itself with browser-like media headers and the provider referer, can fall back to curl-impersonate browser wrappers after protected CDN `HTTP 403` responses, strips short PNG wrappers when present, retries temporary HTTP failures such as `429 Too Many Requests`, optionally paces requests with `KAIZOKU_SEGMENT_DOWNLOAD_DELAY`, then remuxes the local transport stream to MP4. Direct ffmpeg attempts also have a timeout guard so stalled HLS inputs can fall back cleanly. Each episode is written as a temporary `.mp4.part` file and moved into the downloads library after that episode succeeds, so already-finished episodes from a larger batch survive if a later episode fails. Retrying that failed queue job requests only the remaining episodes while keeping the original episode range for display and watchlist sync. Finalization preserves episode numbers from staged `SxxEyy` or `Episode yy` filenames before applying configured season/episode offsets.

+Provider server aliases that resolve to the same URL and request headers are downloaded only once, avoiding duplicate CDN requests and unnecessary rate limiting.
+
 If the primary provider cannot list, resolve, or download a requested episode, Kaizoku searches the same title on the remaining providers and tries the matching episode there. Existing finalization code moves staged files into the configured library layout, preserving data already present in production download folders.

 ## Data Safety
@@ -114,7 +116,7 @@ Kaizoku uses additive SQLite migrations for queue and watchlist schema changes.
 The Anikoto, AniNeko, and AnimePahe parser modules in `providers/extensions/Anime/` are adapted from [TheYogMehta/extensions](https://github.com/TheYogMehta/extensions) and retain their GPL/license headers. The bundled anime provider versions are currently:

 - Anikoto `5.0.5`
- AniNeko `3.0.3`
- AnimePahe `4.0.1`
+- AniNeko `3.0.4`
+- AnimePahe `5.0.1`

 The Config page checks the bundled JavaScript against the upstream `main` branch and reports whether upstream updates are available. If Kaizoku carries a newer local provider patch than upstream, such as the patched Anikoto module, the Config page reports it separately as a local change. Kaizoku acts as a local client-side parser/downloader wrapper and does not host media.
@@ -1 +1 @@
-0.52.32
+0.52.33
@@ -96,6 +96,7 @@ async function resolveAll(provider, episodeId, mode, quality) {
  if (!sources.length) throw new Error("No episode sources were returned.");
  sources.sort((a, b) => qualityScore(b, quality) - qualityScore(a, quality));
  const resolvedSources = [];
+  const resolvedSourceKeys = new Set();
  for (const source of sources) {
    let resolved = null;
    try {
@@ -117,6 +118,12 @@ async function resolveAll(provider, episodeId, mode, quality) {
        }
      } catch (_) {}
      if (!headers.Referer && fallbackReferer) headers.Referer = fallbackReferer;
+      const headerKey = Object.entries(headers)
+        .map(([key, value]) => [String(key).toLowerCase(), String(value)])
+        .sort(([left], [right]) => left.localeCompare(right));
+      const sourceKey = `${resolved.url}\n${JSON.stringify(headerKey)}`;
+      if (resolvedSourceKeys.has(sourceKey)) continue;
+      resolvedSourceKeys.add(sourceKey);
      resolvedSources.push({
        url: resolved.url,
        isM3U8: Boolean(resolved.isM3U8 || String(resolved.url).includes(".m3u8")),
@@ -106,7 +106,7 @@ async function fetchRecentEpisodes(filters = {}) {
  try {
    const page = filters?.page || 1;
    const { data: html } = await global.axios.get(
-      `${baseUrl}/updates?page=${page}`,
+      `${baseUrl}/browse?sort=recently_added&page=${page}`,
    );
    const $ = cheerio.load(html);
    const results = [];
@@ -623,7 +623,7 @@ async function processEmbedServer(server) {

 module.exports = {
  name: "anineko",
-  version: "3.0.3",
+  version: "3.0.4",
  SearchAnime,
  AnimeInfo,
  fetchEpisodeSources,
@@ -28,10 +28,138 @@ const cheerio = require("cheerio");
 // variables
 const baseUrl = "https://animepahe.pw";

+function notifyRenderer(channel, payload) {
+  if (typeof global.sendToRenderer === "function") {
+    try {
+      global.sendToRenderer(channel, payload);
+    } catch (_) {}
+  }
+}
+
+let lastRequestTime = 0;
+const MIN_REQUEST_INTERVAL = 1500;
+
+async function safeGet(url, config = {}, maxRetries = 5) {
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    const now = Date.now();
+    const timeSinceLast = now - lastRequestTime;
+    if (timeSinceLast < MIN_REQUEST_INTERVAL) {
+      await new Promise((resolve) =>
+        setTimeout(resolve, MIN_REQUEST_INTERVAL - timeSinceLast),
+      );
+    }
+    lastRequestTime = Date.now();
+
+    const isApi = url.includes("/api?");
+    const mergedHeaders = {
+      Referer: baseUrl + "/",
+      ...(isApi
+        ? {
+            "X-Requested-With": "XMLHttpRequest",
+            Accept: "application/json, text/javascript, */*; q=0.01",
+          }
+        : {
+            Accept:
+              "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
+          }),
+      ...(config.headers || {}),
+    };
+    const reqConfig = {
+      ...config,
+      headers: mergedHeaders,
+    };
+
+    try {
+      const response = await global.axios.get(url, reqConfig);
+      let data = response?.data;
+
+      if (isApi && typeof data === "string") {
+        try {
+          data = JSON.parse(data);
+          response.data = data;
+        } catch (_) {}
+      }
+
+      const isRateLimited =
+        response?.status === 429 ||
+        data?.status === 429 ||
+        data?.error_code === 1015 ||
+        data?.title?.includes("rate limited") ||
+        (typeof data === "string" &&
+          (data.includes("error code: 1015") || data.includes("rate limited")));
+
+      if (isRateLimited) {
+        if (attempt < maxRetries) {
+          const totalWaitSecs =
+            Math.max(4, Math.ceil(data?.retry_after || 4)) * attempt;
+          console.warn(
+            `[AnimePahe] Rate limited (Attempt ${attempt}/${maxRetries}). Waiting ${totalWaitSecs}s...`,
+          );
+          for (let sec = totalWaitSecs; sec > 0; sec--) {
+            notifyRenderer("catalog-loading-status", {
+              text: `Rate limited by AnimePahe. Retrying in ${sec}s...`,
+            });
+            await new Promise((resolve) => setTimeout(resolve, 1000));
+          }
+          notifyRenderer("catalog-loading-status", {
+            text: "Retrying AnimePahe fetch...",
+          });
+          continue;
+        }
+      }
+
+      notifyRenderer("catalog-loading-status", {
+        text: "",
+      });
+      return response;
+    } catch (err) {
+      const status = err.response?.status;
+      if (status === 404) {
+        notifyRenderer("catalog-loading-status", { text: "" });
+        throw err;
+      }
+      if (status === 403 && attempt < maxRetries) {
+        console.warn(
+          `[AnimePahe] HTTP 403 on attempt ${attempt}/${maxRetries}. Checking clearance...`,
+        );
+        try {
+          if (global.cloudflarebypass) {
+            await global.cloudflarebypass(url, false, baseUrl + "/");
+          }
+        } catch (_) {}
+      }
+      if (attempt < maxRetries) {
+        const totalWaitSecs = status === 429 ? 8 * attempt : 3 * attempt;
+        console.warn(
+          `[AnimePahe] HTTP ${status || "Error"}. Waiting ${totalWaitSecs}s before retry ${attempt}/${maxRetries}...`,
+        );
+        for (let sec = totalWaitSecs; sec > 0; sec--) {
+          notifyRenderer("catalog-loading-status", {
+            text:
+              status === 429
+                ? `Rate limited by AnimePahe. Retrying in ${sec}s...`
+                : `Retrying AnimePahe in ${sec}s...`,
+          });
+          await new Promise((resolve) => setTimeout(resolve, 1000));
+        }
+        notifyRenderer("catalog-loading-status", {
+          text: "Retrying AnimePahe fetch...",
+        });
+
+        continue;
+      }
+      notifyRenderer("catalog-loading-status", {
+        text: "",
+      });
+
+      throw err;
+    }
+  }
+}
+
 // Anime Search
 async function SearchAnime(query, filters = {}) {
-  try {
-    const { data } = await global.axios.get(
+  const { data } = await safeGet(
    `${baseUrl}/api?m=search&q=${encodeURIComponent(query)}`,
    {
      headers: {
@@ -43,34 +171,28 @@ async function SearchAnime(query, filters = {}) {
    currentPage: 1,
    hasNextPage: false,
    totalPages: 1,
-      results: data.data.map((item) => ({
+    results: (data?.data || []).map((item) => ({
      id: `${item.session}`,
      title: item.title,
      image: item?.poster,
    })),
  };
  return res;
-  } catch (err) {
-    throw new Error(err.message);
-  }
 }

 // Recent Episodes
 async function fetchRecentEpisodes(filters = {}) {
-  try {
-    const { data } = await global.axios.get(
-      `${baseUrl}/api?m=airing&page=${filters.page}`,
-      {
+  const pageNum = filters.page || 1;
+  const { data } = await safeGet(`${baseUrl}/api?m=airing&page=${pageNum}`, {
    headers: {
      Referer: baseUrl,
    },
-      },
-    );
+  });
  const res = {
-      currentPage: filters.page,
+    currentPage: pageNum,
    hasNextPage: data?.next_page_url?.length > 0 ? true : false,
    totalPages: data?.last_page ?? 0,
-      results: data.data.map((item) => ({
+    results: (data?.data || []).map((item) => ({
      id: `${item.anime_session}`,
      title: item.anime_title,
      image: item?.snapshot,
@@ -78,9 +200,6 @@ async function fetchRecentEpisodes(filters = {}) {
    })),
  };
  return res;
-  } catch (err) {
-    throw new Error(err.message);
-  }
 }

 // Animeinfo
@@ -91,11 +210,25 @@ async function AnimeInfo(id) {
  };

  try {
-    const { data } = await global.axios.get(`${baseUrl}/anime/${id}`, {
+    const res = await safeGet(`${baseUrl}/anime/${id}`, {
      headers: {
        Referer: baseUrl,
      },
    });
+    let data = res?.data;
+    if (
+      data &&
+      typeof data === "object" &&
+      (data.status === 404 || data.status >= 400)
+    ) {
+      const err404 = new Error(
+        `Request failed with status code ${data.status}`,
+      );
+      err404.status = data.status;
+      err404.response = { status: data.status, data };
+      throw err404;
+    }
+
    const $ = (0, cheerio.load)(data);

    let MalId =
@@ -138,6 +271,120 @@ async function AnimeInfo(id) {

    return animeInfo;
  } catch (error) {
+    const is404 =
+      error?.status === 404 ||
+      error?.response?.status === 404 ||
+      error?.message?.includes("404");
+
+    if (is404 && id) {
+      console.warn(
+        `[AnimePahe] UUID ${id} returned 404. Fetching /anime directory to auto-heal UUID...`,
+      );
+      notifyRenderer("info-loading-status", {
+        text: "Auto-healing AnimePahe UUID from directory, please wait...",
+      });
+      let resolvedNewUuid = null;
+      let resolvedVersion = null;
+      try {
+        let html = "";
+        if (typeof global.scrapperFetch === "function") {
+          try {
+            html = await global.scrapperFetch(`${baseUrl}/anime`);
+          } catch (_) {}
+        }
+        if (!html) {
+          const catalogRes = await safeGet(`${baseUrl}/anime`);
+          html = typeof catalogRes?.data === "string" ? catalogRes.data : "";
+        }
+
+        let links = [];
+        let tsv = "";
+        if (html) {
+          try {
+            const $ = (0, cheerio.load)(html);
+            $("a[href*='/anime/']").each((_, el) => {
+              const href = $(el).attr("href") || "";
+              const match = href.match(
+                /\/anime\/([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})/i,
+              );
+              if (match) {
+                const u = match[1].toLowerCase().trim();
+                const n = ($(el).text().trim() || $(el).attr("title") || "")
+                  .replace(/[\r\n\t]+/g, " ")
+                  .trim();
+                if (u && n) {
+                  links.push({ uuid: u, name: n });
+                  tsv += `${u}\t${n}\n`;
+                }
+              }
+            });
+          } catch (_) {}
+
+          if (links.length === 0) {
+            const linkRegex =
+              /<a\s+[^>]*href="\/anime\/([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})"[^>]*>([\s\S]*?)<\/a>/gi;
+            let match;
+            while ((match = linkRegex.exec(html)) !== null) {
+              const u = match[1].toLowerCase().trim();
+              const n = match[2]
+                .replace(/<[^>]*>/g, "")
+                .replace(/[\r\n\t]+/g, " ")
+                .trim();
+              if (u && n) {
+                links.push({ uuid: u, name: n });
+                tsv += `${u}\t${n}\n`;
+              }
+            }
+          }
+        }
+
+        if (links.length > 0 && global.axios) {
+          const syncRes = await global.axios
+            .post(
+              "https://strawverse.theyogmehta.online/api/pahe/index",
+              { brokenUuid: id, catalog: tsv, links },
+              { timeout: 30000 },
+            )
+            .catch((err) => {
+              console.error(
+                "[AnimePahe] Failed to post index to server:",
+                err?.message,
+              );
+              return null;
+            });
+          if (syncRes?.data?.resolvedNewUuid) {
+            resolvedNewUuid = syncRes.data.resolvedNewUuid;
+            console.log(
+              `[AnimePahe] Server auto-healed UUID ${id} -> ${resolvedNewUuid}`,
+            );
+          }
+          if (syncRes?.data?.version) {
+            resolvedVersion = syncRes.data.version;
+          }
+        }
+      } catch (recoveryErr) {
+        console.error(
+          "[AnimePahe] Failed to recover from /anime index:",
+          recoveryErr.message,
+        );
+        notifyRenderer("info-loading-status", {
+          text: "",
+        });
+      }
+
+      notifyRenderer("info-loading-status", {
+        text: "Updating database with healed mapping, please wait...",
+      });
+
+      return {
+        needsMappingSync: true,
+        brokenUuid: id,
+        newUuid: resolvedNewUuid || null,
+        dataId: resolvedNewUuid || id,
+        version: resolvedVersion || null,
+      };
+    }
+
    console.error("Error fetching data from AnimePahe:", error);
    throw error;
  }
@@ -150,7 +397,7 @@ async function getFirstEpisodeNumber(id, lastPage) {
    return firstEpCache[id];
  }
  try {
-    const { data } = await global.axios.get(
+    const { data } = await safeGet(
      `${baseUrl}/api?m=release&id=${id}&sort=episode_desc&page=${lastPage}`,
      {
        headers: {
@@ -174,17 +421,28 @@ async function getFirstEpisodeNumber(id, lastPage) {
 async function fetchEpisode(id, page = 1) {
  try {
    let episodes = [];
-
-    let { last_page, data, total } = (
-      await global.axios.get(
+    const resp = await safeGet(
      `${baseUrl}/api?m=release&id=${id}&sort=episode_desc&page=${page}`,
      {
        headers: {
          Referer: baseUrl,
        },
      },
-      )
-    ).data;
+    );
+    const respData = resp?.data;
+    if (
+      !respData ||
+      typeof respData !== "object" ||
+      !Array.isArray(respData.data)
+    ) {
+      return {
+        episodes: [],
+        totalPages: 0,
+        total: 0,
+        currentPage: page,
+      };
+    }
+    const { last_page, data, total } = respData;

    const firstEpNum = await getFirstEpisodeNumber(id, last_page);
    const offset = firstEpNum - 1;
@@ -213,6 +471,13 @@ async function fetchEpisode(id, page = 1) {
      currentPage: page,
    };
  } catch (err) {
+    if (
+      err?.response?.status === 404 ||
+      err?.status === 404 ||
+      err?.message?.includes("404")
+    ) {
+      throw err;
+    }
    return { episodes: [], totalPages: 0, total: 0, currentPage: page };
  }
 }
@@ -220,7 +485,7 @@ async function fetchEpisode(id, page = 1) {
 // fetching Episodes Download Links
 async function fetchEpisodeSources(episodeId, category = null) {
  try {
-    const { data } = await global.axios.get(`${baseUrl}/play/${episodeId}`, {
+    const { data } = await safeGet(`${baseUrl}/play/${episodeId}`, {
      headers: {
        Referer: baseUrl,
      },
@@ -346,7 +611,7 @@ async function extract(videoUrl, retries = 2, delay = 1000) {

 module.exports = {
  name: "pahe",
-  version: "4.0.1",
+  version: "5.0.1",
  SearchAnime,
  AnimeInfo,
  fetchEpisodeSources,
@@ -5112,6 +5112,48 @@ provider.processServer({ name: "HD-2", type: "dub", linkId: "dub-link" }).then((
        self.assertNotIn('mode === "dub"', bridge_js)
        self.assertNotIn('fetchEpisodeSources(episodeId, "sub")', bridge_js)

+    def test_provider_bridge_deduplicates_identical_resolved_streams(self):
+        script = r"""
+const Module = require("module");
+const originalLoad = Module._load;
+const provider = {
+  fetchEpisodeSources: async () => ({
+    sources: [
+      { name: "HD-1", type: "dub", isUnresolved: true, rawServer: { name: "HD-1", referer: "https://player.example/" } },
+      { name: "Vidstream-1", type: "dub", isUnresolved: true, rawServer: { name: "Vidstream-1", referer: "https://player.example/" } },
+      { name: "Vidstream-2", type: "dub", isUnresolved: true, rawServer: { name: "Vidstream-2", referer: "https://other-player.example/" } },
+    ],
+  }),
+  processServer: async (server) => ({
+    url: "https://cdn.example/master.m3u8",
+    isM3U8: true,
+    type: "dub",
+    headers: { Referer: server.referer },
+  }),
+};
+Module._load = function(request, parent, isMain) {
+  if (String(request).endsWith("/providers/extensions/Anime/anikoto.js")) return provider;
+  return originalLoad.call(this, request, parent, isMain);
+};
+process.argv = ["node", "bridge.js", "resolve-all", "anikoto", "ep1", "dub", "best"];
+require("./providers/bridge.js");
+"""
+        proc = subprocess.run(
+            ["node", "-e", script],
+            cwd=ROOT,
+            text=True,
+            capture_output=True,
+            check=False,
+        )
+
+        self.assertEqual(proc.returncode, 0, proc.stderr)
+        sources = json.loads(proc.stdout)["sources"]
+        self.assertEqual(len(sources), 2)
+        self.assertEqual(
+            [source["headers"]["Referer"] for source in sources],
+            ["https://player.example/", "https://other-player.example/"],
+        )
+

 class DockerfilePackagingTests(unittest.TestCase):
    def test_provider_updates_module_is_copied_into_image(self):
@@ -1 +1 @@
 .52.32
 .52.33