Fix thumbnail backoff and watchlist fallback matching
This commit is contained in:
Dymas
@@ -1,5 +1,30 @@
|
||||
# Changelog
|
||||
|
||||
## 0.29.0 - 2026-05-16
|
||||
|
||||
- Fixed watchlist thumbnail retry backoff so failed automatic cover lookups now record a retry timestamp and stop hammering AnimeSchedule or AniDB on every later warm-up pass.
|
||||
- Hardened queued-download fallback matching to keep season and part identity intact while still accepting equivalent season notation such as `2nd Season` and `Season 2`.
|
||||
- Made paged watchlist responses clamp out-of-range page requests before fetching rows, keeping the reported `page` value aligned with the returned `items`.
|
||||
- Expanded regression coverage for thumbnail backoff persistence, season-aware fallback matching, and out-of-range watchlist pagination.
|
||||
|
||||
## 0.28.0 - 2026-05-16
|
||||
|
||||
- Bounded watchlist refresh network calls with a shorter dedicated timeout and matched worker-shutdown grace period, making per-show and bulk watchlist teardown behavior more predictable when upstream services are slow.
|
||||
- Improved completed-download watchlist sync failures by preserving the fallback resolution reason and surfacing it in logs and raised errors instead of collapsing everything into one generic unresolved `show_id` message.
|
||||
- Continued the backend split by moving shared title-normalization and title-query helpers into a dedicated `title_matching.py` module.
|
||||
- Tightened the shared browser polling helper so it uses serial `setTimeout` scheduling plus unload cleanup.
|
||||
- Expanded regression coverage for bounded watchlist refresh timeouts, clearer fallback-match failures, and the updated poll-helper cleanup behavior.
|
||||
|
||||
## 0.27.0 - 2026-05-15
|
||||
|
||||
- Made runtime reset stricter by refusing to clear the global runtime state when background workers do not stop cleanly, preventing stale workers from outliving the runtime instance they came from.
|
||||
- Taught per-show watchlist refreshes to stop earlier during shutdown by checking the refresh stop flag before committing later update stages.
|
||||
- Marked bulk watchlist refresh jobs as `interrupted` when shutdown interrupts them instead of letting them look like a normal successful completion.
|
||||
- Hardened the completed-download watchlist sync fallback so it only trusts a fallback search result when the queued title still matches confidently, failing safe instead of risking the wrong anime being marked downloaded.
|
||||
- Reworked the shared browser polling helper to use serial `setTimeout` polling with unload cleanup instead of a bare repeating interval.
|
||||
- Continued the backend split by moving queued-download watchlist identity resolution into a dedicated `watchlist_identity.py` helper module.
|
||||
- Expanded regression coverage for interrupted resets, interrupted bulk watchlist refresh jobs, safer watchlist identity fallback behavior, and browser poll helper cleanup.
|
||||
|
||||
## 0.26.0 - 2026-05-15
|
||||
|
||||
- Made controlled runtime shutdown stricter by canceling active download workers during blocking teardown paths such as app exit and runtime reset, reducing the chance of orphaned background work outliving the server.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
A local web UI for a system-wide `ani-cli` install.
|
||||
|
||||
Current version: `0.26.0`
|
||||
Current version: `0.29.0`
|
||||
|
||||
## Project Layout
|
||||
|
||||
@@ -11,6 +11,8 @@ ani-cli-web/
|
||||
app.py Public entrypoint, runtime wiring, watchlist domain logic, and CLI startup.
|
||||
app_support.py Shared runtime/state/config helpers plus queue and library file utilities.
|
||||
queue_jobs.py Download queue and background watchlist-refresh workers.
|
||||
watchlist_identity.py Safer watchlist identity resolution for queued download jobs.
|
||||
title_matching.py Shared anime-title normalization and matching helpers.
|
||||
http_handler.py HTTP route handling, request parsing, and remote-access guard.
|
||||
web_templates.py Shared Search, Config, and Watchlist page markup aggregation.
|
||||
search_page.py Search page template.
|
||||
@@ -70,7 +72,10 @@ The app currently includes:
|
||||
- The watchlist page now also avoids re-triggering the same unresolved thumbnail warm-up request on every silent poll cycle while a show's thumbnail state is unchanged.
|
||||
- Background `Refresh All` execution with status polling, parallel per-show refresh work, and no inline thumbnail downloads during the bulk job.
|
||||
- Search and Watchlist polling now use a shared serial browser polling helper so slow responses do not cause overlapping poll requests to pile up in the background.
|
||||
- That shared browser polling helper now also tears itself down on page unload and uses chained `setTimeout` scheduling instead of a bare repeating interval.
|
||||
- Controlled shutdown and runtime reset now cancel active download workers when waiting for teardown, reducing the chance of orphaned background downloads outliving the web process.
|
||||
- Runtime reset now also refuses to clear the active global runtime when background workers fail to stop cleanly, preventing stale workers from outliving the runtime object that created them.
|
||||
- Watchlist refresh networking now uses a shorter dedicated timeout, which helps per-show and bulk watchlist shutdown paths settle more predictably when upstream metadata services are slow.
|
||||
- Persistent watchlist refresh history that survives restarts and marks interrupted bulk refresh jobs clearly.
|
||||
- Local thumbnail caching plus manual thumbnail upload when automatic cover lookup fails.
|
||||
- A refreshed glassy dark UI inspired by the Tsuki frontend design language.
|
||||
@@ -80,6 +85,8 @@ The app currently includes:
|
||||
- Search-page queue polling also guards against stale out-of-order refresh responses now.
|
||||
- Queue log writes are batched during active downloads to reduce SQLite churn on noisy jobs.
|
||||
- Queue persistence now stores core job metadata and logs in structured SQLite columns instead of treating the whole job as one large JSON blob.
|
||||
- Failed thumbnail refresh attempts now record a retry timestamp, so unresolved covers back off instead of immediately requerying upstream sources on every later warm-up.
|
||||
- Download-to-watchlist fallback matching now keeps season and part identity intact while still tolerating equivalent season notation such as `2nd Season` vs `Season 2`.
|
||||
- Project-local runtime state under `.ani-cli-web/`.
|
||||
|
||||
## Search Page
|
||||
@@ -189,6 +196,9 @@ Current behavior:
|
||||
- If the anime is already on the watchlist, a successful download marks it as downloaded and keeps its current category.
|
||||
- If the anime is not yet on the watchlist, `ani-cli-web` creates a new watchlist entry in `Finished` after the download succeeds.
|
||||
- Those add and download-sync paths now queue a background refresh for episode counts, AnimeSchedule metadata, and thumbnails instead of waiting for all upstream fetches before returning.
|
||||
- If a queued job ever reaches the older search-based fallback path for `show_id` resolution, the app now only accepts that fallback when the queued title still matches the returned result confidently, preferring to fail safe over marking the wrong anime as downloaded.
|
||||
- That fallback now also preserves season and part distinctions, so `Season 2` downloads do not silently sync against a different season with the same base title.
|
||||
- When that safer fallback still cannot resolve a `show_id`, the failure now keeps the specific reason so logs and error messages are easier to diagnose.
|
||||
- Existing watchlist entries from older versions are migrated into the `Watching` category automatically.
|
||||
|
||||
### Thumbnail Behavior
|
||||
@@ -207,7 +217,11 @@ Current thumbnail behavior:
|
||||
- If a season-labeled title fails as-is, lookup retries with simplified base-title queries.
|
||||
- AniDB is used as a fallback when AnimeSchedule does not resolve a cover.
|
||||
- AniDB fallback reuses the cached parsed title list directly to keep repeated lookups lighter.
|
||||
- Failed fetch attempts no longer refresh the retry window, so a bad lookup does not get stuck in an immediate repeat-404 state.
|
||||
- Failed automatic fetch attempts now start the retry window, so bad lookups back off instead of hammering AnimeSchedule and AniDB on every warm-up pass.
|
||||
|
||||
### Watchlist Pagination
|
||||
|
||||
Watchlist listing responses now clamp out-of-range page requests to the last available page before querying rows, so the returned `page` number and `items` stay consistent even when the browser or API client asks for a page past the end.
|
||||
|
||||
Per-card thumbnail tools:
|
||||
|
||||
|
||||
@@ -9,7 +9,6 @@ import shutil
|
||||
import sqlite3
|
||||
import threading
|
||||
import time
|
||||
import unicodedata
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import xml.etree.ElementTree as ET
|
||||
@@ -45,6 +44,8 @@ from app_support import (
|
||||
WATCHLIST_CATEGORY_CHOICES,
|
||||
WATCHLIST_CATEGORY_LABELS,
|
||||
WATCHLIST_JSON_PATH,
|
||||
WATCHLIST_REFRESH_REQUEST_TIMEOUT_SECONDS,
|
||||
WORKER_SHUTDOWN_TIMEOUT_SECONDS,
|
||||
background_workers_enabled,
|
||||
client_address_is_local,
|
||||
debug_log,
|
||||
@@ -57,6 +58,14 @@ from app_support import (
|
||||
)
|
||||
from http_handler import HandlerContext, build_handler_class, server_host, server_port
|
||||
from queue_jobs import DownloadQueue, WatchlistRefreshJobs
|
||||
from title_matching import (
|
||||
base_title_variants,
|
||||
normalize_title_key,
|
||||
normalize_identity_title_key,
|
||||
title_lookup_queries,
|
||||
title_match_variants,
|
||||
)
|
||||
from watchlist_identity import resolve_job_watchlist_identity as resolve_job_watchlist_identity_impl
|
||||
from web_templates import CONFIG_HTML, INDEX_HTML, PAGE_LINKS, WATCHLIST_HTML, render_page_links, render_sidebar_brand
|
||||
|
||||
RUNTIME_LOCK = threading.RLock()
|
||||
@@ -75,7 +84,7 @@ DOWNLOAD_QUEUE = None
|
||||
WATCHLIST_REFRESH = None
|
||||
|
||||
|
||||
def graph_request(query, variables):
|
||||
def graph_request(query, variables, timeout=25):
|
||||
payload = json.dumps({"variables": variables, "query": query}).encode("utf-8")
|
||||
request = urllib.request.Request(
|
||||
f"{ALLANIME_API}/api",
|
||||
@@ -88,7 +97,7 @@ def graph_request(query, variables):
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=25) as response:
|
||||
with urllib.request.urlopen(request, timeout=timeout) as response:
|
||||
raw = response.read().decode("utf-8")
|
||||
except urllib.error.URLError as exc:
|
||||
raise RuntimeError(f"Could not reach AllAnime: {exc}") from exc
|
||||
@@ -201,64 +210,7 @@ def decode_image_data_url(data_url):
|
||||
return content_type, content
|
||||
|
||||
|
||||
def normalize_title_key(value):
|
||||
text = unicodedata.normalize("NFKD", str(value or ""))
|
||||
text = "".join(char for char in text if not unicodedata.combining(char))
|
||||
text = text.lower()
|
||||
text = re.sub(r"\((19|20)\d{2}\)", " ", text)
|
||||
text = re.sub(r"[^a-z0-9]+", " ", text)
|
||||
return re.sub(r"\s+", " ", text).strip()
|
||||
|
||||
|
||||
def base_title_variants(value):
|
||||
text = str(value or "").strip()
|
||||
variants = []
|
||||
if text:
|
||||
variants.append(text)
|
||||
patterns = [
|
||||
r"\s+\((19|20)\d{2}\)\s*$",
|
||||
r"\s+season\s+\d+\s*$",
|
||||
r"\s+part\s+\d+\s*$",
|
||||
r"\s+cour\s+\d+\s*$",
|
||||
r"\s+\d+(st|nd|rd|th)\s+season\s*$",
|
||||
]
|
||||
changed = True
|
||||
current = text
|
||||
while current and changed:
|
||||
changed = False
|
||||
for pattern in patterns:
|
||||
trimmed = re.sub(pattern, "", current, flags=re.IGNORECASE).strip(" -:")
|
||||
if trimmed and trimmed != current:
|
||||
current = trimmed
|
||||
if current not in variants:
|
||||
variants.append(current)
|
||||
changed = True
|
||||
return variants
|
||||
|
||||
|
||||
def title_match_variants(value):
|
||||
variants = set()
|
||||
for raw_value in base_title_variants(value):
|
||||
normalized = normalize_title_key(raw_value)
|
||||
if normalized:
|
||||
variants.add(normalized)
|
||||
stripped = re.sub(r"\b(19|20)\d{2}\b", " ", normalized)
|
||||
stripped = re.sub(r"\s+", " ", stripped).strip()
|
||||
if stripped:
|
||||
variants.add(stripped)
|
||||
return variants
|
||||
|
||||
|
||||
def title_lookup_queries(value):
|
||||
queries = []
|
||||
for raw_value in base_title_variants(value):
|
||||
text = str(raw_value or "").strip()
|
||||
if text and text not in queries:
|
||||
queries.append(text)
|
||||
return queries
|
||||
|
||||
|
||||
def animeschedule_request(path, params=None):
|
||||
def animeschedule_request(path, params=None, timeout=25):
|
||||
params = params or {}
|
||||
query = "&".join(f"{quote(str(key), safe='')}={quote(str(value), safe='')}" for key, value in params.items())
|
||||
url = f"{ANIMESCHEDULE_API}{path}"
|
||||
@@ -274,7 +226,7 @@ def animeschedule_request(path, params=None):
|
||||
method="GET",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=25) as response:
|
||||
with urllib.request.urlopen(request, timeout=timeout) as response:
|
||||
raw = response.read().decode("utf-8")
|
||||
debug_log(
|
||||
"animeschedule.response",
|
||||
@@ -474,8 +426,8 @@ def find_animeschedule_match(title):
|
||||
return best
|
||||
|
||||
|
||||
def fetch_animeschedule_anime_by_route(route, fallback_title=""):
|
||||
payload = animeschedule_request(f"/anime/{quote(str(route).strip(), safe='')}")
|
||||
def fetch_animeschedule_anime_by_route(route, fallback_title="", timeout=25):
|
||||
payload = animeschedule_request(f"/anime/{quote(str(route).strip(), safe='')}", timeout=timeout)
|
||||
entry = normalize_animeschedule_entry(payload)
|
||||
if not entry:
|
||||
raise RuntimeError("AnimeSchedule anime response was missing required fields")
|
||||
@@ -484,16 +436,16 @@ def fetch_animeschedule_anime_by_route(route, fallback_title=""):
|
||||
return entry
|
||||
|
||||
|
||||
def resolve_animeschedule_anime(title, route=""):
|
||||
def resolve_animeschedule_anime(title, route="", timeout=25):
|
||||
stored_route = str(route or "").strip()
|
||||
if stored_route:
|
||||
try:
|
||||
return fetch_animeschedule_anime_by_route(stored_route, fallback_title=title)
|
||||
return fetch_animeschedule_anime_by_route(stored_route, fallback_title=title, timeout=timeout)
|
||||
except Exception:
|
||||
debug_log("animeschedule.route_refresh_failed", route=stored_route, title=title)
|
||||
|
||||
best = find_animeschedule_match(title)
|
||||
return fetch_animeschedule_anime_by_route(best["route"], fallback_title=best["title"])
|
||||
return fetch_animeschedule_anime_by_route(best["route"], fallback_title=best["title"], timeout=timeout)
|
||||
|
||||
|
||||
def fetch_animeschedule_cover(title):
|
||||
@@ -864,9 +816,9 @@ def numeric_episode_key(value):
|
||||
return key
|
||||
|
||||
|
||||
def fetch_show_episode_snapshot(show_id):
|
||||
def fetch_show_episode_snapshot(show_id, timeout=25):
|
||||
query = "query ($showId: String!) { show(_id: $showId) { _id name availableEpisodesDetail } }"
|
||||
data = graph_request(query, {"showId": show_id})
|
||||
data = graph_request(query, {"showId": show_id}, timeout=timeout)
|
||||
show = data.get("show") or {}
|
||||
detail = show.get("availableEpisodesDetail") or {}
|
||||
episodes = {}
|
||||
@@ -948,9 +900,12 @@ class WatchlistStore:
|
||||
self.refresh_wakeup.set()
|
||||
worker = self.refresh_worker
|
||||
if wait and worker is not None and worker.is_alive():
|
||||
worker.join(timeout=2.5)
|
||||
worker.join(timeout=WORKER_SHUTDOWN_TIMEOUT_SECONDS)
|
||||
return worker is None or not worker.is_alive()
|
||||
|
||||
def _refresh_request_timeout(self):
|
||||
return WATCHLIST_REFRESH_REQUEST_TIMEOUT_SECONDS
|
||||
|
||||
def _next_pending_refresh(self):
|
||||
with self.lock:
|
||||
if not self.refresh_pending:
|
||||
@@ -1256,7 +1211,6 @@ class WatchlistStore:
|
||||
def list(self, page=1, per_page=30, category=None):
|
||||
page = max(1, int(page or 1))
|
||||
per_page = min(60, max(1, int(per_page or 30)))
|
||||
offset = (page - 1) * per_page
|
||||
active_category = normalize_watchlist_category(category) if category else None
|
||||
with self.lock, self._connect() as conn:
|
||||
overall_total = conn.execute("SELECT COUNT(*) AS count FROM watchlist").fetchone()["count"]
|
||||
@@ -1279,6 +1233,9 @@ class WatchlistStore:
|
||||
"SELECT category, COUNT(*) AS count FROM watchlist GROUP BY category"
|
||||
).fetchall()
|
||||
queued_total = conn.execute("SELECT COUNT(*) AS count FROM watchlist WHERE status = 'queued'").fetchone()["count"]
|
||||
pages = max(1, (total + per_page - 1) // per_page)
|
||||
page = min(page, pages)
|
||||
offset = (page - 1) * per_page
|
||||
if active_category:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM watchlist WHERE category = ? ORDER BY title COLLATE NOCASE ASC LIMIT ? OFFSET ?",
|
||||
@@ -1290,14 +1247,13 @@ class WatchlistStore:
|
||||
(per_page, offset),
|
||||
).fetchall()
|
||||
items = [self._row_to_item(row) for row in rows]
|
||||
pages = max(1, (total + per_page - 1) // per_page)
|
||||
category_counts = {key: 0 for key in WATCHLIST_CATEGORY_CHOICES}
|
||||
for row in category_rows:
|
||||
category_counts[normalize_watchlist_category(row["category"])] = int(row["count"] or 0)
|
||||
return {
|
||||
"items": items,
|
||||
"total": total,
|
||||
"page": min(page, pages),
|
||||
"page": page,
|
||||
"pages": pages,
|
||||
"per_page": per_page,
|
||||
"active_category": active_category,
|
||||
@@ -1368,17 +1324,26 @@ class WatchlistStore:
|
||||
existing = self.get(show_id)
|
||||
normalized_show_id = existing["show_id"]
|
||||
now = now_iso()
|
||||
request_timeout = self._refresh_request_timeout()
|
||||
try:
|
||||
snapshot = fetch_show_episode_snapshot(normalized_show_id)
|
||||
snapshot = fetch_show_episode_snapshot(normalized_show_id, timeout=request_timeout)
|
||||
if self.refresh_stop_event.is_set():
|
||||
return None
|
||||
if not self.has_show(normalized_show_id):
|
||||
return None
|
||||
sub_episodes = snapshot["episodes"]["sub"]
|
||||
dub_episodes = snapshot["episodes"]["dub"]
|
||||
schedule = None
|
||||
try:
|
||||
schedule = resolve_animeschedule_anime(snapshot["title"] or existing["title"], route=existing.get("animeschedule_route"))
|
||||
schedule = resolve_animeschedule_anime(
|
||||
snapshot["title"] or existing["title"],
|
||||
route=existing.get("animeschedule_route"),
|
||||
timeout=request_timeout,
|
||||
)
|
||||
except Exception:
|
||||
schedule = None
|
||||
if self.refresh_stop_event.is_set():
|
||||
return None
|
||||
if not self.has_show(normalized_show_id):
|
||||
return None
|
||||
expected_count = parse_nonnegative_int((schedule or {}).get("episodes")) or int(existing.get("expected_count") or 0)
|
||||
@@ -1405,6 +1370,8 @@ class WatchlistStore:
|
||||
normalized_show_id,
|
||||
)
|
||||
except Exception as exc:
|
||||
if self.refresh_stop_event.is_set():
|
||||
return None
|
||||
if not self.has_show(normalized_show_id):
|
||||
return None
|
||||
payload = (
|
||||
@@ -1425,6 +1392,8 @@ class WatchlistStore:
|
||||
)
|
||||
|
||||
with self.lock, self._connect() as conn:
|
||||
if self.refresh_stop_event.is_set():
|
||||
return None
|
||||
cursor = conn.execute(
|
||||
"""
|
||||
UPDATE watchlist
|
||||
@@ -1448,6 +1417,8 @@ class WatchlistStore:
|
||||
)
|
||||
if cursor.rowcount < 1:
|
||||
return None
|
||||
if self.refresh_stop_event.is_set():
|
||||
return None
|
||||
if include_thumbnail and self.has_show(normalized_show_id):
|
||||
try:
|
||||
self.ensure_thumbnail(normalized_show_id)
|
||||
@@ -1653,7 +1624,6 @@ class WatchlistStore:
|
||||
except Exception as exc:
|
||||
debug_log("thumbnail.ensure.failed", show_id=show_id, error=exc)
|
||||
cached_name = None
|
||||
checked_at = item.get("thumbnail_checked_at")
|
||||
|
||||
with self.lock, self._connect() as conn:
|
||||
conn.execute(
|
||||
@@ -1760,39 +1730,47 @@ def upload_watchlist_thumbnail(payload):
|
||||
|
||||
|
||||
def resolve_job_watchlist_identity(job, config=None):
|
||||
show_id = str(job.get("show_id") or "").strip()
|
||||
title = str(job.get("title") or job.get("anime_name") or "").strip()
|
||||
if show_id:
|
||||
return show_id, title
|
||||
try:
|
||||
index = max(1, int(job.get("result_index") or 1))
|
||||
except (TypeError, ValueError):
|
||||
return "", title
|
||||
try:
|
||||
active_config = config or CONFIG or DEFAULT_CONFIG
|
||||
results = search_anime(job.get("query") or title, job.get("mode") or active_config["mode"])
|
||||
except Exception:
|
||||
return "", title
|
||||
if index > len(results):
|
||||
return "", title
|
||||
result = results[index - 1]
|
||||
return str(result.get("id") or "").strip(), str(result.get("title") or title).strip()
|
||||
return resolve_job_watchlist_identity_impl(
|
||||
job,
|
||||
search_fn=search_anime,
|
||||
normalize_title_key_fn=normalize_identity_title_key,
|
||||
base_title_variants_fn=base_title_variants,
|
||||
default_mode=active_config["mode"],
|
||||
)
|
||||
|
||||
|
||||
def sync_downloaded_job_to_watchlist(job):
|
||||
ensure_runtime()
|
||||
show_id, title = resolve_job_watchlist_identity(job)
|
||||
show_id, title, reason = resolve_job_watchlist_identity_impl(
|
||||
job,
|
||||
search_fn=search_anime,
|
||||
normalize_title_key_fn=normalize_identity_title_key,
|
||||
base_title_variants_fn=base_title_variants,
|
||||
default_mode=(CONFIG or DEFAULT_CONFIG)["mode"],
|
||||
return_reason=True,
|
||||
)
|
||||
if not show_id:
|
||||
raise ValueError("Could not resolve a watchlist show_id for the completed download.")
|
||||
debug_log("watchlist.sync.resolve_failed", job=job, reason=reason)
|
||||
raise ValueError(f"Could not resolve a watchlist show_id for the completed download: {reason}.")
|
||||
return WATCHLIST.mark_downloaded(show_id, title=title)
|
||||
|
||||
def prepare_download_job(job):
|
||||
config = ensure_runtime()["config"]
|
||||
show_id, title = resolve_job_watchlist_identity(job, config=config)
|
||||
show_id, title, reason = resolve_job_watchlist_identity_impl(
|
||||
job,
|
||||
search_fn=search_anime,
|
||||
normalize_title_key_fn=normalize_identity_title_key,
|
||||
base_title_variants_fn=base_title_variants,
|
||||
default_mode=config["mode"],
|
||||
return_reason=True,
|
||||
)
|
||||
if show_id:
|
||||
job["show_id"] = show_id
|
||||
if title:
|
||||
job["title"] = title
|
||||
if not show_id and reason:
|
||||
debug_log("watchlist.prepare_job.resolve_skipped", job=job, reason=reason)
|
||||
return job
|
||||
|
||||
|
||||
@@ -1878,23 +1856,29 @@ def shutdown_runtime(wait=False, cancel_active_downloads=None):
|
||||
services = (WATCHLIST, DOWNLOAD_QUEUE, WATCHLIST_REFRESH)
|
||||
watchlist, download_queue, watchlist_refresh = services
|
||||
should_cancel_downloads = bool(wait) if cancel_active_downloads is None else bool(cancel_active_downloads)
|
||||
stopped = True
|
||||
if watchlist is not None:
|
||||
watchlist.shutdown(wait=wait)
|
||||
stopped = watchlist.shutdown(wait=wait) and stopped
|
||||
if download_queue is not None:
|
||||
download_queue.shutdown(wait=wait, cancel_active=should_cancel_downloads)
|
||||
stopped = download_queue.shutdown(wait=wait, cancel_active=should_cancel_downloads) and stopped
|
||||
if watchlist_refresh is not None:
|
||||
watchlist_refresh.shutdown(wait=wait)
|
||||
return True
|
||||
stopped = watchlist_refresh.shutdown(wait=wait) and stopped
|
||||
return stopped
|
||||
|
||||
|
||||
def reset_runtime(wait=False, cancel_active_downloads=None):
|
||||
global CONFIG, WATCHLIST, DOWNLOAD_QUEUE, WATCHLIST_REFRESH
|
||||
shutdown_runtime(wait=wait, cancel_active_downloads=cancel_active_downloads)
|
||||
stopped = shutdown_runtime(wait=wait, cancel_active_downloads=cancel_active_downloads)
|
||||
if not stopped:
|
||||
if wait:
|
||||
raise RuntimeError("Runtime shutdown did not complete cleanly.")
|
||||
return False
|
||||
with RUNTIME_LOCK:
|
||||
CONFIG = None
|
||||
WATCHLIST = None
|
||||
DOWNLOAD_QUEUE = None
|
||||
WATCHLIST_REFRESH = None
|
||||
return True
|
||||
|
||||
|
||||
Handler = build_handler_class(
|
||||
@@ -1944,7 +1928,8 @@ def main():
|
||||
except KeyboardInterrupt:
|
||||
print("\nShutting down server.")
|
||||
finally:
|
||||
shutdown_runtime(wait=True, cancel_active_downloads=True)
|
||||
if not shutdown_runtime(wait=True, cancel_active_downloads=True):
|
||||
print("Warning: background workers did not stop cleanly.")
|
||||
server.server_close()
|
||||
|
||||
|
||||
|
||||
+3
-1
@@ -16,7 +16,7 @@ from uuid import uuid4
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent
|
||||
ANI_CLI = os.environ.get("ANI_CLI_BIN") or shutil.which("ani-cli") or "ani-cli"
|
||||
APP_NAME = "ani-cli-web"
|
||||
VERSION = "0.26.0"
|
||||
VERSION = "0.29.0"
|
||||
ALLANIME_BASE = "allanime.day"
|
||||
ALLANIME_API = f"https://api.{ALLANIME_BASE}"
|
||||
ALLANIME_REFERER = "https://allmanga.to"
|
||||
@@ -40,6 +40,8 @@ WATCHLIST_CATEGORY_LABELS = {
|
||||
WATCHLIST_CATEGORY_CHOICES = tuple(WATCHLIST_CATEGORY_LABELS.keys())
|
||||
MAX_LOG_LINES = 240
|
||||
MAX_JSON_BODY_BYTES = 12 * 1024 * 1024
|
||||
WATCHLIST_REFRESH_REQUEST_TIMEOUT_SECONDS = 8
|
||||
WORKER_SHUTDOWN_TIMEOUT_SECONDS = 10
|
||||
|
||||
|
||||
def env_flag(name, default=False):
|
||||
|
||||
+155
@@ -297,6 +297,27 @@ class WatchlistRefreshJobTests(unittest.TestCase):
|
||||
self.assertEqual(status["errors"], 0)
|
||||
self.assertCountEqual(calls, [("show-a", False), ("show-b", False)])
|
||||
|
||||
def test_refresh_job_marks_interrupted_when_shutdown_requested(self):
|
||||
service = APP.WatchlistRefreshJobs(mock.Mock(), start_worker=False)
|
||||
job = {
|
||||
"id": "refresh-1",
|
||||
"status": "running",
|
||||
"created_at": APP.now_iso(),
|
||||
"updated_at": APP.now_iso(),
|
||||
"completed": 0,
|
||||
"errors": 0,
|
||||
"current_title": None,
|
||||
"message": "",
|
||||
}
|
||||
service.store.all_show_ids = lambda: ["show-a"]
|
||||
service.store.refresh = mock.Mock(return_value={"title": "Show A", "status": "updated"})
|
||||
service.stop_event.set()
|
||||
|
||||
service._run_job(job)
|
||||
|
||||
self.assertEqual(job["status"], "interrupted")
|
||||
self.assertIn("interrupted", job["message"].lower())
|
||||
|
||||
def test_refresh_start_reuses_pending_job(self):
|
||||
class FakeStore:
|
||||
def all_show_ids(self):
|
||||
@@ -407,6 +428,53 @@ class ThumbnailEventRecoveryTests(unittest.TestCase):
|
||||
|
||||
self.assertEqual(store.thumbnail_events, {})
|
||||
|
||||
def test_ensure_thumbnail_records_failed_attempt_time_for_retry_backoff(self):
|
||||
now = APP.now_iso()
|
||||
item = {
|
||||
"show_id": "show-thumb-1",
|
||||
"title": "Broken Cover Show",
|
||||
"category": "watching",
|
||||
"downloaded": False,
|
||||
"status": "tracked",
|
||||
"status_message": "Waiting for thumbnail.",
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
"last_checked": None,
|
||||
"sub_count": 0,
|
||||
"dub_count": 0,
|
||||
"expected_count": 0,
|
||||
"airing_status": None,
|
||||
"sub_latest_episode": None,
|
||||
"dub_latest_episode": None,
|
||||
"animeschedule_route": None,
|
||||
"animeschedule_title": None,
|
||||
"anidb_aid": None,
|
||||
"anidb_title": None,
|
||||
"thumbnail_path": None,
|
||||
"thumbnail_checked_at": None,
|
||||
}
|
||||
with APP.WATCHLIST.lock, APP.WATCHLIST._connect() as conn:
|
||||
APP.WATCHLIST._upsert_conn(conn, item)
|
||||
|
||||
with mock.patch.object(APP, "fetch_animeschedule_cover", side_effect=RuntimeError("lookup failed")) as fetch_cover, mock.patch.object(
|
||||
APP, "fetch_anidb_cover", side_effect=RuntimeError("fallback failed")
|
||||
) as fetch_fallback:
|
||||
result = APP.WATCHLIST.ensure_thumbnail("show-thumb-1")
|
||||
|
||||
self.assertIsNone(result)
|
||||
self.assertEqual(fetch_cover.call_count, 1)
|
||||
self.assertEqual(fetch_fallback.call_count, 1)
|
||||
failed_item = APP.WATCHLIST.get("show-thumb-1")
|
||||
self.assertTrue(failed_item["thumbnail_checked_at"])
|
||||
self.assertFalse(APP.thumbnail_retry_due(failed_item["thumbnail_checked_at"]))
|
||||
|
||||
with mock.patch.object(APP, "fetch_animeschedule_cover", side_effect=AssertionError("retry window should suppress another lookup")), mock.patch.object(
|
||||
APP, "fetch_anidb_cover", side_effect=AssertionError("retry window should suppress fallback lookup")
|
||||
):
|
||||
second = APP.WATCHLIST.ensure_thumbnail("show-thumb-1")
|
||||
|
||||
self.assertIsNone(second)
|
||||
|
||||
|
||||
class WatchlistCompletionTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
@@ -496,6 +564,16 @@ class WatchlistCompletionTests(unittest.TestCase):
|
||||
self.assertEqual(listing["queued_count"], 1)
|
||||
self.assertEqual([item["show_id"] for item in listing["items"]], ["show-1"])
|
||||
|
||||
def test_list_clamps_out_of_range_page_before_querying_rows(self):
|
||||
self.seed_watchlist_item(show_id="show-1", title="Alpha Show", category="watching")
|
||||
self.seed_watchlist_item(show_id="show-2", title="Beta Show", category="watching")
|
||||
|
||||
listing = APP.WATCHLIST.list(page=3, per_page=1, category="watching")
|
||||
|
||||
self.assertEqual(listing["page"], 2)
|
||||
self.assertEqual(listing["pages"], 2)
|
||||
self.assertEqual([item["show_id"] for item in listing["items"]], ["show-2"])
|
||||
|
||||
def test_queue_refresh_marks_item_queued_without_sync_refresh(self):
|
||||
self.seed_watchlist_item(show_id="show-11", title="Refresh Show", category="watching", status="updated")
|
||||
|
||||
@@ -543,6 +621,32 @@ class WatchlistCompletionTests(unittest.TestCase):
|
||||
self.assertEqual(result["item"]["status"], "queued")
|
||||
self.assertIn("Refresh queued", result["message"])
|
||||
|
||||
def test_refresh_uses_bounded_request_timeout(self):
|
||||
self.seed_watchlist_item(show_id="show-1", title="Example Show")
|
||||
snapshot = {
|
||||
"show_id": "show-1",
|
||||
"title": "Example Show",
|
||||
"episodes": {"sub": ["1"], "dub": []},
|
||||
}
|
||||
schedule = {
|
||||
"route": "example-show",
|
||||
"title": "Example Show",
|
||||
"episodes": 12,
|
||||
"status": "Finished",
|
||||
}
|
||||
|
||||
with mock.patch.object(APP, "fetch_show_episode_snapshot", return_value=snapshot) as fetch_snapshot, mock.patch.object(
|
||||
APP, "resolve_animeschedule_anime", return_value=schedule
|
||||
) as resolve_schedule, mock.patch.object(APP.WatchlistStore, "ensure_thumbnail", return_value=None):
|
||||
APP.WATCHLIST.refresh("show-1")
|
||||
|
||||
fetch_snapshot.assert_called_once_with("show-1", timeout=APP.WATCHLIST_REFRESH_REQUEST_TIMEOUT_SECONDS)
|
||||
resolve_schedule.assert_called_once_with(
|
||||
"Example Show",
|
||||
route=None,
|
||||
timeout=APP.WATCHLIST_REFRESH_REQUEST_TIMEOUT_SECONDS,
|
||||
)
|
||||
|
||||
def test_sync_downloaded_job_to_watchlist_resolves_missing_show_id_from_search(self):
|
||||
job = {"query": "Queue Show", "mode": "sub", "result_index": 1, "title": "Queue Show"}
|
||||
|
||||
@@ -562,6 +666,33 @@ class WatchlistCompletionTests(unittest.TestCase):
|
||||
self.assertEqual(prepared["show_id"], "show-42")
|
||||
self.assertEqual(prepared["title"], "Queue Show")
|
||||
|
||||
def test_prepare_download_job_accepts_equivalent_season_notation(self):
|
||||
job = {"query": "Queue Show 2nd Season", "mode": "sub", "result_index": 1, "title": "Queue Show 2nd Season", "show_id": ""}
|
||||
|
||||
with mock.patch.object(APP, "search_anime", return_value=[{"id": "show-42", "title": "Queue Show Season 2"}]):
|
||||
prepared = APP.prepare_download_job(dict(job))
|
||||
|
||||
self.assertEqual(prepared["show_id"], "show-42")
|
||||
self.assertEqual(prepared["title"], "Queue Show Season 2")
|
||||
|
||||
def test_sync_downloaded_job_to_watchlist_rejects_ambiguous_fallback_match(self):
|
||||
job = {"query": "Queue Show", "mode": "sub", "result_index": 1, "title": "Queue Show"}
|
||||
|
||||
with mock.patch.object(APP, "search_anime", return_value=[{"id": "show-42", "title": "Different Anime"}]):
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
APP.sync_downloaded_job_to_watchlist(job)
|
||||
|
||||
self.assertIn("title mismatch", str(ctx.exception).lower())
|
||||
|
||||
def test_sync_downloaded_job_to_watchlist_rejects_cross_season_match(self):
|
||||
job = {"query": "Queue Show Season 2", "mode": "sub", "result_index": 1, "title": "Queue Show Season 2"}
|
||||
|
||||
with mock.patch.object(APP, "search_anime", return_value=[{"id": "show-42", "title": "Queue Show Season 1"}]):
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
APP.sync_downloaded_job_to_watchlist(job)
|
||||
|
||||
self.assertIn("title mismatch", str(ctx.exception).lower())
|
||||
|
||||
|
||||
class WatchlistQueuedRefreshRecoveryTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
@@ -730,6 +861,7 @@ class StartupBehaviorTests(unittest.TestCase):
|
||||
with mock.patch.object(APP, "initialize_runtime") as initialize_runtime, mock.patch.object(
|
||||
APP, "ThreadingHTTPServer", return_value=server
|
||||
), mock.patch.object(APP, "shutdown_runtime") as shutdown_runtime:
|
||||
shutdown_runtime.return_value = True
|
||||
APP.main()
|
||||
|
||||
initialize_runtime.assert_not_called()
|
||||
@@ -741,6 +873,7 @@ class StartupBehaviorTests(unittest.TestCase):
|
||||
APP.initialize_runtime(start_workers=False)
|
||||
try:
|
||||
with mock.patch.object(APP, "shutdown_runtime") as shutdown_runtime:
|
||||
shutdown_runtime.return_value = True
|
||||
APP.reset_runtime(wait=True)
|
||||
|
||||
shutdown_runtime.assert_called_once_with(wait=True, cancel_active_downloads=None)
|
||||
@@ -748,6 +881,25 @@ class StartupBehaviorTests(unittest.TestCase):
|
||||
finally:
|
||||
APP.initialize_runtime(start_workers=False)
|
||||
|
||||
def test_reset_runtime_refuses_to_clear_globals_when_shutdown_does_not_finish(self):
|
||||
APP.initialize_runtime(start_workers=False)
|
||||
try:
|
||||
current_config = APP.CONFIG
|
||||
current_watchlist = APP.WATCHLIST
|
||||
current_queue = APP.DOWNLOAD_QUEUE
|
||||
current_refresh = APP.WATCHLIST_REFRESH
|
||||
with mock.patch.object(APP, "shutdown_runtime", return_value=False):
|
||||
with self.assertRaises(RuntimeError):
|
||||
APP.reset_runtime(wait=True)
|
||||
|
||||
self.assertIs(APP.CONFIG, current_config)
|
||||
self.assertIs(APP.WATCHLIST, current_watchlist)
|
||||
self.assertIs(APP.DOWNLOAD_QUEUE, current_queue)
|
||||
self.assertIs(APP.WATCHLIST_REFRESH, current_refresh)
|
||||
finally:
|
||||
APP.reset_runtime(wait=True)
|
||||
APP.initialize_runtime(start_workers=False)
|
||||
|
||||
def test_save_runtime_config_replaces_runtime_config(self):
|
||||
original = APP.CONFIG
|
||||
try:
|
||||
@@ -876,6 +1028,9 @@ class TemplateHelperTests(unittest.TestCase):
|
||||
self.assertIn("function startSerialPoll(callback, intervalMs)", APP.WATCHLIST_HTML)
|
||||
self.assertIn("startSerialPoll(() => loadQueue(), 2500)", APP.INDEX_HTML)
|
||||
self.assertIn("startSerialPoll(async () => {", APP.WATCHLIST_HTML)
|
||||
self.assertIn("window.clearTimeout(timer);", APP.INDEX_HTML)
|
||||
self.assertIn('window.addEventListener("pagehide", stop, { once: true });', APP.INDEX_HTML)
|
||||
self.assertIn('window.addEventListener("beforeunload", stop, { once: true });', APP.WATCHLIST_HTML)
|
||||
|
||||
def test_watchlist_page_polls_when_queued_items_exist(self):
|
||||
self.assertIn("queuedCount: 0", APP.WATCHLIST_HTML)
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
"""Shared anime title normalization and matching helpers."""
|
||||
|
||||
import re
|
||||
import unicodedata
|
||||
|
||||
|
||||
def normalize_title_key(value):
|
||||
text = unicodedata.normalize("NFKD", str(value or ""))
|
||||
text = "".join(char for char in text if not unicodedata.combining(char))
|
||||
text = text.lower()
|
||||
text = re.sub(r"\((19|20)\d{2}\)", " ", text)
|
||||
text = re.sub(r"[^a-z0-9]+", " ", text)
|
||||
return re.sub(r"\s+", " ", text).strip()
|
||||
|
||||
|
||||
def normalize_identity_title_key(value):
|
||||
text = normalize_title_key(value)
|
||||
if not text:
|
||||
return ""
|
||||
text = re.sub(r"\b(\d+)(st|nd|rd|th)\s+season\b", r"season \1", text)
|
||||
return re.sub(r"\s+", " ", text).strip()
|
||||
|
||||
|
||||
def base_title_variants(value):
|
||||
text = str(value or "").strip()
|
||||
variants = []
|
||||
if text:
|
||||
variants.append(text)
|
||||
patterns = [
|
||||
r"\s+\((19|20)\d{2}\)\s*$",
|
||||
r"\s+season\s+\d+\s*$",
|
||||
r"\s+part\s+\d+\s*$",
|
||||
r"\s+cour\s+\d+\s*$",
|
||||
r"\s+\d+(st|nd|rd|th)\s+season\s*$",
|
||||
]
|
||||
changed = True
|
||||
current = text
|
||||
while current and changed:
|
||||
changed = False
|
||||
for pattern in patterns:
|
||||
trimmed = re.sub(pattern, "", current, flags=re.IGNORECASE).strip(" -:")
|
||||
if trimmed and trimmed != current:
|
||||
current = trimmed
|
||||
if current not in variants:
|
||||
variants.append(current)
|
||||
changed = True
|
||||
return variants
|
||||
|
||||
|
||||
def title_match_variants(value):
|
||||
variants = set()
|
||||
for raw_value in base_title_variants(value):
|
||||
normalized = normalize_title_key(raw_value)
|
||||
if normalized:
|
||||
variants.add(normalized)
|
||||
stripped = re.sub(r"\b(19|20)\d{2}\b", " ", normalized)
|
||||
stripped = re.sub(r"\s+", " ", stripped).strip()
|
||||
if stripped:
|
||||
variants.add(stripped)
|
||||
return variants
|
||||
|
||||
|
||||
def identity_title_variants(value):
|
||||
variants = set()
|
||||
normalized = normalize_identity_title_key(value)
|
||||
if normalized:
|
||||
variants.add(normalized)
|
||||
stripped = re.sub(r"\b(19|20)\d{2}\b", " ", normalized)
|
||||
stripped = re.sub(r"\s+", " ", stripped).strip()
|
||||
if stripped:
|
||||
variants.add(stripped)
|
||||
return variants
|
||||
|
||||
|
||||
def title_lookup_queries(value):
|
||||
queries = []
|
||||
for raw_value in base_title_variants(value):
|
||||
text = str(raw_value or "").strip()
|
||||
if text and text not in queries:
|
||||
queries.append(text)
|
||||
return queries
|
||||
@@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
"""Helpers for resolving watchlist identities from queued download jobs."""
|
||||
|
||||
import re
|
||||
|
||||
|
||||
def _resolution_result(show_id, title, reason, return_reason):
|
||||
if return_reason:
|
||||
return show_id, title, reason
|
||||
return show_id, title
|
||||
|
||||
|
||||
def _normalized_variants(title, normalize_title_key_fn, base_title_variants_fn):
|
||||
del base_title_variants_fn
|
||||
variants = set()
|
||||
normalized_title = normalize_title_key_fn(title)
|
||||
if normalized_title:
|
||||
variants.add(normalized_title)
|
||||
stripped = re.sub(r"\b(19|20)\d{2}\b", " ", normalized_title)
|
||||
stripped = re.sub(r"\s+", " ", stripped).strip()
|
||||
if stripped:
|
||||
variants.add(stripped)
|
||||
return variants
|
||||
|
||||
|
||||
def titles_confidently_match(expected_title, candidate_title, normalize_title_key_fn, base_title_variants_fn):
|
||||
expected_variants = _normalized_variants(expected_title, normalize_title_key_fn, base_title_variants_fn)
|
||||
candidate_variants = _normalized_variants(candidate_title, normalize_title_key_fn, base_title_variants_fn)
|
||||
if not expected_variants or not candidate_variants:
|
||||
return False
|
||||
return bool(expected_variants & candidate_variants)
|
||||
|
||||
|
||||
def resolve_job_watchlist_identity(
|
||||
job,
|
||||
*,
|
||||
search_fn,
|
||||
normalize_title_key_fn,
|
||||
base_title_variants_fn,
|
||||
default_mode,
|
||||
return_reason=False,
|
||||
):
|
||||
show_id = str(job.get("show_id") or "").strip()
|
||||
title = str(job.get("title") or job.get("anime_name") or "").strip()
|
||||
if show_id:
|
||||
return _resolution_result(show_id, title, "", return_reason)
|
||||
try:
|
||||
index = max(1, int(job.get("result_index") or 1))
|
||||
except (TypeError, ValueError):
|
||||
return _resolution_result("", title, "invalid queued result index", return_reason)
|
||||
query = str(job.get("query") or title).strip()
|
||||
if not query or not title:
|
||||
return _resolution_result("", title, "missing queued search query or title", return_reason)
|
||||
try:
|
||||
results = search_fn(query, job.get("mode") or default_mode)
|
||||
except Exception as exc:
|
||||
return _resolution_result("", title, f"search fallback failed: {exc}", return_reason)
|
||||
if index > len(results):
|
||||
return _resolution_result("", title, "queued result index no longer exists in search results", return_reason)
|
||||
result = results[index - 1]
|
||||
candidate_title = str(result.get("title") or "").strip()
|
||||
if not titles_confidently_match(title, candidate_title, normalize_title_key_fn, base_title_variants_fn):
|
||||
return _resolution_result("", title, f"title mismatch with fallback search result: {candidate_title or 'unknown'}", return_reason)
|
||||
resolved_show_id = str(result.get("id") or "").strip()
|
||||
if not resolved_show_id:
|
||||
return _resolution_result("", title, "fallback search result did not include a show_id", return_reason)
|
||||
return _resolution_result(resolved_show_id, candidate_title or title, "", return_reason)
|
||||
Reference in New Issue
Block a user