Compare commits

...
4 Commits
Author SHA1 Message Date
Dymas d0af0258f2 Reduce polling request log noise 2026-05-17 21:20:24 +02:00
Dymas ae090c954b Pace bulk watchlist refresh requests 2026-05-17 21:15:53 +02:00
Dymas a8d7d3de00 Log scheduled refresh lifecycle to stdout 2026-05-17 21:06:48 +02:00
Dymas 6168acb272 Add scheduled watchlist refresh support 2026-05-17 20:38:16 +02:00
9 changed files with 463 additions and 51 deletions
+19
View File
@@ -1,5 +1,24 @@
# Changelog
## 0.32.1 - 2026-05-17
- Suppressed stdout request logging for routine polling endpoints such as `/api/queue` and `/api/watchlist/refresh-status`, reducing noisy access-log spam during normal browser use.
## 0.32.0 - 2026-05-17
- Changed bulk `Refresh All` processing to pace requests with a configurable delay between shows, reducing the chance of flooding upstream episode sources on large watchlists.
- Added a Config page setting for the per-show bulk refresh delay and expanded regression coverage for the new config field and pacing behavior.
## 0.31.1 - 2026-05-17
- Added stdout lifecycle logging for scheduled watchlist refresh jobs so container logs now show when a scheduled run is queued, starts, finishes successfully, finishes with errors, gets interrupted, or fails.
## 0.31.0 - 2026-05-17
- Added scheduled background watchlist refresh support so the app can periodically recheck episode counts for tracked anime.
- Added Config page controls for enabling or disabling the scheduled refresh and setting the refresh period in minutes.
- Reused the existing background `Refresh All` worker for scheduled runs and expanded regression coverage for the new config fields and scheduler timing behavior.
## 0.30.4 - 2026-05-17
- Added browser-friendly remote auth bootstrap: remote users can open the app once with `?token=...`, the server stores the token in an HTTP-only cookie, and then strips the token back out of the URL with a redirect.
+11 -3
View File
@@ -2,7 +2,7 @@
A local web UI for a system-wide `ani-cli` install.
Current version: `0.30.4`
Current version: `0.32.1`
## Project Layout
@@ -156,7 +156,10 @@ The app currently includes:
- Per-show watchlist refresh queueing now also deduplicates in-flight refreshes, so the same show is not fetched twice back-to-back while one refresh is already running.
- The watchlist page now silently refreshes itself while queued per-show refresh work is still being processed, so queued entries settle into their updated state without a manual reload.
- The watchlist page now also avoids re-triggering the same unresolved thumbnail warm-up request on every silent poll cycle while a show's thumbnail state is unchanged.
- Background `Refresh All` execution with status polling, parallel per-show refresh work, and no inline thumbnail downloads during the bulk job.
- Background `Refresh All` execution with status polling, no inline thumbnail downloads during the bulk job, and a configurable delay between shows to reduce upstream request bursts.
- Optional scheduled background watchlist refreshes with Config page controls for enable/disable and refresh period.
- Scheduled background refresh runs now log queue, start, and completion status to stdout so container logs show whether they finished successfully, with errors, or failed.
- Routine browser polling endpoints such as the search-page queue poll and watchlist refresh-status poll now stay quiet in stdout request logs to reduce log spam.
- Search and Watchlist polling now use a shared serial browser polling helper so slow responses do not cause overlapping poll requests to pile up in the background.
- That shared browser polling helper now also tears itself down on page unload and uses chained `setTimeout` scheduling instead of a bare repeating interval.
- Controlled shutdown and runtime reset now cancel active download workers when waiting for teardown, reducing the chance of orphaned background downloads outliving the web process.
@@ -202,6 +205,9 @@ Use it to:
1. Set the default download folder.
2. Choose the default `sub` or `dub` search mode.
3. Choose the default search quality used to prefill the Search page.
4. Enable or disable periodic watchlist refresh checks.
5. Set the refresh period in minutes for that scheduled watchlist job.
6. Set the delay in seconds between shows during bulk watchlist refresh runs.
Those saved defaults are written into the project-local `config.json` and loaded automatically when the Search page opens.
@@ -274,7 +280,7 @@ You can add entries in two ways:
Watchlist features:
1. `Refresh` updates one tracked show.
2. `Refresh All` runs in the background, rejects duplicate queueing while a refresh is already pending or running, preserves interrupted job status across restarts, and refreshes shows in parallel without doing inline thumbnail downloads for every entry.
2. `Refresh All` runs in the background, rejects duplicate queueing while a refresh is already pending or running, preserves interrupted job status across restarts, and refreshes shows one by one with a configurable delay without doing inline thumbnail downloads for every entry.
3. `Open` jumps back to the Search page with the title pre-filled.
4. Clicking the anime title opens the AllManga source page at `bangumi/<show_id>` in a new tab.
5. `Remove` deletes the watchlist entry and clears its cached poster file.
@@ -388,6 +394,8 @@ Environment variables:
Saved defaults from the UI are written into the project-local `config.json`.
The saved config now also includes `watchlist_auto_refresh_enabled`, `watchlist_auto_refresh_minutes`, and `watchlist_refresh_delay_seconds`, which control the periodic background watchlist refresh schedule and the per-show pacing used by bulk refresh jobs.
## Notes
- Importing `app.py` no longer boots the runtime worker threads automatically; runtime setup is deferred until startup or first use.
+1 -1
View File
@@ -1 +1 @@
0.30.4
0.32.1
+48 -9
View File
@@ -57,7 +57,7 @@ from app_support import (
write_json,
)
from http_handler import HandlerContext, build_handler_class, server_host, server_port
from queue_jobs import DownloadQueue, WatchlistRefreshJobs
from queue_jobs import DownloadQueue, WatchlistAutoRefreshScheduler, WatchlistRefreshJobs
from title_matching import (
base_title_variants,
normalize_title_key,
@@ -82,6 +82,7 @@ CONFIG = None
WATCHLIST = None
DOWNLOAD_QUEUE = None
WATCHLIST_REFRESH = None
WATCHLIST_AUTO_REFRESH = None
def graph_request(query, variables, timeout=25):
@@ -1780,10 +1781,14 @@ def get_config_snapshot(fallback=None):
def save_runtime_config(config):
normalized = normalize_config(config)
scheduler = None
with RUNTIME_LOCK:
global CONFIG
CONFIG = dict(normalized)
write_json(CONFIG_PATH, CONFIG)
scheduler = WATCHLIST_AUTO_REFRESH
if scheduler is not None:
scheduler.notify_config_changed()
return dict(CONFIG)
@@ -1793,6 +1798,7 @@ def runtime_state():
"watchlist": WATCHLIST,
"download_queue": DOWNLOAD_QUEUE,
"watchlist_refresh": WATCHLIST_REFRESH,
"watchlist_auto_refresh": WATCHLIST_AUTO_REFRESH,
}
@@ -1808,36 +1814,60 @@ def build_runtime(start_workers=None):
job_prepare_fn=prepare_download_job,
start_worker=worker_state,
)
watchlist_refresh = WatchlistRefreshJobs(watchlist, start_worker=worker_state)
watchlist_refresh = WatchlistRefreshJobs(
watchlist,
config_getter=lambda: get_config_snapshot(config),
start_worker=worker_state,
)
watchlist_auto_refresh = WatchlistAutoRefreshScheduler(
config_getter=lambda: get_config_snapshot(config),
refresh_start_fn=watchlist_refresh.start,
start_worker=worker_state,
)
return {
"config": config,
"watchlist": watchlist,
"download_queue": download_queue,
"watchlist_refresh": watchlist_refresh,
"watchlist_auto_refresh": watchlist_auto_refresh,
}
def initialize_runtime(start_workers=None):
global CONFIG, WATCHLIST, DOWNLOAD_QUEUE, WATCHLIST_REFRESH
global CONFIG, WATCHLIST, DOWNLOAD_QUEUE, WATCHLIST_REFRESH, WATCHLIST_AUTO_REFRESH
with RUNTIME_LOCK:
workers_allowed = background_workers_enabled()
if CONFIG is not None and WATCHLIST is not None and DOWNLOAD_QUEUE is not None and WATCHLIST_REFRESH is not None:
if (
CONFIG is not None
and WATCHLIST is not None
and DOWNLOAD_QUEUE is not None
and WATCHLIST_REFRESH is not None
and WATCHLIST_AUTO_REFRESH is not None
):
if start_workers and workers_allowed:
WATCHLIST.worker_enabled = True
WATCHLIST.ensure_worker()
DOWNLOAD_QUEUE.ensure_worker()
WATCHLIST_REFRESH.ensure_worker()
WATCHLIST_AUTO_REFRESH.ensure_worker()
return runtime_state()
runtime = build_runtime(start_workers=start_workers)
CONFIG = runtime["config"]
WATCHLIST = runtime["watchlist"]
DOWNLOAD_QUEUE = runtime["download_queue"]
WATCHLIST_REFRESH = runtime["watchlist_refresh"]
WATCHLIST_AUTO_REFRESH = runtime["watchlist_auto_refresh"]
return runtime
def ensure_runtime(start_workers=False):
if CONFIG is None or WATCHLIST is None or DOWNLOAD_QUEUE is None or WATCHLIST_REFRESH is None:
if (
CONFIG is None
or WATCHLIST is None
or DOWNLOAD_QUEUE is None
or WATCHLIST_REFRESH is None
or WATCHLIST_AUTO_REFRESH is None
):
return initialize_runtime(start_workers=start_workers)
if start_workers:
return initialize_runtime(start_workers=True)
@@ -1847,10 +1877,16 @@ def ensure_runtime(start_workers=False):
def shutdown_runtime(wait=False, cancel_active_downloads=None):
services = None
with RUNTIME_LOCK:
if CONFIG is None and WATCHLIST is None and DOWNLOAD_QUEUE is None and WATCHLIST_REFRESH is None:
if (
CONFIG is None
and WATCHLIST is None
and DOWNLOAD_QUEUE is None
and WATCHLIST_REFRESH is None
and WATCHLIST_AUTO_REFRESH is None
):
return False
services = (WATCHLIST, DOWNLOAD_QUEUE, WATCHLIST_REFRESH)
watchlist, download_queue, watchlist_refresh = services
services = (WATCHLIST, DOWNLOAD_QUEUE, WATCHLIST_REFRESH, WATCHLIST_AUTO_REFRESH)
watchlist, download_queue, watchlist_refresh, watchlist_auto_refresh = services
should_cancel_downloads = bool(wait) if cancel_active_downloads is None else bool(cancel_active_downloads)
stopped = True
if watchlist is not None:
@@ -1859,11 +1895,13 @@ def shutdown_runtime(wait=False, cancel_active_downloads=None):
stopped = download_queue.shutdown(wait=wait, cancel_active=should_cancel_downloads) and stopped
if watchlist_refresh is not None:
stopped = watchlist_refresh.shutdown(wait=wait) and stopped
if watchlist_auto_refresh is not None:
stopped = watchlist_auto_refresh.shutdown(wait=wait) and stopped
return stopped
def reset_runtime(wait=False, cancel_active_downloads=None):
global CONFIG, WATCHLIST, DOWNLOAD_QUEUE, WATCHLIST_REFRESH
global CONFIG, WATCHLIST, DOWNLOAD_QUEUE, WATCHLIST_REFRESH, WATCHLIST_AUTO_REFRESH
stopped = shutdown_runtime(wait=wait, cancel_active_downloads=cancel_active_downloads)
if not stopped:
if wait:
@@ -1874,6 +1912,7 @@ def reset_runtime(wait=False, cancel_active_downloads=None):
WATCHLIST = None
DOWNLOAD_QUEUE = None
WATCHLIST_REFRESH = None
WATCHLIST_AUTO_REFRESH = None
return True
+38 -1
View File
@@ -16,7 +16,7 @@ from uuid import uuid4
PROJECT_ROOT = Path(__file__).resolve().parent
ANI_CLI = os.environ.get("ANI_CLI_BIN") or shutil.which("ani-cli") or "ani-cli"
APP_NAME = "ani-cli-web"
VERSION = "0.30.4"
VERSION = "0.32.1"
ALLANIME_BASE = "allanime.day"
ALLANIME_API = f"https://api.{ALLANIME_BASE}"
ALLANIME_REFERER = "https://allmanga.to"
@@ -42,6 +42,12 @@ MAX_LOG_LINES = 240
MAX_JSON_BODY_BYTES = 12 * 1024 * 1024
WATCHLIST_REFRESH_REQUEST_TIMEOUT_SECONDS = 8
WORKER_SHUTDOWN_TIMEOUT_SECONDS = 10
WATCHLIST_AUTO_REFRESH_DEFAULT_MINUTES = 60
WATCHLIST_AUTO_REFRESH_MIN_MINUTES = 1
WATCHLIST_AUTO_REFRESH_MAX_MINUTES = 7 * 24 * 60
WATCHLIST_REFRESH_DELAY_DEFAULT_SECONDS = 5
WATCHLIST_REFRESH_DELAY_MIN_SECONDS = 0
WATCHLIST_REFRESH_DELAY_MAX_SECONDS = 300
def env_flag(name, default=False):
@@ -114,6 +120,9 @@ DEFAULT_CONFIG = {
"download_dir": default_download_dir(),
"mode": os.environ.get("ANI_CLI_MODE", "sub"),
"quality": os.environ.get("ANI_CLI_QUALITY", "best"),
"watchlist_auto_refresh_enabled": False,
"watchlist_auto_refresh_minutes": WATCHLIST_AUTO_REFRESH_DEFAULT_MINUTES,
"watchlist_refresh_delay_seconds": WATCHLIST_REFRESH_DELAY_DEFAULT_SECONDS,
}
@@ -254,10 +263,38 @@ def normalize_config(data):
if quality.endswith("p") and quality[:-1].isdigit():
quality = quality[:-1]
download_dir = str(config.get("download_dir") or DEFAULT_CONFIG["download_dir"])
auto_refresh_enabled = config.get("watchlist_auto_refresh_enabled")
auto_refresh_minutes = config.get("watchlist_auto_refresh_minutes")
refresh_delay_seconds = config.get("watchlist_refresh_delay_seconds")
if isinstance(auto_refresh_enabled, str):
auto_refresh_enabled = auto_refresh_enabled.strip().lower() in {"1", "true", "yes", "on"}
else:
auto_refresh_enabled = bool(auto_refresh_enabled)
try:
auto_refresh_minutes = int(str(auto_refresh_minutes).strip() or WATCHLIST_AUTO_REFRESH_DEFAULT_MINUTES)
except (TypeError, ValueError):
auto_refresh_minutes = WATCHLIST_AUTO_REFRESH_DEFAULT_MINUTES
auto_refresh_minutes = max(
WATCHLIST_AUTO_REFRESH_MIN_MINUTES,
min(WATCHLIST_AUTO_REFRESH_MAX_MINUTES, auto_refresh_minutes),
)
try:
refresh_delay_seconds = int(str(refresh_delay_seconds).strip() or WATCHLIST_REFRESH_DELAY_DEFAULT_SECONDS)
except (TypeError, ValueError):
refresh_delay_seconds = WATCHLIST_REFRESH_DELAY_DEFAULT_SECONDS
refresh_delay_seconds = max(
WATCHLIST_REFRESH_DELAY_MIN_SECONDS,
min(WATCHLIST_REFRESH_DELAY_MAX_SECONDS, refresh_delay_seconds),
)
config["mode"] = mode if mode in MODE_CHOICES else "sub"
config["quality"] = quality if quality in QUALITY_CHOICES else "best"
config["download_dir"] = str(Path(download_dir).expanduser())
config["watchlist_auto_refresh_enabled"] = auto_refresh_enabled
config["watchlist_auto_refresh_minutes"] = auto_refresh_minutes
config["watchlist_refresh_delay_seconds"] = refresh_delay_seconds
return config
+42 -2
View File
@@ -306,13 +306,44 @@ CONFIG_HTML = r"""<!doctype html>
<div class="field-hint">Tip: these are the saved defaults only. The active search form can still be adjusted per download.</div>
</section>
<section class="settings settings-main">
<div class="toolbar">
<div>
<h2>Watchlist schedule</h2>
<p class="muted">Periodically queue a background watchlist refresh to check for newly available episodes.</p>
</div>
</div>
<div class="form-grid">
<label>Automatic refresh
<select id="watchlistAutoRefreshEnabled">
<option value="false">Disabled</option>
<option value="true">Enabled</option>
</select>
</label>
<label>Refresh period (minutes)
<input id="watchlistAutoRefreshMinutes" type="number" min="1" max="10080" step="1" value="60">
</label>
<label>Delay between series (seconds)
<input id="watchlistRefreshDelaySeconds" type="number" min="0" max="300" step="1" value="5">
</label>
</div>
<div class="field-hint">When enabled, the server periodically starts the existing background “Refresh All” job using this interval. Bulk refreshes now wait this many seconds between shows to avoid flooding upstream sources.</div>
</section>
<section class="deps" id="deps"></section>
</main>
</div>
<script>
const state = {
config: { mode: "sub", quality: "best", download_dir: "" }
config: {
mode: "sub",
quality: "best",
download_dir: "",
watchlist_auto_refresh_enabled: false,
watchlist_auto_refresh_minutes: 60,
watchlist_refresh_delay_seconds: 5
}
};
const $ = (id) => document.getElementById(id);
@@ -327,6 +358,9 @@ CONFIG_HTML = r"""<!doctype html>
$("downloadDir").value = state.config.download_dir;
$("configMode").value = state.config.mode;
$("configQuality").value = state.config.quality;
$("watchlistAutoRefreshEnabled").value = String(Boolean(state.config.watchlist_auto_refresh_enabled));
$("watchlistAutoRefreshMinutes").value = state.config.watchlist_auto_refresh_minutes;
$("watchlistRefreshDelaySeconds").value = state.config.watchlist_refresh_delay_seconds;
}
async function saveConfig() {
@@ -336,13 +370,19 @@ CONFIG_HTML = r"""<!doctype html>
body: JSON.stringify({
download_dir: $("downloadDir").value,
mode: $("configMode").value,
quality: $("configQuality").value
quality: $("configQuality").value,
watchlist_auto_refresh_enabled: $("watchlistAutoRefreshEnabled").value === "true",
watchlist_auto_refresh_minutes: $("watchlistAutoRefreshMinutes").value,
watchlist_refresh_delay_seconds: $("watchlistRefreshDelaySeconds").value
})
});
state.config = data;
$("downloadDir").value = data.download_dir;
$("configMode").value = data.mode;
$("configQuality").value = data.quality;
$("watchlistAutoRefreshEnabled").value = String(Boolean(data.watchlist_auto_refresh_enabled));
$("watchlistAutoRefreshMinutes").value = data.watchlist_auto_refresh_minutes;
$("watchlistRefreshDelaySeconds").value = data.watchlist_refresh_delay_seconds;
setNotice("Defaults saved.");
} catch (error) {
setNotice(error.message);
+4
View File
@@ -70,6 +70,7 @@ def dependency_status():
class Handler(BaseHTTPRequestHandler):
server_version = "AniCliWeb/1.0"
remote_token_cookie_name = "ani_cli_web_token"
quiet_poll_paths = {"/api/queue", "/api/watchlist/refresh-status"}
def _context(self):
context = getattr(self, "handler_context", None) or getattr(type(self), "handler_context", None)
@@ -467,6 +468,9 @@ class Handler(BaseHTTPRequestHandler):
self.error(HTTPStatus.INTERNAL_SERVER_ERROR, "Internal server error.")
def log_message(self, fmt, *args):
parsed = urlparse(getattr(self, "path", "") or "")
if self.command == "GET" and parsed.path in Handler.quiet_poll_paths:
return
print(f"{self.address_string()} - {fmt % args}")
+133 -32
View File
@@ -2,7 +2,6 @@
"""Queue and background job services for ani-cli-web."""
import concurrent.futures
import json
import os
import shutil
@@ -33,7 +32,6 @@ from app_support import (
LOG_PERSIST_INTERVAL_SECONDS = 0.75
LOG_PERSIST_LINE_BATCH = 8
WATCHLIST_REFRESH_MAX_WORKERS = 4
JOB_STRUCTURED_COLUMNS = (
"show_id",
"title",
@@ -59,8 +57,9 @@ JOB_EXTRA_PAYLOAD_COLUMN = "payload"
class WatchlistRefreshJobs:
def __init__(self, store, start_worker=True):
def __init__(self, store, config_getter=None, start_worker=True):
self.store = store
self.config_getter = config_getter or (lambda: {})
self.lock = threading.RLock()
self.wakeup = threading.Event()
self.stop_event = threading.Event()
@@ -153,6 +152,14 @@ class WatchlistRefreshJobs:
self._upsert_job_conn(conn, job)
self._trim_history_conn(conn)
def _refresh_delay_seconds(self):
config = self.config_getter() or {}
try:
delay = int(str(config.get("watchlist_refresh_delay_seconds", 5)).strip() or "5")
except (TypeError, ValueError):
delay = 5
return max(0, min(300, delay))
def _load_jobs(self):
with self.lock, self._connect() as conn:
rows = conn.execute(
@@ -178,6 +185,9 @@ class WatchlistRefreshJobs:
def _copy_job(self, job):
return json.loads(json.dumps(job))
def _stdout_log(self, message):
print(f"[watchlist-refresh] {message}", flush=True)
def _find_locked(self, job_id):
for job in self.jobs:
if job["id"] == job_id:
@@ -192,10 +202,15 @@ class WatchlistRefreshJobs:
active = bool(job and job["status"] in {"pending", "running"})
return {"job": self._copy_job(job) if job else None, "running": active}
def start(self):
def start(self, source="manual"):
source_name = str(source or "manual").strip().lower() or "manual"
with self.lock:
job = next((candidate for candidate in self.jobs if candidate["status"] in {"pending", "running"}), None)
if job is not None:
if source_name == "scheduled":
self._stdout_log(
f"Scheduled refresh skipped because another refresh is already {job['status']} (job={job['id']})."
)
return {
"message": "Watchlist refresh already queued." if job["status"] == "pending" else "Watchlist refresh already running.",
"job": self._copy_job(job) if job else None,
@@ -205,6 +220,7 @@ class WatchlistRefreshJobs:
job = {
"id": os.urandom(16).hex(),
"status": "pending",
"source": source_name,
"created_at": now,
"updated_at": now,
"started_at": None,
@@ -220,6 +236,8 @@ class WatchlistRefreshJobs:
self._save_job_locked(job)
self.pending.append(job["id"])
self.wakeup.set()
if source_name == "scheduled":
self._stdout_log(f"Scheduled refresh queued (job={job['id']}).")
return {"message": "Watchlist refresh started.", "job": self._copy_job(job), "created": True}
def _next_pending(self):
@@ -243,8 +261,10 @@ class WatchlistRefreshJobs:
def _run_job(self, job):
interrupted = False
source_name = str(job.get("source") or "manual").strip().lower() or "manual"
try:
show_ids = self.store.all_show_ids()
delay_seconds = self._refresh_delay_seconds()
started_at = now_iso()
with self.lock:
job["status"] = "running"
@@ -256,14 +276,17 @@ class WatchlistRefreshJobs:
job["current_title"] = None
job["message"] = f"Refreshing 0/{len(show_ids)} watchlist entries."
self._save_job_locked(job)
if source_name == "scheduled":
self._stdout_log(f"Scheduled refresh started (job={job['id']}, total={len(show_ids)}).")
def refresh_one(show_id):
for index, show_id in enumerate(show_ids, start=1):
if self.stop_event.is_set():
interrupted = True
break
title = show_id
errored = False
skipped = False
error_message = ""
if self.stop_event.is_set():
return {"title": title, "errored": False, "skipped": True, "error": ""}
try:
item = self.store.refresh(show_id, include_thumbnail=False)
if item is None:
@@ -276,34 +299,22 @@ class WatchlistRefreshJobs:
errored = True
error_message = str(exc)
debug_log("watchlist.refresh_all.item_error", show_id=show_id, error=exc)
return {"title": title, "errored": errored, "skipped": skipped, "error": error_message}
max_workers = max(1, min(WATCHLIST_REFRESH_MAX_WORKERS, len(show_ids) or 1))
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
with self.lock:
self.current_executor = executor
futures = [executor.submit(refresh_one, show_id) for show_id in show_ids]
for index, future in enumerate(concurrent.futures.as_completed(futures), start=1):
if self.stop_event.is_set():
job["completed"] = index
job["errors"] += 1 if errored else 0
job["current_title"] = title
job["updated_at"] = now_iso()
if errored and error_message:
job["last_error"] = error_message
if skipped:
job["message"] = f"Refreshing {index}/{len(show_ids)}: removed entry"
else:
job["message"] = f"Refreshing {index}/{len(show_ids)}: {title}"
self._save_job_locked(job)
if index < len(show_ids) and delay_seconds > 0:
if self.stop_event.wait(delay_seconds):
interrupted = True
executor.shutdown(wait=False, cancel_futures=True)
break
result = future.result()
title = result["title"]
with self.lock:
job["completed"] = index
job["errors"] += 1 if result["errored"] else 0
job["current_title"] = title
job["updated_at"] = now_iso()
if result["errored"] and result["error"]:
job["last_error"] = result["error"]
if result["skipped"]:
job["message"] = f"Refreshing {index}/{len(show_ids)}: removed entry"
else:
job["message"] = f"Refreshing {index}/{len(show_ids)}: {title}"
self._save_job_locked(job)
with self.lock:
self.current_executor = None
finished_at = now_iso()
with self.lock:
@@ -320,6 +331,19 @@ class WatchlistRefreshJobs:
else:
job["message"] = f"Refreshed {job['completed']} watchlist entries."
self._save_job_locked(job)
if source_name == "scheduled":
if interrupted:
self._stdout_log(
f"Scheduled refresh interrupted (job={job['id']}, completed={job['completed']}/{job['total']})."
)
elif job["errors"]:
self._stdout_log(
f"Scheduled refresh finished with errors (job={job['id']}, completed={job['completed']}/{job['total']}, errors={job['errors']})."
)
else:
self._stdout_log(
f"Scheduled refresh finished successfully (job={job['id']}, completed={job['completed']}/{job['total']})."
)
except Exception as exc:
finished_at = now_iso()
with self.lock:
@@ -329,12 +353,89 @@ class WatchlistRefreshJobs:
job["current_title"] = None
job["message"] = f"Watchlist refresh failed: {exc}"
self._save_job_locked(job)
if source_name == "scheduled":
self._stdout_log(f"Scheduled refresh failed (job={job['id']}): {exc}")
finally:
with self.lock:
self.current_executor = None
self.current_job_id = None
class WatchlistAutoRefreshScheduler:
def __init__(self, config_getter, refresh_start_fn, start_worker=True):
self.config_getter = config_getter
self.refresh_start_fn = refresh_start_fn
self.lock = threading.RLock()
self.wakeup = threading.Event()
self.stop_event = threading.Event()
self.worker = None
self.last_enabled = None
self.last_interval_seconds = None
self.last_attempt_monotonic = None
if start_worker:
self.ensure_worker()
def ensure_worker(self):
with self.lock:
if self.worker is not None and self.worker.is_alive():
return
self.stop_event.clear()
self.worker = threading.Thread(target=self._run, daemon=True)
self.worker.start()
def shutdown(self, wait=False):
self.stop_event.set()
self.wakeup.set()
worker = self.worker
if wait and worker is not None and worker.is_alive():
worker.join(timeout=WORKER_SHUTDOWN_TIMEOUT_SECONDS)
return worker is None or not worker.is_alive()
def notify_config_changed(self):
self.wakeup.set()
def _config_state(self):
config = self.config_getter() or {}
enabled = bool(config.get("watchlist_auto_refresh_enabled"))
try:
minutes = int(str(config.get("watchlist_auto_refresh_minutes", 60)).strip() or "60")
except (TypeError, ValueError):
minutes = 60
minutes = max(1, min(7 * 24 * 60, minutes))
return enabled, minutes * 60
def tick(self, now_monotonic=None):
now = time.monotonic() if now_monotonic is None else float(now_monotonic)
enabled, interval_seconds = self._config_state()
with self.lock:
changed = enabled != self.last_enabled or interval_seconds != self.last_interval_seconds
if changed:
self.last_enabled = enabled
self.last_interval_seconds = interval_seconds
self.last_attempt_monotonic = now
if not enabled:
return None
if self.last_attempt_monotonic is None:
self.last_attempt_monotonic = now
return interval_seconds
remaining = interval_seconds - (now - self.last_attempt_monotonic)
if remaining > 0:
return remaining
self.last_attempt_monotonic = now
try:
result = self.refresh_start_fn(source="scheduled")
debug_log("watchlist.auto_refresh.tick", result=result)
except Exception as exc:
debug_log("watchlist.auto_refresh.error", error=exc)
return interval_seconds
def _run(self):
while not self.stop_event.is_set():
wait_seconds = self.tick()
self.wakeup.wait(timeout=wait_seconds)
self.wakeup.clear()
class DownloadQueue:
def __init__(self, config_getter, watchlist_sync_fn=None, job_prepare_fn=None, start_worker=True):
self.config_getter = config_getter
+167 -3
View File
@@ -461,6 +461,26 @@ class WatchlistRefreshJobTests(unittest.TestCase):
self.assertEqual(second["job"]["id"], first["job"]["id"])
self.assertEqual(second["job"]["status"], "pending")
def test_scheduled_refresh_logs_queue_and_completion(self):
class FakeStore:
def all_show_ids(self):
return ["show-a"]
def refresh(self, _show_id, include_thumbnail=True):
return {"title": "Show A", "status": "updated"}
service = APP.WatchlistRefreshJobs(FakeStore(), start_worker=False)
with mock.patch("builtins.print") as print_mock:
started = service.start(source="scheduled")
job = service._next_pending()
service._run_job(job)
self.assertTrue(started["created"])
logged = "\n".join(call.args[0] for call in print_mock.call_args_list if call.args)
self.assertIn("Scheduled refresh queued", logged)
self.assertIn("Scheduled refresh started", logged)
self.assertIn("Scheduled refresh finished successfully", logged)
def test_refresh_history_marks_pending_jobs_interrupted_after_restart(self):
class FakeStore:
def all_show_ids(self):
@@ -1048,7 +1068,14 @@ class WatchlistRefreshQueueTests(unittest.TestCase):
class ConfigSnapshotTests(unittest.TestCase):
def test_get_config_snapshot_returns_copy(self):
original = APP.CONFIG
APP.CONFIG = {"mode": "sub", "quality": "best", "download_dir": "/tmp/example"}
APP.CONFIG = {
"mode": "sub",
"quality": "best",
"download_dir": "/tmp/example",
"watchlist_auto_refresh_enabled": False,
"watchlist_auto_refresh_minutes": 60,
"watchlist_refresh_delay_seconds": 5,
}
try:
snapshot = APP.get_config_snapshot()
snapshot["mode"] = "dub"
@@ -1056,6 +1083,85 @@ class ConfigSnapshotTests(unittest.TestCase):
finally:
APP.CONFIG = original
def test_normalize_config_includes_watchlist_schedule_defaults(self):
config = APP.normalize_config({})
self.assertFalse(config["watchlist_auto_refresh_enabled"])
self.assertEqual(config["watchlist_auto_refresh_minutes"], 60)
self.assertEqual(config["watchlist_refresh_delay_seconds"], 5)
def test_normalize_config_parses_watchlist_schedule_fields(self):
config = APP.normalize_config(
{
"watchlist_auto_refresh_enabled": "true",
"watchlist_auto_refresh_minutes": "15",
"watchlist_refresh_delay_seconds": "7",
}
)
self.assertTrue(config["watchlist_auto_refresh_enabled"])
self.assertEqual(config["watchlist_auto_refresh_minutes"], 15)
self.assertEqual(config["watchlist_refresh_delay_seconds"], 7)
class WatchlistAutoRefreshSchedulerTests(unittest.TestCase):
def test_tick_waits_then_triggers_refresh(self):
config = {
"watchlist_auto_refresh_enabled": True,
"watchlist_auto_refresh_minutes": 10,
}
start = mock.Mock(return_value={"created": True})
scheduler = queue_jobs.WatchlistAutoRefreshScheduler(lambda: config, start, start_worker=False)
self.assertEqual(scheduler.tick(now_monotonic=100.0), 600)
self.assertEqual(scheduler.tick(now_monotonic=699.0), 1)
self.assertEqual(scheduler.tick(now_monotonic=700.0), 600)
start.assert_called_once_with(source="scheduled")
def test_refresh_job_waits_between_series_using_configured_delay(self):
class FakeStore:
def all_show_ids(self):
return ["show-a", "show-b", "show-c"]
def refresh(self, show_id, include_thumbnail=True):
return {"title": show_id, "status": "updated"}
service = APP.WatchlistRefreshJobs(
FakeStore(),
config_getter=lambda: {"watchlist_refresh_delay_seconds": 5},
start_worker=False,
)
job = service.start()["job"]
job = service._next_pending()
with mock.patch.object(service.stop_event, "wait", return_value=False) as wait_mock:
service._run_job(job)
self.assertEqual(wait_mock.call_count, 2)
wait_mock.assert_called_with(5)
def test_tick_resets_timer_when_schedule_changes(self):
config = {
"watchlist_auto_refresh_enabled": True,
"watchlist_auto_refresh_minutes": 10,
}
start = mock.Mock(return_value={"created": True})
scheduler = queue_jobs.WatchlistAutoRefreshScheduler(lambda: config, start, start_worker=False)
self.assertEqual(scheduler.tick(now_monotonic=100.0), 600)
config["watchlist_auto_refresh_minutes"] = 5
self.assertEqual(scheduler.tick(now_monotonic=150.0), 300)
start.assert_not_called()
def test_tick_returns_none_when_disabled(self):
config = {
"watchlist_auto_refresh_enabled": False,
"watchlist_auto_refresh_minutes": 10,
}
start = mock.Mock(return_value={"created": True})
scheduler = queue_jobs.WatchlistAutoRefreshScheduler(lambda: config, start, start_worker=False)
self.assertIsNone(scheduler.tick(now_monotonic=100.0))
start.assert_not_called()
class StartupBehaviorTests(unittest.TestCase):
def test_main_does_not_eagerly_initialize_runtime(self):
@@ -1092,6 +1198,7 @@ class StartupBehaviorTests(unittest.TestCase):
current_watchlist = APP.WATCHLIST
current_queue = APP.DOWNLOAD_QUEUE
current_refresh = APP.WATCHLIST_REFRESH
current_auto_refresh = APP.WATCHLIST_AUTO_REFRESH
with mock.patch.object(APP, "shutdown_runtime", return_value=False):
with self.assertRaises(RuntimeError):
APP.reset_runtime(wait=True)
@@ -1100,20 +1207,44 @@ class StartupBehaviorTests(unittest.TestCase):
self.assertIs(APP.WATCHLIST, current_watchlist)
self.assertIs(APP.DOWNLOAD_QUEUE, current_queue)
self.assertIs(APP.WATCHLIST_REFRESH, current_refresh)
self.assertIs(APP.WATCHLIST_AUTO_REFRESH, current_auto_refresh)
finally:
APP.reset_runtime(wait=True)
APP.initialize_runtime(start_workers=False)
def test_save_runtime_config_replaces_runtime_config(self):
original = APP.CONFIG
original_scheduler = APP.WATCHLIST_AUTO_REFRESH
try:
APP.CONFIG = {"mode": "sub", "quality": "best", "download_dir": "/tmp/old"}
saved = APP.save_runtime_config({"mode": "dub", "quality": "720", "download_dir": "/tmp/new"})
APP.CONFIG = {
"mode": "sub",
"quality": "best",
"download_dir": "/tmp/old",
"watchlist_auto_refresh_enabled": False,
"watchlist_auto_refresh_minutes": 60,
"watchlist_refresh_delay_seconds": 5,
}
APP.WATCHLIST_AUTO_REFRESH = mock.Mock()
saved = APP.save_runtime_config(
{
"mode": "dub",
"quality": "720",
"download_dir": "/tmp/new",
"watchlist_auto_refresh_enabled": True,
"watchlist_auto_refresh_minutes": 15,
"watchlist_refresh_delay_seconds": 8,
}
)
self.assertEqual(saved["mode"], "dub")
self.assertEqual(APP.CONFIG["mode"], "dub")
self.assertEqual(APP.CONFIG["quality"], "720")
self.assertTrue(APP.CONFIG["watchlist_auto_refresh_enabled"])
self.assertEqual(APP.CONFIG["watchlist_auto_refresh_minutes"], 15)
self.assertEqual(APP.CONFIG["watchlist_refresh_delay_seconds"], 8)
APP.WATCHLIST_AUTO_REFRESH.notify_config_changed.assert_called_once_with()
finally:
APP.CONFIG = original
APP.WATCHLIST_AUTO_REFRESH = original_scheduler
def test_reset_runtime_clears_runtime_services(self):
APP.initialize_runtime(start_workers=False)
@@ -1123,6 +1254,7 @@ class StartupBehaviorTests(unittest.TestCase):
self.assertIsNone(APP.WATCHLIST)
self.assertIsNone(APP.DOWNLOAD_QUEUE)
self.assertIsNone(APP.WATCHLIST_REFRESH)
self.assertIsNone(APP.WATCHLIST_AUTO_REFRESH)
finally:
APP.initialize_runtime(start_workers=False)
@@ -1138,10 +1270,20 @@ class HandlerRouteTests(unittest.TestCase):
self.assertIsNone(APP.WATCHLIST.refresh_worker)
self.assertIsNone(APP.DOWNLOAD_QUEUE.worker)
self.assertIsNone(APP.WATCHLIST_REFRESH.worker)
self.assertIsNone(APP.WATCHLIST_AUTO_REFRESH.worker)
finally:
APP.reset_runtime(wait=True)
APP.initialize_runtime(start_workers=False)
def test_config_post_accepts_watchlist_schedule_fields(self):
body = b'{"download_dir":"/tmp/example","mode":"sub","quality":"best","watchlist_auto_refresh_enabled":true,"watchlist_auto_refresh_minutes":"25","watchlist_refresh_delay_seconds":"9"}'
handler = DummyHandler("/api/config", body=body, content_length=len(body))
APP.Handler.do_POST(handler)
self.assertEqual(handler.json_status, HTTPStatus.OK)
self.assertTrue(handler.json_payload["watchlist_auto_refresh_enabled"])
self.assertEqual(handler.json_payload["watchlist_auto_refresh_minutes"], 25)
self.assertEqual(handler.json_payload["watchlist_refresh_delay_seconds"], 9)
def test_body_json_rejects_oversized_request(self):
handler = DummyHandler("/api/watchlist/upload-thumbnail", body=b"{}", content_length=APP.MAX_JSON_BODY_BYTES + 1)
with self.assertRaises(APP.HttpError) as ctx:
@@ -1266,6 +1408,28 @@ class HandlerRouteTests(unittest.TestCase):
class TemplateHelperTests(unittest.TestCase):
def test_log_message_skips_queue_poll_requests(self):
handler = object.__new__(APP.Handler)
handler.path = "/api/queue?page=1&per_page=10"
handler.command = "GET"
handler.address_string = lambda: "127.0.0.1"
with mock.patch("builtins.print") as print_mock:
APP.Handler.log_message(handler, '"GET /api/queue?page=1&per_page=10 HTTP/1.1" %s -', 200)
print_mock.assert_not_called()
def test_log_message_keeps_non_poll_requests(self):
handler = object.__new__(APP.Handler)
handler.path = "/api/config"
handler.command = "GET"
handler.address_string = lambda: "127.0.0.1"
with mock.patch("builtins.print") as print_mock:
APP.Handler.log_message(handler, '"GET /api/config HTTP/1.1" %s -', 200)
print_mock.assert_called_once()
def test_page_links_marks_active_page(self):
markup = APP.render_page_links("config")
self.assertIn('class="page-link active" href="/config"', markup)