Release resources 0.3.0 with persistent Queue plugin and connector

This commit is contained in:
Dymas committed 2026-10-29 13:14:12 +01:00
1 parent 108aa9e5b1
commit 494651e118
25 files changed
+962 -25

No files matched your search

+4
View File
@@ -0,0 +1,4 @@
__pycache__/
*.py[cod]
.pytest_cache/
.venv/
+12
View File
@@ -0,0 +1,12 @@
- Always bump `VERSION` unless the only changed files are `Dockerfile`, `*.md`, or `VERSION` itself.
- Always document code and behavior changes in `CHANGELOG.md` unless the only changed files are `Dockerfile`, `*.md`, or `VERSION` itself.
- Update `README.md` whenever user-facing behavior, setup, runtime requirements, Docker usage, or provider state changes.
- Always commit and push changes with a clear commit message.
- when new wersion is published always build a new docker image and push to gitea repo
- always store newly developed plugins or themes in web-framework-resources repo, each plugin and theme will contain a json with metadata desciption.
- When needed, install project tools with `pip` or the platform package manager.
- LAN URLs and `github.com` URLs may be accessed when needed for this project.
- always cleanup after new image is pushed, keep only latest and 5 previouse image version delete older
- Keep README.md concise. Update docs/ with detailed function and behavior changes and synchronize both Gitea wikis with `python tools/sync_wikis.py` before completing a release.
- Keep framework code in web-framework and new resource source in the adjacent web-framework-resources checkout.
+7
View File
@@ -1,3 +1,10 @@
# 0.3.0 — 2026-10-29
- Add Queue 0.1.0 for framework 0.6.0+: durable FIFO job claims, encrypted payloads, renewable leases, progress, bounded job logs, cancellation, retries, and protected server-rendered views.
- Publish complete Queue source and archive with JSON metadata and a catalog checksum.
- Add a deterministic catalog builder preserving previous resource archives.
- Maintain a dedicated adjacent resource checkout, short README, detailed function wiki sources, and a two-repository wiki synchronizer.
# Starter plugins 0.2.0 — 2026-10-26
- Package Notes, Bookmarks, and Workspace status with their complete implementation, metadata, documentation, and namespaced page/widget templates in each plugin folder.
+8 -19
View File
@@ -1,26 +1,15 @@
# Web Framework Resources
Complete plugin packages and theme templates for [Web Framework](https://gitea.coreplay.eu/Dymas/web-framework).
Plugins and themes for [Web Framework](https://gitea.coreplay.eu/Dymas/web-framework).
Default catalog: `https://gitea.coreplay.eu/Dymas/web-framework-resources/raw/branch/main/catalog.json`.
Open **Config → Add resources**, search, and install a plugin or theme. Restart, then enable the plugin or select the theme. The initial packages are also bundled in the framework for offline first startup.
In **Config → Add resources**, install a resource, restart the app, then enable the plugin or select the theme.
## Folder layout
- Plugins: Notes, Bookmarks, Workspace status, and **Queue**.
- Themes: Default, Ocean, and Mint.
- Queue requires framework **0.6.0+**. It adds job progress, logs, cancellation, retries, and a Python connector for plugin workers.
Every resource has its own source folder: `plugins/<id>/` or `themes/<id>/`. Plugin v0.2.0 packages include `plugin.py`, `implementation.py`, `plugin.json`, README, and their page/widget templates. They depend on public framework APIs and the common shell, rather than on the shared starter implementation. Theme folders contain `theme.json` and any accompanying files; the bundled themes are presets using the framework design system.
Keep this checkout beside `web-framework/`. Each plugin/theme lives in its own folder with JSON metadata. Build archives and the catalog with `python tools/build_catalog.py`; preserve previous archives.
Every JSON manifest declares `id`, `name`, `description`, and `version`. Themes also declare `mode` (`dark`, `light`, `system`), `accent` (`violet`, `blue`, `green`), and `density` (`comfortable`, `compact`). Versioned ZIP archives contain the entire resource with files at the archive root. Catalog entries record type, metadata, relative archive path and SHA-256.
Detailed functions and publishing instructions: [Gitea wiki](https://gitea.coreplay.eu/Dymas/web-framework-resources/wiki) · [Local docs](docs/Home.md) · [Queue API](docs/queues.md).
## Persistence and updates
Framework Compose mounts host folders `./plugins` and `./themes` outside the container, with one subfolder per resource. Image rebuilds and restarts preserve existing files. Missing defaults are copied locally from the image; no resource ZIPs are downloaded at startup.
The app checks catalogs in the background on startup and through **Check for updates** on the configuration pages. Checks only compare metadata: they never replace code or download archives. Local versions use PEP 440 comparison; source provenance protects against same-ID packages from another publisher. Operators can disable startup checks with `[resources].check_updates_on_startup = false`.
To update, stop the workspace, back up the resource folder, verify a newer archive against its catalog SHA-256, extract it into that folder, then restart. Keep the mounted resource folders and control-data volume. Saved plugin content and configuration remain in the control store.
## Publish resources
Store newly developed plugins and themes here with all required helpers, templates, and assets in their own subfolder. Increment the resource version, build its complete ZIP, and update `catalog.json` with schema version 1. Commit source, archive, and catalog together. Keep old versioned archives for existing installations. Publish catalogs and archives for unauthenticated read access; never include secrets.
The initial six resources can be regenerated with `PYTHONPATH=. python tools/prepare_resources.py /path/to/web-framework-resources` from the framework repository. The exporter regenerates bundled source directories and their catalog; merge any custom entries before publishing. Only configure trusted repositories: plugin code runs with the application user's privileges after restart.
Update docs with code changes and run `python tools/sync_wikis.py` to publish both repositories' wikis.
+1
View File
@@ -0,0 +1 @@
0.3.0
Binary file not shown.
+15 -6
View File
@@ -1,6 +1,15 @@
{
"schema_version": 1,
"resources": [
{
"id": "bookmarks",
"name": "Bookmarks",
"description": "A shared collection of useful links.",
"version": "0.2.0",
"type": "plugin",
"archive": "archives/plugin-bookmarks-0.2.0.zip",
"sha256": "57d8f8c4407a34722b25c0777844b043485d528eaefaa80ccf1449f2c20385ba"
},
{
"id": "notes",
"name": "Notes",
@@ -11,13 +20,13 @@
"sha256": "5e2e37c1b81aea62c58c79c8f0280a82e2208c08a6066223a430ed443192f507"
},
{
"id": "bookmarks",
"name": "Bookmarks",
"description": "A shared collection of useful links.",
"version": "0.2.0",
"id": "queues",
"name": "Queue",
"description": "Persistent shared job queue with progress, logs, and an internal plugin connector.",
"version": "0.1.0",
"type": "plugin",
"archive": "archives/plugin-bookmarks-0.2.0.zip",
"sha256": "57d8f8c4407a34722b25c0777844b043485d528eaefaa80ccf1449f2c20385ba"
"archive": "archives/plugin-queues-0.1.0.zip",
"sha256": "975ed9e778d4355231166ce20e7ec6770592517a0fba7b05d39a59a5df868f95"
},
{
"id": "status",
+10
View File
@@ -0,0 +1,10 @@
# Web Framework Resources wiki
This repository owns installable plugin and theme source, JSON metadata, versioned ZIP archives, and the catalog consumed by Web Framework.
- [Resource installation and publishing](resources.md)
- [Queue connector functions and page routes](queues.md)
- [Starter plugin and theme functions](starters.md)
- [Documentation and releases](releases.md)
Every newly developed resource belongs here. Keep `README.md` concise; maintain these detailed docs with code changes and run `python tools/sync_wikis.py` before completing publication.
+70
View File
@@ -0,0 +1,70 @@
# Queues plugin function reference
Requires Web Framework 0.6.0 or later. The installable resource ID is `queues`, displayed as **Queue**. Install through the catalog, restart to discover the package, then enable in Config → Plugins. It is optional and is not bundled into the framework image. Producers with a mandatory queue dependency declare `Plugin(..., requires=('queues',))`.
## Connector access
```python
from web_framework.plugins import get_plugins
queue = get_plugins().service('queues', 'queue')
job_id = queue.enqueue('reports', 'Create report', {'month': '2026-10'}, total=10)
job = queue.claim('reports')
if job:
queue.progress(job['id'], job['token'], 5, message='Half complete')
queue.log(job['id'], job['token'], 'Writing output')
queue.complete(job['id'], job['token'])
```
Resolve the service during a request or `with app.app_context():`. The manager lazily creates a single `Queue(app)` per app. Once acquired, it can be used by plugin-owned workers outside a request. Every operation rechecks the provider's enabled state. The connector is a trusted internal Python API; it is not an unauthenticated network ingestion endpoint. Producers must validate their own browser input and permissions.
## Public methods
| Function | Description |
| --- | --- |
| `Queue(app)` | Creates queue job/log tables and indexes in the app's persistent control SQLite database. Stores the owning app/state for later background use. |
| `enqueue(source, title, payload=None, *, total=100)` | Validates a source slug, 1–160 character title, positive integer total up to 2147483647, and JSON payload up to 64 KiB. Encrypts payload, inserts a queued job and initial log atomically, and returns its random ID. Non-finite JSON numbers are rejected. |
| `claim(source, *, lease_seconds=300)` | Atomically claims the oldest queued or expired-running job for the source. Returns metadata, decrypted `payload`, and private `token`, or `None`. Sets running state, increments attempts, logs start/recovery, and sets a 10–86400 second lease. |
| `progress(job_id, token, completed, *, total=None, message=None, lease_seconds=300)` | Verifies an unexpired active claim, validates whole-number `0 <= completed <= total`, updates counts and timestamp, renews lease, and optionally appends a log. Invalid input rolls back the complete update. |
| `log(job_id, token, message, *, level='info')` | Verifies a live claim and appends a 1–4000 character message at info/warning/error level. Retains 500 entries per job. Does not renew the lease. |
| `complete(job_id, token, message='Job completed.')` | Verifies the claim, marks completed, sets completed count equal to total (100%), revokes token/lease, and logs success atomically. |
| `fail(job_id, token, message='Job failed.')` | Verifies the claim, marks failed, preserves partial progress, revokes token/lease, and logs an error. |
| `cancel(job_id)` | Accepts queued/running jobs only; marks cancelled and revokes claim, adding a warning. Cooperatively prevents further worker updates. |
| `retry(job_id)` | Accepts failed/cancelled jobs only; resets progress to zero, queues the same job, and preserves attempts, payload, and log history. The next claim gets a new token. |
| `jobs(*, status=None, source=None, limit=100, offset=0)` | Lists newest jobs first, optionally filtered. Limit 1–200, nonnegative offset. Returns metadata only, including computed progress percentage. |
| `get(job_id)` | Returns one job's metadata or raises `ValueError` for a missing job. Omits payload and claim token. |
| `logs(job_id)` | Returns the retained log entries chronologically as stamp/level/message dictionaries; an unknown job has no entries. |
Metadata contains `id`, `source`, `title`, `status`, `completed`, `total`, `created`, `updated`, `attempts`, and percentage `progress`. All timestamps are Unix seconds; pages render UTC. Status values: queued, running, completed, failed, cancelled. Mutations are serialized through SQLite `BEGIN IMMEDIATE`; SQL parameters bind all user values. Worker methods require the current private claim token and reject expired/revoked/stale tokens with `ValueError`.
## Worker ownership and recovery
Queues persists and coordinates jobs; producer plugins schedule workers and execute their own handlers. A payload alone does not execute code. Use short bounded work units, renew through `progress()` before lease expiry, and make handlers idempotent. Claiming expired work provides at least once delivery and can repeat external side effects. A stale worker must stop when an update rejects its token. Cancellation cannot interrupt arbitrary Python or undo already completed external work. Source groups are conventions for trusted plugin cooperation, not security boundaries.
Jobs remain running after a lease expires until another claim for that source recovers them or an operator cancels them. Failed/cancelled jobs require explicit retry. No automatic retries or worker threads are created by installation. Different app processes can coordinate through the same SQLite file; independent server replicas with separate files cannot.
Payloads are encrypted with the framework master key. Metadata and logs are workspace-visible and must not include secrets. Up to 500 recent log entries per job are retained. Jobs are not automatically purged. Back up the entire control-data directory and key. Queues survives restarts, disable/enable cycles, plugin reinstalls, and image updates when the data volume is retained.
## Page and HTTP functions
`connector()` resolves the enabled service; `selection()` parses status/source/page filters and rejects invalid status or pagination with HTTP 400.
| Route and handler | Behavior |
| --- | --- |
| `GET /plugins/queues/` — `index()` | Renders 50 jobs per page, status/source filters, accessible progress bars, job links and pagination. |
| `GET /plugins/queues/api/jobs` — `jobs_api()` | Same filtered/paginated metadata as JSON. Requires the standard signed-in user and enabled-plugin checks. No payloads or worker tokens. |
| `GET /plugins/queues/jobs/<id>` — `detail()` | Shows job metadata, progress, attempt count and latest 500 chronological log entries; unknown IDs return 404. |
| `POST /plugins/queues/jobs/<id>/cancel` — `action()` | Requires `content.write` and valid CSRF; cancels active work, audits the user action, flashes the result, and redirects to detail. |
| `POST /plugins/queues/jobs/<id>/retry` — `action()` | Same protections; retries failed/cancelled jobs. Other actions return 404. Invalid transitions display an error. |
Signed-in users with `workspace.view` and `plugins.use` can read the queue; viewers cannot mutate it. Disabled-plugin routes return 404. Blueprint CSS/JS use the same activation/authentication gate. Templates escape titles and messages. Active panels refresh every five seconds while the tab is visible and keyboard focus is outside the panel, retaining scroll position and normal navigation. Authentication redirects stop background refresh. JavaScript is optional.
## Internal helpers
`bounded_text(value, label, maximum)` validates strings. `_db()` enforces activation and opens a transactional state connection. `_public(row, payload=False)` excludes worker capabilities and optionally decrypts payload for claim. `_event()` validates/records a log and bounds retention. `_owned()` verifies job state, token and lease. `_lease()` and `_counts()` validate numeric constraints. `_finish()` implements common terminal worker transitions. These helpers are implementation details; other plugins use the public methods above.
Run integration tests from the resource checkout:
```sh
PYTHONPATH=../web-framework ../web-framework/.venv/bin/python -m pytest tests plugins/queues/tests -q
```
+9
View File
@@ -0,0 +1,9 @@
# Resource releases and wiki maintenance
Keep this Git checkout beside `web-framework/`. All new plugins/themes belong here, each in a dedicated folder with JSON metadata (`id`, `name`, `description`, `version`). Independently version changed resources, bump repository `VERSION`, and document behavior in `CHANGELOG.md`. Update `docs/` function references whenever implementations change; keep README files short.
`tools/build_catalog.py` exports `build(root)`: discovers every plugin/theme JSON manifest, builds a deterministic ZIP containing implementation, templates, assets and resource README at archive root, computes SHA-256, and writes schema-version-1 catalog entries. Unchanged packages retain their exact published archive bytes and checksums. It excludes tests, bytecode and cache folders; retains previous versioned archives. Run `python tools/build_catalog.py` from the checkout. Commit source, metadata, archives, docs, catalog and version together, then push.
Run `python tools/sync_wikis.py` to synchronize both repositories' detailed `docs/*.md` pages to their corresponding Gitea wikis. The script accepts optional repository paths. It uses `GITEA_TOKEN` or Git's configured stored credential, enables missing wikis, initializes Home, clones each wiki into temporary storage, copies maintained pages, commits changed content, and pushes without force. It converts local Markdown page links to wiki slugs and preserves unrelated wiki pages and cleans temporary clones. Credentials are never printed or written to Git remotes.
The framework's companion release builds and pushes a new Docker version and `latest`, then retains the current and five previous numeric framework image versions. Resource ZIP archives remain available for installed catalog versions. See the [framework release guide](https://gitea.coreplay.eu/Dymas/web-framework/wiki/releases) for the complete release procedure.
+26
View File
@@ -0,0 +1,26 @@
# Web Framework Resources
Complete plugin packages and theme templates for [Web Framework](https://gitea.coreplay.eu/Dymas/web-framework).
Default catalog: `https://gitea.coreplay.eu/Dymas/web-framework-resources/raw/branch/main/catalog.json`.
Open **Config → Add resources**, search, and install a plugin or theme. Restart, then enable the plugin or select the theme. The six starter packages are also bundled in the framework for offline first startup. Queue is available only through this resource repository and requires framework 0.6.0+.
## Folder layout
Every resource has its own source folder: `plugins/<id>/` or `themes/<id>/`. Plugin v0.2.0 packages include `plugin.py`, `implementation.py`, `plugin.json`, README, and their page/widget templates. They depend on public framework APIs and the common shell, rather than on the shared starter implementation. Theme folders contain `theme.json` and any accompanying files; the bundled themes are presets using the framework design system.
Every JSON manifest declares `id`, `name`, `description`, and `version`. Themes also declare `mode` (`dark`, `light`, `system`), `accent` (`violet`, `blue`, `green`), and `density` (`comfortable`, `compact`). Versioned ZIP archives contain the entire resource with files at the archive root. Catalog entries record type, metadata, relative archive path and SHA-256.
## Persistence and updates
Framework Compose mounts host folders `./plugins` and `./themes` outside the container, with one subfolder per resource. Image rebuilds and restarts preserve existing files. Missing defaults are copied locally from the image; no resource ZIPs are downloaded at startup.
The app checks catalogs in the background on startup and through **Check for updates** on the configuration pages. Checks only compare metadata: they never replace code or download archives. Local versions use PEP 440 comparison; source provenance protects against same-ID packages from another publisher. Operators can disable startup checks with `[resources].check_updates_on_startup = false`.
To update, stop the workspace, back up the resource folder, verify a newer archive against its catalog SHA-256, extract it into that folder, then restart. Keep the mounted resource folders and control-data volume. Saved plugin content and configuration remain in the control store.
## Publish resources
Store newly developed plugins and themes here with all required helpers, templates, and assets in their own subfolder. Increment the resource version, build its complete ZIP, and update `catalog.json` with schema version 1. Commit source, archive, and catalog together. Keep old versioned archives for existing installations. Publish catalogs and archives for unauthenticated read access; never include secrets.
The initial six starter resources can be regenerated with `PYTHONPATH=. python tools/prepare_resources.py /path/to/web-framework-resources` from the framework repository. The exporter refreshes bundled source directories and retains independently developed resources in the catalog. `python tools/build_catalog.py` builds every source resource without replacing bundled source. Only configure trusted repositories: plugin code runs with the application user's privileges after restart.
+13
View File
@@ -0,0 +1,13 @@
# Starter plugin and theme functions
Notes, Bookmarks, and Workspace status are complete independent v0.2.0 plugin packages. Each exports `plugin` from `plugin.py` and includes namespaced templates plus implementation helpers.
`make_collection(plugin_id, name, kind='notes')` constructs a shared collection page and summary widget. Its `index()` lists persistent items on GET. POST requires `content.write` and the framework CSRF token; it validates titles/content and writes or deletes a collection item, records an audit event, flashes the result, and redirects. Notes accept text content; bookmarks additionally require a safe HTTP(S) or root-relative link. The `intro` setting controls the page introduction. The summary widget returns the newest five items.
`make_notes(plugin_id='notes', name='Notes')` and `make_bookmarks(plugin_id='bookmarks', name='Bookmarks')` select the corresponding collection configuration. `make_status(plugin_id='status', name='Workspace status')` builds the status page and summary widget; it reports framework version, uptime, enabled plugin count and configured connection count without resolving database passwords or contacting optional services.
Every plugin's blueprint is registered under `/plugins/<id>/`, protected by authentication, permission, and enabled-state checks. Templates inherit the framework shell. Plugin configuration and collection data persist in the framework control store; restarting the app discovers newly installed Python packages.
Default, Ocean, and Mint are declarative theme presets. Each `theme.json` declares ID, name, description, version, mode, accent and density. Selection uses the framework's `Themes.activate()`; effective appearance uses `Themes.appearance()`. They introduce no executable code or separate server functions.
All resources have JSON metadata and reproducible versioned ZIP archives. See [publishing](resources.md) for package layout, checksum validation and update behavior.
+7
View File
@@ -0,0 +1,7 @@
# Queue
Persistent job queue with progress bars, per-job logs, cancellation, retries, and an internal connector for other plugins. Requires Web Framework 0.6.0+.
Install from Config → Add resources, restart, then enable **Queue**. Producer plugins own their workers and submit/report jobs through `get_plugins().service('queues', 'queue')`.
[Full connector and route reference](https://gitea.coreplay.eu/Dymas/web-framework-resources/wiki/queues).
+190
View File
@@ -0,0 +1,190 @@
"""Durable FIFO queue. Trusted plugin workers own execution and renew leases."""
from contextlib import contextmanager
import json
import secrets
import time
from web_framework.config import IDENTIFIER
STATES = ('queued', 'running', 'completed', 'failed', 'cancelled')
def bounded_text(value, label, maximum):
if not isinstance(value, str) or not value.strip() or len(value) > maximum:
raise ValueError(f'{label} must contain 1–{maximum} characters.')
return value.strip()
class Queue:
"""Acquire via get_plugins().service('queues', 'queue') in an app context.
Each mutation uses the control store's SQLite transaction. Claim tokens are
private capabilities: a stale worker cannot update a reclaimed/cancelled job.
"""
def __init__(self, app):
self.app = app
self.state = app.extensions['state']
with self.state.db() as db:
db.executescript('''
CREATE TABLE IF NOT EXISTS queue_jobs (
sequence INTEGER PRIMARY KEY AUTOINCREMENT,
id TEXT UNIQUE NOT NULL, source TEXT NOT NULL, title TEXT NOT NULL,
payload BLOB NOT NULL, status TEXT NOT NULL DEFAULT 'queued',
completed INTEGER NOT NULL DEFAULT 0, total INTEGER NOT NULL DEFAULT 100,
created REAL NOT NULL, updated REAL NOT NULL,
token TEXT, lease_until REAL, attempts INTEGER NOT NULL DEFAULT 0);
CREATE INDEX IF NOT EXISTS queue_ready ON queue_jobs(status, source, sequence);
CREATE TABLE IF NOT EXISTS queue_logs (
sequence INTEGER PRIMARY KEY AUTOINCREMENT,
job_id TEXT NOT NULL REFERENCES queue_jobs(id) ON DELETE CASCADE,
stamp REAL NOT NULL, level TEXT NOT NULL, message TEXT NOT NULL);
CREATE INDEX IF NOT EXISTS queue_log_lookup ON queue_logs(job_id, sequence);
''')
@contextmanager
def _db(self):
if 'queues' not in self.app.extensions['plugins'].enabled():
raise ValueError('The Queues plugin is disabled.')
with self.state.db() as db:
yield db
def _public(self, row, payload=False):
item = {key: row[key] for key in ('id', 'source', 'title', 'status', 'completed', 'total', 'created', 'updated', 'attempts')}
item['progress'] = round(100 * item['completed'] / item['total'], 1)
if payload:
item['payload'] = json.loads(self.state.cipher.decrypt(row['payload']))
return item
def _event(self, db, job_id, message, level='info'):
message = bounded_text(message, 'Log message', 4000)
if level not in ('info', 'warning', 'error'):
raise ValueError('Choose info, warning, or error.')
db.execute('INSERT INTO queue_logs(job_id,stamp,level,message) VALUES(?,?,?,?)', (job_id, time.time(), level, message))
db.execute('DELETE FROM queue_logs WHERE job_id=? AND sequence NOT IN (SELECT sequence FROM queue_logs WHERE job_id=? ORDER BY sequence DESC LIMIT 500)', (job_id, job_id))
def _owned(self, db, job_id, token):
row = db.execute('SELECT * FROM queue_jobs WHERE id=?', (job_id,)).fetchone()
if not row:
raise ValueError('Job not found.')
if row['status'] != 'running' or not isinstance(token, str) or not row['token'] or not secrets.compare_digest(row['token'], token) or row['lease_until'] <= time.time():
raise ValueError('Job claim expired or is no longer active.')
return row
def enqueue(self, source, title, payload=None, *, total=100):
"""Submit JSON data under the producing plugin's slug; return its job ID."""
if not isinstance(source, str) or not IDENTIFIER.fullmatch(source):
raise ValueError('Source must be a plugin slug.')
title = bounded_text(title, 'Title', 160)
self._counts(0, total)
raw = json.dumps(payload, allow_nan=False).encode()
if len(raw) > 65536:
raise ValueError('Payload exceeds 64 KiB.')
job_id, now = secrets.token_hex(16), time.time()
with self._db() as db:
db.execute('INSERT INTO queue_jobs(id,source,title,payload,total,created,updated) VALUES(?,?,?,?,?,?,?)',
(job_id, source, title, self.state.cipher.encrypt(raw), total, now, now))
self._event(db, job_id, 'Job queued.')
return job_id
def claim(self, source, *, lease_seconds=300):
"""Atomically claim the oldest queued/expired job for a source, or None.
Returned token must accompany worker updates. Expired jobs are delivered
again; handlers must be idempotent because execution is at least once.
"""
self._lease(lease_seconds)
now = time.time()
with self._db() as db:
row = db.execute("SELECT * FROM queue_jobs WHERE source=? AND (status='queued' OR (status='running' AND lease_until<=?)) ORDER BY sequence LIMIT 1", (source, now)).fetchone()
if row is None:
return None
token = secrets.token_hex(32)
db.execute("UPDATE queue_jobs SET status='running',token=?,lease_until=?,updated=?,attempts=attempts+1 WHERE id=?", (token, now + lease_seconds, now, row['id']))
self._event(db, row['id'], 'Expired claim recovered.' if row['status'] == 'running' else 'Job started.')
fresh = db.execute('SELECT * FROM queue_jobs WHERE id=?', (row['id'],)).fetchone()
return {**self._public(fresh, payload=True), 'token': token}
@staticmethod
def _lease(seconds):
if type(seconds) is not int or not 10 <= seconds <= 86400:
raise ValueError('Lease must be 10–86400 whole seconds.')
@staticmethod
def _counts(completed, total):
if type(completed) is not int or type(total) is not int or not 1 <= total <= 2147483647 or not 0 <= completed <= total:
raise ValueError('Progress must be whole numbers with 0 <= completed <= total and total > 0.')
def progress(self, job_id, token, completed, *, total=None, message=None, lease_seconds=300):
"""Report work and renew the lease; optional message appears in job logs."""
self._lease(lease_seconds)
with self._db() as db:
row = self._owned(db, job_id, token)
total = row['total'] if total is None else total
self._counts(completed, total)
now = time.time()
db.execute('UPDATE queue_jobs SET completed=?,total=?,updated=?,lease_until=? WHERE id=?', (completed, total, now, now + lease_seconds, job_id))
if message is not None:
self._event(db, job_id, message)
def log(self, job_id, token, message, *, level='info'):
"""Append a bounded log entry; use progress() to renew the claim."""
with self._db() as db:
self._owned(db, job_id, token)
self._event(db, job_id, message, level)
def _finish(self, job_id, token, status, message):
with self._db() as db:
self._owned(db, job_id, token)
db.execute('UPDATE queue_jobs SET status=?,completed=CASE WHEN ?=\'completed\' THEN total ELSE completed END,updated=?,token=NULL,lease_until=NULL WHERE id=?', (status, status, time.time(), job_id))
self._event(db, job_id, message, 'error' if status == 'failed' else 'info')
def complete(self, job_id, token, message='Job completed.'):
"""Mark success and set progress to 100%."""
self._finish(job_id, token, 'completed', message)
def fail(self, job_id, token, message='Job failed.'):
"""Mark failure, preserving the last reported progress."""
self._finish(job_id, token, 'failed', message)
def cancel(self, job_id):
"""Cancel queued/running work and invalidate its claim cooperatively."""
with self._db() as db:
changed = db.execute("UPDATE queue_jobs SET status='cancelled',token=NULL,lease_until=NULL,updated=? WHERE id=? AND status IN ('queued','running')", (time.time(), job_id)).rowcount
if not changed:
raise ValueError('Only queued or running jobs can be cancelled.')
self._event(db, job_id, 'Job cancelled; worker claim revoked.', 'warning')
def retry(self, job_id):
"""Requeue failed/cancelled jobs with fresh progress; preserve history."""
with self._db() as db:
changed = db.execute("UPDATE queue_jobs SET status='queued',completed=0,updated=? WHERE id=? AND status IN ('failed','cancelled')", (time.time(), job_id)).rowcount
if not changed:
raise ValueError('Only failed or cancelled jobs can be retried.')
self._event(db, job_id, 'Job queued for retry.')
def jobs(self, *, status=None, source=None, limit=100, offset=0):
"""Return paginated metadata, newest first; never expose payloads/tokens."""
if status is not None and status not in STATES:
raise ValueError('Unknown queue status.')
if type(limit) is not int or not 1 <= limit <= 200 or type(offset) is not int or offset < 0:
raise ValueError('Invalid pagination.')
clauses, parameters = [], []
for key, value in (('status', status), ('source', source)):
if value is not None:
clauses.append(key + '=?')
parameters.append(value)
where = ' WHERE ' + ' AND '.join(clauses) if clauses else ''
with self._db() as db:
return [self._public(row) for row in db.execute('SELECT * FROM queue_jobs' + where + ' ORDER BY sequence DESC LIMIT ? OFFSET ?', (*parameters, limit, offset))]
def get(self, job_id):
"""Read metadata; raises ValueError for a missing job."""
with self._db() as db:
row = db.execute('SELECT * FROM queue_jobs WHERE id=?', (job_id,)).fetchone()
if row is None:
raise ValueError('Job not found.')
return self._public(row)
def logs(self, job_id):
"""Return the latest 500 entries in chronological order."""
with self._db() as db:
return [dict(row) for row in db.execute('SELECT stamp,level,message FROM queue_logs WHERE job_id=? ORDER BY sequence', (job_id,))]
+6
View File
@@ -0,0 +1,6 @@
{
"id": "queues",
"name": "Queue",
"description": "Persistent shared job queue with progress, logs, and an internal plugin connector.",
"version": "0.1.0"
}
+68
View File
@@ -0,0 +1,68 @@
"""Queue page and service provider; workers use the internal Python connector."""
from flask import Blueprint, abort, current_app, flash, g, jsonify, redirect, render_template, request, url_for
from web_framework.auth import can
from web_framework.plugins import Plugin, get_plugins
from .connector import Queue, STATES
bp = Blueprint('plugin_queues', __name__, template_folder='templates', static_folder='static')
def connector():
return get_plugins().service('queues', 'queue')
def selection():
status = request.args.get('status') or None
source = request.args.get('source') or None
try:
page = int(request.args.get('page', '1'))
if page < 1 or page > 1000000:
raise ValueError
jobs = connector().jobs(status=status, source=source, limit=50, offset=(page - 1) * 50)
except ValueError as error:
abort(400, description=str(error))
return status, source, page, jobs
@bp.get('/')
def index():
status, source, page, jobs = selection()
return render_template('queues/index.html', title='Queue', jobs=jobs, statuses=STATES, selected_status=status, source=source, page_number=page)
@bp.get('/api/jobs')
def jobs_api():
_, _, page, jobs = selection()
return jsonify(jobs=jobs, page=page)
@bp.get('/jobs/<job_id>')
def detail(job_id):
try:
job = connector().get(job_id)
except ValueError:
abort(404)
return render_template('queues/detail.html', title=job['title'], job=job, logs=connector().logs(job_id))
@bp.post('/jobs/<job_id>/<action>')
def action(job_id, action):
if not can('content.write'):
abort(403)
if action not in ('cancel', 'retry'):
abort(404)
try:
getattr(connector(), action)(job_id)
current_app.extensions['state'].audit(g.user['username'], 'queues.' + action, job_id)
flash('Job ' + ('cancelled.' if action == 'cancel' else 'queued for retry.'), 'success')
except ValueError as error:
flash(str(error), 'error')
return redirect(url_for('plugin_queues.detail', job_id=job_id))
plugin = Plugin(
'queues', 'Queue', version='0.1.0',
description='Persistent shared job queue with progress, logs, and an internal plugin connector.',
blueprint=bp, endpoint='plugin_queues.index',
services={'queue': lambda: Queue(current_app._get_current_object())},
)
+23
View File
@@ -0,0 +1,23 @@
.queue-panel { margin-bottom: 24px; }
.queue-filters { display: flex; align-items: end; flex-wrap: wrap; gap: 12px; }
.queue-filters label { display: grid; gap: 8px; flex: 1 1 180px; font-size: 12px; }
.queue-refresh { margin: 16px 0; font-size: 12px; }
.queue-panel progress { width: 100%; min-width: 100px; height: 12px; accent-color: var(--accent); }
.queue-panel small { display: block; margin-top: 6px; color: var(--muted); }
.queue-progress { margin: 18px 0 10px; }
.queue-actions { margin: 20px 0; }
.queue-pagination { display: flex; align-items: center; gap: 16px; margin-top: 20px; }
.queue-logs { list-style: none; padding: 0; margin: 18px 0; max-height: 50vh; overflow: auto; }
.queue-logs li { display: flex; flex-wrap: wrap; gap: 12px; padding: 12px 0; border-bottom: 1px solid var(--line); white-space: pre-wrap; overflow-wrap: anywhere; }
.queue-logs time { color: var(--muted); font-size: 12px; }
.queue-panel th, .queue-panel td { white-space: nowrap; }
.queue-panel td:first-child { white-space: normal; min-width: 160px; }
@media (max-width: 640px) {
.queue-panel thead { display: none; }
.queue-panel tbody { display: grid; gap: 14px; }
.queue-panel tr { display: grid; gap: 12px; padding: 16px 0; border-top: 1px solid var(--line); }
.queue-panel td { display: grid; grid-template-columns: 65px minmax(0, 1fr); gap: 12px; padding: 0; border: 0; white-space: normal; font-size: 12px; }
.queue-panel td::before { content: attr(data-label); color: var(--muted); }
.queue-panel .badge { width: fit-content; }
}
+15
View File
@@ -0,0 +1,15 @@
// Replace the queue panel only; retain keyboard focus and scroll position.
const refreshQueue = async () => {
const panel = document.querySelector('[data-queue-refresh]');
if (!panel || document.hidden || panel.contains(document.activeElement)) return;
try {
const response = await fetch(window.location.href, {redirect: 'error'});
if (!response.ok) return;
const documentNext = new DOMParser().parseFromString(await response.text(), 'text/html');
const nextPanel = documentNext.querySelector('.queue-panel');
if (nextPanel) panel.replaceWith(nextPanel);
} catch (_) {
// Manual refresh remains available when a connection or session expires.
}
};
setInterval(refreshQueue, 5000);
@@ -0,0 +1,26 @@
{% extends 'base.html' %}
{% block head %}
<link rel="stylesheet" href="{{ url_for('plugin_queues.static', filename='queues.css') }}">
<script src="{{ url_for('plugin_queues.static', filename='queues.js') }}" defer></script>
{% endblock %}
{% block content %}
<div class="page-heading"><div><p class="eyebrow">QUEUE JOB</p><h1>{{ job.title }}</h1><p class="muted">{{ job.source }}</p></div><a class="button" href="{{ url_for('plugin_queues.index') }}">Back to Queue</a></div>
<section class="panel queue-panel" {% if job.status in ('queued', 'running') %}data-queue-refresh{% endif %}>
<p class="muted"><span class="badge">{{ job.status }}</span> · Attempt {{ job.attempts }}</p>
<h2>Progress</h2>
<progress class="queue-progress" value="{{ job.completed }}" max="{{ job.total }}" aria-label="Job progress">{{ job.progress }}%</progress>
<p>{{ job.progress }}% · {{ job.completed }} of {{ job.total }} completed</p>
<p class="muted">Created {{ job.created|timestamp }} · Updated {{ job.updated|timestamp }}</p>
{% if can('content.write') and job.status != 'completed' %}
<form class="queue-actions" method="post" action="{{ url_for('plugin_queues.action', job_id=job.id, action='cancel' if job.status in ('queued','running') else 'retry') }}">
<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button class="button" type="submit">{{ 'Cancel job' if job.status in ('queued','running') else 'Retry job' }}</button>
</form>
{% endif %}
<h2>Job log</h2>
<ol class="queue-logs" aria-label="Job log">
{% for log in logs %}<li><time>{{ log.stamp|timestamp }}</time> <span class="badge">{{ log.level }}</span> <span>{{ log.message }}</span></li>{% else %}<li>No log entries yet.</li>{% endfor %}
</ol>
<p class="muted">Showing the latest 500 entries. <a href="{{ request.url }}">Refresh now</a></p>
</section>
{% endblock %}
@@ -0,0 +1,32 @@
{% extends 'base.html' %}
{% block head %}
<link rel="stylesheet" href="{{ url_for('plugin_queues.static', filename='queues.css') }}">
<script src="{{ url_for('plugin_queues.static', filename='queues.js') }}" defer></script>
{% endblock %}
{% block content %}
<div class="page-heading"><div><p class="eyebrow">PLUGIN JOBS</p><h1>Queue</h1><p class="muted">Follow queued work, progress, and job logs across your plugins.</p></div></div>
<section class="panel queue-panel" data-queue-refresh>
<form method="get" class="queue-filters">
<label>Status<select name="status"><option value="">All statuses</option>{% for status in statuses %}<option value="{{ status }}" {% if selected_status == status %}selected{% endif %}>{{ status|title }}</option>{% endfor %}</select></label>
<label>Plugin<input name="source" value="{{ source or '' }}" placeholder="All plugins"></label>
<button class="button" type="submit">Filter</button>
<a class="button" href="{{ url_for('plugin_queues.index') }}">Reset</a>
</form>
<p class="muted queue-refresh" role="status">Updates every 5 seconds. <a href="{{ request.url }}">Refresh now</a></p>
{% if jobs %}
<div class="table-scroll"><table><thead><tr><th>Job</th><th>Plugin</th><th>Status</th><th>Progress</th><th>Updated</th></tr></thead><tbody>
{% for job in jobs %}<tr>
<td data-label="Job"><a href="{{ url_for('plugin_queues.detail', job_id=job.id) }}">{{ job.title }}</a></td>
<td data-label="Plugin">{{ job.source }}</td><td data-label="Status"><span class="badge">{{ job.status }}</span></td>
<td data-label="Progress"><div><progress max="{{ job.total }}" value="{{ job.completed }}" aria-label="{{ job.title }} progress">{{ job.progress }}%</progress><small>{{ job.progress }}% · {{ job.completed }}/{{ job.total }}</small></div></td>
<td data-label="Updated">{{ job.updated|timestamp }}</td>
</tr>{% endfor %}
</tbody></table></div>
{% else %}<div class="empty-state"><h2>No jobs here yet</h2><p>Jobs submitted by connected plugins will appear here. Try clearing your filters.</p></div>{% endif %}
<nav class="queue-pagination" aria-label="Queue pages">
{% if page_number > 1 %}<a class="button" href="{{ url_for('plugin_queues.index', page=page_number-1, status=selected_status or '', source=source or '') }}">Previous</a>{% endif %}
<span class="muted">Page {{ page_number }}</span>
{% if jobs|length == 50 %}<a class="button" href="{{ url_for('plugin_queues.index', page=page_number+1, status=selected_status or '', source=source or '') }}">Next</a>{% endif %}
</nav>
</section>
{% endblock %}
+86
View File
@@ -0,0 +1,86 @@
"""Exercise Queue in Chromium, including polling, subpath, and mobile layout."""
import logging
import os
from pathlib import Path
import tempfile
import threading
from playwright.sync_api import expect, sync_playwright
from werkzeug.serving import make_server
from web_framework import create_app
def main():
logging.getLogger('werkzeug').setLevel(logging.ERROR)
with tempfile.TemporaryDirectory(prefix='queues-browser-') as temporary:
root = Path(temporary)
server = make_server('127.0.0.1', 0, lambda environ, start_response: [])
base = f'http://127.0.0.1:{server.server_port}/projectx'
app = create_app(overrides={
'app': {'base_url': base},
'runtime': {'data_dir': str(root / 'state'), 'plugin_dir': str(Path(__file__).resolve().parents[2]), 'theme_dir': str(root / 'themes')},
'plugins': {'enabled': ['queues']},
'resources': {'check_updates_on_startup': False},
'logging': {'console': False},
})
server.app = app
worker = threading.Thread(target=server.serve_forever, daemon=True)
worker.start()
try:
queue = app.extensions['plugins'].service('queues', 'queue')
job_id = queue.enqueue('reports', 'Monthly report', total=10)
with sync_playwright() as playwright:
options = {'headless': True}
if os.environ.get('PLAYWRIGHT_CHROMIUM_EXECUTABLE'):
options['executable_path'] = os.environ['PLAYWRIGHT_CHROMIUM_EXECUTABLE']
browser = playwright.chromium.launch(**options)
page = browser.new_page()
errors = []
page.on('pageerror', lambda error: errors.append(str(error)))
page.on('console', lambda message: errors.append(message.text) if message.type == 'error' else None)
page.goto(base + '/plugins/queues/')
page.get_by_label('Username', exact=True).fill('admin')
password = (root / 'state/initial-admin-password').read_text().strip()
page.get_by_label('Password', exact=True).fill(password)
page.get_by_role('button', name='Sign in', exact=True).click()
page.get_by_label('Current password', exact=True).fill(password)
page.get_by_label('New password', exact=True).fill('Queue-browser-password!42')
page.get_by_label('Confirm new password', exact=True).fill('Queue-browser-password!42')
page.get_by_role('button', name='Change password', exact=True).click()
page.goto(base + '/plugins/queues/')
expect(page.get_by_role('link', name='Monthly report')).to_be_visible()
page.get_by_role('link', name='Monthly report').click()
expect(page.get_by_text('Job queued.', exact=True)).to_be_visible()
job = queue.claim('reports')
queue.progress(job_id, job['token'], 5, message='Halfway through')
expect(page.get_by_text('Halfway through', exact=True)).to_be_visible(timeout=10000)
expect(page.get_by_role('progressbar')).to_have_attribute('value', '5')
queue.complete(job_id, job['token'])
expect(page.get_by_text('Job completed.', exact=True)).to_be_visible(timeout=10000)
expect(page.locator('.queue-panel .badge').first).to_have_text('completed')
assert not page.locator('[data-queue-refresh]').count()
failed = queue.enqueue('reports', 'Retry this report')
claimed = queue.claim('reports')
queue.fail(failed, claimed['token'], 'Example failure')
page.goto(base + '/plugins/queues/jobs/' + failed)
page.get_by_role('button', name='Retry job', exact=True).click()
expect(page.get_by_role('button', name='Cancel job')).to_be_visible()
page.get_by_role('button', name='Cancel job', exact=True).click()
expect(page.get_by_role('button', name='Retry job')).to_be_visible()
assert queue.get(failed)['status'] == 'cancelled'
page.set_viewport_size({'width': 390, 'height': 844})
page.goto(base + '/plugins/queues/?status=completed&source=reports')
expect(page.get_by_role('link', name='Monthly report')).to_be_visible()
expect(page.get_by_role('link', name='Retry this report')).to_have_count(0)
assert page.evaluate('document.documentElement.scrollWidth <= window.innerWidth')
page.screenshot(path='/tmp/queue-mobile-preview.png', full_page=True)
assert not errors, errors
browser.close()
finally:
server.shutdown()
worker.join()
server.server_close()
print('Queue browser checks passed: polling, completion, logs, retry/cancel, filters, mobile, subpath, CSP.')
if __name__ == '__main__':
main()
+146
View File
@@ -0,0 +1,146 @@
"""Connector, durable worker claims, and browser access integration."""
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import pytest
from web_framework import create_app
@pytest.fixture
def app(tmp_path, monkeypatch):
monkeypatch.setenv('WEB_FRAMEWORK_DATA_DIR', str(tmp_path / 'state'))
app = create_app(overrides={
'runtime': {'plugin_dir': str(Path(__file__).resolve().parents[2]), 'theme_dir': str(tmp_path / 'themes')},
'plugins': {'enabled': ['queues']},
'resources': {'check_updates_on_startup': False},
})
app.testing = True
return app
@pytest.fixture
def queue(app):
return app.extensions['plugins'].service('queues', 'queue')
def authenticate(app):
client = app.test_client()
password = (app.extensions['state'].directory / 'initial-admin-password').read_text().strip()
client.get('/login')
with client.session_transaction() as session:
token = session['csrf']
client.post('/login', data={'username': 'admin', 'password': password, 'csrf_token': token})
client.post('/account', data={'current_password': password, 'new_password': 'A-secure-password!42', 'confirm_password': 'A-secure-password!42', 'csrf_token': csrf(client)})
return client
def csrf(client):
with client.session_transaction() as session:
return session['csrf']
def test_lifecycle_payload_progress_logs_and_restart(app, queue):
job_id = queue.enqueue('reports', '<script>Report</script>', {'rows': [1, 2]}, total=2)
claimed = queue.claim('reports')
assert claimed['id'] == job_id and claimed['payload'] == {'rows': [1, 2]}
assert claimed['attempts'] == 1
assert queue.claim('reports') is None
queue.progress(job_id, claimed['token'], 1, message='<script>half done</script>')
assert queue.get(job_id)['progress'] == 50
queue.complete(job_id, claimed['token'])
assert queue.get(job_id)['status'] == 'completed'
assert queue.get(job_id)['progress'] == 100
assert len(queue.logs(job_id)) == 4
restarted = create_app(overrides=app.extensions['workspace']['config'])
restored = restarted.extensions['plugins'].service('queues', 'queue')
assert restored.get(job_id)['status'] == 'completed'
assert restored.logs(job_id) == queue.logs(job_id)
with pytest.raises(ValueError):
queue.progress(job_id, claimed['token'], 0)
with pytest.raises(ValueError):
queue.retry(job_id)
client = authenticate(app)
response = client.get('/plugins/queues/jobs/' + job_id)
assert b'&lt;script&gt;' in response.data and b'<script>half' not in response.data
metadata = client.get('/plugins/queues/api/jobs').json['jobs'][0]
assert 'token' not in metadata and 'payload' not in metadata
assert b'value="2" max="2"' in response.data
def test_fifo_concurrent_claims_and_sources(queue):
first = queue.enqueue('one', 'First')
second = queue.enqueue('one', 'Second')
other = queue.enqueue('two', 'Other')
with ThreadPoolExecutor(max_workers=4) as pool:
jobs = list(pool.map(lambda _: queue.claim('one'), range(4)))
assert {job['id'] for job in jobs if job} == {first, second}
assert len([job for job in jobs if job]) == 2
assert queue.claim('two')['id'] == other
assert queue.jobs(source='one', limit=1, offset=1)[0]['id'] == first
def test_lease_expiry_recovery_and_stale_worker(queue):
job_id = queue.enqueue('worker', 'Recover me')
original = queue.claim('worker')
with queue.state.db() as db:
db.execute('UPDATE queue_jobs SET lease_until=0 WHERE id=?', (job_id,))
with pytest.raises(ValueError, match='expired'):
queue.complete(job_id, original['token'])
recovered = queue.claim('worker')
assert recovered['attempts'] == 2 and recovered['token'] != original['token']
with pytest.raises(ValueError):
queue.fail(job_id, original['token'])
queue.fail(job_id, recovered['token'], 'Worker failed.')
queue.retry(job_id)
fresh = queue.claim('worker')
queue.cancel(job_id)
with pytest.raises(ValueError):
queue.complete(job_id, fresh['token'])
assert queue.get(job_id)['status'] == 'cancelled'
def test_permissions_csrf_filtering_and_disabled_handle(app, queue):
job_id = queue.enqueue('worker', 'Pending')
anonymous = app.test_client()
assert anonymous.get('/plugins/queues/').status_code == 302
client = authenticate(app)
assert client.get('/plugins/queues/').status_code == 200
assert client.get('/plugins/queues/?status=invalid').status_code == 400
assert client.get('/plugins/queues/?page=-1').status_code == 400
assert client.get('/plugins/queues/jobs/missing').status_code == 404
endpoint = f'/plugins/queues/jobs/{job_id}/cancel'
assert client.post(endpoint).status_code == 400
with queue.state.db() as db:
db.execute("UPDATE users SET role='viewer'")
assert client.get('/plugins/queues/').status_code == 200
assert client.post(endpoint, data={'csrf_token': csrf(client)}).status_code == 403
queue.state.set('plugins.enabled', [])
assert client.get('/plugins/queues/').status_code == 404
with pytest.raises(ValueError, match='disabled'):
queue.enqueue('worker', 'Must not run')
with pytest.raises(ValueError, match='disabled'):
app.extensions['plugins'].service('queues', 'queue')
@pytest.mark.parametrize('completed,total', [(-1, 10), (11, 10), (0, 0), (True, 10), (1, 1.2)])
def test_invalid_progress_is_atomic(queue, completed, total):
job_id = queue.enqueue('worker', 'Validate')
job = queue.claim('worker')
with pytest.raises(ValueError):
queue.progress(job_id, job['token'], completed, total=total)
assert queue.get(job_id)['completed'] == 0
def test_logs_bounded_payload_encrypted_and_failed_update_rollback(queue):
job_id = queue.enqueue('worker', 'Secret payload', {'secret': 'payload-private-marker'})
assert b'payload-private-marker' not in queue.state.path.read_bytes()
job = queue.claim('worker')
with pytest.raises(ValueError):
queue.progress(job_id, job['token'], 50, message='')
assert queue.get(job_id)['completed'] == 0
with queue._db() as db:
for i in range(505):
queue._event(db, job_id, str(i))
logs = queue.logs(job_id)
assert len(logs) == 500 and logs[0]['message'] == '5'
with pytest.raises(ValueError):
queue.enqueue('worker', 'Too large', 'x' * 65537)
+50
View File
@@ -0,0 +1,50 @@
import hashlib
import importlib.util
import json
from pathlib import Path
import zipfile
spec = importlib.util.spec_from_file_location('resource_catalog', Path(__file__).resolve().parents[1] / 'tools/build_catalog.py')
builder = importlib.util.module_from_spec(spec)
spec.loader.exec_module(builder)
def source(root):
directory = root / 'plugins/example'
directory.mkdir(parents=True)
(root / 'themes').mkdir()
info = {'id': 'example', 'name': 'Example', 'description': 'Test resource', 'version': '0.1.0'}
(directory / 'plugin.json').write_text(json.dumps(info))
(directory / 'plugin.py').write_text('# plugin source\n')
(directory / 'tests').mkdir()
(directory / 'tests/test_example.py').write_text('# excluded\n')
return directory
def test_builder_checksums_exclusions_and_reproducibility(tmp_path):
source(tmp_path)
entries = builder.build(tmp_path)
archive = tmp_path / entries[0]['archive']
first = archive.read_bytes()
with zipfile.ZipFile(archive) as result:
assert set(result.namelist()) == {'plugin.json', 'plugin.py'}
assert hashlib.sha256(first).hexdigest() == entries[0]['sha256']
assert builder.build(tmp_path) == entries
assert archive.read_bytes() == first
def test_builder_preserves_published_bytes_and_previous_versions(tmp_path):
directory = source(tmp_path)
archives = tmp_path / 'archives'
archives.mkdir()
older = archives / 'plugin-example-0.0.1.zip'
older.write_bytes(b'previous published version')
published = archives / 'plugin-example-0.1.0.zip'
with zipfile.ZipFile(published, 'w') as output:
for name in ('plugin.py', 'plugin.json'):
output.writestr(name, (directory / name).read_bytes())
original = published.read_bytes()
entries = builder.build(tmp_path)
assert published.read_bytes() == original
assert older.read_bytes() == b'previous published version'
assert entries[0]['sha256'] == hashlib.sha256(original).hexdigest()
+43
View File
@@ -0,0 +1,43 @@
"""Build reproducible archives and catalog entries for every source resource."""
import hashlib
import json
from pathlib import Path
import zipfile
def build(root):
root = Path(root)
archives = root / 'archives'
archives.mkdir(exist_ok=True)
entries = []
for kind in ('plugin', 'theme'):
for directory in sorted((root / (kind + 's')).iterdir()):
manifest = directory / (kind + '.json')
if not manifest.is_file():
continue
info = json.loads(manifest.read_text())
archive = archives / f"{kind}-{info['id']}-{info['version']}.zip"
files = {}
for path in sorted(directory.rglob('*')):
relative = path.relative_to(directory)
if path.is_file() and not {'__pycache__', 'tests', '.pytest_cache'}.intersection(relative.parts) and path.suffix != '.pyc':
files[relative.as_posix()] = path.read_bytes()
same = False
if archive.exists():
with zipfile.ZipFile(archive) as existing:
same = {name: existing.read(name) for name in existing.namelist()} == files
# Retain exact published bytes and checksums for unchanged versions.
if not same:
with zipfile.ZipFile(archive, 'w', zipfile.ZIP_DEFLATED) as output:
for name, content in files.items():
item = zipfile.ZipInfo(name)
item.compress_type = zipfile.ZIP_DEFLATED
item.external_attr = 0o100644 << 16
output.writestr(item, content)
entries.append({**info, 'type': kind, 'archive': archive.relative_to(root).as_posix(), 'sha256': hashlib.sha256(archive.read_bytes()).hexdigest()})
(root / 'catalog.json').write_text(json.dumps({'schema_version': 1, 'resources': entries}, indent=2) + '\n')
return entries
if __name__ == '__main__':
build(Path(__file__).resolve().parents[1])
+95
View File
@@ -0,0 +1,95 @@
"""Synchronize maintained docs into both adjacent Gitea repository wikis."""
import argparse
import base64
import json
import os
import re
from pathlib import Path
import subprocess
import tempfile
from urllib.error import HTTPError
from urllib.parse import urlsplit
from urllib.request import Request, urlopen
def git(*args, cwd=None, env=None, capture=False):
return subprocess.run(['git', *args], cwd=cwd, env=env, check=True, text=True,
stdout=subprocess.PIPE if capture else None).stdout
def credentials(host):
token = os.environ.get('GITEA_TOKEN')
if token:
return os.environ.get('GITEA_USER', 'Dymas'), token
result = subprocess.run(['git', 'credential', 'fill'], input=f'protocol=https\nhost={host}\n\n',
text=True, capture_output=True, check=True)
values = dict(line.split('=', 1) for line in result.stdout.splitlines() if '=' in line)
return values['username'], values['password']
def sync(repository):
repository = Path(repository).resolve()
origin = git('remote', 'get-url', 'origin', cwd=repository, capture=True).strip()
remote = urlsplit(origin)
if remote.scheme != 'https' or remote.username or remote.password or not remote.hostname:
raise ValueError('Wiki sync requires an HTTPS origin without embedded credentials.')
owner, name = remote.path.removesuffix('.git').strip('/').split('/')
username, token = credentials(remote.netloc)
authorization = 'Basic ' + base64.b64encode(f'{username}:{token}'.encode()).decode()
base = f'{remote.scheme}://{remote.netloc}'
def api(path, method='GET', body=None):
data = None if body is None else json.dumps(body).encode()
request = Request(base + '/api/v1' + path, data=data, method=method,
headers={'Authorization': authorization, 'Content-Type': 'application/json'})
with urlopen(request, timeout=30) as response:
raw = response.read()
return json.loads(raw) if raw else None
path = f'/repos/{owner}/{name}'
if not api(path)['has_wiki']:
api(path, 'PATCH', {'has_wiki': True})
try:
api(path + '/wiki/page/Home')
except HTTPError as error:
if error.code != 404:
raise
home = (repository / 'docs/Home.md').read_bytes()
api(path + '/wiki/new', 'POST', {'title': 'Home', 'content_base64': base64.b64encode(home).decode()})
env = os.environ.copy()
# Keep credentials out of process arguments, files, and repository remotes.
count = int(env.get('GIT_CONFIG_COUNT', '0'))
env.update({f'GIT_CONFIG_KEY_{count}': f'http.{base}/.extraHeader',
f'GIT_CONFIG_VALUE_{count}': 'Authorization: ' + authorization,
'GIT_CONFIG_COUNT': str(count + 1)})
with tempfile.TemporaryDirectory(prefix=f'{name}-wiki-') as temporary:
git('clone', '--quiet', base + f'/{owner}/{name}.wiki.git', temporary, env=env)
pages = sorted((repository / 'docs').glob('*.md'))
if not pages:
raise ValueError(f'No documentation pages in {repository}')
for page in pages:
# Wiki routes use page slugs; local documentation links keep .md.
content = re.sub(r'(\]\()([A-Za-z0-9_-]+)\.md(?=[)#])', r'\1\2', page.read_text())
(Path(temporary) / page.name).write_text(content)
git('add', '--', *[page.name for page in pages], cwd=temporary)
changed = subprocess.run(['git', 'diff', '--cached', '--quiet'], cwd=temporary)
if changed.returncode == 1:
git('commit', '-m', f'Update {name} function and operations documentation', cwd=temporary)
git('push', 'origin', 'HEAD', cwd=temporary, env=env)
elif changed.returncode != 0:
raise RuntimeError('Unable to inspect wiki changes')
print(f'Synchronized {name} wiki ({len(pages)} maintained pages).')
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('repositories', nargs='*', type=Path)
args = parser.parse_args()
workspace = Path(__file__).resolve().parents[2]
repositories = args.repositories or [workspace / 'web-framework', workspace / 'web-framework-resources']
for repository in repositories:
sync(repository)
if __name__ == '__main__':
main()