71 lines
7.6 KiB
Markdown
71 lines
7.6 KiB
Markdown
# Queues plugin function reference
|
||
|
||
Requires Web Framework 0.6.0 or later. The installable resource ID is `queues`, displayed as **Queue**. Install through the catalog, restart to discover the package, then enable in Config → Plugins. It is optional and is not bundled into the framework image. Producers with a mandatory queue dependency declare `Plugin(..., requires=('queues',))`.
|
||
|
||
## Connector access
|
||
|
||
```python
|
||
from web_framework.plugins import get_plugins
|
||
|
||
queue = get_plugins().service('queues', 'queue')
|
||
job_id = queue.enqueue('reports', 'Create report', {'month': '2026-10'}, total=10)
|
||
job = queue.claim('reports')
|
||
if job:
|
||
queue.progress(job['id'], job['token'], 5, message='Half complete')
|
||
queue.log(job['id'], job['token'], 'Writing output')
|
||
queue.complete(job['id'], job['token'])
|
||
```
|
||
|
||
Resolve the service during a request or `with app.app_context():`. The manager lazily creates a single `Queue(app)` per app. Once acquired, it can be used by plugin-owned workers outside a request. Every operation rechecks the provider's enabled state. The connector is a trusted internal Python API; it is not an unauthenticated network ingestion endpoint. Producers must validate their own browser input and permissions.
|
||
|
||
## Public methods
|
||
|
||
| Function | Description |
|
||
| --- | --- |
|
||
| `Queue(app)` | Creates queue job/log tables and indexes in the app's persistent control SQLite database. Stores the owning app/state for later background use. |
|
||
| `enqueue(source, title, payload=None, *, total=100)` | Validates a source slug, 1–160 character title, positive integer total up to 2147483647, and JSON payload up to 64 KiB. Encrypts payload, inserts a queued job and initial log atomically, and returns its random ID. Non-finite JSON numbers are rejected. |
|
||
| `claim(source, *, lease_seconds=300)` | Atomically claims the oldest queued or expired-running job for the source. Returns metadata, decrypted `payload`, and private `token`, or `None`. Sets running state, increments attempts, logs start/recovery, and sets a 10–86400 second lease. |
|
||
| `progress(job_id, token, completed, *, total=None, message=None, lease_seconds=300)` | Verifies an unexpired active claim, validates whole-number `0 <= completed <= total`, updates counts and timestamp, renews lease, and optionally appends a log. Invalid input rolls back the complete update. |
|
||
| `log(job_id, token, message, *, level='info')` | Verifies a live claim and appends a 1–4000 character message at info/warning/error level. Retains 500 entries per job. Does not renew the lease. |
|
||
| `complete(job_id, token, message='Job completed.')` | Verifies the claim, marks completed, sets completed count equal to total (100%), revokes token/lease, and logs success atomically. |
|
||
| `fail(job_id, token, message='Job failed.')` | Verifies the claim, marks failed, preserves partial progress, revokes token/lease, and logs an error. |
|
||
| `cancel(job_id)` | Accepts queued/running jobs only; marks cancelled and revokes claim, adding a warning. Cooperatively prevents further worker updates. |
|
||
| `retry(job_id)` | Accepts failed/cancelled jobs only; resets progress to zero, queues the same job, and preserves attempts, payload, and log history. The next claim gets a new token. |
|
||
| `jobs(*, status=None, source=None, limit=100, offset=0)` | Lists newest jobs first, optionally filtered. Limit 1–200, nonnegative offset. Returns metadata only, including computed progress percentage. |
|
||
| `get(job_id)` | Returns one job's metadata or raises `ValueError` for a missing job. Omits payload and claim token. |
|
||
| `logs(job_id)` | Returns the retained log entries chronologically as stamp/level/message dictionaries; an unknown job has no entries. |
|
||
|
||
Metadata contains `id`, `source`, `title`, `status`, `completed`, `total`, `created`, `updated`, `attempts`, and percentage `progress`. All timestamps are Unix seconds; pages render UTC. Status values: queued, running, completed, failed, cancelled. Mutations are serialized through SQLite `BEGIN IMMEDIATE`; SQL parameters bind all user values. Worker methods require the current private claim token and reject expired/revoked/stale tokens with `ValueError`.
|
||
|
||
## Worker ownership and recovery
|
||
|
||
Queues persists and coordinates jobs; producer plugins schedule workers and execute their own handlers. A payload alone does not execute code. Use short bounded work units, renew through `progress()` before lease expiry, and make handlers idempotent. Claiming expired work provides at least once delivery and can repeat external side effects. A stale worker must stop when an update rejects its token. Cancellation cannot interrupt arbitrary Python or undo already completed external work. Source groups are conventions for trusted plugin cooperation, not security boundaries.
|
||
|
||
Jobs remain running after a lease expires until another claim for that source recovers them or an operator cancels them. Failed/cancelled jobs require explicit retry. No automatic retries or worker threads are created by installation. Different app processes can coordinate through the same SQLite file; independent server replicas with separate files cannot.
|
||
|
||
Payloads are encrypted with the framework master key. Metadata and logs are workspace-visible and must not include secrets. Up to 500 recent log entries per job are retained. Jobs are not automatically purged. Back up the entire control-data directory and key. Queues survives restarts, disable/enable cycles, plugin reinstalls, and image updates when the data volume is retained.
|
||
|
||
## Page and HTTP functions
|
||
|
||
`connector()` resolves the enabled service; `selection()` parses status/source/page filters and rejects invalid status or pagination with HTTP 400.
|
||
|
||
| Route and handler | Behavior |
|
||
| --- | --- |
|
||
| `GET /plugins/queues/` — `index()` | Renders 50 jobs per page, status/source filters, accessible progress bars, job links and pagination. |
|
||
| `GET /plugins/queues/api/jobs` — `jobs_api()` | Same filtered/paginated metadata as JSON. Requires the standard signed-in user and enabled-plugin checks. No payloads or worker tokens. |
|
||
| `GET /plugins/queues/jobs/<id>` — `detail()` | Shows job metadata, progress, attempt count and latest 500 chronological log entries; unknown IDs return 404. |
|
||
| `POST /plugins/queues/jobs/<id>/cancel` — `action()` | Requires `content.write` and valid CSRF; cancels active work, audits the user action, flashes the result, and redirects to detail. |
|
||
| `POST /plugins/queues/jobs/<id>/retry` — `action()` | Same protections; retries failed/cancelled jobs. Other actions return 404. Invalid transitions display an error. |
|
||
|
||
Signed-in users with `workspace.view` and `plugins.use` can read the queue; viewers cannot mutate it. Disabled-plugin routes return 404. Blueprint CSS/JS use the same activation/authentication gate. Templates escape titles and messages. Active panels refresh every five seconds while the tab is visible and keyboard focus is outside the panel, retaining scroll position and normal navigation. Authentication redirects stop background refresh. JavaScript is optional.
|
||
|
||
## Internal helpers
|
||
|
||
`bounded_text(value, label, maximum)` validates strings. `_db()` enforces activation and opens a transactional state connection. `_public(row, payload=False)` excludes worker capabilities and optionally decrypts payload for claim. `_event()` validates/records a log and bounds retention. `_owned()` verifies job state, token and lease. `_lease()` and `_counts()` validate numeric constraints. `_finish()` implements common terminal worker transitions. These helpers are implementation details; other plugins use the public methods above.
|
||
|
||
Run integration tests from the resource checkout:
|
||
|
||
```sh
|
||
PYTHONPATH=../web-framework ../web-framework/.venv/bin/python -m pytest tests plugins/queues/tests -q
|
||
```
|