docs: archive historical plans, remove duplicates

- Move migration-roadmap.md, epub3_upgrade_plan.md, entities_step_overhaul_plan.md to docs/archive/
- Remove duplicate tts-plugin-architecture.md

docs/archive/entities_step_overhaul_plan.md

@@ -0,0 +1,152 @@
+# Entities Step Overhaul Plan
+
+## Requirements Recap
+- Integrate part-of-speech (POS) tagging to detect proper nouns with better precision.
+- Rename Step 3 of the wizard from **Speakers** to **Entities** everywhere (routes, templates, copy, JS).
+- Introduce a sub-navigation immediately below the step indicators with three tabs: **People**, **Entities**, **Manual Overrides**.
+- Populate tabs with appropriate data:
+  - **People**: characters with dialogue/speech evidence.
+  - **Entities**: non-person proper nouns (organizations, places, artefacts, etc.).
+  - **Manual Overrides**: user-added entries with search-driven selection, pronunciation editing, and voice assignment tools.
+- Allow manual overrides to:
+  - Search for tokens present in the uploaded manuscript/EPUB.
+  - Configure pronunciations and pick a voice (defaulting to narrator voice).
+  - Trigger previews using the same audio preview logic as other steps.
+- Provide voice selection dropdowns (with auto-generate, browse, clear, etc.) for People and Manual Override rows.
+- Tighten extraction logic so only proper nouns surface (no "The", "That", etc.).
+- Normalise detected names by removing titles ("Mr.", "Dr.") and possessives ("Bob's" -> "Bob").
+- Retain expandable sample paragraphs for context ("Preview full text" pattern) in the People tab and wherever excerpts appear.
+- Persist pronunciation overrides in a shared store so recurring entities automatically preload past settings.
+- Apply pronunciation overrides to every preview request and final conversion so TTS always respects user inputs.
+- Add a help page documenting phonetic spelling techniques (inspired by the CMU guide) and surface it via a contextual tooltip/icon inside Step 3.
+
+## Additional Considerations & Assumptions
+- POS tagging scope is English-only for the initial release; spaCy will process the manuscript once and cache results so repeated visits to Step 3 reuse the parsed doc.
+- spaCy core is MIT-licensed while the bundled `en_core_web_sm` model is CC BY-SA 3.0; we must include attribution and ensure redistribution remains compliant with the share-alike terms when packaging the model.
+- spaCy may surface unusual proper nouns (e.g., fantasy names); users can leave them unchanged or override as desired.
+- Manual overrides should persist with the pending job so that they can influence subsequent steps and final conversion. We likely need to extend pending job JSON storage and final job payloads.
+- People tab currently depends on `pending.speakers` generated in `speaker_analysis.py`. Re-architecting should avoid breaking existing downstream behaviour (e.g., queueing with selected voices).
+- Entities tab is new; we need to decide what metadata to display (count, first occurrence, sample sentences) and how it affects conversion (e.g., optional pronunciations, tags?). For now, assume read-only insights with optional pronunciation overrides similar to People.
+- Voice preview/generation flows already live in `prepare.js`; ensure refactors keep a single source of truth to avoid duplication.
+
+## Linguistic & Data Strategy
+1. **POS Tagging Research & Adoption**
+   - Leverage **spaCy** (>=3.5) for tokenisation, POS tagging, and named entity recognition (NER). It offers:
+     - Accurate POS tags for proper nouns (`PROPN`).
+     - Entity type labels (`PERSON`, `ORG`, `GPE`, etc.) that can help route to People vs Entities.
+   - Add `spacy` to dependencies and document model installation (`en_core_web_sm` minimum). Provide fallbacks:
+     - If model missing, prompt friendly error and skip advanced detection rather than failing job.
+     - Future extension: allow language-specific models per job language (English default, warn otherwise).
+
+2. **Proper Noun Filtering Logic**
+   - Process each chapter/chunk through spaCy pipeline.
+   - For each token / entity:
+     - Keep tokens tagged `PROPN` or NER labelled as proper nouns.
+     - Discard stopwords and determiners even if mislabelled (helps avoid "The", "That").
+     - Normalise by removing leading titles (`Mr.`, `Dr.`, `Lady`, etc.) and trailing possessives (`'s`, `’s`).
+     - Merge contiguous proper nouns into multi-word names (spaCy entity spans help).
+   - Build frequency map; attach contextual snippets (e.g., surrounding sentence) for each.
+   - Classify as Person vs Entity:
+     - If entity label `PERSON` or strongly associated with dialogue attribution (existing heuristics), treat as **Person**.
+     - Otherwise, map to **Entity**; optionally infer subtypes (Org, Place) for later enhancements.
+
+3. **Integration with Existing Speaker Analysis**
+   - Reuse dialogue-based detection (`speaker_analysis.py`) for People to keep gender heuristics and sample quotes.
+   - Align IDs: ensure People tab entries map to existing speaker IDs so voice selections propagate to final job.
+   - Entities tab can draw from new data structure, decoupled from `speaker_analysis` but referencing chapter/chunk indices.
+
+4. **Manual Overrides Workflow**
+   - Backend:
+     - Maintain `pending.manual_overrides` list containing `token`, `normalised_label`, `pronunciation`, `voice`, `notes`, `context`, while syncing to a persistent overrides table (e.g., SQLite) keyed by normalised token + language so history is reused across projects. Manual entries do not require spaCy detection—users can add arbitrary tokens.
+  - On load, hydrate the pending list with any matching historical overrides before rendering Step 3.
+  - Provide API endpoints:
+       1. `GET` suggestions for a search query (scan processed tokens + raw text indexes).
+       2. `POST` create/update override entries.
+       3. `DELETE` override.
+   - Frontend:
+     - Search input with debounced calls to suggestion endpoint; results list to choose target word/phrase.
+     - Once selected, show pronunciation input, voice picker (reusing component from People), preview buttons.
+    - Allow manual entry of custom tokens when no suggestion matches (spaCy not required).
+    - Persist changes via AJAX (same pattern as existing speaker updates if possible) or within form submission when continuing.
+
+## Implementation Plan
+1. **Backend Enhancements**
+   - Add spaCy dependency and lazy-load model in `speaker_analysis.py` or a new `entity_analysis.py` module.
+  - Cache parsed spaCy documents per pending job (disk-backed or memoized) so repeated analysis reuses existing results without reprocessing the manuscript.
+   - Implement `extract_entities(chapters, language, config)` returning structure:
+     ```python
+     {
+       "people": [
+         {"id": "speaker_1", "label": "Bob", "count": 12, "samples": [...], ...}
+       ],
+       "entities": [
+         {"id": "entity_1", "label": "Starfleet", "kind": "ORG", "count": 5, "snippets": [...]}
+       ],
+       "index": {...}  # for search/autocomplete
+     }
+     ```
+   - Enhance normalisation function to strip titles/possessives and collapse whitespace/diacritics consistently.
+   - Integrate entity output into pending job serialization so Step 3 view can render tabs without recomputation.
+   - Update job finalisation logic to include manual overrides and entity-derived metadata (for future TTS improvements).
+  - Introduce a persistent pronunciation overrides repository (SQLite via SQLAlchemy layer) shared across jobs/instances, with migrations and CRUD helpers.
+  - Apply pronunciation overrides to preview/conversion pipelines by substituting text prior to TTS synthesis (covering narrator defaults, People tab assignments, Entities tab items, and manual overrides on every TTS run).
+
+2. **Template & UI Updates**
+   - Rename Step 3 to **Entities** in all templates (`prepare_speakers.html`, upload modal partial, step indicator macros).
+   - Refactor `prepare_speakers.html` to:
+     - Wrap content in tabbed interface (likely `<div role="tablist">` + panels).
+     - Tab panels:
+       1. **People**: existing speaker list; adjust headings and copy.
+       2. **Entities**: new list/grid showing non-person entities with counts and sample context; include optional pronunciation/voice controls if relevant.
+       3. **Manual Overrides**: search box, selected override editing form, table of current overrides.
+     - Ensure sample paragraphs remain behind a collapsible disclosure control (link + `<details>` as today).
+       - Place a help icon near pronunciation inputs; focusing/hovering reveals tooltip text summarising phonetic spelling tips and links to the full guide.
+     - Update CSS to style tabs consistent with modal aesthetic, including tooltip styling for the help icon.
+     - Add a dedicated phonetic spelling help page (e.g., `phonetic-pronunciation.html`) sourced from the CMU reference with attribution, linked from the tooltip and main help menu.
+
+3. **Frontend Logic (`prepare.js`)**
+   - Introduce tab controller managing focus and ARIA attributes.
+   - Wire People tab voice dropdowns to existing preview logic; extend to manual overrides entries.
+   - Implement search suggestions for manual overrides (debounce, fetch, render list, handle selection).
+   - Ensure previews use existing `data-role="speaker-preview"` pipeline; extend dataset attributes as needed.
+   - Persist override edits either via hidden inputs or asynchronous saves; align with form submission semantics.
+
+4. **APIs & Routing**
+   - Add Flask routes under `routes.py` or `web/service.py` for:
+     - `/pending/<id>/entities` (fetch processed entity data if not already included in template context).
+     - `/pending/<id>/overrides` (CRUD operations for manual overrides).
+   - Ensure permissions and CSRF tokens align with existing patterns.
+
+5. **Data Persistence**
+   - Expand pending job model (likely stored in `queue_manager_gui.py` / `queued_item.py`) to keep:
+     - `entity_summary` snapshot (people/entities lists).
+     - `manual_overrides` list with user edits.
+     - Cached spaCy doc metadata (hash of source + serialized parse) to avoid reprocessing unchanged texts.
+  - Introduce persistent `pronunciation_overrides` table (SQLite) keyed by normalised token + language, storing pronunciation, preferred voice, notes, and usage metadata for reuse across projects.
+  - On finalise, merge overrides into job metadata so downstream conversion can honour pronunciations/voices and sync any changes back to the shared table.
+
+6. **Testing Strategy**
+   - Unit tests for new normalisation and POS filtering functions (ensure "The", "That" excluded; "Bob's" normalised).
+   - Integration tests to confirm People tab still flows, manual overrides persist, Entities tab populates expected data.
+   - Add regression tests ensuring Step 3 rename does not break existing forms (e.g., `test_prepare_form.py`).
+   - Consider snapshot tests for API JSON structures.
+  - Add automated checks that pronunciation overrides apply to preview playback and conversion payloads for People and Entities entries alike.
+
+7. **Documentation & Ops**
+   - Update README / docs with new Step 3 name and manual override instructions.
+   - Provide guidance for installing spaCy model (e.g., `python -m spacy download en_core_web_sm`).
+  - Document spaCy/model licensing obligations (MIT for core, CC BY-SA for small model) and add attribution in app credits/help page.
+  - Publish phonetic spelling help page content and link it from the tooltip/icon in Step 3 and support docs.
+
+## Open Questions / Follow-Ups
+- Should Entities tab allow voice assignments that influence TTS, or is it informational only? Yes, it should include voice assignments that influence TTS.
+- Manual override search scope: entire text vs detected proper nouns? Current plan searches raw text and entity index.
+- Performance: confirm caching strategy (e.g., store spaCy Doc pickles vs. rebuilding from serialized spans) to balance speed and storage.
+
+## Next Steps
+1. Validate spaCy dependency choice and licensing obligations (MIT core, CC BY-SA model) with stakeholders.
+2. Finalise data contracts for entities, overrides, and the persistent pronunciation history schema.
+3. Implement backend entity extraction, cached spaCy parsing, override hydration, and the TTS substitution pipeline.
+4. Refactor frontend Step 3 UI with tabs, help icon/tooltip, and updated voice controls.
+5. Build manual override search/edit UX wired to the shared overrides store and preview flow.
+6. Update documentation (including phonetic guide) and expand automated tests.

docs/archive/epub3_upgrade_plan.md

@@ -0,0 +1,208 @@
+# EPUB 3 Upgrade Plan
+
+## Overview
+Elevate Abogen to produce rich EPUB 3 packages with synchronized narration, configurable TTS chunking, and groundwork for multi-speaker voice assignment. This document records the objectives, architectural adjustments, data model changes, UI flows, and implementation phases required to deliver the upgrade.
+
+## Goals
+- Generate EPUB 3 output that preserves source metadata and embeds audio narration via media overlays.
+- Allow users to choose the chunking granularity (paragraph vs. sentence) used for TTS synthesis and media-overlay alignment.
+- Introduce speaker assignments for every chunk, starting with a single narrator but paving the way for multi-speaker control.
+- Prototype practical, lightweight strategies for detecting likely speakers and estimating their dialogue frequency.
+
+## Non-goals / Out-of-scope
+- Full multi-speaker editing UI (beyond gating the option).
+- Automatic voice-casting or LLM-based dialogue attribution.
+- Desktop GUI resurrection (web UI remains primary).
+
+## Current Architecture Snapshot
+| Area | Notes |
+| --- | --- |
+| Text ingestion | `abogen/text_extractor.py` outputs `ExtractionResult` with chapter-level text.
+| Job prep UI | `web/routes.py` builds `PendingJob` objects and renders chapter selection.
+| Audio pipeline | `web/conversion_runner.py` creates per-job audio artifacts; chunking is effectively paragraph-level.
+| Metadata | `ExtractionResult.metadata` feeds into FF metadata and output tagging, but not yet into EPUB packaging.
+
+## Feature 1 – EPUB 3 Output with Narration
+### Requirements
+- Preserve original EPUB metadata (Dublin Core entries, TOC, cover art).
+- Package synthesized audio and SMIL media overlays aligned to chosen chunk granularity.
+- Provide EPUB as an additional selectable output alongside current audio/subtitle formats.
+
+### Proposed Components
+1. **`abogen/epub3/exporter.py`** (new module)
+   - Responsibilities: build XHTML spine with IDs, generate overlay SMIL files, write OPF manifest/spine, assemble zip package.
+   - Status: **Implemented** — `build_epub3_package` emits EPUB 3 archives with media overlays driven by chunk metadata.
+   - Dependencies: reuse `ebooklib` for reading source metadata; use `zipfile` for packaging; optional `lxml` for DOM manipulation.
+2. **`EPUB3PackageBuilder` class**
+   - Inputs: extraction payload, chunk collection (with IDs, speaker mapping, timing metadata), audio asset paths, source metadata.
+   - Outputs: path to generated EPUB.
+3. **Metadata preservation**
+   - Copy from source `ExtractionResult.metadata` and EPUB navigation if available.
+   - Ensure custom fields (e.g., chapter count) survive.
+4. **Media overlay generation**
+   - Create one SMIL per content doc or per chapter, depending on chunk count.
+   - `<par>` nodes reference chunk IDs and audio clip times.
+5. **Configuration surface**
+   - Add “EPUB 3 (audio + text)” to output format selector (or a dedicated toggle under project settings).
+
+### Data Flow
+```
+extract_from_path -> Chapter payload
+              |-> chunker (sentence/paragraph)
+              |-> chunk IDs + audio segments (timestamps from runner)
+Conversion runner -> audio files + timing index
+EPUB3PackageBuilder -> manifest, spine, SMIL, zip
+```
+
+### Open Questions
+- Should we embed audio inside the EPUB or link externally? (Plan: embed to comply with spec.)
+- How to handle very large audio assets? Consider splitting per chapter to keep file sizes manageable.
+
+## Feature 2 – Configurable Chunking
+### Requirements
+- Users select chunking level (paragraph or sentence) before audio generation.
+- Pipeline produces stable, unique IDs for each chunk regardless of level.
+- Provide chunk metadata (text, speaker, offsets) to both TTS and EPUB exporter.
+
+### Proposed Architecture
+1. **Chunk Model**
+   ```python
+   @dataclass
+   class Chunk:
+       id: str
+       chapter_index: int
+       order: int
+       level: Literal["paragraph", "sentence"]
+       text: str
+       speaker_id: str
+       approx_characters: int
+   ```
+2. **Chunker Service (`abogen/chunking.py`)**
+   - Accepts chapter text and desired level.
+   - Uses spaCy (already bundled via `en-core-web-sm`) for sentence segmentation; fallback to regex when model unavailable.
+   - Emits `Chunk` objects with deterministic IDs (e.g., `chap{chapter_index:04d}_para{paragraph_idx:03d}_sent{sentence_idx:03d}`).
+3. **Integration points**
+   - `web/routes.py` -> apply chunker when building `PendingJob` instead of storing raw paragraphs only.
+   - `PendingJob` / `Job` dataclasses -> include `chunks` list and `chunk_level` enum.
+   - `conversion_runner` -> iterate over `chunks` when synthesizing audio, producing per-chunk audio and capturing actual duration for overlay.
+4. **Settings persistence**
+   - Extend config with `chunking_level` default; expose in UI (radio buttons or select).
+
+### Testing
+- Unit tests for chunk splitting across languages, punctuation, abbreviations.
+- Property-based tests ensuring concatenated chunks reproduce original text (except whitespace normalization).
+
+## Feature 3 – Speaker Assignment Foundations
+### Requirements
+- Every chunk must carry a `speaker_id` (default `narrator`).
+- UI offers new option: “Single Speaker” (proceeds) vs. “Multi-Speaker (Coming Soon)” (blocks and shows message).
+- Data model anticipates future multi-speaker support.
+
+### Implementation Outline
+1. **Data Model Changes**
+   - `Chunk.speaker_id` default `"narrator"`.
+   - `PendingJob` & `Job` store `speakers` metadata (dictionary of speaker descriptors).
+   - `JobResult` optionally includes `chunk_speakers.json` artifact for downstream use.
+2. **UI Adjustments**
+   - On upload form (`index.html` / JS), add selector for speaker mode.
+   - If “Multi-Speaker” chosen, display tooltip/modal: “Coming soon; please choose Single Speaker to continue.” disable submission.
+   - In `prepare_job.html`, display speaker info column (read-only for now).
+3. **Serialization**
+   - Update JSON API routes to include speaker data.
+   - Update queue/job detail templates to show chunk level & speaker summary.
+
+### Testing
+- Add web route tests ensuring multi-speaker path blocks progression.
+- Verify job persistence includes `speaker_id` fields.
+
+## Feature 4 – Speaker Detection Strategies
+### Objectives
+Build groundwork for lightweight, deterministic speaker inference to inform future multi-speaker mode.
+
+### User Stories
+1. **As a producer**, I can run an automated analysis on a book to see the list of likely speakers and how often they talk, so I can decide where multiple voices make sense.
+   - _Acceptance_: System outputs a JSON report containing speaker IDs/names, occurrence counts, representative excerpts, and confidence tier. Report stored with job artifacts and downloadable from job detail page.
+2. **As a producer**, I can set a minimum occurrence threshold so that infrequent speakers automatically fall back to the narrator voice.
+   - _Acceptance_: Analysis respects configurable threshold; speakers below it are tagged as `default_narrator` in the report.
+3. **As a developer/operator**, I can trigger the analysis via CLI or background task without blocking the main conversion pipeline.
+   - _Acceptance_: Command `abogen analyze-speakers <input>` (or background queue hook) runs in isolation, returns exit code 0 on success, emits metrics/logs for CI.
+
+### Strategy Ideas
+1. **Quotation-bound heuristic**
+   - Split paragraphs on dialogue quotes.
+   - Use verb cues ("said", "asked") to associate names preceding/following quotes.
+2. **Name detection via NER**
+   - Use spaCy’s entity recognition to spot `PERSON` entities inside dialogue spans.
+   - Maintain frequency counts per name.
+3. **Speaker dictionary**
+   - Pre-build mapping of common narrator cues ("he said", "Mary replied") to propagate speaker assignment across adjacent sentences.
+4. **Pronoun fallback with gender hints**
+   - Map pronouns to most recent speaker mention; degrade gracefully when ambiguous.
+5. **Thresholding mechanism**
+   - After counting occurrences, expose a threshold slider (future UI) to decide when to allocate unique voices vs. default narrator.
+6. **Diagnostics**
+   - Provide summary report: top N speaker candidates, counts, unresolved dialogue segments.
+
+### Implementation Staging
+1. **Phase 1 – Analysis Engine (Backend)**
+   - Build `speaker_analysis.py` module implementing heuristics, returning structured results.
+   - Add CLI entry point `abogen-speaker-analyze` for standalone runs.
+   - Persist analysis artifacts (`speakers.json`, `speaker_excerpts.csv`) alongside job data when invoked post-extraction.
+   - Tests: unit tests for heuristic functions; snapshot tests for sample novels.
+2. **Phase 2 – Configuration & Thresholding**
+   - Extend settings UI with optional “speaker analysis threshold” control (numeric).
+   - Update analysis module to accept threshold; mark low-frequency speakers as narrator.
+   - Emit summary digest (top speakers, narrator fallback count) in job logs.
+3. **Phase 3 – UI Surfacing**
+   - Display analysis summary on job detail page (charts/table).
+   - Offer download link for raw JSON/CSV artifacts.
+   - Provide warning banner when analysis confidence is low (e.g., high unmatched dialogue percentage).
+4. **Phase 4 – Integration Hooks**
+   - Wire analysis output into chunk speaker assignments (without yet enabling multi-speaker playback).
+   - Store mapping in `Job.speakers` metadata for future voice routing.
+
+### Technical Notes
+- Reuse spaCy `en_core_web_sm` for entity recognition; allow pluggable models per language.
+- Maintain rolling context window to resolve pronouns (e.g., last two named speakers).
+- Provide instrumentation (timings, counts) to assess heuristic accuracy on sample corpora.
+- Design analysis output schema versioning (`speaker_analysis_version`) to support iterative improvements.
+
+## UI & Configuration Updates
+| Screen | Update |
+| --- | --- |
+| Upload form (`index.html`) | Add chunking level selector and speaker mode buttons. |
+| Prepare job (`prepare_job.html`) | Display chunk level, IDs, speaker column; allow future editing hooks. |
+| Settings modal | Persist defaults for chunking level and speaker mode. |
+
+## Data Model Checklist
+- [x] Update `PendingJob` and `Job` dataclasses with `chunk_level`, `chunks`, `speakers` metadata.
+- [x] Ensure serialization persists these fields in queue state file.
+- [x] Persist chunk timing metadata from TTS (start/end timestamps).
+
+## Testing Strategy
+- Unit tests for chunker and speaker heuristics.
+- Integration tests: enqueue job with sentence-level chunking, assert chunk IDs and speaker metadata.
+- Regression tests: ensure existing paragraph-level jobs still succeed.
+- Acceptance tests for EPUB exporter: validate manifest, spine, and SMIL structure against schema (use `epubcheck` in CI if feasible).
+
+## Migration & Compat
+- Bump state version in `ConversionService` when augmenting job schema; include migration logic for legacy queues.
+- Provide CLI flag to reprocess older jobs without speaker metadata.
+- Document new dependencies (e.g., `lxml`, optional spaCy models for languages beyond English).
+
+## Implementation Phases
+1. **Foundation** – Introduce chunk model, chunker service, speaker defaults.
+2. **Pipeline integration** – Update job lifecycle and TTS runner to work with chunks.
+3. **EPUB exporter** – Build packaging module, connect to pipeline.
+4. **UI polish** – Expose settings, guard multi-speaker path, surface diagnostics.
+5. **Speaker analysis tool** – Prototype heuristics and reporting.
+
+## Open Questions
+- How to handle non-EPUB inputs (PDF/TXT) when exporting EPUB 3? (Possible: generate synthetic XHTML with normalized chapters.)
+- Storage impact of embedding per-chunk audio – do we need compression or streaming strategies?
+- Internationalization: sentence segmentation quality varies; need language-specific models.
+
+## Next Steps
+- Review plan with stakeholders for scope confirmation.
+- Break down Phase 1 into actionable tickets (chunker, data model migration, UI toggle).
+- Estimate resource requirements for EPUB packaging and testing (including epubcheck integration).