abogen/docs/archive/entities_step_overhaul_plan.md

Entities Step Overhaul Plan

Requirements Recap

• Integrate part-of-speech (POS) tagging to detect proper nouns with better precision.
• Rename Step 3 of the wizard from Speakers to Entities everywhere (routes, templates, copy, JS).
• Introduce a sub-navigation immediately below the step indicators with three tabs: People, Entities, Manual Overrides.
• Populate tabs with appropriate data:
  • People: characters with dialogue/speech evidence.
  • Entities: non-person proper nouns (organizations, places, artefacts, etc.).
  • Manual Overrides: user-added entries with search-driven selection, pronunciation editing, and voice assignment tools.
• Allow manual overrides to:
  • Search for tokens present in the uploaded manuscript/EPUB.
  • Configure pronunciations and pick a voice (defaulting to narrator voice).
  • Trigger previews using the same audio preview logic as other steps.
• Provide voice selection dropdowns (with auto-generate, browse, clear, etc.) for People and Manual Override rows.
• Tighten extraction logic so only proper nouns surface (no "The", "That", etc.).
• Normalise detected names by removing titles ("Mr.", "Dr.") and possessives ("Bob's" -> "Bob").
• Retain expandable sample paragraphs for context ("Preview full text" pattern) in the People tab and wherever excerpts appear.
• Persist pronunciation overrides in a shared store so recurring entities automatically preload past settings.
• Apply pronunciation overrides to every preview request and final conversion so TTS always respects user inputs.
• Add a help page documenting phonetic spelling techniques (inspired by the CMU guide) and surface it via a contextual tooltip/icon inside Step 3.

Additional Considerations & Assumptions

• POS tagging scope is English-only for the initial release; spaCy will process the manuscript once and cache results so repeated visits to Step 3 reuse the parsed doc.
• spaCy core is MIT-licensed while the bundled en_core_web_sm model is CC BY-SA 3.0; we must include attribution and ensure redistribution remains compliant with the share-alike terms when packaging the model.
• spaCy may surface unusual proper nouns (e.g., fantasy names); users can leave them unchanged or override as desired.
• Manual overrides should persist with the pending job so that they can influence subsequent steps and final conversion. We likely need to extend pending job JSON storage and final job payloads.
• People tab currently depends on pending.speakers generated in speaker_analysis.py. Re-architecting should avoid breaking existing downstream behaviour (e.g., queueing with selected voices).
• Entities tab is new; we need to decide what metadata to display (count, first occurrence, sample sentences) and how it affects conversion (e.g., optional pronunciations, tags?). For now, assume read-only insights with optional pronunciation overrides similar to People.
• Voice preview/generation flows already live in prepare.js; ensure refactors keep a single source of truth to avoid duplication.

Linguistic & Data Strategy

1. POS Tagging Research & Adoption

   • Leverage spaCy (>=3.5) for tokenisation, POS tagging, and named entity recognition (NER). It offers:
     • Accurate POS tags for proper nouns (PROPN).
     • Entity type labels (PERSON, ORG, GPE, etc.) that can help route to People vs Entities.
   • Add spacy to dependencies and document model installation (en_core_web_sm minimum). Provide fallbacks:
     • If model missing, prompt friendly error and skip advanced detection rather than failing job.
     • Future extension: allow language-specific models per job language (English default, warn otherwise).

2. Proper Noun Filtering Logic

   • Process each chapter/chunk through spaCy pipeline.
   • For each token / entity:
     • Keep tokens tagged PROPN or NER labelled as proper nouns.
     • Discard stopwords and determiners even if mislabelled (helps avoid "The", "That").
     • Normalise by removing leading titles (Mr., Dr., Lady, etc.) and trailing possessives ('s, ’s).
     • Merge contiguous proper nouns into multi-word names (spaCy entity spans help).
   • Build frequency map; attach contextual snippets (e.g., surrounding sentence) for each.
   • Classify as Person vs Entity:
     • If entity label PERSON or strongly associated with dialogue attribution (existing heuristics), treat as Person.
     • Otherwise, map to Entity; optionally infer subtypes (Org, Place) for later enhancements.

3. Integration with Existing Speaker Analysis

   • Reuse dialogue-based detection (speaker_analysis.py) for People to keep gender heuristics and sample quotes.
   • Align IDs: ensure People tab entries map to existing speaker IDs so voice selections propagate to final job.
   • Entities tab can draw from new data structure, decoupled from speaker_analysis but referencing chapter/chunk indices.

4. Manual Overrides Workflow

   • Backend:
     • Maintain pending.manual_overrides list containing token, normalised_label, pronunciation, voice, notes, context, while syncing to a persistent overrides table (e.g., SQLite) keyed by normalised token + language so history is reused across projects. Manual entries do not require spaCy detection—users can add arbitrary tokens.

• On load, hydrate the pending list with any matching historical overrides before rendering Step 3.
• Provide API endpoints:
  1. GET suggestions for a search query (scan processed tokens + raw text indexes).
  2. POST create/update override entries.
  3. DELETE override.
• Frontend:
  • Search input with debounced calls to suggestion endpoint; results list to choose target word/phrase.
  • Once selected, show pronunciation input, voice picker (reusing component from People), preview buttons. - Allow manual entry of custom tokens when no suggestion matches (spaCy not required). - Persist changes via AJAX (same pattern as existing speaker updates if possible) or within form submission when continuing.

Implementation Plan

1. Backend Enhancements
   • Add spaCy dependency and lazy-load model in speaker_analysis.py or a new entity_analysis.py module.

• Cache parsed spaCy documents per pending job (disk-backed or memoized) so repeated analysis reuses existing results without reprocessing the manuscript.
• Implement extract_entities(chapters, language, config) returning structure:

  {
    "people": [
      {"id": "speaker_1", "label": "Bob", "count": 12, "samples": [...], ...}
    ],
    "entities": [
      {"id": "entity_1", "label": "Starfleet", "kind": "ORG", "count": 5, "snippets": [...]}
    ],
    "index": {...}  # for search/autocomplete
  }
  

• Enhance normalisation function to strip titles/possessives and collapse whitespace/diacritics consistently.
• Integrate entity output into pending job serialization so Step 3 view can render tabs without recomputation.
• Update job finalisation logic to include manual overrides and entity-derived metadata (for future TTS improvements).
• Introduce a persistent pronunciation overrides repository (SQLite via SQLAlchemy layer) shared across jobs/instances, with migrations and CRUD helpers.
• Apply pronunciation overrides to preview/conversion pipelines by substituting text prior to TTS synthesis (covering narrator defaults, People tab assignments, Entities tab items, and manual overrides on every TTS run).

2. Template & UI Updates

   • Rename Step 3 to Entities in all templates (prepare_speakers.html, upload modal partial, step indicator macros).
   • Refactor prepare_speakers.html to:
     • Wrap content in tabbed interface (likely <div role="tablist"> + panels).
     • Tab panels:
       1. People: existing speaker list; adjust headings and copy.
       2. Entities: new list/grid showing non-person entities with counts and sample context; include optional pronunciation/voice controls if relevant.
       3. Manual Overrides: search box, selected override editing form, table of current overrides.
     • Ensure sample paragraphs remain behind a collapsible disclosure control (link + <details> as today).
       • Place a help icon near pronunciation inputs; focusing/hovering reveals tooltip text summarising phonetic spelling tips and links to the full guide.
     • Update CSS to style tabs consistent with modal aesthetic, including tooltip styling for the help icon.
     • Add a dedicated phonetic spelling help page (e.g., phonetic-pronunciation.html) sourced from the CMU reference with attribution, linked from the tooltip and main help menu.

3. Frontend Logic (prepare.js)

   • Introduce tab controller managing focus and ARIA attributes.
   • Wire People tab voice dropdowns to existing preview logic; extend to manual overrides entries.
   • Implement search suggestions for manual overrides (debounce, fetch, render list, handle selection).
   • Ensure previews use existing data-role="speaker-preview" pipeline; extend dataset attributes as needed.
   • Persist override edits either via hidden inputs or asynchronous saves; align with form submission semantics.

4. APIs & Routing

   • Add Flask routes under routes.py or web/service.py for:
     • /pending/<id>/entities (fetch processed entity data if not already included in template context).
     • /pending/<id>/overrides (CRUD operations for manual overrides).
   • Ensure permissions and CSRF tokens align with existing patterns.

5. Data Persistence

   • Expand pending job model (likely stored in queue_manager_gui.py / queued_item.py) to keep:
     • entity_summary snapshot (people/entities lists).
     • manual_overrides list with user edits.
     • Cached spaCy doc metadata (hash of source + serialized parse) to avoid reprocessing unchanged texts.

• Introduce persistent pronunciation_overrides table (SQLite) keyed by normalised token + language, storing pronunciation, preferred voice, notes, and usage metadata for reuse across projects.
• On finalise, merge overrides into job metadata so downstream conversion can honour pronunciations/voices and sync any changes back to the shared table.

6. Testing Strategy
   • Unit tests for new normalisation and POS filtering functions (ensure "The", "That" excluded; "Bob's" normalised).
   • Integration tests to confirm People tab still flows, manual overrides persist, Entities tab populates expected data.
   • Add regression tests ensuring Step 3 rename does not break existing forms (e.g., test_prepare_form.py).
   • Consider snapshot tests for API JSON structures.

• Add automated checks that pronunciation overrides apply to preview playback and conversion payloads for People and Entities entries alike.

7. Documentation & Ops
   • Update README / docs with new Step 3 name and manual override instructions.
   • Provide guidance for installing spaCy model (e.g., python -m spacy download en_core_web_sm).

• Document spaCy/model licensing obligations (MIT for core, CC BY-SA for small model) and add attribution in app credits/help page.
• Publish phonetic spelling help page content and link it from the tooltip/icon in Step 3 and support docs.

Open Questions / Follow-Ups

• Should Entities tab allow voice assignments that influence TTS, or is it informational only? Yes, it should include voice assignments that influence TTS.
• Manual override search scope: entire text vs detected proper nouns? Current plan searches raw text and entity index.
• Performance: confirm caching strategy (e.g., store spaCy Doc pickles vs. rebuilding from serialized spans) to balance speed and storage.

Next Steps

1. Validate spaCy dependency choice and licensing obligations (MIT core, CC BY-SA model) with stakeholders.
2. Finalise data contracts for entities, overrides, and the persistent pronunciation history schema.
3. Implement backend entity extraction, cached spaCy parsing, override hydration, and the TTS substitution pipeline.
4. Refactor frontend Step 3 UI with tabs, help icon/tooltip, and updated voice controls.
5. Build manual override search/edit UX wired to the shared overrides store and preview flow.
6. Update documentation (including phonetic guide) and expand automated tests.