abogen/docs/archive/epub3_upgrade_plan.md

EPUB 3 Upgrade Plan

Overview

Elevate Abogen to produce rich EPUB 3 packages with synchronized narration, configurable TTS chunking, and groundwork for multi-speaker voice assignment. This document records the objectives, architectural adjustments, data model changes, UI flows, and implementation phases required to deliver the upgrade.

Goals

• Generate EPUB 3 output that preserves source metadata and embeds audio narration via media overlays.
• Allow users to choose the chunking granularity (paragraph vs. sentence) used for TTS synthesis and media-overlay alignment.
• Introduce speaker assignments for every chunk, starting with a single narrator but paving the way for multi-speaker control.
• Prototype practical, lightweight strategies for detecting likely speakers and estimating their dialogue frequency.

Non-goals / Out-of-scope

• Full multi-speaker editing UI (beyond gating the option).
• Automatic voice-casting or LLM-based dialogue attribution.
• Desktop GUI resurrection (web UI remains primary).

Current Architecture Snapshot

Area	Notes
Text ingestion	abogen/text_extractor.py outputs ExtractionResult with chapter-level text.
Job prep UI	web/routes.py builds PendingJob objects and renders chapter selection.
Audio pipeline	web/conversion_runner.py creates per-job audio artifacts; chunking is effectively paragraph-level.
Metadata	ExtractionResult.metadata feeds into FF metadata and output tagging, but not yet into EPUB packaging.

Feature 1 – EPUB 3 Output with Narration

Requirements

• Preserve original EPUB metadata (Dublin Core entries, TOC, cover art).
• Package synthesized audio and SMIL media overlays aligned to chosen chunk granularity.
• Provide EPUB as an additional selectable output alongside current audio/subtitle formats.

Proposed Components

1. abogen/epub3/exporter.py (new module)
   • Responsibilities: build XHTML spine with IDs, generate overlay SMIL files, write OPF manifest/spine, assemble zip package.
   • Status: Implemented — build_epub3_package emits EPUB 3 archives with media overlays driven by chunk metadata.
   • Dependencies: reuse ebooklib for reading source metadata; use zipfile for packaging; optional lxml for DOM manipulation.
2. EPUB3PackageBuilder class
   • Inputs: extraction payload, chunk collection (with IDs, speaker mapping, timing metadata), audio asset paths, source metadata.
   • Outputs: path to generated EPUB.
3. Metadata preservation
   • Copy from source ExtractionResult.metadata and EPUB navigation if available.
   • Ensure custom fields (e.g., chapter count) survive.
4. Media overlay generation
   • Create one SMIL per content doc or per chapter, depending on chunk count.
   • <par> nodes reference chunk IDs and audio clip times.
5. Configuration surface
   • Add “EPUB 3 (audio + text)” to output format selector (or a dedicated toggle under project settings).

Data Flow

extract_from_path -> Chapter payload
              |-> chunker (sentence/paragraph)
              |-> chunk IDs + audio segments (timestamps from runner)
Conversion runner -> audio files + timing index
EPUB3PackageBuilder -> manifest, spine, SMIL, zip

Open Questions

• Should we embed audio inside the EPUB or link externally? (Plan: embed to comply with spec.)
• How to handle very large audio assets? Consider splitting per chapter to keep file sizes manageable.

Feature 2 – Configurable Chunking

Requirements

• Users select chunking level (paragraph or sentence) before audio generation.
• Pipeline produces stable, unique IDs for each chunk regardless of level.
• Provide chunk metadata (text, speaker, offsets) to both TTS and EPUB exporter.

Proposed Architecture

1. Chunk Model

   @dataclass
   class Chunk:
       id: str
       chapter_index: int
       order: int
       level: Literal["paragraph", "sentence"]
       text: str
       speaker_id: str
       approx_characters: int
   

2. Chunker Service (abogen/chunking.py)
   • Accepts chapter text and desired level.
   • Uses spaCy (already bundled via en-core-web-sm) for sentence segmentation; fallback to regex when model unavailable.
   • Emits Chunk objects with deterministic IDs (e.g., chap{chapter_index:04d}_para{paragraph_idx:03d}_sent{sentence_idx:03d}).
3. Integration points
   • web/routes.py -> apply chunker when building PendingJob instead of storing raw paragraphs only.
   • PendingJob / Job dataclasses -> include chunks list and chunk_level enum.
   • conversion_runner -> iterate over chunks when synthesizing audio, producing per-chunk audio and capturing actual duration for overlay.
4. Settings persistence
   • Extend config with chunking_level default; expose in UI (radio buttons or select).

Testing

• Unit tests for chunk splitting across languages, punctuation, abbreviations.
• Property-based tests ensuring concatenated chunks reproduce original text (except whitespace normalization).

Feature 3 – Speaker Assignment Foundations

Requirements

• Every chunk must carry a speaker_id (default narrator).
• UI offers new option: “Single Speaker” (proceeds) vs. “Multi-Speaker (Coming Soon)” (blocks and shows message).
• Data model anticipates future multi-speaker support.

Implementation Outline

1. Data Model Changes
   • Chunk.speaker_id default "narrator".
   • PendingJob & Job store speakers metadata (dictionary of speaker descriptors).
   • JobResult optionally includes chunk_speakers.json artifact for downstream use.
2. UI Adjustments
   • On upload form (index.html / JS), add selector for speaker mode.
   • If “Multi-Speaker” chosen, display tooltip/modal: “Coming soon; please choose Single Speaker to continue.” disable submission.
   • In prepare_job.html, display speaker info column (read-only for now).
3. Serialization
   • Update JSON API routes to include speaker data.
   • Update queue/job detail templates to show chunk level & speaker summary.

Testing

• Add web route tests ensuring multi-speaker path blocks progression.
• Verify job persistence includes speaker_id fields.

Feature 4 – Speaker Detection Strategies

Objectives

Build groundwork for lightweight, deterministic speaker inference to inform future multi-speaker mode.

User Stories

1. As a producer, I can run an automated analysis on a book to see the list of likely speakers and how often they talk, so I can decide where multiple voices make sense.
   • Acceptance: System outputs a JSON report containing speaker IDs/names, occurrence counts, representative excerpts, and confidence tier. Report stored with job artifacts and downloadable from job detail page.
2. As a producer, I can set a minimum occurrence threshold so that infrequent speakers automatically fall back to the narrator voice.
   • Acceptance: Analysis respects configurable threshold; speakers below it are tagged as default_narrator in the report.
3. As a developer/operator, I can trigger the analysis via CLI or background task without blocking the main conversion pipeline.
   • Acceptance: Command abogen analyze-speakers <input> (or background queue hook) runs in isolation, returns exit code 0 on success, emits metrics/logs for CI.

Strategy Ideas

1. Quotation-bound heuristic
   • Split paragraphs on dialogue quotes.
   • Use verb cues ("said", "asked") to associate names preceding/following quotes.
2. Name detection via NER
   • Use spaCy’s entity recognition to spot PERSON entities inside dialogue spans.
   • Maintain frequency counts per name.
3. Speaker dictionary
   • Pre-build mapping of common narrator cues ("he said", "Mary replied") to propagate speaker assignment across adjacent sentences.
4. Pronoun fallback with gender hints
   • Map pronouns to most recent speaker mention; degrade gracefully when ambiguous.
5. Thresholding mechanism
   • After counting occurrences, expose a threshold slider (future UI) to decide when to allocate unique voices vs. default narrator.
6. Diagnostics
   • Provide summary report: top N speaker candidates, counts, unresolved dialogue segments.

Implementation Staging

1. Phase 1 – Analysis Engine (Backend)
   • Build speaker_analysis.py module implementing heuristics, returning structured results.
   • Add CLI entry point abogen-speaker-analyze for standalone runs.
   • Persist analysis artifacts (speakers.json, speaker_excerpts.csv) alongside job data when invoked post-extraction.
   • Tests: unit tests for heuristic functions; snapshot tests for sample novels.
2. Phase 2 – Configuration & Thresholding
   • Extend settings UI with optional “speaker analysis threshold” control (numeric).
   • Update analysis module to accept threshold; mark low-frequency speakers as narrator.
   • Emit summary digest (top speakers, narrator fallback count) in job logs.
3. Phase 3 – UI Surfacing
   • Display analysis summary on job detail page (charts/table).
   • Offer download link for raw JSON/CSV artifacts.
   • Provide warning banner when analysis confidence is low (e.g., high unmatched dialogue percentage).
4. Phase 4 – Integration Hooks
   • Wire analysis output into chunk speaker assignments (without yet enabling multi-speaker playback).
   • Store mapping in Job.speakers metadata for future voice routing.

Technical Notes

• Reuse spaCy en_core_web_sm for entity recognition; allow pluggable models per language.
• Maintain rolling context window to resolve pronouns (e.g., last two named speakers).
• Provide instrumentation (timings, counts) to assess heuristic accuracy on sample corpora.
• Design analysis output schema versioning (speaker_analysis_version) to support iterative improvements.

UI & Configuration Updates

Screen	Update
Upload form (index.html)	Add chunking level selector and speaker mode buttons.
Prepare job (prepare_job.html)	Display chunk level, IDs, speaker column; allow future editing hooks.
Settings modal	Persist defaults for chunking level and speaker mode.

Data Model Checklist

• Update PendingJob and Job dataclasses with chunk_level, chunks, speakers metadata.
• Ensure serialization persists these fields in queue state file.
• Persist chunk timing metadata from TTS (start/end timestamps).

Testing Strategy

• Unit tests for chunker and speaker heuristics.
• Integration tests: enqueue job with sentence-level chunking, assert chunk IDs and speaker metadata.
• Regression tests: ensure existing paragraph-level jobs still succeed.
• Acceptance tests for EPUB exporter: validate manifest, spine, and SMIL structure against schema (use epubcheck in CI if feasible).

Migration & Compat

• Bump state version in ConversionService when augmenting job schema; include migration logic for legacy queues.
• Provide CLI flag to reprocess older jobs without speaker metadata.
• Document new dependencies (e.g., lxml, optional spaCy models for languages beyond English).

Implementation Phases

1. Foundation – Introduce chunk model, chunker service, speaker defaults.
2. Pipeline integration – Update job lifecycle and TTS runner to work with chunks.
3. EPUB exporter – Build packaging module, connect to pipeline.
4. UI polish – Expose settings, guard multi-speaker path, surface diagnostics.
5. Speaker analysis tool – Prototype heuristics and reporting.

Open Questions

• How to handle non-EPUB inputs (PDF/TXT) when exporting EPUB 3? (Possible: generate synthetic XHTML with normalized chapters.)
• Storage impact of embedding per-chunk audio – do we need compression or streaming strategies?
• Internationalization: sentence segmentation quality varies; need language-specific models.

Next Steps

• Review plan with stakeholders for scope confirmation.
• Break down Phase 1 into actionable tickets (chunker, data model migration, UI toggle).
• Estimate resource requirements for EPUB packaging and testing (including epubcheck integration).