Compare commits

Author SHA1 Message Date
Juraj Borza 268de365f5 Update README.md to include Supertonic TTS in PyQt6 Desktop GUI features - Supertonic 3 support
Fixes #163
2026-05-25 20:45:28 +02:00
Deniz Şafak e193a686c6 Fix syntax errors in supertonic_total_steps retrieval in service.py and new_job_step_book.html 2026-05-24 15:27:23 +03:00
Deniz ŞafakandGitHub e9ce1942fe Merge pull request #166 from jborza/feature/supertonic-3
Feature/supertonic 3
2026-05-24 15:04:10 +03:00
Juraj Borza 0f0cd86dd4 Add onnxruntime-gpu installation for Supertonic GPU acceleration - #163 2026-05-24 13:32:49 +02:00
Juraj Borza ef0af3e0d7 fixed supertonic voice selection for preview - #163 2026-05-24 13:16:05 +02:00
Juraj Borza 8163666dd9 increased supertonic max chunk length so longer sentences won't be split #163 2026-05-24 13:09:23 +02:00
Juraj Borza 9cfe799a8e supertonic picker for web version - #163 2026-05-24 11:11:23 +02:00
Juraj Borza 502a2d5a2e set default steps to 8, updated supertonic rate from 24000 to 44100 - #163 2026-05-24 11:11:05 +02:00
Juraj Borza 08bcb3f492 Refactor flag icon handling and update language code mappings for SuperTonic and Kokoro 2026-05-24 08:07:05 +02:00
Juraj Borza 81937b7a40 Add Supertonic TTS provider support and configuration options 2026-05-23 21:12:05 +02:00
Deniz ŞafakandGitHub 9fa81fbe1e Merge pull request #160 from JoaGamo/main
Fix #152 : preview buttons on webUI
2026-04-30 13:05:44 +03:00
JoaGamo 9fd9fad238 Fall-back to CPU if no compatible device is available 2026-04-21 22:50:59 -03:00
Deniz ŞafakandGitHub ca5c5ee62d Merge pull request #146 from olandir/131wVoiceTags
Voice Tags and Word Substitution Added to Main Script
2026-03-07 01:48:59 +03:00
olandir e51be95bc1 Update .gitignore 2026-02-28 21:26:57 -05:00
olandirandClaude Sonnet 4.6 2223f46c9e Port voice marker and word substitution features to upstream refactored structure
The upstream project moved PyQt code to abogen/pyqt/ subdirectory, making the
original feature commits non-mergeable. This commit re-applies both features
to the new file locations.

Voice Marker feature (<<VOICE:voice_name>> syntax):
- subtitle_utils.py: Added _VOICE_MARKER_PATTERN, _VOICE_MARKER_SEARCH_PATTERN,
  validate_voice_name(), split_text_by_voice_markers() (with valid/invalid counts)
- pyqt/conversion.py: Added load_voice_cached(), voice marker pre-processing before
  chapter loop, inner voice segment loop wrapping spaCy+TTS block, updated imports
- pyqt/gui.py: Added Insert Voice Marker button and insert_voice_marker() to TextboxDialog

Word Substitution feature (text preprocessing before TTS):
- word_substitution.py: New module (word replacements, ALL CAPS, numerals, punctuation)
- pyqt/conversion.py: apply_word_substitutions() call after clean_text()
- pyqt/gui.py: WordSubstitutionsDialog, word_sub_combo, Settings button,
  on_word_sub_changed(), show_word_sub_dialog(), config persistence, queue restore
- pyqt/queued_item.py: 6 new word substitution fields
- pyqt/queue_manager_gui.py: 6 fields added to OVERRIDE_FIELDS and get_current_attributes()

Note: num2words>=0.5.13 was already added to pyproject.toml by upstream.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 21:19:39 -05:00
Deniz ŞafakandGitHub 8322f7f416 Merge pull request #128 from vladimir-sol/fix-add-missing-pauses
Ensure appropriate speech pauses by adding newlines at epub processing
2026-02-19 14:57:06 +03:00
Vladimir Sol 2c15d2f78a Ensure appropriate speech pauses by adding newlines at epub processing 2026-02-17 19:59:03 -08:00
Deniz ŞafakandGitHub cc9c2a22ba Update GitHub Sponsors usernames in FUNDING.yml 2026-02-10 16:14:05 +03:00
Deniz ŞafakandGitHub d1366b445d Merge pull request #139 from abenea/crash
Fix importing chapters in the PyQt UI.
2026-02-08 17:51:56 +03:00
Andrei Benea c224cdbb56 Fix importing chapters in the PyQt UI.
The app was crashing after importing a .txt and clicking convert because of a missing import. Fixed the imports and removed the legacy abogen.conversion module which doesn't seem necessary anymore.
2026-02-08 11:01:31 +01:00
Deniz Şafak d30415ffe7 Update project version from 1.3.0 to 1.3.1. 2026-02-07 00:23:08 +03:00
Deniz ŞafakandGitHub 083f1eb09b Update CHANGELOG for version 1.3.0
Removed unreleased section and updated version 1.3.0 details.
2026-02-07 00:04:03 +03:00
Deniz ŞafakandGitHub 30929e8f4e Format pyproject.toml 2026-02-07 00:03:20 +03:00
Deniz ŞafakandGitHub ded73843c9 Merge pull request #136 from denizsafak/webui
Merge webui with main

Huge thanks to @jeremiahsb!!
2026-02-06 23:46:03 +03:00
7b1f4f54ee Update README.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-06 23:45:13 +03:00
8cbfc15028 Update CHANGELOG.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-06 23:44:52 +03:00
Deniz Şafak 79957d09ed Fix c10.dll issue in installer script 2026-02-06 23:36:32 +03:00
Deniz Şafak 925ef51edc Add webui photo 2026-02-06 23:26:16 +03:00
Deniz Şafak be7c7baa75 Update readme 2026-02-06 23:20:04 +03:00
Deniz Şafak 6ab863e65f Update README and version 2026-02-06 23:09:49 +03:00
Deniz Şafak 950eb317f0 Fixed all pytest errors, enhanced book parser with context management and resource cleanup, update tests for proper parser closure 2026-01-10 00:09:04 +03:00
Deniz Şafak bd30939d27 Fix "span_text not defined" warning 2026-01-09 22:39:49 +03:00
Deniz ŞafakandGitHub ace1022da4 Merge pull request #126 from mohangk/webui
Further refactoring of book_handler logic into PDFParser
2026-01-09 20:50:33 +03:00
Mohan Krishnan eb7d135bc3 Further refactoring of book_handler logic into PDFParser
Also deletes the unnencessary abogen/book_handler and just use pyqt/book_hanlder
2026-01-09 08:44:33 +08:00
Deniz Şafak 5ae153f841 Reformat using black 2026-01-09 01:36:14 +03:00
Deniz Şafak 3c800df450 Performance improvements for voice mixer 2026-01-09 01:30:38 +03:00
Deniz Şafak fc0af420c6 Handle empty voice list initially 2026-01-09 01:19:39 +03:00
Deniz Şafak 7677f5a1e2 Update installation instructions in README to use editable mode for development 2026-01-08 01:30:43 +03:00
Deniz ŞafakandGitHub 0e726b97c4 Merge pull request #120 from jeremiahsb/main
feat: Supertonic TTS integration, Docker/GPU enhancements, PyQt support, and comprehensive test suite
2026-01-08 01:03:35 +03:00
JB b5b157879f Resolve book_handler conflict: keep PyQt in pyqt/ 2026-01-06 16:40:22 -08:00
JB 27041c6475 Merge branch 'feature/additive-merge-webui' into integrate/webui-into-main-20260106-1629 2026-01-06 16:32:03 -08:00
JB 7df4345486 Merge upstream/main into feature branch
Resolved conflicts:
- CHANGELOG.md: Kept Unreleased section + upstream version history
- README.md: Kept web-first documentation rewrite
- abogen/Dockerfile: Removed (replaced by webui/Dockerfile)
- abogen/{book_handler,conversion,gui,queue_manager_gui,voice_formula_gui}.py: Kept stubs
- abogen/main.py: Kept web UI launcher
- abogen/utils.py: Kept optional chardet imports
- pyproject.toml: Merged dependencies (Flask, gpustat, httpx + pip from upstream)
2025-12-22 12:39:05 -08:00
JB 5b174737ad Merge upstream/main into main
Resolved conflicts:
- CHANGELOG.md: Kept Unreleased section + upstream version history
- README.md: Kept web-first documentation rewrite
- abogen/Dockerfile: Removed (replaced by webui/Dockerfile)
- abogen/{book_handler,conversion,gui,queue_manager_gui,voice_formula_gui}.py: Kept stubs (implementations in pyqt/)
- abogen/main.py: Kept web UI launcher
- abogen/utils.py: Kept version with optional chardet imports
- pyproject.toml: Merged dependencies (kept Flask, gpustat, httpx for web UI + pip from upstream)
2025-12-22 11:50:47 -08:00
JB 1219b408b5 feat: Update Dockerfile and entrypoint script for CUDA diagnostics and web server startup; adjust PyTorch version and URL handling 2025-12-22 08:54:03 -08:00
JB b2058ef3ee fix: Adjust URL handling in CalibreOPDSClient to support base URLs without trailing slashes 2025-12-22 07:27:17 -08:00
JB d1e0e0a536 feat: Refactor Audiobookshelf and Calibre OPDS integration routes to use helper functions for settings extraction 2025-12-22 06:56:25 -08:00
JB 4272c847f7 feat: Add network mode configuration for Docker in .env and docker-compose 2025-12-22 06:42:00 -08:00
JB 89645682f1 fix: Use abogen-web command in Docker for headless web UI
The 'abogen' command now launches the PyQt6 desktop GUI which requires
display libraries not available in Docker containers. Changed the Docker
CMD to use 'abogen-web' which launches the Flask web UI.
2025-12-22 06:17:40 -08:00
JB e2b2f610a6 feat: Additive merge of webui branch with PyQt GUI support
- Add abogen/pyqt/ package with full PyQt6 desktop GUI
- Restore PyQt GUI files (gui.py, book_handler.py, queue_manager_gui.py, voice_formula_gui.py, conversion.py)
- Add new shared modules from webui: book_parser.py, subtitle_utils.py, spacy_utils.py
- Add Linux libraries (libxcb-cursor) for Qt platform plugin support
- Add new epub parsing tests from webui branch
- Update pyproject.toml with dual entry points:
  - abogen: PyQt6 desktop GUI
  - abogen-web: Flask Web UI
- Add PyQt6 to dependencies
- Re-export PyQt classes from root modules for backwards compatibility
- Merge CHANGELOG.md entries (1.2.0-1.2.5 from webui)
- Update README.md with dual interface documentation

Implements #26 - shared core with separate UI folders
2025-12-22 05:51:21 -08:00
Deniz ŞafakandGitHub 76cab1192c Merge pull request #119 from mohangk/main
Extracts book_parser.py from book_handler.py
2025-12-22 11:23:24 +03:00
Mohan Krishnan 1fdd1c8540 Extracts book_parser.py from book_handler.py
Extracts out a BaseBookParser, PDFParser and MarkdownParser class from book_handler logic

The basic contract is that the parser classes populate 4 attributes that are used by the book_handler logic
- content_texts
- content_lengths
- book_metatdata
- processed_nav_structures (used by the method get_chapters on the BaseBookParser class)

Adds tests to validate the changes as well in tests/
Run tests from the abogen dir as follows

```bash
python -m unittest discover
```
2025-12-22 13:32:40 +08:00
JB ede5343e0c feat: Update job detail and logs routes to return friendly pages instead of 404 errors 2025-12-21 16:21:45 -08:00
JB 63d179ba19 feat: Update GPU configuration to prioritize CUDA and remove TensorRT dependency 2025-12-21 09:04:19 -08:00
JB 90786d2bed feat: Update Supertonic GPU configuration to include loader patching for ONNX providers 2025-12-21 08:58:51 -08:00
JB c2209eeb2a feat: Add GPU configuration for Supertonic to enable acceleration support 2025-12-21 08:51:05 -08:00
JB 899c9f5aa5 feat: Add USE_GPU argument to Dockerfile and docker-compose for GPU support 2025-12-21 08:31:51 -08:00
JB 938e122166 feat: Refactor code structure to move web-related components to the webui module and update references accordingly 2025-12-21 08:12:48 -08:00
JB 5303dcf681 Refactored code to move into the webui folder in order to prep for merging the branch. 2025-12-21 08:06:15 -08:00
JB b47952b857 feat: Enhance series metadata handling in Calibre OPDS integration and Epub extraction 2025-12-20 20:22:05 -08:00
JB 0fe8ee0ad4 feat: Add pending ID handling and pronunciation/manual overrides to audio preview generation 2025-12-20 17:40:05 -08:00
JB 5dd53354a1 feat: Add _chunk_text_for_tts function to prioritize raw text for TTS synthesis and implement related tests 2025-12-20 17:18:46 -08:00
JB eb57744533 feat: Implement unsupported character handling in Supertonic pipeline and add related tests 2025-12-20 12:18:27 -08:00
JB de8debb6b1 feat: Add support for saved speaker references in voice selection and enhance related tests 2025-12-20 09:11:35 -08:00
JB e47536e2ab feat: Enhance voice resolution handling and support mixed-provider conversions 2025-12-20 09:09:37 -08:00
JB 32f616e90b feat: Improve SuperTonic voice resolution and add tests for voice formula handling 2025-12-20 08:35:33 -08:00
JB 19e98c3ad6 feat: Add subtitle support in OPDSEntry and enhance metadata handling in API routes 2025-12-20 08:27:47 -08:00
JB eabfe87ffb feat: Enhance Supertonic voice mixer controls with range inputs and display labels 2025-12-20 07:36:34 -08:00
JB 9a72c209b3 feat: Enhance provider picker modal handling and improve action event delegation 2025-12-20 07:19:02 -08:00
JB 888c737293 feat: Add provider selection modal and enhance voice profile handling for Supertonic integration 2025-12-20 07:12:19 -08:00
JB 77eb58bdff Refactor voice profiles to support Supertonic integration
- Introduced normalization functions for Supertonic voice profiles.
- Updated profile serialization to include provider information.
- Enhanced API endpoints to handle Supertonic profiles, including import/export functionality.
- Modified settings to allow selection of default speakers and removed deprecated options.
- Updated front-end to manage speaker profiles, including UI changes for Supertonic settings.
- Improved handling of voice mixing and preview functionalities for both Kokoro and Supertonic providers.
2025-12-20 06:49:13 -08:00
JB 95d5921e67 feat: Integrate Supertonic TTS provider with configuration options and UI updates; enhance voice handling and settings management 2025-12-20 06:20:33 -08:00
JB 08ebedc177 feat: Update year normalization logic to reflect US-style pronunciation for years 1100-1999; adjust related tests for consistency 2025-12-16 08:52:03 -08:00
Deniz ŞafakandGitHub 1c33a99554 Merge pull request #115 from cedarhillgoods/patch-1
Fixed typo (termminal > terminal)
2025-12-16 02:46:29 -08:00
JB 015435adb6 feat: Enhance text normalization with support for internet slang expansion, currency formatting, and date handling; update debug WAVs interface and settings 2025-12-15 17:27:08 -08:00
JB 0afaaf561b feat: Implement voice profile resolution in debug TTS and add corresponding tests 2025-12-15 16:28:30 -08:00
JB fdf95dbae7 feat: Implement language aliasing in debug TTS and add debug WAVs page with artifact handling 2025-12-15 16:17:55 -08:00
JB 05d7c28128 feat: Expand debug TTS samples with additional cases and implement validation test for minimum sample counts 2025-12-15 15:41:41 -08:00
JB aa71783e5a feat: Add debug TTS functionality with EPUB generation, WAV artifact creation, and settings integration 2025-12-15 13:46:13 -08:00
cedarhillgoodsandGitHub 0c8a7cdf81 Fixed typo (termminal > terminal) 2025-12-15 12:05:10 -08:00
JB daf4d78766 feat: Implement heteronym handling with extraction, UI integration, and processing logic 2025-12-15 06:41:13 -08:00
Deniz Şafak 66a97835f0 Update README.md and pyproject.toml for improved installation instructions and development dependencies 2025-12-13 23:03:51 +03:00
Deniz Şafak 0d097f562a Update installation instructions for AMD GPUs and CUDA versions in README.md 2025-12-13 21:02:29 +03:00
Deniz Şafak a7ae13558c Add missing load_config import to subtitle_utils.py 2025-12-13 20:32:33 +03:00
Deniz Şafak 5735f05be0 Improve WINDOWS_INSTALL.bat to support version selection and install dependencies using uv 2025-12-13 20:27:48 +03:00
Deniz ŞafakandGitHub c3673b6dc7 Merge pull request #113 from mohangk/refactor
Extracts out subtitle_utils from conversion.py
2025-12-13 08:19:10 -08:00
Your Name 675ed437c8 Extracts out subtitle_utils from conversion.py
The goal of this change is to move any non PyQt
related logic from conversion.py into its own
subtitle_utils.py. Together with this change there
was an opportunity to pull together some duplicate
text processing that was also in utils.py
2025-12-13 17:08:22 +08:00
Deniz ŞafakandGitHub 5010616e85 Add PyPI total downloads badge to README
Added a badge for total downloads from PyPI.
2025-12-12 04:33:39 +03:00
Deniz Şafak 65eecbcfee Update installation instructions in README for Silicon and Intel Mac support 2025-12-10 23:50:43 +03:00
Deniz Şafak c9b17b6f49 Update installation commands in README to include index strategy for CUDA and ROCm 2025-12-10 05:59:58 +03:00
Deniz Şafak 7722863435 Update installation commands in README to include extra index URLs for CUDA and ROCm 2025-12-10 05:40:16 +03:00
Deniz Şafak 6a1928483b Fix numpy error 2025-12-10 04:22:30 +03:00
Deniz Şafak 2a20dfab6f Update readme 2025-12-10 04:19:12 +03:00
Deniz Şafak a2f186e7a1 Update installation command in README to include Kokoro dependency 2025-12-10 04:17:18 +03:00
JB ef2b045b69 feat: Enhance number normalization logic to distinguish between addresses and years 2025-12-02 12:21:46 -08:00
JB 196e2cdf2e feat: Update metadata handling in new job step book template for improved data access 2025-12-02 11:33:33 -08:00
JB a501e96b12 feat: Add metadata fields for title, subtitle, author, series, publisher, and description in book form 2025-12-02 10:51:51 -08:00
JB 1e13c901fe feat: Implement normalization settings UI with dynamic groups and options 2025-12-02 09:58:47 -08:00
JB 609af66748 feat: Enhance currency normalization to support magnitude and fractional amounts 2025-12-02 07:43:08 -08:00
JB 2e1e9af995 feat: Implement migration from legacy SQLite database to JSON for pronunciation overrides 2025-12-02 06:55:24 -08:00
JB 40eb294fec feat: Refactor pronunciation storage from SQLite to JSON for improved simplicity and performance 2025-12-02 06:43:08 -08:00
JB b502ff9068 feat: Update entities route to use '/overrides' prefix and enhance filtering options for pronunciation overrides 2025-12-02 05:15:24 -08:00
JB a14469c1d0 feat: Add get_override_stats function to retrieve statistics for pronunciation overrides 2025-12-01 20:08:01 -08:00
JB aa53438acf feat: Add normalization settings for currency conversion and footnote removal in UI 2025-12-01 20:00:56 -08:00
JB baef01ad45 feat: Add analysis and override fields to Job class for enhanced processing capabilities 2025-12-01 19:27:49 -08:00
JB 7eceb601f8 feat: Implement footnote removal and URL normalization in text processing; enhance manual override token normalization 2025-12-01 06:23:30 -08:00
JB b3aaa94831 feat: Add estimated time remaining display and duration formatting to job progress cards 2025-12-01 05:26:50 -08:00
JB c7dd8cf13a feat: Add job statistics overview to dashboard with styling enhancements 2025-11-30 15:28:37 -08:00
JB 7db1779ca5 fix: Enhance author handling in metadata payload and add currency conversion support in normalization 2025-11-30 15:13:51 -08:00
JB 76b3aae341 fix: Update _normalize_metadata_casefold and _first_nonempty to handle various data types 2025-11-30 12:36:58 -08:00
JB 07c78255e8 feat: Enhance entity tabs with global loading spinner and status text for improved user feedback 2025-11-30 11:55:32 -08:00
JB 040bce1bc1 fix: Implement garbage collection in run_conversion_job to prevent memory accumulation and add resource limits in docker-compose for better performance 2025-11-30 05:37:40 -08:00
JB 540b191d5b fix: Simplify audiobookshelf availability check and improve error handling in prepare.js 2025-11-30 05:31:25 -08:00
JB 079c36702e fix: Update redirect keys in wizard functions and add next_step to form data 2025-11-29 19:25:19 -08:00
JB 59c8568348 fix: Include pending_id in redirect URL and form data when available 2025-11-29 12:47:51 -08:00
JB f2ba8f692c fix: Enhance wizard_upload to handle pending jobs and new file uploads 2025-11-29 12:17:56 -08:00
JB 2e1f38a98b fix: Update redirect step in api_calibre_opds_import to point to 'book' instead of 'chapters' 2025-11-29 12:16:09 -08:00
JB 3d073e8e55 fix: Rename redirect key to redirect_url in api_calibre_opds_import response 2025-11-29 12:09:06 -08:00
JB cc641cec78 refactor: Remove redundant checks for activeLetter in find_books.js 2025-11-29 12:00:47 -08:00
JB aa45228844 refactor: Replace feed_to_dict with feed.to_dict in CalibreOPDSClient integration 2025-11-29 11:49:17 -08:00
JB 6cc2b4e8a4 feat: Enhance CalibreOPDSClient with improved search scoring and OpenSearch template fetching 2025-11-29 10:21:23 -08:00
JB 252be6d4b7 feat: Add text normalization for improved search functionality in CalibreOPDSClient 2025-11-29 06:02:04 -08:00
JB bce1419d92 feat: Enhance CalibreOPDSClient to start feed from the first link if available 2025-11-29 05:57:03 -08:00
JB b0a3e1dbd9 feat: Enhance search functionality in CalibreOPDSClient to support contextual search with start_href 2025-11-29 05:54:13 -08:00
JB 8a2bd7ec50 feat: Add Calibre OPDS feed and import endpoints for enhanced integration 2025-11-29 05:39:11 -08:00
JB a5f2bf7fbe feat: Refactor integration settings loading to use environment variable fallbacks for Calibre OPDS 2025-11-28 20:16:34 -08:00
JB 4bfce0f900 feat: Update Calibre integration to use 'calibre_opds' for settings retrieval 2025-11-28 19:49:41 -08:00
JB 5b548cc155 feat: Implement Calibre OPDS import endpoint for downloading books 2025-11-28 19:43:13 -08:00
JB c69e5af54d feat: Improve integration settings loading by ensuring proper mapping and type validation 2025-11-28 19:03:33 -08:00
JB 48e9534b68 feat: Preserve password and API token in integration settings loading 2025-11-28 18:57:53 -08:00
JB a9489dec2d feat: Update Audiobookshelf and Calibre OPDS integration settings with new parameters and improved handling 2025-11-28 18:28:36 -08:00
JB 94983c39bb feat: Enhance Audiobookshelf integration with improved error handling and add Calibre OPDS test route 2025-11-28 18:12:33 -08:00
JB 963b020c0f feat: Add normalization settings for contraction and year style options 2025-11-28 17:50:59 -08:00
JB 83e0841274 feat: Refactor settings page routing to handle form submissions and improve code structure 2025-11-28 17:27:09 -08:00
JB 726decfaf4 feat: Update form action to correctly submit settings changes 2025-11-28 16:55:11 -08:00
JB 319037fe8c feat: Add integration settings loading to settings page 2025-11-28 16:38:31 -08:00
JB c00cbaef69 feat: Enhance entity and job management with new routes and template updates 2025-11-28 16:00:16 -08:00
JB 124e5b33db feat: Update wizard routing and improve pending job handling in modal 2025-11-28 15:31:45 -08:00
JB fd93e1c9e9 feat: Add entity pronunciation preview API and enhance job management pages 2025-11-28 15:20:22 -08:00
JB d08cbcfdc9 Add voice management functionality and voice synthesis preview
- Implemented voice management routes in `voices.py` for listing, saving, and deleting speaker configurations.
- Added a test endpoint for voice synthesis preview, allowing users to test voice settings with provided text and speed.
- Introduced utility functions in `voice.py` for building voice catalogs, resolving voice settings, and synthesizing audio from normalized text.
- Enhanced speaker roster building and configuration application logic to support dynamic voice settings.
2025-11-28 14:57:23 -08:00
JB 0a2b3533f4 feat: Improve logging error handling in job processing to capture failures in stderr 2025-11-28 13:44:15 -08:00
JB ad70c630c7 feat: Enhance security by adding user permissions in Dockerfile, parameterized queries in pronunciation_store, and secure filename handling in routes 2025-11-28 13:19:53 -08:00
JB 39628453de feat: Update Audiobookshelf timeout settings to 3600 seconds for improved upload reliability 2025-11-28 13:00:58 -08:00
JB 7cc60aacf6 feat: Add bottom navigation for OPDS modal and enhance styling for better visibility 2025-11-28 12:13:37 -08:00
JB 775e05f94c feat: Enhance heading processing by stripping numeric prefixes from titles 2025-11-27 19:41:40 -08:00
JB 550fdbf537 feat: Enhance normalization settings with additional options and UI elements 2025-11-27 10:49:29 -08:00
JB d0532347be feat: Improve query matching in OPDS entries for enhanced search functionality 2025-11-12 08:38:06 -08:00
JB 3ef95900cc feat: Enhance OPDS client with pagination support for search and browsing by letter 2025-11-12 06:12:05 -08:00
JB 71dfbd49b6 feat: Add browsing functionality by alphabet letter in Calibre OPDS client 2025-11-11 18:44:16 -08:00
JB 166162931c feat: Implement local search functionality with alphabet filtering in OPDS feed 2025-11-11 18:09:58 -08:00
JB aba72524ce feat: Implement filtering of OPDS feed entries and preserve navigation links 2025-11-11 17:37:11 -08:00
JB 028384e6ee feat: Add stubs for soundfile and static_ffmpeg modules to facilitate testing 2025-11-11 17:00:58 -08:00
JB 504b5ab5e5 feat: Enhance entity management with filtering and preview functionality 2025-11-11 09:57:44 -08:00
JB 95f0307be1 feat: Add handling for author-series name collisions in Audiobookshelf metadata 2025-11-11 07:56:51 -08:00
JB 67453ed17c feat: Implement filtering of unsupported download formats in CalibreOPDSClient 2025-11-11 07:29:13 -08:00
JB 4491674fac feat: Enhance URL handling in CalibreOPDSClient to preserve catalog prefix for relative paths 2025-11-11 06:57:27 -08:00
JB 387d324ea0 feat: Add year pronunciation handling and roman numeral normalization to text processing 2025-11-11 05:25:29 -08:00
JB acede32559 feat: Add series sequence handling and normalization for Audiobookshelf metadata 2025-11-03 05:20:05 -08:00
JB 4e06601b04 feat: Add support for decimal number normalization in text processing 2025-11-03 05:10:02 -08:00
JB f2ab62aeab feat: Enhance Audiobookshelf metadata handling to support additional series and book number tags 2025-11-02 07:04:29 -08:00
JB 72b2ada9ac feat: Implement existing item lookup and overwrite handling for Audiobookshelf uploads 2025-11-02 06:25:14 -08:00
JB af90da0b99 feat: Add contraction handling options and UI for normalization settings 2025-11-02 05:47:20 -08:00
JB d238524cbb feat: Enhance Calibre OPDS integration to extract and handle tags, ratings, and publication dates from metadata 2025-11-01 07:27:56 -07:00
JB 2d61674e90 feat: Implement folder browsing modal for Audiobookshelf integration and enhance folder selection UI 2025-11-01 06:49:43 -07:00
JB 28652d9e78 feat: Add folder browsing functionality to Audiobookshelf integration and enhance related settings UI 2025-11-01 06:07:29 -07:00
JB 1d9dcf8800 feat: Update Audiobookshelf integration to support folder name or ID input and enhance related documentation 2025-10-31 14:41:07 -07:00
JB 5322c6406d feat: Implement Folder ID support in Audiobookshelf integration and update related documentation 2025-10-31 14:12:08 -07:00
JB 1d4316ad18 feat: Add Folder ID support for Audiobookshelf integration in settings and configuration 2025-10-31 12:49:46 -07:00
JB 8aaf183f75 feat: Update base URL handling in Audiobookshelf client for consistent API requests 2025-10-31 05:12:49 -07:00
JB eb28b01c06 feat: Refactor API path handling in Audiobookshelf client for improved endpoint management 2025-10-30 15:54:10 -07:00
JB 65ec77180d feat: Normalize Audiobookshelf base URL input and update placeholder instructions 2025-10-30 15:20:54 -07:00
JB 0906b590a6 feat: Enhance series metadata extraction from categories in OPDS feed 2025-10-30 15:15:47 -07:00
JB b9e3fb5e1c feat: Add series index and label formatting for improved entry metadata display 2025-10-30 12:54:05 -07:00
JB fe5419565d feat: Add support for series metadata and closing outro in audio conversion 2025-10-30 12:43:24 -07:00
JB 688d550f13 feat: Add outro text generation and voice resolution for audio conversion jobs 2025-10-30 12:22:19 -07:00
JB c7356338c2 feat: Implement contraction resolution using spaCy for ambiguous contractions 2025-10-30 09:36:23 -07:00
JB 13c6b120c9 feat: Add position handling in OPDSEntry and update navigation button behavior in the UI 2025-10-30 08:27:21 -07:00
JB 9acec3c309 fix: Refactor regex patterns for number ranges and fractions to improve accuracy 2025-10-30 06:54:33 -07:00
JB 013c5c2dbb feat: Enhance find books page with template options and settings integration 2025-10-30 06:35:33 -07:00
JB e7f6f0221d feat: Implement apostrophe normalization settings and overrides in the UI and backend 2025-10-30 06:25:08 -07:00
JB 963ef71647 feat: Add support for numeric ranges and simple fractions in text normalization 2025-10-30 05:34:02 -07:00
JB b8a6ca7091 feat: Update conversion button labels and status messages for clarity 2025-10-29 10:17:15 -07:00
JB cf6ccf0171 feat: Revamp Calibre OPDS integration with improved UI, search functionality, and modal handling 2025-10-29 06:52:23 -07:00
JB eb4704bc26 feat: Enhance OPDS browser with navigation link handling and update UI for Calibre catalog 2025-10-29 06:01:21 -07:00
JB 0fc32e9f3a Add Calibre OPDS and Audiobookshelf integration features
- Implemented functions to handle Calibre OPDS settings and test connections.
- Added Audiobookshelf settings handling and test connections.
- Enhanced the UI to allow users to test connections for both integrations.
- Created new API endpoints for importing books from Calibre OPDS and testing Audiobookshelf connections.
- Updated the find_books.html template to include an OPDS browser for searching Calibre catalogs.
- Added JavaScript functionality for handling OPDS browsing and importing books.
- Updated settings.html to include fields for Audiobookshelf configuration.
- Enhanced job management to allow sending jobs to Audiobookshelf.
2025-10-29 05:15:49 -07:00
JB 67f4493d06 feat: Reset wizard state upon successful redirection 2025-10-28 04:38:45 -07:00
JB de0f17cd57 feat: Add Audiobookshelf and Calibre OPDS integration
- Implemented Audiobookshelf integration for uploading audiobooks with metadata, cover, chapters, and subtitles.
- Added configuration options for Audiobookshelf in the settings page, including base URL, API token, library ID, and upload preferences.
- Introduced Calibre OPDS integration for fetching and downloading resources from a Calibre OPDS catalog.
- Enhanced job processing to include post-completion hooks for Audiobookshelf uploads.
- Updated settings template to include new integration options and fields.
- Added utility functions for metadata normalization and chapter extraction.
- Included HTTP client functionality for both Audiobookshelf and Calibre OPDS interactions.
- Updated dependencies to include httpx for HTTP requests.
2025-10-27 17:10:48 -07:00
JB 6e536c6e3b feat: Implement normalize chapter opening caps feature and update related settings 2025-10-27 15:58:34 -07:00
JB 9d35e39e89 feat: Add support for reading book title and authors before narration 2025-10-26 10:49:51 -07:00
JB a81ed70b14 feat: Update LLM context handling and prompt template for improved clarity and legacy support 2025-10-26 10:04:36 -07:00
JB 7951a4d992 feat: Update LLM context mode to use sentence-level context and enhance prompt for regex replacements 2025-10-26 08:42:41 -07:00
JB 10cd2c993c feat: Add environment variables for voice cache and Hugging Face directory in Docker setup 2025-10-26 08:03:40 -07:00
JB 0259963eb8 feat: Update LLM base URL configuration and enhance model selection logic 2025-10-26 07:57:18 -07:00
JB 6b5255a80d Implement LLM client and normalization settings
- Added LLMClient class for handling requests to LLM API, including methods for listing models and generating completions.
- Introduced LLMConfiguration dataclass for managing LLM configuration settings.
- Created normalization_settings module to manage normalization configurations and environment variable overrides.
- Developed JavaScript functionality for the settings interface, including model fetching and preview generation for LLM and normalization.
- Enhanced user experience with status messages and error handling in the settings UI.
2025-10-26 07:42:12 -07:00
JB 0a6d09445d feat: Add original text preservation in chunking and export processes 2025-10-15 05:41:51 -07:00
JB 6ae8b827d2 feat: Update chapter title formatting and enhance settings layout 2025-10-14 07:37:28 -07:00
JB 6248dfdc0c feat: Enhance chapter navigation and add chapter panel functionality 2025-10-14 06:56:54 -07:00
JB 7ca030d67d feat: Add auto-prefix option for chapter titles and enhance reader functionality
- Introduced `auto_prefix_chapter_titles` setting in Job and PendingJob classes to control prefixing of chapter titles with "Chapter".
- Updated job detail and settings templates to display and configure the new option.
- Enhanced reader.js to manage playback controls, including chapter navigation and playback speed adjustments.
- Implemented a new prepare_chapters.html template for chapter selection and configuration during job preparation.
- Added tests for chapter title formatting and heading equivalence to ensure correct behavior of the new feature.
2025-10-14 06:24:15 -07:00
JB bccfd9f5c5 feat: Enhance spine href resolution and normalization in chapter handling 2025-10-13 17:33:45 -07:00
JB f7541388e6 feat: Simplify EPUB chapter href normalization and enhance spine href logging 2025-10-13 17:20:33 -07:00
JB 62d4acc4d6 feat: Improve EPUB path normalization by deduplicating segments and handling backslashes 2025-10-13 16:07:00 -07:00
JB 0922ad4727 feat: Normalize chapter hrefs by removing leading slashes and enhancing href handling 2025-10-13 15:55:11 -07:00
JB a5543528ba feat: Normalize base directory handling in EPUB path normalization 2025-10-13 15:10:55 -07:00
JB 14cc7103cf feat: Enhance chapter navigation with improved canonical URL handling and deduplication 2025-10-13 15:04:28 -07:00
JB 1f0cfef310 feat: Add dynamic loading of JSZip library with error handling for EPUB fetching 2025-10-13 14:50:03 -07:00
JB 5971630803 feat: Enhance EPUB fetching with improved error handling and status messaging 2025-10-13 14:16:28 -07:00
JB b3dc73cb15 feat: Implement EPUB asset fetching with improved error handling and status messaging 2025-10-13 13:42:13 -07:00
JB 0f549a56e4 feat: Improve EPUB loading status messaging and error handling 2025-10-13 12:58:57 -07:00
JB cab5221c46 feat: Enhance reader toolbar with improved button states and status messaging 2025-10-13 12:30:03 -07:00
JB d114ae60fe feat: Add pronunciation override handling with compilation and application logic 2025-10-13 07:29:29 -07:00
JB 3e54007baa feat: Add status messaging and override management functionality with improved UI elements 2025-10-13 06:57:09 -07:00
JB ded405ff70 feat: Enhance entity manual override handling and improve tagline styling 2025-10-13 06:22:24 -07:00
JB eb78347a85 feat: Enhance manual override functionality with status updates and improved UI elements 2025-10-12 17:36:37 -07:00
JB a2c08b1b4d feat: Add normalization for voice preview and improve voice resolution logic 2025-10-12 17:00:49 -07:00
JB 9ee53fc886 feat: Update entity preview button to use new data attributes and improve accessibility 2025-10-12 12:20:15 -07:00
JB 82d780db0d feat: Enhance entity processing and UI with new filters and manual override options 2025-10-12 11:38:06 -07:00
JB 17534d7890 feat: Add filters for entity and people summaries with minimum mention options 2025-10-12 10:01:43 -07:00
JB b4c9a1ced8 feat: Enhance wizard functionality with initial step reset and loading states 2025-10-12 08:17:56 -07:00
JB 8e9c9e6077 feat: Implement entity recognition settings and UI updates across multiple components 2025-10-12 07:29:28 -07:00
JB 7d11ebc338 refactor: Remove speaker mode handling from various components 2025-10-12 06:59:39 -07:00
JB e6d2649d5d Refactor job preparation templates and routes
- Removed `prepare_chapters.html` and `prepare_entities.html` templates as they are no longer needed.
- Updated `routes.py` to remove references to the removed templates and adjusted job preparation logic.
- Simplified response handling in job preparation routes by removing unnecessary parameters.
- Consolidated job preparation logic to improve maintainability and clarity.
2025-10-12 06:16:26 -07:00
JB 091a3785e6 fix: Normalize wizard step handling in finalize_job function 2025-10-11 18:54:05 -07:00
JB ec6dc25e23 feat: Add missing JavaScript modules for dashboard functionality 2025-10-11 18:36:04 -07:00
JB f8d2e860fe feat: Add styles for wizard card stages and forms 2025-10-11 17:07:22 -07:00
JB b8b0aea890 Add new job step templates for book, chapters, and entities
- Implemented `new_job_step_book.html` to handle manuscript and subtitle settings, including language selection, subtitle mode, and narrator defaults.
- Created `new_job_step_chapters.html` for managing detected chapters, allowing users to toggle chapters, rename them, and set voice overrides.
- Developed `new_job_step_entities.html` to configure speaker settings, manage entity pronunciations, and handle manual overrides for voice assignments.
- Enhanced user experience with dynamic forms and validation, ensuring smooth navigation through the job creation process.
2025-10-11 16:36:44 -07:00
JB 7203037a42 refactor: Remove unnecessary step indicators and warnings from upload and chapter preparation templates 2025-10-11 14:35:20 -07:00
JB 93fe48f601 feat: Implement pronunciation store with SQLite backend
- Added a new module for managing pronunciation overrides using SQLite.
- Implemented functions to load, save, search, and delete pronunciation overrides.
- Introduced schema for storing overrides and metadata.
- Added thread-safe access to the database with RLock.
- Created a utility for normalizing tokens for consistent storage and retrieval.

refactor: Overhaul entities step in the preparation wizard

- Renamed Step 3 from "Speakers" to "Entities" across all templates and routes.
- Introduced sub-navigation with tabs for "People", "Entities", and "Manual Overrides".
- Enhanced UI to display detected entities and allow manual overrides for pronunciations.
- Implemented search functionality for manual overrides with AJAX support.
- Updated frontend logic to manage tab interactions and voice selections.

docs: Add detailed plan for entities step overhaul

- Documented requirements, implementation strategies, and testing plans for the entities step.
- Outlined the integration of POS tagging and entity recognition using spaCy.
- Provided a comprehensive overview of the manual overrides workflow and data persistence strategies.
2025-10-11 14:14:19 -07:00
JB 610091c0d9 feat: Implement upload modal event dispatching and enhance wizard modal navigation 2025-10-11 12:29:52 -07:00
JB 6266819670 refactor: Update upload modal inclusion to use context blocks for better readability 2025-10-11 12:04:33 -07:00
JB 4a62363064 feat: Enhance upload modal and chapter preparation templates for improved user experience 2025-10-11 11:57:50 -07:00
JB ccf5a11222 Enhance speaker analysis and web templates
- Added gender inference improvements in speaker_analysis.py, including handling of titles and diacritics.
- Updated analyze_speakers function to include sample excerpts with context paragraphs.
- Modified routes.py to skip suppressed speakers in the speaker roster.
- Enhanced prepare.js to manage speaker samples and pronunciation previews more effectively.
- Refined prepare_chapters.html and prepare_speakers.html templates for better navigation and user experience.
- Added tests for speaker analysis to ensure proper handling of stopwords and threshold suppression.
2025-10-11 11:14:52 -07:00
JB 8ed5202ab8 Add chapter and speaker preparation templates for audiobook conversion workflow
- Created `prepare_chapters.html` to allow users to select and configure chapters for conversion, including options for voice profiles and chunk granularity.
- Developed `prepare_speakers.html` for assigning voices to detected speakers, auditioning samples, and applying saved speaker configurations.
- Implemented step indicators and navigation between chapters and speakers in the UI.
- Added error and notice handling for user feedback during the preparation process.
- Included scripts for voice catalog and language mapping to enhance user experience.
2025-10-11 09:31:49 -07:00
JB 4b0aa50da6 feat: Add 'Find Books' page and integrate links to Standard Ebooks and Project Gutenberg 2025-10-11 08:58:14 -07:00
JB 97fd8b85fc feat: Refactor reader modal implementation and enhance dashboard event handling 2025-10-11 07:13:59 -07:00
JB 5be4da1cce feat: Refactor event handling in dashboard for improved modal interactions and event resolution 2025-10-11 06:33:47 -07:00
JB 3e7bbd648c feat: Enhance job handling with download options for M4B, EPUB 3, and improved UI for speaker selection 2025-10-11 06:04:44 -07:00
JB e15d2b12a3 feat: Update Dockerfile to include mutagen dependency and enhance dashboard event handling 2025-10-11 05:41:44 -07:00
JB aed0df1b09 feat: Enhance dashboard and prepare wizard functionality with speaker step handling and UI updates 2025-10-10 19:18:26 -07:00
JB 60fe7e7ffb feat: Enhance upload form layout with flexible display and improved section handling 2025-10-10 18:48:01 -07:00
JB 20327faf0c feat: Enhance file upload experience with drag-and-drop support and improved UI for upload dropzone 2025-10-10 18:07:18 -07:00
JB be37f03109 Add number normalization and enhance UI for voice preview
- Implemented number conversion to words for grouped numbers in text normalization.
- Added configuration options for number conversion in ApostropheConfig.
- Updated the normalize_apostrophes function to include number normalization.
- Enhanced the dashboard UI with new sections for manuscript and narrator defaults.
- Improved voice preview functionality with better handling of audio playback and status updates.
- Refactored HTML templates for cleaner structure and added new fields for speaker settings.
- Updated CSS styles for improved layout and responsiveness.
- Added tests to ensure correct spelling of grouped numbers in the normalization process.
- Included num2words as a new dependency in pyproject.toml.
2025-10-10 16:12:51 -07:00
JB f35b35c7a9 feat: Enhance EPUB export by grouping chunks for rendering and adding group ID support 2025-10-10 14:43:28 -07:00
JB 283651e7dd fix: Simplify HTML rendering in chunk output by removing unnecessary line breaks 2025-10-10 12:50:48 -07:00
JB 4f46eb74f4 feat: Add original text preservation in chunk overlays and enhance whitespace handling in EPUB export 2025-10-10 12:08:22 -07:00
JB 15c1220ba3 feat: Enhance chunking logic to include display text and preserve whitespace in sentences 2025-10-10 11:17:14 -07:00
JB 443dee09b6 feat: Integrate roman numeral normalization in chapter titles and enhance related tests 2025-10-10 09:31:27 -07:00
JB 3a91e79cb6 feat: Enhance text normalization and chunking logic to preserve original whitespace and handle abbreviations 2025-10-10 08:38:00 -07:00
JB 258c3549f7 feat: Enhance reader styling and accessibility with improved layout and focus indicators 2025-10-10 07:42:13 -07:00
JB e7408a06b8 feat: Implement EPUB reader modal with navigation and audio playback features 2025-10-10 06:48:59 -07:00
JB e37ef99856 feat: Add speaker mode selection and skip logic for speakers step in the wizard 2025-10-09 14:20:36 -07:00
JB 01a6267c7a feat: Implement voice fallback logic and enhance voice resolution tests for custom mixes 2025-10-09 13:58:21 -07:00
JB f0b6976d12 feat: Enhance voice formula parsing and validation, implement voice asset caching, and add tests for new functionality 2025-10-09 13:37:36 -07:00
JB 6bd301b707 feat: Add static job logs view and enhance logging functionality with improved UI and error handling 2025-10-09 12:29:38 -07:00
JB 205c435e72 feat: Add retry functionality for jobs and update UI to support job retries 2025-10-09 11:46:38 -07:00
JB 9828730061 feat: Enhance error logging in conversion job and service with detailed traceback information 2025-10-09 11:01:31 -07:00
JB da0c453205 feat: Refactor speaker configuration and UI components, enhance custom mix functionality, and improve styling for better usability 2025-10-09 10:12:11 -07:00
JB 63f9474741 feat: Implement logging filter for successful HTTP access logs and enhance modal styling for better usability 2025-10-09 05:51:45 -07:00
JB b0875c7486 Enhance audiobook workflow UI with modal and styling updates
- Updated styles.css to introduce new modal and card styles for improved layout and responsiveness.
- Modified index.html to implement a modal for uploading files and settings, enhancing user experience.
- Refactored prepare_job.html to support a wizard-like interface for preparing jobs, including step indicators and dynamic content updates.
- Added functionality for gender selection and voice mixing in the speaker preparation section.
- Improved accessibility and usability with better hints and instructions throughout the forms.
2025-10-08 15:15:10 -07:00
JB 881717c4cb Refactor templates for audiobook preparation and settings
- Updated navigation in base.html to include a link to the queue section.
- Modified index.html to change step labels and added a jobs panel for real-time updates.
- Enhanced prepare_job.html with a wizard-style step navigation and improved speaker configuration options.
- Added new fields for speaker analysis settings and chapter options in prepare_job.html.
- Introduced a voice selection modal for better user experience in speaker selection.
- Updated settings.html to allow selection of randomizer languages for speakers.
- Cleaned up speakers.html by removing unnecessary language selection fields and ensuring consistency in speaker configuration.
2025-10-08 12:50:50 -07:00
JB d01887f31b feat: Add speaker configuration management and UI enhancements
- Introduced a new speaker configuration page with the ability to create, edit, and delete speaker presets.
- Added a step indicator to guide users through the audiobook workflow.
- Enhanced the audiobook creation process by allowing users to select speaker presets and configure individual speaker settings.
- Implemented dynamic UI elements for managing speaker rows, including adding and removing speakers.
- Updated existing templates to integrate speaker configuration features and improve user experience.
- Added JavaScript functionality for managing speaker rows and ensuring proper form handling.
- Created a new module for handling speaker configuration data storage and retrieval.
2025-10-08 11:19:52 -07:00
JB 1bae37477b feat: Enhance speaker analysis with gender inference and update related tests 2025-10-08 07:09:12 -07:00
JB 3b07df9708 feat: Add title and suffix abbreviation expansion, ensure terminal punctuation, and enhance speaker analysis functionality 2025-10-08 05:43:49 -07:00
JB b0cfd8d687 fix: Remove unnecessary newline in _reassign function 2025-10-07 18:08:28 -07:00
JB 41f56a8491 feat: Implement speaker analysis and EPUB 3 export functionality
- Added speaker analysis module to infer speaker identities from text chunks.
- Introduced SpeakerGuess and SpeakerAnalysis data classes for managing speaker data.
- Developed functions for analyzing speaker occurrences and confidence levels.
- Created EPUB 3 exporter to generate EPUB packages with synchronized narration and media overlays.
- Implemented configurable chunking options for TTS synthesis and EPUB alignment.
- Enhanced JavaScript for speaker preview functionality in the web interface.
- Added comprehensive tests for chunking and EPUB exporting features.
- Documented upgrade plan for transitioning to EPUB 3 with multi-speaker support.
2025-10-07 17:57:53 -07:00
JB bacf1b2f9e feat: Enhance chapter preselection logic and add scoring for supplemental titles 2025-10-07 15:29:33 -07:00
JB 6181f12bd4 Refactor code to remove legacy UI, transitioning to a straight webapp 2025-10-07 15:04:28 -07:00
JB da56966247 feat: Refactor supplemental section detection and enhance auto-selection logic in HandlerDialog 2025-10-07 14:51:21 -07:00
JB 783dbcf8f2 feat: Add supplemental section detection for improved content selection in HandlerDialog 2025-10-07 14:19:49 -07:00
JB b35ab7b002 feat: Implement dynamic state path determination and migration for queue state file 2025-10-07 13:49:53 -07:00
JB 4b3aa227b7 feat: Update metadata handling to include chapter count and normalize keys for ffmpeg arguments 2025-10-07 11:04:00 -07:00
JB 42334e92a4 feat: Add format specification for m4b, mp4, and m4a files in metadata embedding 2025-10-07 10:40:42 -07:00
JB a51fd25271 feat: Enhance chapter selection and metadata handling in conversion process 2025-10-07 10:23:42 -07:00
JB 02da72434b feat: Implement chapter embedding in m4b files using mutagen and update dependencies 2025-10-07 08:47:18 -07:00
JB 0dd74412d1 feat: Add chapter intro delay setting and implement in conversion process 2025-10-07 07:07:08 -07:00
JB ea1c7bd93e feat: Add ffmetadata rendering and writing functions with tests for chapter inclusion 2025-10-07 06:01:29 -07:00
JB 718a3fa1c0 feat: Enhance job management UI and add apostrophe normalization
- Updated styles for job cards to include a paused state and improved title styling.
- Modified job card template to display job status and progress more clearly, including pause/resume functionality.
- Introduced a new script for managing chapter row states in the prepare job form, allowing for dynamic enabling/disabling of inputs.
- Created a new template for preparing jobs, featuring a summary of metadata and chapter details.
- Added a comprehensive apostrophe normalization module to handle various cases of apostrophe usage in text.
2025-10-07 05:34:53 -07:00
JB 85310ad916 feat: Implement chapter overrides and metadata merging in conversion process
- Added `_coerce_truthy` function to handle truthy value coercion.
- Introduced `_apply_chapter_overrides` to apply chapter modifications based on provided overrides.
- Implemented `_merge_metadata` to combine extracted metadata with overrides, ensuring proper handling of None values.
- Updated `run_conversion_job` to utilize new chapter override and metadata merging functionalities.
- Modified `Job` class to store chapters as dictionaries for better flexibility.
- Enhanced `ConversionService` to normalize chapter input and metadata tags.
- Added comprehensive tests for chapter overrides and metadata merging to ensure functionality and correctness.
2025-10-06 18:05:12 -07:00
JB c8e9eb6fd2 feat: Update audio sink to manage ffmpeg cache by platform 2025-10-06 17:05:50 -07:00
JB 43bee0b76e feat: Enhance audio sink with internal ffmpeg cache directory management 2025-10-06 16:50:40 -07:00
JB fd8ede318f feat: Implement internal cache path management for improved resource handling 2025-10-06 16:34:01 -07:00
JB e76701ab32 feat: Add allow-direct-references setting to hatch metadata 2025-10-06 16:23:57 -07:00
JB 0d74171bb5 feat: Update en-core-web-sm dependency to use direct URL for installation 2025-10-06 16:19:28 -07:00
JB 477c5055b4 feat: Update environment variables and Docker configuration for cache management and temporary directory settings 2025-10-06 16:14:49 -07:00
JB 523e55d8a4 fix: Correct indentation for environment variables in docker-compose.yaml 2025-10-06 15:52:27 -07:00
JB c19050261c feat: Configure cache environment variables for Hugging Face and update Docker settings 2025-10-06 15:50:32 -07:00
JB dc7a115e2e feat: Update environment variables and Docker configuration for improved directory management 2025-10-06 15:26:17 -07:00
JB 153d5ba92c feat: Update environment configuration for Docker to include temporary directory settings 2025-10-06 14:42:54 -07:00
JB 26e0e764db feat: Add UID/GID configuration to .env.example and update README for container user settings 2025-10-06 14:30:55 -07:00
JB 7d132e6fcc feat: Enhance voice resolution and logging in conversion job for improved feedback and performance 2025-10-06 14:04:17 -07:00
JB b75e1c1b2e feat: Refactor queue page to use jobs panel rendering for improved performance 2025-10-06 12:55:44 -07:00
JB fc4c41c7cf feat: Implement job management features with improved UI for active and finished jobs 2025-10-06 12:41:22 -07:00
JB f3aaeda37b feat: Update voice field visibility and enhance CSS for improved layout and accessibility 2025-10-06 11:59:44 -07:00
JB 323ab08f38 feat: Enhance dashboard and styles for improved layout and accessibility 2025-10-06 11:42:15 -07:00
JB 54bc632b2e feat: Improve UI consistency and accessibility across dashboard and settings pages 2025-10-06 10:37:34 -07:00
JB 5497697741 feat: Enhance voice selection and settings UI with default voice option and improved layout 2025-10-06 09:20:33 -07:00
JB 1b907be322 feat: Adjust voice editor layout for improved alignment and spacing 2025-10-06 08:13:05 -07:00
JB e44ba1e903 feat: Enhance voice mixer UI with new preview speed control and improved layout for profile actions 2025-10-06 07:56:42 -07:00
JB 97d81d78e2 feat: Refactor environment loading to use explicit path and find_dotenv for better configuration management 2025-10-06 07:27:04 -07:00
JB 01209f6878 feat: Add gender filter to voice mixer and enhance UI for better user experience
feat: Implement dynamic settings page with form handling and default values
feat: Create a queue page to display ongoing jobs with auto-refresh
feat: Revamp dashboard with live text preview and character/word count
fix: Update navigation links in base template for active state indication
2025-10-06 06:59:16 -07:00
JB 2e402f6b5b feat: Enhance voice mixer functionality with language filtering and improved directory management 2025-10-06 06:00:17 -07:00
JB 0c47067cb8 feat: Revamp voice mixer UI with new layout and enhanced voice management features 2025-10-06 05:32:12 -07:00
JB b718dae1b3 feat: Implement voice mixer UI and functionality
- Added new styles for the voice mixer components in styles.css.
- Updated base.html to include a block for scripts.
- Refactored voices.html to create a structured voice mixer interface with profile management features.
- Introduced voices.js to handle voice mixer logic, including profile creation, editing, and previewing.
- Implemented actions for importing and exporting voice profiles.
- Enhanced user experience with loading states and status messages.
2025-10-06 05:10:32 -07:00
JB 9ba2362528 fix: Correct link to voice mixer in index.html 2025-10-05 16:24:00 -07:00
JB 1629d3e80c feat: Update application to use port 8808 instead of 8000 in README, Dockerfile, app.py, and docker-compose.yaml 2025-10-05 16:18:05 -07:00
JB 66a0679e18 feat: Update Docker configuration for GPU support and remove deprecated compose file 2025-10-05 16:05:16 -07:00
JB 338ff104e8 feat: Implement conversion service with job management and logging
- Added `ConversionService` class to handle job queuing, processing, and cancellation.
- Introduced `Job`, `JobLog`, and `JobResult` data classes to manage job details and results.
- Implemented job status tracking with enums for better state management.
- Created a web interface with HTML templates for job submission and monitoring.
- Developed CSS styles for a modern UI layout and responsive design.
- Added functionality for voice profile management in the voice mixer.
- Implemented a Docker Compose configuration for GPU support.
- Wrote unit tests for the conversion service to ensure job processing works as expected.
2025-10-05 15:53:33 -07:00
201 changed files with 55486 additions and 8831 deletions
+44
View File
@@ -0,0 +1,44 @@
# Copy this file to `.env` and customize the paths to match your environment.
# Relative paths are resolved from the repository root when running locally.
# Each `*_DIR` value below points to a directory on the host. Docker Compose
# mounts it into the container at the standard path noted in the comment.
# Host directory that stores JSON settings. Mounted to /config in Docker.
ABOGEN_SETTINGS_DIR=./config
# Host directory for rendered audio/subtitle files. Mounted to /data/outputs
# in Docker.
ABOGEN_OUTPUT_DIR=./storage/output
# Temporary working directory. When running in Docker, keep this inside the
# mounted data volume (or another writable host path) so non-root users can
# write to it. Only audio conversion scratch files are staged here by default;
# other library caches remain inside the container volume. For local
# (non-Docker) usage, change this to a path that makes sense on your machine or
# comment it out to fall back to the OS cache directory. Mounted to /data/cache
# in Docker.
ABOGEN_TEMP_DIR=./storage/tmp
# UID/GID used when running the Docker container. 1000:1000 matches most Linux hosts.
# Find your current values with:
# id -u # UID
# id -g # GID
ABOGEN_UID=1000
ABOGEN_GID=1000
# Network mode for the Docker container. Options:
# bridge (default) - Isolated container network, uses port mapping
# host - Container uses host's network directly, required for
# accessing LAN resources like Calibre OPDS servers
# ABOGEN_NETWORK_MODE=host
# Optional: Seed the web UI with working defaults for the LLM-powered
# text normalization features. Leave these blank to configure everything
# from the Settings page.
ABOGEN_LLM_BASE_URL=http://localhost:11434 # Supply the server root; /v1 is added automatically.
ABOGEN_LLM_API_KEY=ollama
ABOGEN_LLM_MODEL=llama3.1:8b
ABOGEN_LLM_TIMEOUT=45
ABOGEN_LLM_CONTEXT_MODE=sentence
# For custom prompts, keep the text on a single line or escape newlines.
#ABOGEN_LLM_PROMPT=Provide regex replacements for any apostrophes in {{sentence}} using apply_regex_replacements.
+15
View File
@@ -0,0 +1,15 @@
# These are supported funding model platforms
github: [jborza, jeremiahsb, mohangk]
patreon: # Replace with a single Patreon username
open_collective: # Replace with a single Open Collective username
ko_fi: # Replace with a single Ko-fi username
tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
liberapay: # Replace with a single Liberapay username
issuehunt: # Replace with a single IssueHunt username
lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
polar: # Replace with a single Polar username
buy_me_a_coffee: # Replace with a single Buy Me a Coffee username
thanks_dev: # Replace with a single thanks.dev username
custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']
+6
View File
@@ -19,6 +19,7 @@ __pycache__/
env/
venv/
.env/
.env
.venv/
test/
@@ -30,6 +31,11 @@ python_embedded/
# abogen
*config.json
config/
storage/
build/
dist/
.old/
test_assets/
dev_notes/
.claude/
+13 -2
View File
@@ -1,3 +1,14 @@
# 1.3.0
- Special thanks to [@jeremiahsb](https://github.com/jeremiahsb) for his [massive contribution](https://github.com/denizsafak/abogen/pull/120) (>55k lines!) that brought the Web UI, EPUB 3 pipeline, and core architectural improvements to life.
- Added an EPUB 3 packaging pipeline that builds media-overlay EPUBs from generated audio and chunk metadata.
- Persisted chunk timing metadata in job artifacts and exercised the exporter with automated tests.
- Added Flask-based Web UI (`abogen-web`) for Docker and headless server deployments.
- Reorganized codebase to support both PyQt6 desktop GUI and Web UI from a shared core.
- Added Supertonic TTS engine support with GPU acceleration.
- Added entity analysis and pronunciation override system for proper nouns.
- Added speaker/role assignment for multi-voice "theatrical" audiobooks.
- Added Calibre OPDS and Audiobookshelf integration.
# 1.2.5
- Added new option: `Override item settings with current selection` in the queue manager. When enabled, all items in the queue will be processed using the current global settings selected in the main GUI, overriding their individual settings. When disabled, each item will retain its own specific settings.
- Fixed `Error "Could not load the Qt platform plugin "xcb"` error that occurred in some Linux distributions due to missing `libxcb-cursor0` library by conditionally loading the bundled library when the system version is unavailable, issue mentioned by @bmcgonag in #101.
@@ -53,7 +64,7 @@
- Fixed `/` and `\` path display by normalizing paths.
- Fixed book reprocessing issue where books were being processed every time the chapters window was opened, improving performance when reopening the same book.
- Fixed taskbar icon not appearing correctly in Windows.
- Fixed Go to folder button not opening the chapter output directory when only separate chapters were generated.
- Fixed "Go to folder" button not opening the chapter output directory when only separate chapters were generated.
- Improvements in code and documentation.
# 1.1.9
@@ -182,7 +193,7 @@
- Improved invalid profile handling in the voice mixer.
# v1.0.3
- Added voice mixing, allowing multiple voices to be combined into a single Mixed Voice, a feature mentioned by @PulsarFTW in #1. Special thanks to @jborza for making this possible through his contributions in #5.
- Added voice mixing, allowing multiple voices to be combined into a single "Mixed Voice", a feature mentioned by @PulsarFTW in #1. Special thanks to @jborza for making this possible through his contributions in #5.
- Added profile system to voice mixer, allowing users to create and manage multiple voice profiles.
- Improvements in the voice mixer, mostly for organizing controls and enhancing user experience.
- Added icons for flags and genders in the GUI, making it easier to identify different options.
+232 -68
View File
@@ -4,6 +4,7 @@
[![GitHub Release](https://img.shields.io/github/v/release/denizsafak/abogen)](https://github.com/denizsafak/abogen/releases/latest)
[![Abogen PyPi Python Versions](https://img.shields.io/pypi/pyversions/abogen)](https://pypi.org/project/abogen/)
[![Operating Systems](https://img.shields.io/badge/os-windows%20%7C%20linux%20%7C%20macos%20-blue)](https://github.com/denizsafak/abogen/releases/latest)
[![PyPi Total Downloads](https://img.shields.io/pepy/dt/abogen?label=downloads%20(pypi)&color=blue)](https://pypi.org/project/abogen/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![License: MIT](https://img.shields.io/badge/License-MIT-maroon.svg)](https://opensource.org/licenses/MIT)
@@ -17,7 +18,7 @@ Abogen is a powerful text-to-speech conversion tool that makes it easy to turn e
https://github.com/user-attachments/assets/094ba3df-7d66-494a-bc31-0e4b41d0b865
> This demo was generated in just 5 seconds, producing 1 minute of audio with perfectly synced subtitles. To create a similar video, see [the demo guide](https://github.com/denizsafak/abogen/tree/main/demo).
> This demo was generated in just 5 seconds, producing 1 minute of audio with perfectly synced subtitles. To create a similar video, see [the demo guide](https://github.com/denizsafak/abogen/tree/main/demo).
## `How to install?` <a href="https://pypi.org/project/abogen/" target="_blank"><img src="https://img.shields.io/pypi/pyversions/abogen" alt="Abogen Compatible PyPi Python Versions" align="right" style="margin-top:6px;"></a>
@@ -37,17 +38,19 @@ This method handles everything automatically - installing all dependencies inclu
#### <b>OPTION 2: Install using uv</b>
First, [install uv](https://docs.astral.sh/uv/getting-started/installation/) if you haven't already.
The CUDA extras install both GPU-accelerated Kokoro (via PyTorch) and Supertonic (via onnxruntime-gpu).
```bash
# For NVIDIA GPUs (CUDA 12.8) - Recommended
uv tool install --python 3.12 abogen[cuda]
uv tool install --python 3.12 abogen[cuda] --extra-index-url https://download.pytorch.org/whl/cu128 --index-strategy unsafe-best-match
# For NVIDIA GPUs (CUDA 12.6) - Older drivers
uv tool install --python 3.12 abogen[cuda126]
uv tool install --python 3.12 abogen[cuda126] --extra-index-url https://download.pytorch.org/whl/cu126 --index-strategy unsafe-best-match
# For NVIDIA GPUs (CUDA 13.0) - Newer drivers
uv tool install --python 3.12 abogen[cuda130]
uv tool install --python 3.12 abogen[cuda130] --extra-index-url https://download.pytorch.org/whl/cu130 --index-strategy unsafe-best-match
# For AMD GPUs or without GPU (CPU) - ROCm is not available on Windows. Use Linux if you have AMD GPU
# For AMD GPUs or without GPU - If you have AMD GPU, you need to use Linux for GPU acceleration, because ROCm is not available on Windows.
uv tool install --python 3.12 abogen
```
@@ -64,6 +67,9 @@ venv\Scripts\activate
# We need to use an older version of PyTorch (2.8.0) until this issue is fixed: https://github.com/pytorch/pytorch/issues/166628
pip install torch==2.8.0+cu128 torchvision==0.23.0+cu128 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128
# Also install onnxruntime-gpu for Supertonic GPU acceleration:
pip install onnxruntime-gpu
# For AMD GPUs:
# Not supported yet, because ROCm is not available on Windows. Use Linux if you have AMD GPU.
@@ -81,8 +87,11 @@ First, [install uv](https://docs.astral.sh/uv/getting-started/installation/) if
# Install espeak-ng
brew install espeak-ng
# Install abogen (Automatically handles Silicon Mac/MPS support)
uv tool install --python 3.12 abogen
# For Silicon Mac (M1, M2 etc.)
uv tool install --python 3.13 abogen --with "kokoro @ git+https://github.com/hexgrad/kokoro.git,numpy<2"
# For Intel Mac
uv tool install --python 3.12 abogen --with "kokoro @ git+https://github.com/hexgrad/kokoro.git,numpy<2"
```
<details>
@@ -117,11 +126,11 @@ sudo apt install espeak-ng # Ubuntu/Debian
sudo pacman -S espeak-ng # Arch Linux
sudo dnf install espeak-ng # Fedora
# For NVIDIA GPUs or without GPU (CPU) - No need to include [CUDA] in here.
# For NVIDIA GPUs or without GPU - No need to include [cuda] in here.
uv tool install --python 3.12 abogen
# For AMD GPUs (ROCm 6.4)
uv tool install --python 3.12 abogen[rocm]
uv tool install --python 3.12 abogen[rocm] --extra-index-url https://download.pytorch.org/whl/nightly/rocm6.4 --index-strategy unsafe-best-match
```
<details>
@@ -162,13 +171,30 @@ pip3 install --pre torch torchvision torchaudio --index-url https://download.pyt
> Special thanks to [@hg000125](https://github.com/hg000125) for his contribution in [#23](https://github.com/denizsafak/abogen/issues/23). AMD GPU support is possible thanks to his work.
## Interfaces
Abogen offers **two interfaces**, but currently they have different feature sets. The **Web UI** contains newer features that are still being integrated into the desktop application.
| Command | Interface | Features |
|---------|-----------|----------|
| `abogen` | PyQt6 Desktop GUI | Stable core features + **Supertonic TTS**|
| `abogen-web` | Flask Web UI | Core features + **Supertonic TTS**, **LLM Normalization**, **Audiobookshelf Integration** and more! |
> **Note:** The Web UI is under active development. We are working to integrate these new features into the PyQt desktop app. until then, the Web UI provides the most feature-rich experience.
> Special thanks to [@jeremiahsb](https://github.com/jeremiahsb) for making this possible! I was honestly surprised by his [massive contribution](https://github.com/denizsafak/abogen/pull/120) (>55,000 lines!) that brought the entire Web UI to life.
# 🖥️ Desktop Application (PyQt)
## `How to run?`
You can simply run this command to start Abogen:
You can simply run this command to start Abogen Desktop GUI:
```bash
abogen
```
> [!TIP]
> If you installed Abogen using the Windows installer `(WINDOWS_INSTALL.bat)`, It should have created a shortcut in the same folder, or your desktop. You can run it from there. If you lost the shortcut, Abogen is located in `python_embedded/Scripts/abogen.exe`. You can run it from there directly.
@@ -185,7 +211,7 @@ abogen
## `In action`
<img title="Abogen in action" src='https://raw.githubusercontent.com/denizsafak/abogen/refs/heads/main/demo/abogen.gif'>
Heres Abogen in action: in this demo, it processes 3,000 characters of text in just 11 seconds and turns it into 3 minutes and 28 seconds of audio, and I have a low-end **RTX 2060 Mobile laptop GPU**. Your results may vary depending on your hardware.
Heres Abogen in action: in this demo, it processes 3,000 characters of text in just 11 seconds and turns it into 3 minutes and 28 seconds of audio, and I have a low-end **RTX 2060 Mobile laptop GPU**. Your results may vary depending on your hardware.
## `Configuration`
@@ -258,6 +284,172 @@ Abogen will process each item in the queue automatically, saving outputs as conf
> Special thanks to [@jborza](https://github.com/jborza) for adding queue mode in PR [#35](https://github.com/denizsafak/abogen/pull/35)
---
# 🌐 Web Application (WebUI)
## `How to run?`
Run this command to start the Web UI:
```bash
abogen-web
```
Then open http://localhost:8808 and drag in your documents. Jobs run in the background worker and the browser updates automatically.
<img title="Abogen in action" src='https://raw.githubusercontent.com/denizsafak/abogen/refs/heads/main/demo/abogen-webui.png'>
## `Using the web UI`
1. Upload a document (drag & drop or use the upload button).
2. Choose voice, language, speed, subtitle style, and output format.
3. Click **Create job**. The job immediately appears in the queue.
4. Watch progress and logs update live. Download audio/subtitle assets when complete.
5. Cancel or delete jobs any time. Download logs for troubleshooting.
Multiple jobs can run sequentially; the worker processes them in order.
## `Container image`
You can build a lightweight container image directly from the repository root:
```bash
docker build -t abogen .
mkdir -p ~/abogen-data/uploads ~/abogen-data/outputs
docker run --rm \
-p 8808:8808 \
-v ~/abogen-data:/data \
--name abogen \
abogen
```
Browse to http://localhost:8808. Uploaded source files are stored in `/data/uploads` and rendered audio/subtitles appear in `/data/outputs`.
### Container environment variables
| Variable | Default | Purpose |
|----------|---------|---------|
| `ABOGEN_HOST` | `0.0.0.0` | Bind address for the Flask server |
| `ABOGEN_PORT` | `8808` | HTTP port |
| `ABOGEN_DEBUG` | `false` | Enable Flask debug mode |
| `ABOGEN_UPLOAD_ROOT` | `/data/uploads` | Directory where uploaded files are stored |
| `ABOGEN_OUTPUT_ROOT` | `/data/outputs` | Directory for generated audio and subtitles (legacy alias of `ABOGEN_OUTPUT_DIR`) |
| `ABOGEN_OUTPUT_DIR` | `/data/outputs` | Container path for rendered audio/subtitles |
| `ABOGEN_SETTINGS_DIR` | `/config` | Container path for JSON settings/configuration |
| `ABOGEN_TEMP_DIR` | `/data/cache` (Docker) or platform cache dir | Container path for temporary audio working files |
| `ABOGEN_UID` | `1000` | UID that the container should run as (matches host user) |
| `ABOGEN_GID` | `1000` | GID that the container should run as (matches host group) |
| `ABOGEN_LLM_BASE_URL` | `""` | OpenAI-compatible endpoint used to seed the Settings → LLM panel |
| `ABOGEN_LLM_API_KEY` | `""` | API key passed to the endpoint above |
| `ABOGEN_LLM_MODEL` | `""` | Default model selected when you refresh the model list |
| `ABOGEN_LLM_TIMEOUT` | `30` | Timeout (seconds) for server-side LLM requests |
| `ABOGEN_LLM_CONTEXT_MODE` | `sentence` | Default prompt context window (`sentence`, `paragraph`, `document`) |
| `ABOGEN_LLM_PROMPT` | `""` | Custom normalization prompt template seeded into the UI |
Set any of these with `-e VAR=value` when starting the container.
To discover your local UID/GID for matching file permissions inside the container, run:
```bash
id -u
id -g
```
Use those values to populate `ABOGEN_UID` / `ABOGEN_GID` in your `.env` file.
When running via Docker Compose, set `ABOGEN_SETTINGS_DIR`,
`ABOGEN_OUTPUT_DIR`, and `ABOGEN_TEMP_DIR` in your `.env` file to the host
directories you want mounted into the container. Compose maps them to
`/config`, `/data/outputs`, and `/data/cache` respectively while exporting
those in-container paths to the application. Non-audio caches (e.g., Hugging
Face downloads) stick to the container's internal cache under `/tmp/abogen-home/.cache`
by default, so only conversion scratch data touches the mounted `ABOGEN_TEMP_DIR`.
Ensure each host directory exists and is writable by the UID/GID you configure
before starting the stack.
### Docker Compose (GPU by default)
The repo includes `docker-compose.yaml`, which targets GPU hosts out of the box. Install the NVIDIA Container Toolkit and run:
```bash
docker compose up -d --build
```
Key build/runtime knobs:
- `TORCH_VERSION` pin a specific PyTorch release that matches your driver (leave blank for the latest on the configured index).
- `TORCH_INDEX_URL` swap out the PyTorch download index when targeting a different CUDA build.
- `ABOGEN_DATA` host path that stores uploads/outputs (defaults to `./data`).
CPU-only deployment: comment out the `deploy.resources.reservations.devices` block (and the optional `runtime: nvidia` line) inside the compose file. Compose will then run without requesting a GPU. If you prefer the classic CLI:
```bash
docker build -f abogen/Dockerfile -t abogen-gpu .
docker run --rm \
--gpus all \
-p 8808:8808 \
-v ~/abogen-data:/data \
abogen-gpu
```
## `LLM-assisted text normalization`
Abogen can hand tricky apostrophes and contractions to an OpenAI-compatible large language model. Configure it from **Settings → LLM**:
1. Enter the base URL for your endpoint (Ollama, OpenAI proxy, etc.) and an API key if required. Use the server root (for Ollama: `http://localhost:11434`)—Abogen appends `/v1/...` automatically, but it also accepts inputs that already end in `/v1`.
2. Click **Refresh models** to load the catalog, pick a default model, and adjust the timeout or prompt template.
3. Use the preview box to test the prompt, then save the settings. The Normalization panel can synthesize a short audio preview with the current configuration.
When you are running inside Docker or a CI pipeline, seed the form automatically with `ABOGEN_LLM_*` variables in your `.env` file. The `.env.example` file includes sample values for a local Ollama server.
## `Audiobookshelf integration`
Abogen can push finished audiobooks directly into Audiobookshelf. Configure this under **Settings → Integrations → Audiobookshelf** by providing:
- **Base URL** the HTTPS origin (and optional path prefix) where your Audiobookshelf server is reachable, for example `https://abs.example.com` or `https://media.example.com/abs`. Do **not** append `/api`.
- **Library ID** the identifier of the target Audiobookshelf library (copy it from the librarys settings page in ABS).
- **Folder (name or ID)** the destination folder inside that library. Enter the folder name exactly as it appears in Audiobookshelf (Abogen resolves it to the correct ID automatically), paste the raw `folderId`, or click **Browse folders** to fetch the available folders and populate the field.
- **API token** a personal access token generated in Audiobookshelf under *Account → API tokens*.
You can enable automatic uploads for future jobs or trigger individual uploads from the queue once the connection succeeds.
### Reverse proxy checklist (Nginx Proxy Manager)
When Audiobookshelf sits behind Nginx Proxy Manager (NPM), make sure the API paths and headers reach the backend untouched:
1. Create a **Proxy Host** that points to your ABS container or host (default forward port `13378`).
2. Under the **SSL** tab, enable your certificate and tick **Force SSL** if you want HTTPS only.
3. In the **Advanced** tab, append the snippet below so bearer tokens, client IPs, and large uploads survive the proxy hop:
```nginx
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header Authorization $http_authorization;
client_max_body_size 5g;
proxy_read_timeout 300s;
proxy_connect_timeout 300s;
```
4. Disable **Block Common Exploits** (it strips Authorization headers in some NPM builds).
5. Enable **Websockets Support** on the main proxy screen (Audiobookshelf uses it for the web UI, and it keeps the reverse proxy configuration consistent).
6. If you publish Audiobookshelf under a path prefix (for example `/abs`), add a **Custom Location** with `Location: /abs/` and set the **Forward Path** to `/`. That rewrite strips the `/abs` prefix before traffic reaches Audiobookshelf so `/abs/api/...` on the internet becomes `/api/...` on the backend. Use the same prefixed URL in Abogens “Base URL” field.
After saving the proxy host, test the API from the machine running Abogen:
```bash
curl -i "https://abs.example.com/api/libraries" \
-H "Authorization: Bearer YOUR_API_TOKEN"
```
If you still receive `Cannot GET /api/...`, the proxy is rewriting paths. Double-check the **Custom Locations** table (the `Forward Path` column should be empty for `/abs/`) and review the NPM access/error logs while issuing the curl request to confirm the backend sees the full `/api/libraries` URL.
A JSON response confirming the libraries list means the proxy is routing API calls correctly. You can then use **Browse folders** to confirm the library contents, run **Test connection** in Abogens settings (it verifies the library and resolves the folder), and use the “Send to Audiobookshelf” button on completed jobs.
## `JSON endpoints`
Need machine-readable status updates? The dashboard calls a small set of helper endpoints you can reuse:
- `GET /api/jobs/<id>` returns job metadata, progress, and log lines in JSON.
- `GET /partials/jobs` renders the live job list as HTML (htmx uses this for polling).
- `GET /partials/jobs/<id>/logs` renders just the log window.
More automation hooks are planned; contributions are very welcome if you need additional routes.
---
# Core Features (Available in Both)
## `About Chapter Markers`
When you process ePUB, PDF or markdown files, Abogen converts them into text files stored in your cache directory. When you click "Edit," you're actually modifying these converted text files. In these text files, you'll notice tags that look like this:
@@ -331,6 +523,9 @@ For a complete list of supported languages and voices, refer to Kokoro's [VOICES
> See [How to fix Japanese audio not working?](#japanese-audio-not-working)
---
# Guides & Troubleshooting
## `MPV Config`
I highly recommend using [MPV](https://mpv.io/installation/) to play your audio files, as it supports displaying subtitles even without a video track. Here's my `mpv.conf`:
```
@@ -349,54 +544,6 @@ audio-samplerate=48000
volume-max=200
```
## `Docker Guide`
If you want to run Abogen in a Docker container:
1) [Download the repository](https://github.com/denizsafak/abogen/archive/refs/heads/main.zip) and extract, or clone it using git.
2) Go to `abogen` folder. You should see `Dockerfile` there.
3) Open your termminal in that directory and run the following commands:
```bash
# Build the Docker image:
docker build --progress plain -t abogen .
# Note that building the image may take a while.
# After building is complete, run the Docker container:
# Windows
docker run --name abogen -v %cd%:/shared -p 5800:5800 -p 5900:5900 --gpus all abogen
# Linux
docker run --name abogen -v $(pwd):/shared -p 5800:5800 -p 5900:5900 --gpus all abogen
# MacOS
docker run --name abogen -v $(pwd):/shared -p 5800:5800 -p 5900:5900 abogen
# We expose port 5800 for use by a web browser, 5900 if you want to connect with a VNC client.
```
Abogen launches automatically inside the container.
- You can access it via a web browser at [http://localhost:5800](http://localhost:5800) or connect to it using a VNC client at `localhost:5900`.
- You can use `/shared` directory to share files between your host and the container.
- For later use, start it with `docker start abogen` and stop it with `docker stop abogen`.
- Pass in `-e WEB_AUDIO="1"` for `docker run` to enable audio.
Known issues:
- Audio preview is not working inside container (ALSA error) if using a VNC client.
- `Open cache directory` and `Open configuration directory` options in settings not working. (Tried pcmanfm, did not work with Abogen).
> Special thanks to [@geo38](https://www.reddit.com/user/geo38/) from Reddit, who provided the Dockerfile and instructions in [this comment](https://www.reddit.com/r/selfhosted/comments/1k8x1yo/comment/mpe0bz8/).
## `🌐 Web Application`
A web-based version of Abogen has been developed by [@jeremiahsb](https://github.com/jeremiahsb).
**Access the repository here:** [jeremiahsb/abogen](https://github.com/jeremiahsb/abogen)
> [!NOTE]
> I intend to merge this implementation into the main repository in the future once existing conflicts are resolved. Until then, please be aware that the web version is maintained independently and may not always be in sync with the latest updates in this repository.
> Special thanks to [@jeremiahsb](https://github.com/jeremiahsb) for implementing the web app!
## `Similar Projects`
Abogen is a standalone project, but it is inspired by and shares some similarities with other projects. Here are a few:
- [audiblez](https://github.com/santinic/audiblez): Generate audiobooks from e-books. **(Has CLI and GUI support)**
@@ -461,11 +608,12 @@ This will start Abogen in command-line mode and display detailed error messages.
>
> If you used `uv` to install Abogen, you can uninstall and try reinstalling with another CUDA version:
> ```bash
> # First uninstall Abogen
> uv tool uninstall abogen
> # First, try CUDA 13.0 for newer drivers
> uv tool install --python 3.12 abogen[cuda130]
> # If that doesn't work, try CUDA 12.6 for older drivers
> uv tool install --python 3.12 abogen[cuda126]
> # Try CUDA 12.6 for older drivers
> uv tool install --python 3.12 abogen[cuda126] --extra-index-url https://download.pytorch.org/whl/cu126 --index-strategy unsafe-best-match
> # If that doesn't work, try CUDA 13.0 for newer drivers
> uv tool install --python 3.12 abogen[cuda130] --extra-index-url https://download.pytorch.org/whl/cu130 --index-strategy unsafe-best-match
> ```
</details>
@@ -500,7 +648,7 @@ This will start Abogen in command-line mode and display detailed error messages.
> ```
> If you installed Abogen using pip, open your terminal in the virtual environment and run:
> ```bash
> pip install torch==2.8.0 torchaudio==2.8.0 torchvision==0.23.0
> pip install torch==2.8.0 torchaudio==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128
> ```
</details>
@@ -542,14 +690,30 @@ I welcome contributions! If you have ideas for new features, improvements, or bu
If you'd like to modify the code and contribute to development, you can [download the repository](https://github.com/denizsafak/abogen/archive/refs/heads/main.zip), extract it and run the following commands to build **or** install the package:
```bash
# Go to the directory where you extracted the repository and run:
pip install -e . # Installs the package in editable mode
pip install build # Install the build package
python -m build # Builds the package in dist folder (optional)
abogen # Opens the GUI
pip install -e .[dev] # Installs the package in editable mode with build dependencies
python -m build # Builds the package in dist folder (optional)
abogen # Opens the GUI
```
> Make sure you are using Python 3.10 to 3.12. You need to create a virtual environment if needed.
<details>
<summary><b>Alternative: Using uv (click to expand)</b></summary>
```bash
# Go to the directory where you extracted the repository and run:
uv venv --python 3.12 # Creates a virtual environment with Python 3.12
# After activating the virtual environment, run:
uv pip install -e . # Installs the package in editable mode
uv build # Builds the package in dist folder (optional)
abogen # Opens the GUI
```
</details>
Feel free to explore the code and make any changes you like.
## `Credits`
- Web UI implementation by [@jeremiahsb](https://github.com/jeremiahsb)
- Abogen uses [Kokoro](https://github.com/hexgrad/kokoro) for its high-quality, natural-sounding text-to-speech synthesis. Huge thanks to the Kokoro team for making this possible.
- Thanks to the [spaCy](https://spacy.io/) project for its sentence-segmentation tools, which help Abogen produce cleaner, more natural sentence segmentation.
- Thanks to [@wojiushixiaobai](https://github.com/wojiushixiaobai) for [Embedded Python](https://github.com/wojiushixiaobai/Python-Embed-Win64) packages. These modified packages include pip pre-installed, enabling Abogen to function as a standalone application without requiring users to separately install Python in Windows.
+69 -20
View File
@@ -20,6 +20,7 @@ set MISAKI_LANG=en
for /f "delims=: tokens=*" %%A in ('findstr /b ::: "%~f0"') do @echo(%%A
set CURRENT_DIR="%CD%"
set "UV_CACHE_DIR=%~dp0.uv_cache"
set NAME=abogen
set PROJECTFOLDER=abogen
set RUN=python_embedded\Scripts\abogen.exe
@@ -29,6 +30,28 @@ set refrenv=%PROJECTFOLDER%\refrenv.bat
set PYTHON_PATH=python_embedded\pythonw.exe
set PYTHON_CONSOLE_PATH=python_embedded\python.exe
:: ---------------------------------------------------------
:: Version Selection
:: ---------------------------------------------------------
echo.
echo Select installation version:
echo [1] Stable (PyPI) - Safer, recommended for most users.
echo [2] Dev (Local) - Install from current folder (may include commits after the latest release).
echo.
choice /C 12 /M "Your choice"
if errorlevel 2 (
set INSTALL_SOURCE=dev
echo.
echo Selected: Dev - Local Editable
) else (
set INSTALL_SOURCE=pypi
echo.
echo Selected: Stable - PyPI
)
echo.
:: ---------------------------------------------------------
:: Check for updates
echo Checking for updates...
set VERSION_FILE=%PROJECTFOLDER%\VERSION
@@ -197,18 +220,19 @@ if not "%~1"=="" (
echo Open with: "%~1"
)
:: Update pip
echo Updating pip...
:: Update pip and install uv
echo Updating pip and installing uv...
%PYTHON_CONSOLE_PATH% -m pip install --upgrade pip --no-warn-script-location
%PYTHON_CONSOLE_PATH% -m pip install uv --no-warn-script-location
if errorlevel 1 (
echo Failed to update pip.
echo Failed to install uv.
pause
exit /b
)
:: Install docopt's fixed version
echo Installing fixed version of docopt...
%PYTHON_CONSOLE_PATH% -m pip install --force-reinstall https://github.com/denizsafak/abogen/raw/refs/heads/main/abogen/resources/docopt-0.6.2-py2.py3-none-any.whl --no-warn-script-location
%PYTHON_CONSOLE_PATH% -m uv pip install --system --force-reinstall https://github.com/denizsafak/abogen/raw/refs/heads/main/abogen/resources/docopt-0.6.2-py2.py3-none-any.whl
if errorlevel 1 (
echo Failed to install fixed version of docopt.
pause
@@ -217,7 +241,7 @@ if errorlevel 1 (
:: Install progress's fixed version
echo Installing fixed version of progress...
%PYTHON_CONSOLE_PATH% -m pip install --force-reinstall https://github.com/denizsafak/abogen/raw/refs/heads/main/abogen/resources/progress-1.6-py3-none-any.whl --no-warn-script-location
%PYTHON_CONSOLE_PATH% -m uv pip install --system --force-reinstall https://github.com/denizsafak/abogen/raw/refs/heads/main/abogen/resources/progress-1.6-py3-none-any.whl
if errorlevel 1 (
echo Failed to install fixed version of progress.
pause
@@ -226,7 +250,7 @@ if errorlevel 1 (
:: Install setup requirements
echo Installing setup requirements...
%PYTHON_CONSOLE_PATH% -m pip install --upgrade setuptools setuptools-scm wheel sphinx hatchling editables --no-warn-script-location
%PYTHON_CONSOLE_PATH% -m uv pip install --system --upgrade setuptools setuptools-scm wheel sphinx hatchling editables
if errorlevel 1 (
echo Failed to install setup requirements.
pause
@@ -235,32 +259,43 @@ if errorlevel 1 (
:: Install gpustat
echo Installing gpustat...
%PYTHON_CONSOLE_PATH% -m pip install gpustat --no-warn-script-location
%PYTHON_CONSOLE_PATH% -m uv pip install --system gpustat
if errorlevel 1 (
echo Failed to install gpustat.
pause
exit /b
)
:: Install project and dependencies from pyproject.toml
echo Checking and installing project dependencies...
if exist %PYPROJECT_FILE% (
echo Installing project from pyproject.toml...
%PYTHON_CONSOLE_PATH% -m pip install -e . --no-warn-script-location
:: Install project based on user selection
if "%INSTALL_SOURCE%"=="pypi" (
echo Installing stable version from PyPI...
%PYTHON_CONSOLE_PATH% -m uv pip install --system abogen
if errorlevel 1 (
echo Failed to install from pyproject.toml.
echo Failed to install abogen from PyPI.
pause
exit /b
)
) else (
echo Warning: pyproject.toml not found in current directory.
pause
echo Checking and installing project dependencies...
if exist %PYPROJECT_FILE% (
echo Installing project from pyproject.toml using uv...
:: Using uv pip install --system --editable
%PYTHON_CONSOLE_PATH% -m uv pip install --system --editable .
if errorlevel 1 (
echo Failed to install from pyproject.toml.
pause
exit /b
)
) else (
echo Warning: pyproject.toml not found in current directory.
pause
)
)
:: Install misaki again if MISAKI_LANG is not set to "en"
:: Install misaki again if MISAKI_LANG is not set to "en" via uv
if "%MISAKI_LANG%" NEQ "en" (
echo Configuring language pack: %MISAKI_LANG%
%PYTHON_CONSOLE_PATH% -m pip install misaki[%MISAKI_LANG%] --upgrade --no-warn-script-location
%PYTHON_CONSOLE_PATH% -m uv pip install --system misaki[%MISAKI_LANG%] --upgrade
if errorlevel 1 (
echo Failed to install misaki language pack.
pause
@@ -275,19 +310,26 @@ for /f %%i in ('%PYTHON_CONSOLE_PATH% -c "from abogen.is_nvidia import check; pr
echo.
echo Checking CUDA availability...
if /I "%IS_NVIDIA%"=="true" (
for /f %%i in ('%PYTHON_CONSOLE_PATH% -c "from torch.cuda import is_available; print(is_available())"') do set cuda_available=%%i
for /f %%i in ('%PYTHON_CONSOLE_PATH% %PROJECTFOLDER%\check_cuda.py') do set cuda_available=%%i
if "%cuda_available%"=="False" (
echo "Installing PyTorch with CUDA (12.8) support..."
:: We need to use an older version of PyTorch (2.8.0) until this issue is fixed: https://github.com/pytorch/pytorch/issues/166628
:: Solution mentioned by @mazenemam19 in #99:
%PYTHON_CONSOLE_PATH% -m pip install torch==2.8.0+cu128 torchvision==0.23.0+cu128 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128 --no-warn-script-location
%PYTHON_CONSOLE_PATH% -m uv pip install --system torch==2.8.0+cu128 torchvision==0.23.0+cu128 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128
echo.
if errorlevel 1 (
echo Failed to install PyTorch.
pause
exit /b
)
echo Installing onnxruntime-gpu for Supertonic GPU acceleration...
%PYTHON_CONSOLE_PATH% -m uv pip install --system onnxruntime-gpu
if errorlevel 1 (
echo Failed to install onnxruntime-gpu.
pause
exit /b
)
) else (
echo CUDA is available on NVIDIA GPU.
)
@@ -307,12 +349,19 @@ if /I "%IS_NVIDIA%"=="true" (
echo "Installing PyTorch with CUDA (12.8) support..."
:: We need to use an older version of PyTorch (2.8.0) until this issue is fixed: https://github.com/pytorch/pytorch/issues/166628
:: Solution mentioned by @mazenemam19 in #99:
%PYTHON_CONSOLE_PATH% -m pip install torch==2.8.0+cu128 torchvision==0.23.0+cu128 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128 --no-warn-script-location
%PYTHON_CONSOLE_PATH% -m uv pip install --system torch==2.8.0+cu128 torchvision==0.23.0+cu128 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128
if errorlevel 1 (
echo Failed to install PyTorch.
pause
exit /b
)
echo Installing onnxruntime-gpu for Supertonic GPU acceleration...
%PYTHON_CONSOLE_PATH% -m uv pip install --system onnxruntime-gpu
if errorlevel 1 (
echo Failed to install onnxruntime-gpu.
pause
exit /b
)
)
)
-44
View File
@@ -1,44 +0,0 @@
# Special thanks to @geo38 from Reddit, who provided this Dockerfile:
# https://www.reddit.com/r/selfhosted/comments/1k8x1yo/comment/mpe0bz8/
# Use a docker base image that runs a window manager that can be viewed
# outside the image with a web browser or VNC client.
# https://github.com/jlesage/docker-baseimage-gui
FROM jlesage/baseimage-gui:debian-12-v4
# Load stuff needed by abogen
RUN apt-get update \
&& apt-get install -y \
python3 \
python3-venv \
python3-pip \
python3-pyqt6 \
espeak-ng \
libxcb-cursor0 \
libgl1 \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# The base image will run /startapp.sh on launch.
#
# The base image runs that script as user 'app' uid=1000. That user
# does not exist in the base image but is created at run time.
#
# We need to install abogen in python venv (requirement of newer python3).
#
# The python venv has to be writable by the 'app' user as abogen dynamically
# installs python packages, so create the venv as that user
#
# We intend to share the /shared directory with the host using a bind volume
# in order to access any source files and the created files.
RUN echo '#!/bin/bash\nsource /app/venv/bin/activate\nexec abogen' > /startapp.sh \
&& chmod 555 /startapp.sh \
&& mkdir /app /shared \
&& chown 1000:1000 /app /shared \
&& chmod 755 /app /shared
USER 1000:1000
RUN python3 -m venv /app/venv
RUN /bin/bash -c "source /app/venv/bin/activate && pip install abogen"
# Change back to user ROOT as the startup scripts inside base image needs it
USER root
+1 -1
View File
@@ -1 +1 @@
1.2.5
1.3.1
Binary file not shown.

Before

Width:  |  Height:  |  Size: 571 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 992 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 786 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 877 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.0 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 832 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 521 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 868 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 808 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 372 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 883 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 465 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 885 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 381 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 855 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 917 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 441 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 856 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 837 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 875 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 617 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 843 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 875 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 891 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.0 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 851 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 431 B

File diff suppressed because it is too large Load Diff
+30
View File
@@ -0,0 +1,30 @@
import sys
import os
import platform
import ctypes
import importlib.util
def check_cuda_with_fix():
"""
Check if CUDA is available, with a fix for PyTorch DLL loading issue
([WinError 1114]) on Windows.
"""
# Fix PyTorch DLL loading issue ([WinError 1114]) on Windows
try:
if platform.system() == "Windows":
spec = importlib.util.find_spec("torch")
if spec and spec.origin:
dll_path = os.path.join(os.path.dirname(spec.origin), "lib", "c10.dll")
if os.path.exists(dll_path):
ctypes.CDLL(os.path.normpath(dll_path))
except Exception:
pass
try:
from torch.cuda import is_available
print(is_available())
except ImportError:
print("False")
if __name__ == "__main__":
check_cuda_with_fix()
+275
View File
@@ -0,0 +1,275 @@
from __future__ import annotations
from dataclasses import dataclass
from typing import Dict, Iterable, Iterator, List, Literal, Optional, Tuple
from typing import Pattern
import re
from abogen.kokoro_text_normalization import ApostropheConfig, normalize_for_pipeline
from abogen.normalization_settings import build_apostrophe_config, get_runtime_settings
ChunkLevel = Literal["paragraph", "sentence"]
_SENTENCE_SPLIT_REGEX = re.compile(r"(?<!\b[A-Z])[.!?][\s\n]+")
_WHITESPACE_REGEX = re.compile(r"\s+")
_PARAGRAPH_SPLIT_REGEX = re.compile(r"(?:\r?\n){2,}")
_ABBREVIATION_END_RE = re.compile(
r"\b(?:Mr|Mrs|Ms|Dr|Prof|Rev|Sr|Jr|St|Gen|Lt|Col|Sgt|Capt|Adm|Cmdr|vs|etc)\.$",
re.IGNORECASE,
)
_PIPELINE_APOSTROPHE_CONFIG = ApostropheConfig()
@dataclass(frozen=True)
class Chunk:
id: str
chapter_index: int
chunk_index: int
level: ChunkLevel
text: str
speaker_id: str = "narrator"
voice: Optional[str] = None
voice_profile: Optional[str] = None
voice_formula: Optional[str] = None
display_text: Optional[str] = None
def as_dict(self) -> Dict[str, object]:
return {
"id": self.id,
"chapter_index": self.chapter_index,
"chunk_index": self.chunk_index,
"level": self.level,
"text": self.text,
"speaker_id": self.speaker_id,
"voice": self.voice,
"voice_profile": self.voice_profile,
"voice_formula": self.voice_formula,
"display_text": self.display_text,
}
def _iter_paragraphs(text: str) -> Iterator[str]:
for raw_segment in _PARAGRAPH_SPLIT_REGEX.split(text.strip()):
normalized = raw_segment.strip()
if normalized:
yield normalized
def _iter_sentences(paragraph: str) -> Iterator[Tuple[str, str]]:
if not paragraph:
return
start = 0
for match in _SENTENCE_SPLIT_REGEX.finditer(paragraph):
end = match.end()
raw_segment = paragraph[start:end]
candidate = raw_segment.strip()
if candidate:
yield candidate, raw_segment
start = match.end()
tail_raw = paragraph[start:]
tail = tail_raw.strip()
if tail:
yield tail, tail_raw
def _normalize_whitespace(value: str) -> str:
return _WHITESPACE_REGEX.sub(" ", value).strip()
def _normalize_chunk_text(value: str) -> str:
settings = get_runtime_settings()
config = build_apostrophe_config(
settings=settings, base=_PIPELINE_APOSTROPHE_CONFIG
)
normalized = normalize_for_pipeline(value, config=config, settings=settings)
return _normalize_whitespace(normalized)
def _split_sentences(paragraph: str) -> List[Tuple[str, str]]:
sentences = list(_iter_sentences(paragraph))
if not sentences:
return []
merged: List[Tuple[str, str]] = []
buffer_norm: List[str] = []
buffer_raw: List[str] = []
for normalized_sentence, raw_sentence in sentences:
if buffer_norm:
buffer_norm.append(normalized_sentence)
buffer_raw.append(raw_sentence)
else:
buffer_norm = [normalized_sentence]
buffer_raw = [raw_sentence]
if _ABBREVIATION_END_RE.search(normalized_sentence.rstrip()):
continue
merged.append((" ".join(buffer_norm), "".join(buffer_raw)))
buffer_norm = []
buffer_raw = []
if buffer_norm:
merged.append((" ".join(buffer_norm), "".join(buffer_raw)))
return merged
def chunk_text(
*,
chapter_index: int,
chapter_title: str,
text: str,
level: ChunkLevel,
speaker_id: str = "narrator",
voice: Optional[str] = None,
voice_profile: Optional[str] = None,
voice_formula: Optional[str] = None,
chunk_prefix: Optional[str] = None,
) -> List[Dict[str, object]]:
"""Split text into ordered chunk dictionaries."""
prefix = chunk_prefix or f"chap{chapter_index:04d}"
chunks: List[Dict[str, object]] = []
if level == "paragraph":
paragraphs = list(_iter_paragraphs(text)) or [text.strip()]
for para_index, paragraph in enumerate(paragraphs):
normalized = _normalize_whitespace(paragraph)
if not normalized:
continue
chunk_id = f"{prefix}_p{para_index:04d}"
payload = Chunk(
id=chunk_id,
chapter_index=chapter_index,
chunk_index=len(chunks),
level=level,
text=normalized,
speaker_id=speaker_id,
voice=voice,
voice_profile=voice_profile,
voice_formula=voice_formula,
).as_dict()
payload["normalized_text"] = _normalize_chunk_text(paragraph)
payload["original_text"] = paragraph
chunks.append(payload)
_attach_display_text(text, chunks)
return chunks
# Sentence level flatten paragraphs into individual sentences
sentence_index = 0
for para_index, paragraph in enumerate(
list(_iter_paragraphs(text)) or [text.strip()]
):
normalized_para = _normalize_whitespace(paragraph)
if not normalized_para:
continue
sentence_pairs = _split_sentences(paragraph) or [(normalized_para, paragraph)]
for sent_local_index, (normalized_sentence, raw_sentence) in enumerate(
sentence_pairs
):
normalized_sentence = _normalize_whitespace(normalized_sentence)
if not normalized_sentence:
continue
chunk_id = f"{prefix}_p{para_index:04d}_s{sent_local_index:04d}"
payload = Chunk(
id=chunk_id,
chapter_index=chapter_index,
chunk_index=sentence_index,
level=level,
text=normalized_sentence,
speaker_id=speaker_id,
voice=voice,
voice_profile=voice_profile,
voice_formula=voice_formula,
).as_dict()
payload["normalized_text"] = _normalize_chunk_text(raw_sentence)
payload["display_text"] = raw_sentence
payload["original_text"] = raw_sentence
chunks.append(payload)
sentence_index += 1
_attach_display_text(text, chunks)
return chunks
_DISPLAY_PATTERN_CACHE: Dict[str, Pattern[str]] = {}
def _build_display_pattern(text: str) -> Pattern[str]:
cached = _DISPLAY_PATTERN_CACHE.get(text)
if cached is not None:
return cached
escaped = re.escape(text)
escaped = escaped.replace(r"\ ", r"\s+")
pattern = re.compile(r"(\s*" + escaped + r"\s*)", re.DOTALL)
_DISPLAY_PATTERN_CACHE[text] = pattern
return pattern
def _search_source_span(
source: str, normalized: str, start: int
) -> Optional[Tuple[int, int]]:
if not normalized:
return None
pattern = _build_display_pattern(normalized)
match = pattern.search(source, start)
if not match:
return None
return match.start(1), match.end(1)
def _attach_display_text(source: str, chunks: List[Dict[str, object]]) -> None:
if not source or not chunks:
return
cursor = 0
for chunk in chunks:
candidate = str(chunk.get("display_text") or chunk.get("text") or "")
if not candidate:
continue
match = _search_source_span(source, candidate, cursor)
if match is None and cursor:
match = _search_source_span(source, candidate, 0)
if match is None:
chunk.setdefault("display_text", candidate)
chunk.setdefault("original_text", chunk.get("display_text") or candidate)
continue
start, end = match
chunk["display_text"] = source[start:end]
chunk["original_text"] = source[start:end]
cursor = end
def build_chunks_for_chapters(
chapters: Iterable[Dict[str, object]],
*,
level: ChunkLevel,
speaker_id: str = "narrator",
) -> List[Dict[str, object]]:
"""Generate chunk dictionaries for a sequence of chapter payloads."""
all_chunks: List[Dict[str, object]] = []
for chapter_index, entry in enumerate(chapters):
if not isinstance(entry, dict): # defensive
continue
text = str(entry.get("text", "") or "").strip()
if not text:
continue
voice = entry.get("voice")
voice_profile = entry.get("voice_profile")
voice_formula = entry.get("voice_formula")
prefix = entry.get("id") or f"chap{chapter_index:04d}"
chapter_chunks = chunk_text(
chapter_index=chapter_index,
chapter_title=str(entry.get("title") or f"Chapter {chapter_index + 1}"),
text=text,
level=level,
speaker_id=speaker_id,
voice=str(voice) if voice else None,
voice_profile=str(voice_profile) if voice_profile else None,
voice_formula=str(voice_formula) if voice_formula else None,
chunk_prefix=str(prefix),
)
all_chunks.extend(chapter_chunks)
return all_chunks
+51
View File
@@ -29,6 +29,57 @@ LANGUAGE_DESCRIPTIONS = {
"z": "Mandarin Chinese",
}
# Mapping from Kokoro single-letter language codes to ISO 3166-1 alpha-2 country codes
# Used for loading flag icons
KOKORO_LANG_TO_COUNTRY = {
"a": "us", # American English -> United States
"b": "gb", # British English -> United Kingdom
"e": "es", # Spanish -> Spain
"f": "fr", # French -> France
"h": "in", # Hindi -> India
"i": "it", # Italian -> Italy
"j": "jp", # Japanese -> Japan
"p": "br", # Brazilian Portuguese -> Brazil
"z": "cn", # Mandarin Chinese -> China
}
# Mapping from Supertonic ISO 639-1 language codes to ISO 3166-1 alpha-2 country codes
# Used for loading flag icons in the Supertonic language picker
SUPERTONIC_LANG_TO_COUNTRY = {
"en": "gb",
"ko": "kr",
"ja": "jp",
"ar": "ae",
"bg": "bg",
"cs": "cz",
"da": "dk",
"de": "de",
"el": "gr",
"es": "es",
"et": "ee",
"fi": "fi",
"fr": "fr",
"hi": "in",
"hr": "hr",
"hu": "hu",
"id": "id",
"it": "it",
"lt": "lt",
"lv": "lv",
"nl": "nl",
"pl": "pl",
"pt": "pt",
"ro": "ro",
"ru": "ru",
"sk": "sk",
"sl": "si",
"sv": "se",
"tr": "tr",
"uk": "ua",
"vi": "vn",
"na": "na",
}
# Supported sound formats
SUPPORTED_SOUND_FORMATS = [
"wav",
+390
View File
@@ -0,0 +1,390 @@
from __future__ import annotations
import tempfile
import zipfile
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable, List, Sequence
MARKER_PREFIX = "[[ABOGEN-DBG:"
MARKER_SUFFIX = "]]"
@dataclass(frozen=True)
class DebugTTSSample:
code: str
label: str
text: str
DEBUG_TTS_SAMPLES: Sequence[DebugTTSSample] = (
DebugTTSSample(
code="APOS_001",
label="Apostrophes & contractions (1)",
text="It's a beautiful day, isn't it? Let's see what we'll do.",
),
DebugTTSSample(
code="APOS_002",
label="Apostrophes & contractions (2)",
text="I'm sure you're ready; we'd better go before it's too late.",
),
DebugTTSSample(
code="APOS_003",
label="Apostrophes & contractions (3)",
text="He'll say it's fine, but I can't promise it'll work.",
),
DebugTTSSample(
code="APOS_004",
label="Apostrophes & contractions (4)",
text="They've done it, and I'd agree they've earned it.",
),
DebugTTSSample(
code="APOS_005",
label="Apostrophes & contractions (5)",
text="She's here, we're late, they're waiting, and you're right.",
),
DebugTTSSample(
code="POS_001",
label="Plural possessives (1)",
text="The dogs' bowls were empty, but the boss's office was quiet.",
),
DebugTTSSample(
code="POS_002",
label="Plural possessives (2)",
text="The teachers' lounge was closed during the students' exams.",
),
DebugTTSSample(
code="POS_003",
label="Plural possessives (3)",
text="The actresses' roles changed, and the directors' notes piled up.",
),
DebugTTSSample(
code="POS_004",
label="Plural possessives (4)",
text="The Joneses' car was parked by the neighbors' fence.",
),
DebugTTSSample(
code="POS_005",
label="Plural possessives (5)",
text="The bosses' meeting ended before the witnesses' statements began.",
),
DebugTTSSample(
code="NUM_001",
label="Grouped numbers (1)",
text="There are 1,234 apples, 56 oranges, and 7.89 liters of juice.",
),
DebugTTSSample(
code="NUM_002",
label="Grouped numbers (2)",
text="The population is 10,000,000 and the area is 123.45 square miles.",
),
DebugTTSSample(
code="NUM_003",
label="Grouped numbers (3)",
text="Set the timer for 0.5 seconds, then wait 2.0 minutes.",
),
DebugTTSSample(
code="NUM_004",
label="Grouped numbers (4)",
text="We measured 3.1415 radians and wrote down 2,718.28 as well.",
),
DebugTTSSample(
code="NUM_005",
label="Grouped numbers (5)",
text="The sequence is 1, 2, 3, 4, 5, and then 13.",
),
DebugTTSSample(
code="YEAR_001",
label="Years and decades (1)",
text="In 1999, people said the '90s were over.",
),
DebugTTSSample(
code="YEAR_002",
label="Years and decades (2)",
text="In 2001, the show premiered; by 2010 it was everywhere.",
),
DebugTTSSample(
code="YEAR_003",
label="Years and decades (3)",
text="The 1980s were loud, and the 1970s were groovy.",
),
DebugTTSSample(
code="YEAR_004",
label="Years and decades (4)",
text="She loved the '80s, but he preferred the '60s.",
),
DebugTTSSample(
code="YEAR_005",
label="Years and decades (5)",
text="In 2024, we looked back at 2020 and planned for 2030.",
),
DebugTTSSample(
code="DATE_001",
label="Dates (1)",
text="On 2023-01-01, we celebrated the new year.",
),
DebugTTSSample(
code="DATE_002",
label="Dates (2)",
text="The deadline is 1999-12-31 at midnight.",
),
DebugTTSSample(
code="DATE_003",
label="Dates (3)",
text="Leap day happens on 2024-02-29.",
),
DebugTTSSample(
code="DATE_004",
label="Dates (4)",
text="Some formats look like 01/02/2003 and can be ambiguous.",
),
DebugTTSSample(
code="DATE_005",
label="Dates (5)",
text="We met on March 5, 2020 and again on Apr. 7, 2021.",
),
DebugTTSSample(
code="CUR_001",
label="Currency symbols (1)",
text="The price is $10.50, but it was £8.00 yesterday.",
),
DebugTTSSample(
code="CUR_002",
label="Currency symbols (2)",
text="Tickets cost €12, and the fine was $0.99.",
),
DebugTTSSample(
code="CUR_003",
label="Currency symbols (3)",
text="The bill was ¥500 and the refund was $-3.25.",
),
DebugTTSSample(
code="CUR_004",
label="Currency symbols (4)",
text="He paid £1,234.56 for the instrument.",
),
DebugTTSSample(
code="CUR_005",
label="Currency symbols (5)",
text="The subscription is $5 per month, or $50 per year.",
),
DebugTTSSample(
code="TITLE_001",
label="Titles and abbreviations (1)",
text="Dr. Smith lives on Elm St. near the U.S. border.",
),
DebugTTSSample(
code="TITLE_002",
label="Titles and abbreviations (2)",
text="Mr. and Mrs. Doe met Prof. Adams at 5 p.m.",
),
DebugTTSSample(
code="TITLE_003",
label="Titles and abbreviations (3)",
text="Gen. Smith spoke to Sgt. Rivera on Main St.",
),
DebugTTSSample(
code="TITLE_004",
label="Titles and abbreviations (4)",
text="The report came from the U.K. office, not the U.S.A. team.",
),
DebugTTSSample(
code="TITLE_005",
label="Titles and abbreviations (5)",
text="St. John's is different from St. Louis.",
),
DebugTTSSample(
code="PUNC_001",
label="Terminal punctuation (1)",
text="This sentence ends without punctuation",
),
DebugTTSSample(
code="PUNC_002",
label="Terminal punctuation (2)",
text="An ellipsis is already present...",
),
DebugTTSSample(
code="PUNC_003",
label="Terminal punctuation (3)",
text="A question without a mark",
),
DebugTTSSample(
code="PUNC_004",
label="Terminal punctuation (4)",
text="An exclamation without a bang",
),
DebugTTSSample(
code="PUNC_005",
label="Terminal punctuation (5)",
text='A quote ends here"',
),
DebugTTSSample(
code="QUOTE_001",
label="ALL CAPS inside quotes (1)",
text='He shouted, "THIS IS IMPORTANT!" and then whispered, "ok."',
),
DebugTTSSample(
code="QUOTE_002",
label="ALL CAPS inside quotes (2)",
text='She said, "NO WAY", but he replied, "maybe".',
),
DebugTTSSample(
code="QUOTE_003",
label="ALL CAPS inside quotes (3)",
text='The sign read "DO NOT ENTER" and the note read "pls knock".',
),
DebugTTSSample(
code="QUOTE_004",
label="ALL CAPS inside quotes (4)",
text='He muttered, "OK", then yelled, "STOP!"',
),
DebugTTSSample(
code="QUOTE_005",
label="ALL CAPS inside quotes (5)",
text='They chanted, "USA!" and someone wrote "idk".',
),
DebugTTSSample(
code="FOOT_001",
label="Footnote indicators (1)",
text="This is a sentence with a footnote[1] and another[12].",
),
DebugTTSSample(
code="FOOT_002",
label="Footnote indicators (2)",
text="Some books use multiple footnotes like this[2][3] in a row.",
),
DebugTTSSample(
code="FOOT_003",
label="Footnote indicators (3)",
text="A footnote can appear mid-sentence[4] and continue afterward.",
),
DebugTTSSample(
code="FOOT_004",
label="Footnote indicators (4)",
text="Edge cases include [0] or very large indices like [1234].",
),
DebugTTSSample(
code="FOOT_005",
label="Footnote indicators (5)",
text="Sometimes a footnote follows punctuation.[5] Sometimes it doesn't[6]",
),
)
def marker_for(code: str) -> str:
return f"{MARKER_PREFIX}{code}{MARKER_SUFFIX}"
def build_debug_epub(dest_path: Path, *, title: str = "abogen debug samples") -> Path:
"""Create a tiny EPUB containing all debug samples.
The text includes stable marker codes so developers can report failures
precisely.
"""
dest_path = Path(dest_path)
dest_path.parent.mkdir(parents=True, exist_ok=True)
chapter_lines: List[str] = [
'<?xml version="1.0" encoding="utf-8"?>',
"<!DOCTYPE html>",
'<html xmlns="http://www.w3.org/1999/xhtml">',
"<head>",
f" <title>{title}</title>",
' <meta charset="utf-8" />',
"</head>",
"<body>",
f" <h1>{title}</h1>",
" <p>Each paragraph begins with a stable debug code marker.</p>",
]
for sample in DEBUG_TTS_SAMPLES:
safe_label = sample.label.replace("&", "and")
chapter_lines.append(f" <h2>{safe_label}</h2>")
chapter_lines.append(
" <p><strong>"
+ marker_for(sample.code)
+ "</strong> "
+ sample.text
+ "</p>"
)
chapter_lines += ["</body>", "</html>"]
chapter_xhtml = "\n".join(chapter_lines)
container_xml = """<?xml version="1.0" encoding="UTF-8"?>
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
<rootfiles>
<rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/>
</rootfiles>
</container>
"""
content_opf = """<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://www.idpf.org/2007/opf" unique-identifier="bookid" version="3.0">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
<dc:identifier id="bookid">abogen-debug-samples</dc:identifier>
<dc:title>abogen debug samples</dc:title>
<dc:language>en</dc:language>
</metadata>
<manifest>
<item id="chapter" href="chapter.xhtml" media-type="application/xhtml+xml" />
<item id="nav" href="nav.xhtml" media-type="application/xhtml+xml" properties="nav" />
</manifest>
<spine>
<itemref idref="chapter" />
</spine>
</package>
"""
nav_xhtml = """<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Navigation</title>
<meta charset="utf-8" />
</head>
<body>
<nav epub:type="toc" id="toc">
<h2>Table of Contents</h2>
<ol>
<li><a href="chapter.xhtml">Debug samples</a></li>
</ol>
</nav>
</body>
</html>
"""
with tempfile.TemporaryDirectory() as tmp:
tmp_path = Path(tmp)
(tmp_path / "mimetype").write_text("application/epub+zip", encoding="utf-8")
meta_inf = tmp_path / "META-INF"
meta_inf.mkdir(parents=True, exist_ok=True)
(meta_inf / "container.xml").write_text(container_xml, encoding="utf-8")
oebps = tmp_path / "OEBPS"
oebps.mkdir(parents=True, exist_ok=True)
(oebps / "content.opf").write_text(content_opf, encoding="utf-8")
(oebps / "chapter.xhtml").write_text(chapter_xhtml, encoding="utf-8")
(oebps / "nav.xhtml").write_text(nav_xhtml, encoding="utf-8")
# Per EPUB spec: mimetype must be the first entry and stored (no compression).
with zipfile.ZipFile(dest_path, "w") as zf:
zf.write(
tmp_path / "mimetype", "mimetype", compress_type=zipfile.ZIP_STORED
)
for source in (
meta_inf / "container.xml",
oebps / "content.opf",
oebps / "chapter.xhtml",
oebps / "nav.xhtml",
):
arcname = str(source.relative_to(tmp_path)).replace("\\", "/")
zf.write(source, arcname, compress_type=zipfile.ZIP_DEFLATED)
return dest_path
def iter_expected_codes() -> Iterable[str]:
for sample in DEBUG_TTS_SAMPLES:
yield sample.code
+489
View File
@@ -0,0 +1,489 @@
from __future__ import annotations
import hashlib
import os
import re
import threading
import time
from collections import Counter
from dataclasses import dataclass, field
from typing import Any, Dict, Iterable, List, Mapping, Optional, Tuple
try: # pragma: no cover - fallback when spaCy not available during tests
import spacy # type: ignore[import-not-found]
except Exception: # pragma: no cover - spaCy optional during runtime bootstrap
spacy = None
_Language = Any # type: ignore[misc,assignment]
Doc = Any # type: ignore[misc,assignment]
Span = Any # type: ignore[misc,assignment]
_TITLE_PREFIXES = (
"mr",
"mrs",
"ms",
"miss",
"dr",
"prof",
"sir",
"madam",
"lady",
"lord",
"capt",
"captain",
"col",
"colonel",
"maj",
"major",
"sgt",
"sergeant",
"rev",
"father",
"mother",
"brother",
"sister",
)
_STOP_LABELS = {
"the",
"that",
"this",
"those",
"these",
"there",
"here",
"then",
"and",
"but",
"or",
"nor",
"so",
"yet",
"dr",
"mr",
"mrs",
"ms",
"miss",
"sir",
"madam",
"lady",
"lord",
}
_EXCLUDED_NER_LABELS = {
"CARDINAL",
"DATE",
"ORDINAL",
"PERCENT",
"TIME",
"LAW",
"MONEY",
"QUANTITY",
}
_TITLE_PATTERN = re.compile(
r"^(?:" + "|".join(re.escape(prefix) for prefix in _TITLE_PREFIXES) + r")\.?\s+",
re.IGNORECASE,
)
_POSSESSIVE_PATTERN = re.compile(r"(?:'s|s|\u2019s)$", re.IGNORECASE)
_NON_WORD_PATTERN = re.compile(r"[^\w\s'-]+")
_MULTI_SPACE_PATTERN = re.compile(r"\s+")
_SUFFIX_PATTERN = re.compile(
r",?\s+(?:jr|sr|ii|iii|iv|v|vi|md|phd|esq|esquire|dds|dvm)\.?$",
re.IGNORECASE,
)
@dataclass(slots=True)
class EntityRecord:
key: Tuple[str, str]
label: str
kind: str
category: str
count: int = 0
samples: List[Dict[str, Any]] = field(default_factory=list)
chapter_indices: set[int] = field(default_factory=set)
forms: Counter = field(default_factory=Counter)
first_position: Optional[Tuple[int, int]] = None
def register(
self, *, chapter_index: int, position: int, text: str, sentence: Optional[str]
) -> None:
self.count += 1
self.chapter_indices.add(chapter_index)
self.forms[text] += 1
if self.first_position is None:
self.first_position = (chapter_index, position)
if sentence and len(self.samples) < 5:
payload = {
"excerpt": sentence.strip(),
"chapter_index": chapter_index,
}
if payload not in self.samples:
self.samples.append(payload)
def as_dict(self, ordinal: int) -> Dict[str, Any]:
chapter_indices = sorted(self.chapter_indices)
first_chapter = chapter_indices[0] if chapter_indices else None
return {
"id": f"{self.category}_{ordinal}",
"label": self.label,
"normalized": self.key[1],
"category": self.category,
"kind": self.kind,
"count": self.count,
"samples": list(self.samples),
"chapter_indices": chapter_indices,
"first_chapter": first_chapter,
"forms": self.forms.most_common(6),
}
@dataclass(slots=True)
class EntityExtractionResult:
summary: Dict[str, Any]
cache_key: str
elapsed: float
errors: List[str]
class EntityModelError(RuntimeError):
pass
_MODEL_CACHE: Dict[str, Any] = {}
_MODEL_LOCK = threading.RLock()
def _resolve_model_name(language: str) -> str:
override = os.environ.get("ABOGEN_SPACY_MODEL")
if override:
return override.strip()
lowered = language.strip().lower()
if lowered.startswith("en"):
return "en_core_web_sm"
return "en_core_web_sm"
def _load_model(language: str) -> Any:
if spacy is None:
raise EntityModelError(
"spaCy is not available. Install spaCy to enable entity extraction."
)
model_name = _resolve_model_name(language)
cache_key = model_name.lower()
with _MODEL_LOCK:
if cache_key in _MODEL_CACHE:
return _MODEL_CACHE[cache_key]
try:
nlp = spacy.load(model_name) # type: ignore[arg-type]
except OSError as exc: # pragma: no cover - external dependency failure
raise EntityModelError(
f"spaCy model '{model_name}' is not installed. Download it with "
"`python -m spacy download en_core_web_sm`."
) from exc
nlp.max_length = max(nlp.max_length, 2_000_000)
_MODEL_CACHE[cache_key] = nlp
return nlp
def _normalize_label(text: str) -> str:
if not text:
return ""
stripped = text.strip().strip("\"'`“”’")
if not stripped:
return ""
stripped = _TITLE_PATTERN.sub("", stripped)
stripped = _SUFFIX_PATTERN.sub("", stripped)
stripped = _POSSESSIVE_PATTERN.sub("", stripped)
stripped = _NON_WORD_PATTERN.sub(" ", stripped)
stripped = _MULTI_SPACE_PATTERN.sub(" ", stripped)
stripped = stripped.strip()
if not stripped or stripped.lower() in _STOP_LABELS:
return ""
parts = stripped.split()
if not parts:
return ""
if len(parts) == 1 and len(parts[0]) <= 1:
return ""
# Normalise casing: preserve uppercase abbreviations, otherwise title case.
normalized_parts = []
for index, part in enumerate(parts):
if part.isupper():
normalized_parts.append(part)
elif part[:1].isupper():
normalized_parts.append(part[:1].upper() + part[1:])
elif index == 0:
normalized_parts.append(part[:1].upper() + part[1:])
else:
normalized_parts.append(part)
normalized = " ".join(normalized_parts).strip()
if normalized.lower() in _STOP_LABELS:
return ""
return normalized
def _token_key(value: str) -> str:
return _MULTI_SPACE_PATTERN.sub(" ", value.lower().strip()).strip()
def _iter_named_entities(doc: Any) -> Iterable[Any]: # type: ignore[override]
for ent in getattr(doc, "ents", ()):
if ent.label_ == "":
continue
yield ent
def _extract_propn_tokens(doc: Any) -> Iterable[Any]: # type: ignore[override]
seen: set[Tuple[int, int]] = set()
for ent in getattr(doc, "ents", ()): # guard multi-token spans
seen.add((ent.start, ent.end))
for token in doc:
if token.pos_ != "PROPN":
continue
span_key = (token.i, token.i + 1)
if span_key in seen:
continue
if token.is_stop:
continue
text = token.text.strip()
if not text:
continue
if token.ent_type_:
continue
yield doc[token.i : token.i + 1]
def _empty_result(
cache_key: str, error: Optional[str] = None
) -> EntityExtractionResult:
payload = {
"people": [],
"entities": [],
"index": {"tokens": []},
"stats": {
"tokens": 0,
"chapters": 0,
"processed": False,
},
"model": None,
}
errors = [error] if error else []
return EntityExtractionResult(
summary=payload, cache_key=cache_key, elapsed=0.0, errors=errors
)
def extract_entities(
chapters: Iterable[Mapping[str, Any]],
*,
language: str = "en",
) -> EntityExtractionResult:
start = time.perf_counter()
normalized_language = language or "en"
combined_hasher = hashlib.sha1()
chapter_texts: List[Tuple[int, str]] = []
for idx, chapter in enumerate(chapters):
text = chapter.get("text") if isinstance(chapter, Mapping) else None
text_value = str(text or "")
original_index = idx
if isinstance(chapter, Mapping):
try:
original_index = int(chapter.get("index", idx))
except (TypeError, ValueError):
original_index = idx
chapter_texts.append((original_index, text_value))
if text_value:
combined_hasher.update(text_value.encode("utf-8", "ignore"))
combined_hasher.update(str(original_index).encode("utf-8", "ignore"))
cache_key = combined_hasher.hexdigest()
if not chapter_texts:
return _empty_result(cache_key)
try:
nlp = _load_model(normalized_language)
except EntityModelError as exc:
return _empty_result(cache_key, str(exc))
records: Dict[Tuple[str, str], EntityRecord] = {}
tokens_for_index: Dict[str, Dict[str, Any]] = {}
processed_tokens = 0
for chapter_index, text in chapter_texts:
trimmed = text.strip()
if not trimmed:
continue
if len(trimmed) + 1024 > nlp.max_length:
nlp.max_length = len(trimmed) + 1024
doc = nlp(trimmed)
def _register_span(span: Any, category_hint: Optional[str] = None) -> None:
nonlocal processed_tokens
if category_hint is None and span.label_ in _EXCLUDED_NER_LABELS:
return
cleaned = _normalize_label(span.text)
if not cleaned:
return
key = _token_key(cleaned)
if not key:
return
category = category_hint or (
"people" if span.label_ == "PERSON" else "entities"
)
record_key = (category, key)
record = records.get(record_key)
if record is None:
record = EntityRecord(
key=record_key,
label=cleaned,
kind=span.label_
or ("PROPN" if category == "entities" else "PERSON"),
category=category,
)
records[record_key] = record
sentence = (
span.sent.text
if hasattr(span, "sent") and span.sent is not None
else None
)
record.register(
chapter_index=chapter_index,
position=span.start,
text=span.text,
sentence=sentence,
)
processed_tokens += 1
index_entry = tokens_for_index.get(key)
if index_entry is None:
index_entry = {
"token": record.label,
"normalized": key,
"category": category,
"count": 0,
"samples": [],
}
tokens_for_index[key] = index_entry
index_entry["count"] += 1
if sentence and len(index_entry["samples"]) < 3:
if sentence not in index_entry["samples"]:
index_entry["samples"].append(sentence)
for ent in _iter_named_entities(doc):
_register_span(ent)
for span in _extract_propn_tokens(doc):
_register_span(span, category_hint="entities")
elapsed = time.perf_counter() - start
people_records = [
record for record in records.values() if record.category == "people"
]
people_keys = {record.key[1] for record in people_records}
entity_records = [
record
for record in records.values()
if record.category == "entities"
and record.key[1] not in people_keys
and record.kind != "PERSON"
]
people_records.sort(key=lambda rec: (-rec.count, rec.label))
entity_records.sort(key=lambda rec: (-rec.count, rec.label))
people_payload = [
record.as_dict(index + 1) for index, record in enumerate(people_records)
]
entity_payload = [
record.as_dict(index + 1) for index, record in enumerate(entity_records)
]
index_payload = sorted(
tokens_for_index.values(), key=lambda item: (-item["count"], item["token"])
)
summary = {
"people": people_payload,
"entities": entity_payload,
"index": {"tokens": index_payload},
"stats": {
"tokens": processed_tokens,
"chapters": len(chapter_texts),
"processed": True,
"people": len(people_payload),
"entities": len(entity_payload),
},
"model": {
"name": getattr(nlp, "meta", {}).get("name", "unknown"),
"version": getattr(nlp, "meta", {}).get("version", "unknown"),
"lang": getattr(nlp, "meta", {}).get("lang", normalized_language),
},
}
return EntityExtractionResult(
summary=summary, cache_key=cache_key, elapsed=elapsed, errors=[]
)
def search_tokens(
index: Mapping[str, Any], query: str, *, limit: int = 15
) -> List[Dict[str, Any]]:
tokens = index.get("tokens") if isinstance(index, Mapping) else None
if not isinstance(tokens, list) or not query:
return []
normalized = query.strip().lower()
if not normalized:
return tokens[:limit]
results: List[Dict[str, Any]] = []
for entry in tokens:
token_label = str(entry.get("token", ""))
normalized_label = token_label.lower()
if normalized in normalized_label or normalized in str(
entry.get("normalized", "")
):
results.append(entry)
if len(results) >= limit:
break
return results
def merge_override(
summary: Mapping[str, Any], overrides: Mapping[str, Mapping[str, Any]]
) -> Dict[str, Any]:
if not isinstance(summary, Mapping):
return {"people": [], "entities": []}
merged_summary: Dict[str, Any] = dict(summary)
for key in ("people", "entities"):
items = summary.get(key)
if not isinstance(items, list):
continue
merged_items: List[Dict[str, Any]] = []
for entry in items:
if not isinstance(entry, Mapping):
continue
normalized = _token_key(
str(entry.get("normalized") or entry.get("label") or "")
)
merged = dict(entry)
if normalized and normalized in overrides:
merged_override = dict(overrides[normalized])
merged["override"] = merged_override
merged_items.append(merged)
merged_summary[key] = merged_items
return merged_summary
def normalize_token(token: str) -> str:
return _token_key(_normalize_label(token))
def normalize_manual_override_token(token: str) -> str:
if not token:
return ""
stripped = token.strip().strip("\"'`“”’")
if not stripped:
return ""
return _MULTI_SPACE_PATTERN.sub(" ", stripped.lower()).strip()
+3
View File
@@ -0,0 +1,3 @@
from .exporter import EPUB3PackageBuilder, build_epub3_package
__all__ = ["EPUB3PackageBuilder", "build_epub3_package"]
+910
View File
@@ -0,0 +1,910 @@
from __future__ import annotations
import html
import re
import shutil
import uuid
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from tempfile import TemporaryDirectory
from typing import Any, Dict, Iterable, List, Optional, Pattern, Sequence, Tuple
import zipfile
from abogen.text_extractor import ExtractedChapter, ExtractionResult
@dataclass(slots=True)
class ChunkOverlay:
id: str
text: str
original_text: Optional[str]
start: Optional[float]
end: Optional[float]
speaker_id: str
voice: Optional[str]
level: Optional[str] = None
group_id: Optional[str] = None
@dataclass(slots=True)
class ChapterDocument:
index: int # zero-based
title: str
xhtml_name: str
smil_name: str
chunks: List[ChunkOverlay]
start: Optional[float]
end: Optional[float]
class EPUB3PackageBuilder:
"""Constructs an EPUB 3 package with media overlays."""
def __init__(
self,
*,
output_path: Path,
book_id: str,
extraction: ExtractionResult,
metadata_tags: Dict[str, Any],
chapter_markers: Sequence[Dict[str, Any]],
chunk_markers: Sequence[Dict[str, Any]],
chunks: Iterable[Dict[str, Any]],
audio_path: Path,
speaker_mode: str = "single",
cover_image_path: Optional[Path] = None,
cover_image_mime: Optional[str] = None,
) -> None:
self.output_path = output_path
self.book_id = book_id or str(uuid.uuid4())
self.extraction = extraction
self.metadata_tags = _normalize_metadata(metadata_tags)
self.chapter_markers = list(chapter_markers or [])
self.chunk_markers = list(chunk_markers or [])
self.chunks = list(chunks or [])
self.audio_path = audio_path
self.speaker_mode = speaker_mode or "single"
self.cover_image_path = cover_image_path if cover_image_path and cover_image_path.exists() else None
self.cover_image_mime = cover_image_mime
self._combined_metadata = _combine_metadata(extraction.metadata, self.metadata_tags)
self._title = self._combined_metadata.get("title") or self._fallback_title()
self._authors = _split_authors(self._combined_metadata)
self._language = self._determine_language()
self._publisher = self._combined_metadata.get("publisher") or ""
self._description = self._combined_metadata.get("comment")
self._duration = _calculate_total_duration(self.chunk_markers, self.chapter_markers)
self._modified = _utc_now_iso()
def build(self) -> Path:
if not self.audio_path or not self.audio_path.exists():
raise FileNotFoundError(f"Audio asset missing: {self.audio_path}")
chapter_documents = self._build_chapter_documents()
with TemporaryDirectory() as tmp_dir:
root = Path(tmp_dir)
oebps = root / "OEBPS"
text_dir = oebps / "text"
smil_dir = oebps / "smil"
audio_dir = oebps / "audio"
image_dir = oebps / "images"
stylesheet_dir = oebps / "styles"
for directory in (oebps, text_dir, smil_dir, audio_dir, stylesheet_dir):
directory.mkdir(parents=True, exist_ok=True)
if self.cover_image_path:
image_dir.mkdir(parents=True, exist_ok=True)
_write_mimetype(root)
_write_container_xml(root)
audio_filename = self.audio_path.name
embedded_audio = audio_dir / audio_filename
shutil.copy2(self.audio_path, embedded_audio)
if self.cover_image_path:
shutil.copy2(self.cover_image_path, image_dir / self.cover_image_path.name)
stylesheet_path = stylesheet_dir / "style.css"
stylesheet_path.write_text(_DEFAULT_STYLESHEET, encoding="utf-8")
for chapter in chapter_documents:
chapter_path = text_dir / chapter.xhtml_name
chapter_path.write_text(
self._render_chapter_xhtml(chapter),
encoding="utf-8",
)
smil_path = smil_dir / chapter.smil_name
smil_path.write_text(
self._render_chapter_smil(chapter, f"audio/{audio_filename}"),
encoding="utf-8",
)
nav_path = oebps / "nav.xhtml"
nav_path.write_text(self._render_nav(chapter_documents), encoding="utf-8")
opf_path = oebps / "content.opf"
opf_path.write_text(
self._render_opf(
chapter_documents,
audio_filename,
has_cover=self.cover_image_path is not None,
stylesheet_path=stylesheet_path.relative_to(oebps),
),
encoding="utf-8",
)
self.output_path.parent.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile(self.output_path, "w", compression=zipfile.ZIP_DEFLATED) as archive:
# Ensure mimetype is the first entry and stored without compression
mimetype_path = root / "mimetype"
info = zipfile.ZipInfo("mimetype")
info.compress_type = zipfile.ZIP_STORED
archive.writestr(info, mimetype_path.read_bytes())
for file_path in sorted(root.rglob("*")):
if file_path == mimetype_path or file_path.is_dir():
continue
archive.write(file_path, file_path.relative_to(root))
return self.output_path
# ------------------------------------------------------------------
def _build_chapter_documents(self) -> List[ChapterDocument]:
chunk_lookup = _build_chunk_lookup(self.chunks)
markers_by_chapter = _group_markers_by_chapter(self.chunk_markers)
chapter_meta = {int(entry.get("index", idx + 1)) - 1: dict(entry) for idx, entry in enumerate(self.chapter_markers)}
documents: List[ChapterDocument] = []
for chapter_index, chapter in enumerate(self.extraction.chapters):
markers = markers_by_chapter.get(chapter_index, [])
if not markers and chunk_lookup.by_chapter.get(chapter_index):
markers = [
{
"id": item.get("id"),
"chapter_index": chapter_index,
"chunk_index": item.get("chunk_index"),
"start": None,
"end": None,
"speaker_id": item.get("speaker_id", "narrator"),
"voice": item.get("voice"),
}
for item in chunk_lookup.by_chapter.get(chapter_index, [])
]
if not markers:
markers = [
{
"id": f"chap{chapter_index:04d}_auto0000",
"chapter_index": chapter_index,
"chunk_index": 0,
"start": chapter_meta.get(chapter_index, {}).get("start"),
"end": chapter_meta.get(chapter_index, {}).get("end"),
"speaker_id": "narrator",
"voice": None,
}
]
overlays = self._build_overlays_for_chapter(
chapter_index,
markers,
chunk_lookup,
)
xhtml_name = f"chapter_{chapter_index + 1:04d}.xhtml"
smil_name = f"chapter_{chapter_index + 1:04d}.smil"
chapter_start = chapter_meta.get(chapter_index, {}).get("start")
chapter_end = chapter_meta.get(chapter_index, {}).get("end")
documents.append(
ChapterDocument(
index=chapter_index,
title=chapter.title or f"Chapter {chapter_index + 1}",
xhtml_name=xhtml_name,
smil_name=smil_name,
chunks=overlays,
start=chapter_start,
end=chapter_end,
)
)
return documents
def _build_overlays_for_chapter(
self,
chapter_index: int,
markers: Sequence[Dict[str, Any]],
chunk_lookup: "ChunkLookup",
) -> List[ChunkOverlay]:
overlays: List[ChunkOverlay] = []
used_ids: set[str] = set()
chapter_chunks = list(chunk_lookup.by_chapter.get(chapter_index, []))
chapter_chunks.sort(key=lambda entry: _safe_int(entry.get("chunk_index")))
for position, marker in enumerate(markers):
chunk_id = marker.get("id")
chunk_entry = None
if chunk_id and chunk_id in chunk_lookup.by_id:
chunk_entry = chunk_lookup.by_id[chunk_id]
else:
candidate_index = _safe_int(marker.get("chunk_index"))
chunk_entry = _find_chunk_by_index(chapter_chunks, candidate_index)
if chunk_entry is None and chapter_chunks and position < len(chapter_chunks):
chunk_entry = chapter_chunks[position]
level = None
if chunk_entry is None:
text = self.extraction.chapters[chapter_index].text
speaker_id = str(marker.get("speaker_id") or "narrator")
voice = marker.get("voice")
else:
display_text = chunk_entry.get("display_text")
text = str(chunk_entry.get("text") or "")
speaker_id = str(chunk_entry.get("speaker_id") or marker.get("speaker_id") or "narrator")
voice = chunk_entry.get("voice") or chunk_entry.get("resolved_voice") or marker.get("voice")
level = chunk_entry.get("level") or None
if chunk_entry is None:
level = None
normalized_id = _normalize_chunk_id(chunk_id) if chunk_id else None
if not normalized_id:
normalized_id = f"chap{chapter_index:04d}_chunk{position:04d}"
while normalized_id in used_ids:
normalized_id = f"{normalized_id}_dup"
used_ids.add(normalized_id)
raw_group_key = chunk_entry.get("id") if chunk_entry else chunk_id
group_id = _derive_group_id(raw_group_key, level)
normalized_group_id = _normalize_chunk_id(group_id) if group_id else None
original_text = None
if chunk_entry is not None:
original_text = chunk_entry.get("original_text") or chunk_entry.get("display_text")
overlays.append(
ChunkOverlay(
id=normalized_id,
text=text or self.extraction.chapters[chapter_index].text,
original_text=str(original_text) if original_text is not None else None,
start=_safe_float(marker.get("start")),
end=_safe_float(marker.get("end")),
speaker_id=speaker_id,
voice=str(voice) if voice else None,
level=str(level) if level else None,
group_id=normalized_group_id,
)
)
chapter_text = ""
if 0 <= chapter_index < len(self.extraction.chapters):
chapter_entry = self.extraction.chapters[chapter_index]
chapter_text = getattr(chapter_entry, "text", "") or ""
_restore_original_chunk_text(chapter_text, overlays)
return overlays
def _render_chapter_xhtml(self, chapter: ChapterDocument) -> str:
language = html.escape(self._language or "en")
title = html.escape(chapter.title)
grouped_chunks = _group_chunks_for_render(chapter.chunks)
chunk_html = "\n".join(
_render_chunk_group_html(group_id, items) for group_id, items in grouped_chunks
)
if not chunk_html:
chunk_html = "<p></p>"
original_block = ""
if chapter.chunks:
original_text = "".join((chunk.original_text if chunk.original_text is not None else (chunk.text or "")) for chunk in chapter.chunks)
if original_text:
safe_original = html.escape(original_text)
original_block = (
" <pre class=\"chapter-original\" hidden=\"hidden\" aria-hidden=\"true\">\n"
f"{safe_original}\n"
" </pre>"
)
return (
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n"
"<html xmlns=\"http://www.w3.org/1999/xhtml\" xmlns:epub=\"http://www.idpf.org/2007/ops\" xml:lang=\"{lang}\" lang=\"{lang}\">\n"
" <head>\n"
" <title>{title}</title>\n"
" <meta charset=\"utf-8\"/>\n"
" <link rel=\"stylesheet\" type=\"text/css\" href=\"styles/style.css\"/>\n"
" </head>\n"
" <body>\n"
" <section epub:type=\"chapter\" id=\"chapter-{index:04d}\">\n"
" <h1>{title}</h1>\n"
" {chunks}\n"
"{original_block}"
" </section>\n"
" </body>\n"
"</html>\n"
).format(
lang=language,
title=title,
index=chapter.index + 1,
chunks=chunk_html,
original_block=("" if not original_block else f"{original_block}\n"),
)
def _render_chapter_smil(self, chapter: ChapterDocument, audio_href: str) -> str:
par_lines = []
for chunk in chapter.chunks:
par_lines.append(
" <par id=\"par-{chunk_id}\">\n"
" <text src=\"text/{xhtml}#{chunk_id}\"/>\n"
" <audio src=\"{audio}\" clipBegin=\"{start}\" clipEnd=\"{end}\"/>\n"
" </par>".format(
chunk_id=html.escape(chunk.id),
xhtml=html.escape(chapter.xhtml_name),
audio=html.escape(audio_href),
start=_format_smil_time(chunk.start),
end=_format_smil_time(chunk.end),
)
)
return (
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n"
"<smil xmlns=\"http://www.w3.org/2001/SMIL20/Language\" xmlns:epub=\"http://www.idpf.org/2007/ops\">\n"
" <head>\n"
" <meta name=\"dc:title\" content=\"{title}\"/>\n"
" <meta name=\"dtb:uid\" content=\"{book_id}\"/>\n"
" <meta name=\"dtb:generator\" content=\"Abogen\"/>\n"
" </head>\n"
" <body>\n"
" <seq id=\"seq-{index:04d}\" epub:textref=\"text/{xhtml}\">\n"
"{pars}\n"
" </seq>\n"
" </body>\n"
"</smil>\n"
).format(
title=html.escape(chapter.title),
book_id=html.escape(self.book_id),
index=chapter.index + 1,
xhtml=html.escape(chapter.xhtml_name),
pars="\n".join(par_lines) if par_lines else " <par/>",
)
def _render_nav(self, chapters: Sequence[ChapterDocument]) -> str:
items = []
for chapter in chapters:
href = f"text/{chapter.xhtml_name}"
items.append(
" <li><a href=\"{href}\">{title}</a></li>".format(
href=html.escape(href),
title=html.escape(chapter.title),
)
)
return (
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n"
"<html xmlns=\"http://www.w3.org/1999/xhtml\" xmlns:epub=\"http://www.idpf.org/2007/ops\" xml:lang=\"{lang}\">\n"
" <head>\n"
" <title>Navigation</title>\n"
" <meta charset=\"utf-8\"/>\n"
" </head>\n"
" <body>\n"
" <nav epub:type=\"toc\" id=\"toc\">\n"
" <h1>{title}</h1>\n"
" <ol>\n"
"{items}\n"
" </ol>\n"
" </nav>\n"
" </body>\n"
"</html>\n"
).format(
lang=html.escape(self._language or "en"),
title=html.escape(self._title),
items="\n".join(items) if items else " <li><a href=\"text/chapter_0001.xhtml\">Chapter 1</a></li>",
)
def _render_opf(
self,
chapters: Sequence[ChapterDocument],
audio_filename: str,
*,
has_cover: bool,
stylesheet_path: Path,
) -> str:
manifest_items = []
spine_refs = []
for chapter in chapters:
item_id = f"chap{chapter.index + 1:04d}"
overlay_id = f"mo-{chapter.index + 1:04d}"
manifest_items.append(
" <item id=\"{item_id}\" href=\"text/{href}\" media-type=\"application/xhtml+xml\" media-overlay=\"{overlay_id}\"/>".format(
item_id=item_id,
href=html.escape(chapter.xhtml_name),
overlay_id=overlay_id,
)
)
manifest_items.append(
" <item id=\"{overlay_id}\" href=\"smil/{smil}\" media-type=\"application/smil+xml\"/>".format(
overlay_id=overlay_id,
smil=html.escape(chapter.smil_name),
)
)
spine_refs.append(f" <itemref idref=\"{item_id}\"/>")
audio_item_id = "primary-audio"
manifest_items.append(
" <item id=\"{item_id}\" href=\"audio/{href}\" media-type=\"{mime}\"/>".format(
item_id=audio_item_id,
href=html.escape(audio_filename),
mime=_detect_audio_mime(audio_filename),
)
)
manifest_items.append(
" <item id=\"nav\" href=\"nav.xhtml\" media-type=\"application/xhtml+xml\" properties=\"nav\"/>"
)
manifest_items.append(
" <item id=\"style\" href=\"{href}\" media-type=\"text/css\"/>".format(
href=html.escape(str(stylesheet_path).replace("\\", "/")),
)
)
if has_cover and self.cover_image_path:
cover_id = "cover-image"
manifest_items.append(
" <item id=\"{item_id}\" href=\"images/{href}\" media-type=\"{mime}\" properties=\"cover-image\"/>".format(
item_id=cover_id,
href=html.escape(self.cover_image_path.name),
mime=self.cover_image_mime or _detect_image_mime(self.cover_image_path.suffix),
)
)
metadata_elements = _render_metadata_xml(
self._title,
self._authors,
self._language,
self.book_id,
duration=self._duration,
publisher=self._publisher,
description=self._description,
speaker_mode=self.speaker_mode,
modified=self._modified,
)
return (
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n"
"<package xmlns=\"http://www.idpf.org/2007/opf\" version=\"3.0\" unique-identifier=\"book-id\">\n"
" <metadata xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:opf=\"http://www.idpf.org/2007/opf\" xmlns:media=\"http://www.idpf.org/epub/vocab/mediaoverlays/#\" xmlns:abogen=\"https://abogen.app/ns#\" xmlns:dcterms=\"http://purl.org/dc/terms/\">\n"
"{metadata}\n"
" </metadata>\n"
" <manifest>\n"
"{manifest}\n"
" </manifest>\n"
" <spine>\n"
"{spine}\n"
" </spine>\n"
"</package>\n"
).format(
metadata="\n".join(metadata_elements),
manifest="\n".join(manifest_items),
spine="\n".join(spine_refs) if spine_refs else " <itemref idref=\"chap0001\"/>",
)
def _fallback_title(self) -> str:
if self.extraction.chapters:
first_title = self.extraction.chapters[0].title
if first_title:
return first_title
return "Generated Audiobook"
def _determine_language(self) -> str:
language = self._combined_metadata.get("language")
if language:
return language
return "en"
def build_epub3_package(
*,
output_path: Path,
book_id: str,
extraction: ExtractionResult,
metadata_tags: Dict[str, Any],
chapter_markers: Sequence[Dict[str, Any]],
chunk_markers: Sequence[Dict[str, Any]],
chunks: Iterable[Dict[str, Any]],
audio_path: Path,
speaker_mode: str = "single",
cover_image_path: Optional[Path] = None,
cover_image_mime: Optional[str] = None,
) -> Path:
builder = EPUB3PackageBuilder(
output_path=output_path,
book_id=book_id,
extraction=extraction,
metadata_tags=metadata_tags,
chapter_markers=chapter_markers,
chunk_markers=chunk_markers,
chunks=chunks,
audio_path=audio_path,
speaker_mode=speaker_mode,
cover_image_path=cover_image_path,
cover_image_mime=cover_image_mime,
)
return builder.build()
# ---------------------------------------------------------------------------
# Helpers
@dataclass
class ChunkLookup:
by_id: Dict[str, Dict[str, Any]]
by_chapter: Dict[int, List[Dict[str, Any]]]
def _normalize_metadata(metadata: Optional[Dict[str, Any]]) -> Dict[str, str]:
normalized: Dict[str, str] = {}
for key, value in (metadata or {}).items():
if value is None:
continue
normalized[str(key).lower()] = str(value)
return normalized
def _combine_metadata(*sources: Dict[str, Any]) -> Dict[str, str]:
combined: Dict[str, str] = {}
for source in sources:
for key, value in (source or {}).items():
if value is None:
continue
combined[str(key).lower()] = str(value)
return combined
def _split_authors(metadata: Dict[str, str]) -> List[str]:
candidates = []
for key in ("artist", "author", "authors", "album_artist", "creator"):
value = metadata.get(key)
if value:
candidates.extend(part.strip() for part in value.replace(";", ",").split(","))
return [author for author in candidates if author]
def _calculate_total_duration(
chunk_markers: Sequence[Dict[str, Any]],
chapter_markers: Sequence[Dict[str, Any]],
) -> Optional[float]:
candidates: List[float] = []
for marker in chunk_markers or []:
end_value = _safe_float(marker.get("end"))
if end_value is not None:
candidates.append(end_value)
for marker in chapter_markers or []:
end_value = _safe_float(marker.get("end"))
if end_value is not None:
candidates.append(end_value)
if not candidates:
return None
return max(candidates)
def _write_mimetype(root: Path) -> None:
(root / "mimetype").write_text("application/epub+zip", encoding="utf-8")
def _write_container_xml(root: Path) -> None:
meta_inf = root / "META-INF"
meta_inf.mkdir(parents=True, exist_ok=True)
container = meta_inf / "container.xml"
container.write_text(
(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
"<container version=\"1.0\" xmlns=\"urn:oasis:names:tc:opendocument:xmlns:container\">\n"
" <rootfiles>\n"
" <rootfile full-path=\"OEBPS/content.opf\" media-type=\"application/oebps-package+xml\"/>\n"
" </rootfiles>\n"
"</container>\n"
),
encoding="utf-8",
)
def _build_chunk_lookup(chunks: Iterable[Dict[str, Any]]) -> ChunkLookup:
by_id: Dict[str, Dict[str, Any]] = {}
by_chapter: Dict[int, List[Dict[str, Any]]] = {}
for entry in chunks or []:
if not isinstance(entry, dict):
continue
chunk_id = entry.get("id")
if chunk_id:
by_id[str(chunk_id)] = dict(entry)
chapter_index = _safe_int(entry.get("chapter_index"))
by_chapter.setdefault(chapter_index, []).append(dict(entry))
return ChunkLookup(by_id=by_id, by_chapter=by_chapter)
def _group_markers_by_chapter(markers: Iterable[Dict[str, Any]]) -> Dict[int, List[Dict[str, Any]]]:
grouped: Dict[int, List[Dict[str, Any]]] = {}
for entry in markers or []:
if not isinstance(entry, dict):
continue
chapter_index = _safe_int(entry.get("chapter_index"))
grouped.setdefault(chapter_index, []).append(dict(entry))
for chapter_index, items in grouped.items():
items.sort(key=lambda payload: (_safe_int(payload.get("chunk_index")), _safe_float(payload.get("start")) or 0.0))
return grouped
def _find_chunk_by_index(
chapter_chunks: Sequence[Dict[str, Any]],
chunk_index: Optional[int],
) -> Optional[Dict[str, Any]]:
if chunk_index is None:
return None
for entry in chapter_chunks:
if _safe_int(entry.get("chunk_index")) == chunk_index:
return entry
return None
def _normalize_chunk_id(chunk_id: Optional[Any]) -> Optional[str]:
if chunk_id is None:
return None
text = str(chunk_id).strip()
if not text:
return None
safe = "".join(ch if ch.isalnum() or ch in {"_", "-"} else "_" for ch in text)
return safe[:120]
def _derive_group_id(chunk_id: Optional[Any], level: Optional[Any]) -> Optional[str]:
if chunk_id is None:
return None
text = str(chunk_id).strip()
if not text:
return None
if str(level or "").lower() == "sentence":
match = re.match(r"(.+?)_s\d+(?:_.*)?$", text)
if match:
return match.group(1)
return text
def _group_chunks_for_render(chunks: Sequence[ChunkOverlay]) -> List[Tuple[Optional[str], List[ChunkOverlay]]]:
groups: List[Tuple[Optional[str], List[ChunkOverlay]]] = []
current_key: Optional[str] = None
current_items: List[ChunkOverlay] = []
for chunk in chunks:
key = chunk.group_id or chunk.id
if current_items and key != current_key:
groups.append((current_key, current_items))
current_items = []
if not current_items:
current_key = key
current_items.append(chunk)
if current_items:
groups.append((current_key, current_items))
return groups
def _render_chunk_inline(chunk: ChunkOverlay) -> str:
escaped_id = html.escape(chunk.id)
speaker_attr = f" data-speaker=\"{html.escape(chunk.speaker_id)}\"" if chunk.speaker_id else ""
voice_attr = f" data-voice=\"{html.escape(chunk.voice)}\"" if chunk.voice else ""
level_attr = f" data-level=\"{html.escape(chunk.level)}\"" if chunk.level else ""
raw_text = chunk.text or ""
escaped_text = html.escape(raw_text)
if not escaped_text:
escaped_text = "&nbsp;"
return (
f"<span class=\"chunk\" id=\"{escaped_id}\"{speaker_attr}{voice_attr}{level_attr}>"
f"{escaped_text}"
"</span>"
)
def _render_chunk_group_html(group_id: Optional[str], chunks: Sequence[ChunkOverlay]) -> str:
if not chunks:
return ""
group_attr = f" data-group=\"{html.escape(group_id)}\"" if group_id else ""
inline_html = "".join(_render_chunk_inline(chunk) for chunk in chunks)
if not inline_html:
inline_html = "&nbsp;"
return f" <p class=\"chunk-group\"{group_attr}>{inline_html}</p>"
def _format_smil_time(value: Optional[float]) -> str:
if value is None or value < 0:
value = 0.0
total_ms = int(round(value * 1000))
hours, remainder = divmod(total_ms, 3600_000)
minutes, remainder = divmod(remainder, 60_000)
seconds, milliseconds = divmod(remainder, 1000)
return f"{hours:02d}:{minutes:02d}:{seconds:02d}.{milliseconds:03d}"
def _safe_int(value: Any, default: int = 0) -> int:
try:
return int(value)
except (TypeError, ValueError):
return default
def _safe_float(value: Any) -> Optional[float]:
if value is None:
return None
try:
return float(value)
except (TypeError, ValueError):
return None
def _restore_original_chunk_text(chapter_text: str, overlays: List[ChunkOverlay]) -> None:
if not chapter_text or not overlays:
return
cursor = 0
for chunk in overlays:
if chunk.original_text is not None:
prepared = _prepare_display_text(chunk.original_text)
chunk.text = prepared
continue
candidate = chunk.text or ""
if not candidate:
continue
match = _search_original_span(chapter_text, candidate, cursor)
if match is None and cursor:
match = _search_original_span(chapter_text, candidate, 0)
if match is None:
if chunk.original_text is None:
chunk.original_text = chunk.text
chunk.text = _prepare_display_text(chunk.text or "")
continue
start, end = match
segment = chapter_text[start:end]
chunk.original_text = segment
chunk.text = _prepare_display_text(segment)
cursor = end
def _prepare_display_text(value: str) -> str:
if not value:
return ""
cleaned = re.sub(r"(?:[ \t]*\r?\n)+\Z", "", value)
return cleaned if cleaned else ""
def _search_original_span(source: str, normalized: str, start: int) -> Optional[Tuple[int, int]]:
if not normalized:
return None
pattern = _build_chunk_pattern(normalized)
match = pattern.search(source, start)
if not match:
return None
return match.start(1), match.end(1)
_CHUNK_REGEX_CACHE: Dict[str, Pattern[str]] = {}
def _build_chunk_pattern(text: str) -> Pattern[str]:
cached = _CHUNK_REGEX_CACHE.get(text)
if cached is not None:
return cached
escaped = re.escape(text)
escaped = escaped.replace(r"\ ", r"\s+")
pattern = re.compile(r"(\s*" + escaped + r"\s*)", re.DOTALL)
_CHUNK_REGEX_CACHE[text] = pattern
return pattern
def _render_metadata_xml(
title: str,
authors: Sequence[str],
language: str,
book_id: str,
*,
duration: Optional[float],
publisher: Optional[str],
description: Optional[str],
speaker_mode: Optional[str],
modified: Optional[str],
) -> List[str]:
elements = [
f" <dc:identifier id=\"book-id\">{html.escape(book_id)}</dc:identifier>",
f" <dc:title>{html.escape(title)}</dc:title>",
f" <dc:language>{html.escape(language or 'en')}</dc:language>",
]
for author in authors or ["Unknown"]:
elements.append(f" <dc:creator>{html.escape(author)}</dc:creator>")
if publisher:
elements.append(f" <dc:publisher>{html.escape(publisher)}</dc:publisher>")
if description:
elements.append(f" <dc:description>{html.escape(description)}</dc:description>")
if duration is not None:
elements.append(f" <meta property=\"media:duration\">{_format_iso_duration(duration)}</meta>")
if speaker_mode:
elements.append(
" <meta property=\"abogen:speakerMode\">{}</meta>".format(
html.escape(str(speaker_mode))
)
)
if modified:
elements.append(f" <meta property=\"dcterms:modified\">{html.escape(modified)}</meta>")
return elements
def _format_iso_duration(value: float) -> str:
total_seconds = int(value)
remainder = value - total_seconds
hours, remainder_seconds = divmod(total_seconds, 3600)
minutes, seconds = divmod(remainder_seconds, 60)
seconds_with_fraction = seconds + remainder
if seconds_with_fraction.is_integer():
seconds_text = f"{int(seconds_with_fraction)}"
else:
seconds_text = f"{seconds_with_fraction:.3f}".rstrip("0").rstrip(".")
return f"PT{hours}H{minutes}M{seconds_text}S"
def _detect_audio_mime(audio_filename: str) -> str:
suffix = Path(audio_filename).suffix.lower()
return {
".mp3": "audio/mpeg",
".m4a": "audio/mp4",
".m4b": "audio/mp4",
".aac": "audio/aac",
".wav": "audio/wav",
".flac": "audio/flac",
".ogg": "audio/ogg",
".opus": "audio/ogg",
}.get(suffix, "audio/mpeg")
def _detect_image_mime(suffix: str) -> str:
normalized = suffix.lower()
return {
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
".png": "image/png",
".gif": "image/gif",
".webp": "image/webp",
}.get(normalized, "image/jpeg")
def _utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
_DEFAULT_STYLESHEET = """
body {
font-family: 'Georgia', serif;
line-height: 1.6;
margin: 1.5em;
}
h1 {
font-size: 1.5em;
margin-bottom: 0.5em;
}
.chunk-group {
margin: 0.5em 0;
}
.chunk-group .chunk {
white-space: pre-wrap;
}
"""
+6 -4040
View File
File diff suppressed because it is too large Load Diff
+304
View File
@@ -0,0 +1,304 @@
from __future__ import annotations
import hashlib
import re
from dataclasses import dataclass
from typing import Any, Dict, Iterable, List, Mapping, Optional, Sequence, Tuple
try: # pragma: no cover - optional dependency
import spacy # type: ignore
except Exception: # pragma: no cover - spaCy may be unavailable in minimal environments
spacy = None
@dataclass(frozen=True)
class HeteronymVariant:
key: str
label: str
replacement_token: str
example_sentence: str
@dataclass(frozen=True)
class HeteronymSpec:
token: str
variants: Tuple[HeteronymVariant, HeteronymVariant]
def default_choice_for_token(self, spacy_token: Any) -> str:
"""Return the most likely variant key for this token."""
pos = (getattr(spacy_token, "pos_", "") or "").upper()
tag = (getattr(spacy_token, "tag_", "") or "").upper()
token_lower = self.token.casefold()
if token_lower == "wind":
# VERB => /waɪnd/, NOUN => /wɪnd/
return "verb" if pos == "VERB" else "noun"
if token_lower == "read":
# VBD/VBN => /rɛd/
return "past" if tag in {"VBD", "VBN"} else "present"
if token_lower == "tear":
return "verb" if pos == "VERB" else "noun"
if token_lower == "close":
return "verb" if pos == "VERB" else "adj"
if token_lower == "lead":
# Default to verb unless POS suggests noun.
return "metal" if pos == "NOUN" else "verb"
return self.variants[0].key
# Minimal, high-confidence starter set.
# NOTE: These replacements intentionally prioritize speech output.
# Some replacements may not be appropriate for subtitles/text exports.
_HETERONYM_SPECS: Dict[str, HeteronymSpec] = {
"wind": HeteronymSpec(
token="wind",
variants=(
HeteronymVariant(
key="noun",
label="Noun (the wind)",
replacement_token="wind",
example_sentence="Listen to the wind.",
),
HeteronymVariant(
key="verb",
label="Verb (to wind)",
replacement_token="wynd",
example_sentence="I need to wind the watch.",
),
),
),
"read": HeteronymSpec(
token="read",
variants=(
HeteronymVariant(
key="present",
label="Present (I read every day)",
replacement_token="read",
example_sentence="I read every day.",
),
HeteronymVariant(
key="past",
label="Past (I read it yesterday)",
replacement_token="red",
example_sentence="I read it yesterday.",
),
),
),
"tear": HeteronymSpec(
token="tear",
variants=(
HeteronymVariant(
key="noun",
label="Noun (a tear /crying/)",
replacement_token="tier",
example_sentence="A tear rolled down her cheek.",
),
HeteronymVariant(
key="verb",
label="Verb (to tear /rip/)",
replacement_token="tear",
example_sentence="Please don't tear the page.",
),
),
),
"close": HeteronymSpec(
token="close",
variants=(
HeteronymVariant(
key="adj",
label="Adjective (close /near/)",
replacement_token="close",
example_sentence="We are close to the station.",
),
HeteronymVariant(
key="verb",
label="Verb (close /klohz/)",
replacement_token="cloze",
example_sentence="Please close the door.",
),
),
),
"lead": HeteronymSpec(
token="lead",
variants=(
HeteronymVariant(
key="verb",
label="Verb (to lead)",
replacement_token="lead",
example_sentence="They will lead the way.",
),
HeteronymVariant(
key="metal",
label="Noun (lead /metal/)",
replacement_token="led",
example_sentence="The pipe was made of lead.",
),
),
),
}
def _hash_id(*parts: str) -> str:
digest = hashlib.sha1("\n".join(parts).encode("utf-8")).hexdigest()
return digest[:12]
_WORD_BOUNDARY_CACHE: Dict[str, re.Pattern[str]] = {}
def _word_boundary_pattern(token: str) -> re.Pattern[str]:
key = token.casefold()
cached = _WORD_BOUNDARY_CACHE.get(key)
if cached is not None:
return cached
escaped = re.escape(token)
pattern = re.compile(
rf"(?i)(?<!\w){escaped}(?P<possessive>'s|\u2019s|\u2019)?(?!\w)"
)
_WORD_BOUNDARY_CACHE[key] = pattern
return pattern
def _preserve_case(replacement: str, original: str) -> str:
if not replacement:
return replacement
if original.isupper():
return replacement.upper()
if original[:1].isupper():
return replacement[:1].upper() + replacement[1:]
return replacement
def _build_replacement_sentence(
sentence: str, token: str, replacement_token: str
) -> str:
pattern = _word_boundary_pattern(token)
def _repl(match: re.Match[str]) -> str:
matched = match.group(0) or ""
suffix = match.group("possessive") or ""
base = matched[: len(matched) - len(suffix)] if suffix else matched
return _preserve_case(replacement_token, base) + suffix
return pattern.sub(_repl, sentence)
def _load_spacy(language: str) -> Any:
if spacy is None:
return None
# English only for now.
# Use installed small model; keep it simple.
lang = (language or "en").lower()
if lang.startswith("en"):
try:
return spacy.load("en_core_web_sm")
except Exception:
return spacy.blank("en")
return spacy.blank("xx")
def extract_heteronym_overrides(
chapters: Sequence[Mapping[str, Any]],
*,
language: str,
existing: Optional[Iterable[Mapping[str, Any]]] = None,
) -> List[Dict[str, Any]]:
"""Extract distinct heteronym-containing sentences from chapters.
Returns entries shaped for persistence + UI.
Each entry contains:
- id
- token
- sentence
- options: [{key,label,replacement_token,replacement_sentence,example_sentence}]
- default_choice
- choice
"""
lang = (language or "en").lower()
if not lang.startswith("en"):
return []
if spacy is None:
return []
nlp = _load_spacy(lang)
if nlp is None:
return []
previous_choices: Dict[str, str] = {}
if existing:
for item in existing:
if not isinstance(item, Mapping):
continue
entry_id = str(item.get("id") or "").strip()
choice = str(item.get("choice") or "").strip()
if entry_id and choice:
previous_choices[entry_id] = choice
results: List[Dict[str, Any]] = []
seen: set[tuple[str, str]] = set()
for chapter in chapters:
if not isinstance(chapter, Mapping):
continue
text = str(chapter.get("text") or "")
if not text.strip():
continue
doc = nlp(text)
for sent in getattr(doc, "sents", []):
sentence = str(getattr(sent, "text", "") or "").strip()
if not sentence:
continue
for token in sent:
token_text = str(getattr(token, "text", "") or "")
if not token_text:
continue
token_key = token_text.casefold()
spec = _HETERONYM_SPECS.get(token_key)
if not spec:
continue
dedupe_key = (token_key, sentence)
if dedupe_key in seen:
continue
seen.add(dedupe_key)
entry_id = _hash_id(token_key, sentence)
default_choice = spec.default_choice_for_token(token)
choice = previous_choices.get(entry_id, default_choice)
options: List[Dict[str, Any]] = []
for variant in spec.variants:
replacement_sentence = _build_replacement_sentence(
sentence,
token=spec.token,
replacement_token=variant.replacement_token,
)
options.append(
{
"key": variant.key,
"label": variant.label,
"replacement_token": variant.replacement_token,
"replacement_sentence": replacement_sentence,
"example_sentence": variant.example_sentence,
}
)
results.append(
{
"id": entry_id,
"token": token_text,
"token_lower": token_key,
"sentence": sentence,
"options": options,
"default_choice": default_choice,
"choice": choice,
}
)
return results
+1
View File
@@ -0,0 +1 @@
"""Integration clients for external services."""
+680
View File
@@ -0,0 +1,680 @@
from __future__ import annotations
import json
import logging
import math
import mimetypes
import re
from contextlib import ExitStack
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Dict, Iterable, List, Mapping, Optional, Sequence, Tuple
import httpx
logger = logging.getLogger(__name__)
class AudiobookshelfUploadError(RuntimeError):
"""Raised when an upload to Audiobookshelf fails."""
@dataclass(frozen=True)
class AudiobookshelfConfig:
base_url: str
api_token: str
library_id: Optional[str] = None
collection_id: Optional[str] = None
folder_id: Optional[str] = None
verify_ssl: bool = True
send_cover: bool = True
send_chapters: bool = True
send_subtitles: bool = True
timeout: float = 3600.0
def normalized_base_url(self) -> str:
base = (self.base_url or "").strip()
if not base:
raise ValueError("Audiobookshelf base URL is required")
normalized = base.rstrip("/")
# The web UI historically suggested including '/api' in the base URL; trim
# it here so we can safely append `/api/...` endpoints below.
if normalized.lower().endswith("/api"):
normalized = normalized[:-4]
return normalized or base
class AudiobookshelfClient:
"""Client for the legacy Audiobookshelf multipart upload endpoint."""
def __init__(self, config: AudiobookshelfConfig) -> None:
if not config.api_token:
raise ValueError("Audiobookshelf API token is required")
# library_id is now optional for discovery
self._config = config
normalized = config.normalized_base_url() or ""
self._base_url = normalized.rstrip("/") or normalized
self._client_base_url = f"{self._base_url}/"
self._folder_cache: Optional[Tuple[str, str, str]] = None
def get_libraries(self) -> List[Dict[str, Any]]:
"""Fetch all libraries from the Audiobookshelf server."""
route = self._api_path("libraries")
try:
with self._open_client() as client:
response = client.get(route)
response.raise_for_status()
data = response.json()
# data['libraries'] is a list of library objects
return data.get("libraries", [])
except httpx.HTTPError as exc:
raise AudiobookshelfUploadError(f"Failed to fetch libraries: {exc}") from exc
def _api_path(self, suffix: str = "") -> str:
"""Join the API prefix with the provided suffix without losing proxies."""
clean_suffix = suffix.lstrip("/")
return f"api/{clean_suffix}" if clean_suffix else "api"
def upload_audiobook(
self,
audio_path: Path,
*,
metadata: Dict[str, Any],
cover_path: Optional[Path] = None,
chapters: Optional[Iterable[Dict[str, Any]]] = None,
subtitles: Optional[Iterable[Path]] = None,
) -> Dict[str, Any]:
if not audio_path.exists():
raise AudiobookshelfUploadError(f"Audio path does not exist: {audio_path}")
form_fields = self._build_upload_fields(audio_path, metadata, chapters)
file_entries = self._build_file_entries(audio_path, cover_path, subtitles)
route = self._api_path("upload")
try:
with self._open_client() as client, ExitStack() as stack:
files_payload = self._open_file_handles(file_entries, stack)
response = client.post(route, data=form_fields, files=files_payload)
response.raise_for_status()
except httpx.HTTPStatusError as exc:
status = exc.response.status_code
detail = (exc.response.text or "").strip()
if detail:
detail = detail[:200]
message = f"Audiobookshelf upload failed with status {status}: {detail}"
else:
message = f"Audiobookshelf upload failed with status {status}"
raise AudiobookshelfUploadError(
message
) from exc
except httpx.HTTPError as exc:
raise AudiobookshelfUploadError(f"Audiobookshelf upload failed: {exc}") from exc
return {}
def _open_client(self) -> httpx.Client:
headers = {
"Authorization": f"Bearer {self._config.api_token}",
"Accept": "application/json",
}
return httpx.Client(
base_url=self._client_base_url,
headers=headers,
timeout=self._config.timeout,
verify=self._config.verify_ssl,
)
def _build_upload_fields(
self,
audio_path: Path,
metadata: Dict[str, Any],
chapters: Optional[Iterable[Dict[str, Any]]],
) -> Dict[str, str]:
folder_id, _, _ = self._ensure_folder()
title = self._extract_title(metadata, audio_path)
author = self._extract_author(metadata)
series = self._extract_series(metadata)
series_sequence = self._extract_series_sequence(metadata)
fields: Dict[str, str] = {
"library": self._config.library_id,
"folder": folder_id,
"title": title,
}
if author:
fields["author"] = author
if series:
fields["series"] = series
if series_sequence:
fields["seriesSequence"] = series_sequence
if self._config.collection_id:
fields["collectionId"] = self._config.collection_id
metadata_payload: Dict[str, Any] = metadata or {}
if chapters and self._config.send_chapters:
metadata_payload = dict(metadata_payload)
metadata_payload["chapters"] = list(chapters)
if metadata_payload:
# Ensure authors is a list of strings in the JSON payload if it exists
if "authors" in metadata_payload:
authors_val = metadata_payload["authors"]
if isinstance(authors_val, str):
metadata_payload["authors"] = [a.strip() for a in authors_val.split(",") if a.strip()]
elif isinstance(authors_val, list):
metadata_payload["authors"] = [str(a).strip() for a in authors_val if str(a).strip()]
try:
fields["metadata"] = json.dumps(metadata_payload, ensure_ascii=False)
except (TypeError, ValueError):
logger.debug("Failed to serialize Audiobookshelf metadata payload")
return fields
def _build_file_entries(
self,
audio_path: Path,
cover_path: Optional[Path],
subtitles: Optional[Iterable[Path]],
) -> List[Tuple[str, Path]]:
entries: List[Tuple[str, Path]] = [("file0", audio_path)]
index = 1
if cover_path and self._config.send_cover and cover_path.exists():
entries.append((f"file{index}", cover_path))
index += 1
if subtitles and self._config.send_subtitles:
for subtitle in subtitles:
if subtitle.exists():
entries.append((f"file{index}", subtitle))
index += 1
return entries
def _open_file_handles(
self,
entries: Sequence[Tuple[str, Path]],
stack: ExitStack,
) -> List[Tuple[str, Tuple[str, Any, str]]]:
files: List[Tuple[str, Tuple[str, Any, str]]] = []
for field_name, path in entries:
mime_type, _ = mimetypes.guess_type(path.name)
mime_type = mime_type or "application/octet-stream"
handle = stack.enter_context(path.open("rb"))
files.append((field_name, (path.name, handle, mime_type)))
return files
def find_existing_items(
self,
title: str,
*,
folder_id: Optional[str] = None,
) -> List[Mapping[str, Any]]:
normalized_title = self._normalize_title_value(title)
if not normalized_title:
return []
folder_hint = folder_id or self._config.folder_id
target_folders = set()
if folder_hint:
folder_token = str(folder_hint).strip().lower()
if folder_token:
target_folders.add(folder_token)
requests = self._candidate_search_requests(title, folder_hint)
if not requests:
return []
matches: List[Mapping[str, Any]] = []
try:
with self._open_client() as client:
for route, params in requests:
try:
response = client.get(route, params=params)
except httpx.HTTPError as exc:
logger.debug("Audiobookshelf lookup failed for %s: %s", route, exc)
continue
if response.status_code == 404:
continue
try:
response.raise_for_status()
except httpx.HTTPStatusError as exc:
status = exc.response.status_code
if status in {401, 403}:
raise AudiobookshelfUploadError(
"Audiobookshelf authentication failed while checking for existing items."
) from exc
logger.debug("Audiobookshelf lookup error %s for %s", status, route)
continue
try:
payload = response.json()
except ValueError:
continue
candidates = self._extract_candidate_items(payload)
for item in candidates:
item_title = self._normalize_item_title(item)
if not item_title or item_title != normalized_title:
continue
if target_folders:
item_folder = self._normalize_folder_id(item)
if item_folder and item_folder not in target_folders:
continue
matches.append(item)
if matches:
break
except AudiobookshelfUploadError:
raise
except Exception:
logger.debug(
"Unexpected error while checking Audiobookshelf for existing items",
exc_info=True,
)
return matches
def delete_items(self, items: Iterable[Mapping[str, Any] | str]) -> None:
to_delete: List[str] = []
for entry in items:
if isinstance(entry, Mapping):
item_id = self._extract_item_id(entry)
else:
item_id = str(entry).strip()
if item_id:
to_delete.append(item_id)
if not to_delete:
return
with self._open_client() as client:
for item_id in to_delete:
self._delete_single_item(client, item_id)
def _candidate_search_requests(
self,
title: str,
folder_id: Optional[str],
) -> List[Tuple[str, Dict[str, Any]]]:
query = (title or "").strip()
if not query:
return []
library_id = self._config.library_id
folder_token = (folder_id or self._config.folder_id or "").strip()
requests: List[Tuple[str, Dict[str, Any]]] = []
seen_routes: set[str] = set()
def _append(route: str, params: Dict[str, Any]) -> None:
if route in seen_routes:
return
seen_routes.add(route)
requests.append((route, params))
if folder_token:
_append(
self._api_path(f"folders/{folder_token}/items"),
{"library": library_id, "search": query},
)
_append(self._api_path(f"libraries/{library_id}/items"), {"search": query})
_append(self._api_path("items"), {"library": library_id, "search": query})
_append(
self._api_path("search"),
{"query": query, "library": library_id, "media": "audiobook"},
)
return requests
def _delete_single_item(self, client: httpx.Client, item_id: str) -> None:
routes = [
self._api_path(f"items/{item_id}"),
self._api_path(f"libraries/{self._config.library_id}/items/{item_id}"),
]
for route in routes:
try:
response = client.delete(route)
except httpx.HTTPError as exc:
logger.debug("Audiobookshelf delete failed for %s: %s", route, exc)
continue
if response.status_code in (200, 202, 204):
return
if response.status_code == 404:
continue
try:
response.raise_for_status()
except httpx.HTTPStatusError as exc:
raise AudiobookshelfUploadError(
f"Failed to delete Audiobookshelf item '{item_id}': {exc}"
) from exc
logger.debug("Audiobookshelf item %s could not be confirmed deleted", item_id)
def resolve_folder(self) -> Tuple[str, str, str]:
"""Return the resolved folder (id, name, library name)."""
return self._ensure_folder()
def list_folders(self) -> List[Dict[str, str]]:
"""Return all folders for the configured library."""
library_name, folders = self._load_library_metadata()
results: List[Dict[str, str]] = []
for folder in folders:
folder_id = str(folder.get("id") or "").strip()
if not folder_id:
continue
name = self._folder_display_name(folder)
path = self._select_folder_path(folder)
results.append(
{
"id": folder_id,
"name": name,
"path": path,
"library": library_name,
}
)
results.sort(key=lambda entry: (entry.get("path") or entry.get("name") or entry.get("id") or "").lower())
return results
def _ensure_folder(self) -> Tuple[str, str, str]:
if self._folder_cache:
return self._folder_cache
identifier = (self._config.folder_id or "").strip()
if not identifier:
raise AudiobookshelfUploadError(
"Audiobookshelf folder is required; enter the folder name or ID in Settings."
)
identifier_norm = self._normalize_identifier(identifier)
library_name, folders = self._load_library_metadata()
# direct ID match
for folder in folders:
folder_id = str(folder.get("id") or "").strip()
if folder_id and folder_id == identifier:
folder_name = self._folder_display_name(folder) or folder_id
self._folder_cache = (folder_id, folder_name, library_name)
return self._folder_cache
has_path_component = "/" in identifier_norm
for folder in folders:
folder_id = str(folder.get("id") or "").strip()
if not folder_id:
continue
folder_name = self._folder_display_name(folder)
name_norm = self._normalize_identifier(folder_name)
if name_norm and name_norm == identifier_norm:
self._folder_cache = (folder_id, folder_name or folder_id, library_name)
return self._folder_cache
for candidate in self._folder_path_candidates(folder):
candidate_norm = self._normalize_identifier(candidate)
if not candidate_norm:
continue
if candidate_norm == identifier_norm:
self._folder_cache = (folder_id, folder_name or folder_id, library_name)
return self._folder_cache
if has_path_component and candidate_norm.endswith(identifier_norm):
self._folder_cache = (folder_id, folder_name or folder_id, library_name)
return self._folder_cache
if not has_path_component:
tail = candidate_norm.split("/")[-1]
if tail and tail == identifier_norm:
self._folder_cache = (folder_id, folder_name or folder_id, library_name)
return self._folder_cache
raise AudiobookshelfUploadError(
f"Folder '{identifier}' was not found in library '{library_name}'. "
"Enter the folder name exactly as it appears in Audiobookshelf, a trailing path segment, or paste the folder ID."
)
def _load_library_metadata(self) -> Tuple[str, List[Mapping[str, Any]]]:
try:
with self._open_client() as client:
response = client.get(self._api_path(f"libraries/{self._config.library_id}"))
response.raise_for_status()
payload = response.json()
except httpx.HTTPStatusError as exc:
status = exc.response.status_code
if status == 404:
message = f"Audiobookshelf library '{self._config.library_id}' not found."
else:
detail = (exc.response.text or "").strip()
if detail:
detail = detail[:200]
message = (
f"Failed to load Audiobookshelf library '{self._config.library_id}' "
f"(status {status}): {detail}"
)
else:
message = (
f"Failed to load Audiobookshelf library '{self._config.library_id}' "
f"(status {status})."
)
raise AudiobookshelfUploadError(message) from exc
except httpx.HTTPError as exc:
raise AudiobookshelfUploadError(
f"Failed to reach Audiobookshelf library '{self._config.library_id}': {exc}"
) from exc
if not isinstance(payload, Mapping):
return self._config.library_id, []
library_name = str(payload.get("name") or payload.get("label") or self._config.library_id)
raw_folders = payload.get("libraryFolders") or payload.get("folders") or []
folders = [entry for entry in raw_folders if isinstance(entry, Mapping)]
return library_name, folders
@staticmethod
def _folder_path_candidates(folder: Mapping[str, Any]) -> List[str]:
candidates: List[str] = []
for key in ("fullPath", "fullpath", "path", "folderPath", "virtualPath"):
value = folder.get(key)
if isinstance(value, str) and value.strip():
candidates.append(value)
return candidates
@staticmethod
def _folder_display_name(folder: Mapping[str, Any]) -> str:
name = str(folder.get("name") or folder.get("label") or "").strip()
if name:
return name
path = AudiobookshelfClient._select_folder_path(folder)
if path:
tail = path.strip("/ ")
tail = tail.split("/")[-1] if tail else ""
if tail:
return tail
return str(folder.get("id") or "").strip()
@staticmethod
def _select_folder_path(folder: Mapping[str, Any]) -> str:
for candidate in AudiobookshelfClient._folder_path_candidates(folder):
normalized = candidate.replace("\\", "/").strip()
if normalized:
return normalized
return ""
@staticmethod
def _normalize_identifier(value: str) -> str:
token = (value or "").strip()
token = token.replace("\\", "/")
if len(token) > 1 and token[1] == ":":
token = token[2:]
token = token.strip("/ ")
return token.lower()
@staticmethod
def _normalize_title_value(value: Optional[str]) -> str:
if not isinstance(value, str):
return ""
normalized = re.sub(r"\s+", " ", value).strip()
return normalized.casefold() if normalized else ""
@staticmethod
def _normalize_item_title(item: Mapping[str, Any]) -> str:
if not isinstance(item, Mapping):
return ""
for key in ("title", "name", "label"):
candidate = item.get(key)
if isinstance(candidate, str) and candidate.strip():
return AudiobookshelfClient._normalize_title_value(candidate)
library_item = item.get("libraryItem")
if isinstance(library_item, Mapping):
return AudiobookshelfClient._normalize_item_title(library_item)
return ""
@staticmethod
def _normalize_folder_id(item: Mapping[str, Any]) -> Optional[str]:
if not isinstance(item, Mapping):
return None
for key in ("folderId", "libraryFolderId", "folder_id", "folder"):
value = item.get(key)
if isinstance(value, str) and value.strip():
return value.strip().lower()
if isinstance(value, (int, float)):
return str(value).strip().lower()
library_item = item.get("libraryItem")
if isinstance(library_item, Mapping):
return AudiobookshelfClient._normalize_folder_id(library_item)
return None
@staticmethod
def _extract_item_id(item: Mapping[str, Any]) -> Optional[str]:
if not isinstance(item, Mapping):
return None
for key in ("id", "libraryItemId", "itemId"):
value = item.get(key)
if isinstance(value, str) and value.strip():
return value.strip()
if isinstance(value, (int, float)):
return str(value).strip()
library_item = item.get("libraryItem")
if isinstance(library_item, Mapping):
return AudiobookshelfClient._extract_item_id(library_item)
return None
@staticmethod
def _extract_candidate_items(payload: Any) -> List[Mapping[str, Any]]:
items: List[Mapping[str, Any]] = []
seen_ids: set[str] = set()
visited: set[int] = set()
def _visit(obj: Any) -> None:
if isinstance(obj, Mapping):
obj_id = id(obj)
if obj_id in visited:
return
visited.add(obj_id)
title = AudiobookshelfClient._normalize_item_title(obj)
item_id = AudiobookshelfClient._extract_item_id(obj)
if title and item_id:
key = item_id.strip().lower()
if key not in seen_ids:
seen_ids.add(key)
items.append(obj)
for value in obj.values():
_visit(value)
elif isinstance(obj, list):
for entry in obj:
_visit(entry)
_visit(payload)
return items
@staticmethod
def _extract_title(metadata: Mapping[str, Any], audio_path: Path) -> str:
title = metadata.get("title") if isinstance(metadata, Mapping) else None
candidate = str(title).strip() if isinstance(title, str) else ""
if candidate:
return candidate
return audio_path.stem or audio_path.name
@staticmethod
def _extract_author(metadata: Mapping[str, Any]) -> str:
authors = metadata.get("authors") if isinstance(metadata, Mapping) else None
if isinstance(authors, str):
candidate = authors.strip()
return candidate
if isinstance(authors, Iterable) and not isinstance(authors, (str, Mapping)):
names = [str(entry).strip() for entry in authors if isinstance(entry, str) and entry.strip()]
if names:
# ABS expects a comma-separated string for multiple authors.
return ", ".join(names)
return ""
@staticmethod
def _extract_series(metadata: Mapping[str, Any]) -> str:
series_name = metadata.get("seriesName") if isinstance(metadata, Mapping) else None
if isinstance(series_name, str) and series_name.strip():
return series_name.strip()
return ""
@staticmethod
def _extract_series_sequence(metadata: Mapping[str, Any]) -> str:
if not isinstance(metadata, Mapping):
return ""
preferred_keys = (
"seriesSequence",
"series_sequence",
"seriesIndex",
"series_index",
"seriesNumber",
"series_number",
"bookNumber",
"book_number",
)
for key in preferred_keys:
if key not in metadata:
continue
normalized = AudiobookshelfClient._normalize_series_sequence(metadata.get(key))
if normalized:
return normalized
return ""
@staticmethod
def _normalize_series_sequence(raw: Any) -> str:
if raw is None:
return ""
if isinstance(raw, (int, float)):
if isinstance(raw, float) and (math.isnan(raw) or math.isinf(raw)):
return ""
text = str(raw)
else:
text = str(raw).strip()
if not text:
return ""
candidate = text.replace(",", ".")
match = re.search(r"\d+(?:\.\d+)?", candidate)
if not match:
return ""
normalized = match.group(0)
if "." in normalized:
normalized = normalized.rstrip("0").rstrip(".")
if not normalized:
normalized = "0"
return normalized
try:
return str(int(normalized))
except ValueError:
cleaned = normalized.lstrip("0")
return cleaned or "0"
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+210
View File
@@ -0,0 +1,210 @@
from __future__ import annotations
import json
from dataclasses import dataclass
from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple
from urllib import error, parse, request
class LLMClientError(RuntimeError):
"""Raised when an LLM request fails."""
@dataclass(frozen=True)
class LLMConfiguration:
base_url: str
api_key: str
model: str
timeout: float = 30.0
def is_configured(self) -> bool:
return bool(self.base_url.strip() and self.model.strip())
@dataclass(frozen=True)
class LLMToolCall:
name: str
arguments: str
@dataclass(frozen=True)
class LLMCompletion:
content: Optional[str]
tool_calls: Tuple[LLMToolCall, ...]
_DEFAULT_HEADERS = {
"Content-Type": "application/json",
"Accept": "application/json",
}
def _normalized_base_url(base_url: str) -> str:
trimmed = (base_url or "").strip()
if not trimmed:
raise LLMClientError("LLM base URL is required")
if not trimmed.endswith("/"):
trimmed += "/"
return trimmed
def _build_url(base_url: str, path: str) -> str:
normalized = _normalized_base_url(base_url)
trimmed_path = path.lstrip("/")
parsed = parse.urlparse(normalized)
if parsed.path.rstrip("/").lower().endswith("/v1") and trimmed_path.startswith(
"v1/"
):
trimmed_path = trimmed_path[len("v1/") :]
return parse.urljoin(normalized, trimmed_path)
def _build_headers(api_key: str) -> Dict[str, str]:
headers = dict(_DEFAULT_HEADERS)
token = (api_key or "").strip()
if token and token.lower() != "ollama":
headers["Authorization"] = f"Bearer {token}"
return headers
def _perform_request(
method: str,
url: str,
*,
headers: Optional[Mapping[str, str]] = None,
payload: Optional[Mapping[str, Any]] = None,
timeout: float = 30.0,
) -> Any:
data_bytes: Optional[bytes] = None
if payload is not None:
data_bytes = json.dumps(payload).encode("utf-8")
request_headers = dict(headers or {})
req = request.Request(
url, data=data_bytes, headers=request_headers, method=method.upper()
)
try:
with request.urlopen(req, timeout=timeout) as response:
body = response.read()
except error.HTTPError as exc: # pragma: no cover - defensive network guard
message = exc.read().decode("utf-8", "ignore") if exc.fp else exc.reason
raise LLMClientError(f"LLM request failed ({exc.code}): {message}") from exc
except error.URLError as exc: # pragma: no cover - defensive network guard
raise LLMClientError(f"LLM request failed: {exc.reason}") from exc
except Exception as exc: # pragma: no cover - defensive network guard
raise LLMClientError("LLM request failed") from exc
if not body:
return None
try:
return json.loads(body.decode("utf-8"))
except json.JSONDecodeError as exc:
raise LLMClientError("LLM response was not valid JSON") from exc
def list_models(configuration: LLMConfiguration) -> List[Dict[str, str]]:
if not configuration.is_configured() and not configuration.base_url.strip():
raise LLMClientError("LLM configuration is incomplete")
url = _build_url(configuration.base_url, "v1/models")
headers = _build_headers(configuration.api_key)
payload = _perform_request(
"GET", url, headers=headers, timeout=configuration.timeout
)
if not isinstance(payload, Mapping):
raise LLMClientError("Unexpected response when listing models")
data = payload.get("data")
if not isinstance(data, list):
return []
models: List[Dict[str, str]] = []
for entry in data:
if not isinstance(entry, Mapping):
continue
identifier = str(entry.get("id") or "").strip()
if not identifier:
continue
description = str(entry.get("name") or entry.get("description") or identifier)
models.append({"id": identifier, "label": description})
return models
def generate_completion(
configuration: LLMConfiguration,
*,
system_message: str,
user_message: str,
temperature: float = 0.2,
max_tokens: Optional[int] = None,
tools: Optional[Sequence[Mapping[str, Any]]] = None,
tool_choice: Optional[Mapping[str, Any]] = None,
response_format: Optional[Mapping[str, Any]] = None,
) -> LLMCompletion:
if not configuration.is_configured():
raise LLMClientError("LLM configuration is incomplete")
url = _build_url(configuration.base_url, "v1/chat/completions")
headers = _build_headers(configuration.api_key)
payload: Dict[str, Any] = {
"model": configuration.model,
"messages": [
{"role": "system", "content": system_message},
{"role": "user", "content": user_message},
],
"temperature": temperature,
}
if max_tokens is not None:
payload["max_tokens"] = max_tokens
if tools:
payload["tools"] = list(tools)
if tool_choice:
payload["tool_choice"] = dict(tool_choice)
if response_format:
payload["response_format"] = dict(response_format)
response = _perform_request(
"POST", url, headers=headers, payload=payload, timeout=configuration.timeout
)
if not isinstance(response, Mapping):
raise LLMClientError("Unexpected response from LLM")
choices = response.get("choices")
if not isinstance(choices, list) or not choices:
raise LLMClientError("LLM response did not include choices")
first = choices[0]
if not isinstance(first, Mapping):
raise LLMClientError("LLM response choice was invalid")
message = first.get("message")
content: Optional[str] = None
tool_calls: List[LLMToolCall] = []
if isinstance(message, Mapping):
content = message.get("content")
if isinstance(content, str):
stripped = content.strip()
if stripped:
content = stripped
else:
content = None
tool_call_entries = message.get("tool_calls")
if isinstance(tool_call_entries, list):
for entry in tool_call_entries:
if not isinstance(entry, Mapping):
continue
fn = entry.get("function")
if not isinstance(fn, Mapping):
continue
name = str(fn.get("name") or "").strip()
if not name:
continue
args = fn.get("arguments", "")
if isinstance(args, (dict, list)):
arguments = json.dumps(args)
else:
arguments = str(args)
tool_calls.append(LLMToolCall(name=name, arguments=arguments))
if content:
return LLMCompletion(content=content, tool_calls=tuple(tool_calls))
text = first.get("text")
if isinstance(text, str):
stripped = text.strip()
if stripped:
content = stripped
if content or tool_calls:
return LLMCompletion(content=content, tool_calls=tuple(tool_calls))
raise LLMClientError("LLM response did not include text content")
+27 -165
View File
@@ -1,116 +1,36 @@
import os
import sys
import platform
"""Backwards-compatible entry point that now launches the web UI."""
from __future__ import annotations
import atexit
import os
import platform
import signal
from abogen.utils import get_resource_path, load_config, prevent_sleep_end
import sys
from abogen.utils import load_config, prevent_sleep_end
from abogen.webui.app import main as _run_web_ui
# Fix PyTorch DLL loading issue ([WinError 1114]) on Windows before importing PyQt6
if platform.system() == "Windows":
import ctypes
from importlib.util import find_spec
try:
if (
(spec := find_spec("torch"))
and spec.origin
and os.path.exists(
dll_path := os.path.join(os.path.dirname(spec.origin), "lib", "c10.dll")
)
):
ctypes.CDLL(os.path.normpath(dll_path))
except Exception:
pass
# Qt platform plugin detection (fixes #59)
try:
from PyQt6.QtCore import QLibraryInfo
# Get the path to the plugins directory
plugins = QLibraryInfo.path(QLibraryInfo.LibraryPath.PluginsPath)
# Normalize path to use the OS-native separators and absolute path
platform_dir = os.path.normpath(os.path.join(plugins, "platforms"))
# Ensure we work with an absolute path for clarity
platform_dir = os.path.abspath(platform_dir)
if os.path.isdir(platform_dir):
os.environ["QT_QPA_PLATFORM_PLUGIN_PATH"] = platform_dir
print("QT_QPA_PLATFORM_PLUGIN_PATH set to:", platform_dir)
else:
print("PyQt6 platform plugins not found at", platform_dir)
except ImportError:
print("PyQt6 not installed.")
# Pre-load "libxcb-cursor" on Linux (fixes #101)
if platform.system() == "Linux":
arch = platform.machine().lower()
lib_filename = {"x86_64": "libxcb-cursor-amd64.so.0", "amd64": "libxcb-cursor-amd64.so.0", "aarch64": "libxcb-cursor-arm64.so.0", "arm64": "libxcb-cursor-arm64.so.0"}.get(arch)
if lib_filename:
import ctypes
try:
# Try to load the system libxcb-cursor.so.0 first
ctypes.CDLL('libxcb-cursor.so.0', mode=ctypes.RTLD_GLOBAL)
except OSError:
# System lib not available, load the bundled version
lib_path = get_resource_path('abogen.libs', lib_filename)
if lib_path:
try:
ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
except OSError:
# If it fails (e.g. wrong glibc version on very old systems),
# we simply ignore it and hope the system has the library.
pass
# Set application ID for Windows taskbar icon
if platform.system() == "Windows":
try:
from abogen.constants import PROGRAM_NAME, VERSION
import ctypes
app_id = f"{PROGRAM_NAME}.{VERSION}"
ctypes.windll.shell32.SetCurrentProcessExplicitAppUserModelID(app_id)
except Exception as e:
print("Warning: failed to set AppUserModelID:", e)
from PyQt6.QtWidgets import QApplication
from PyQt6.QtGui import QIcon
from PyQt6.QtCore import (
QLibraryInfo,
qInstallMessageHandler,
QtMsgType,
)
# Add the directory to Python path
sys.path.insert(0, os.path.join(os.path.dirname(__file__)))
# Set Hugging Face Hub environment variables
os.environ["HF_HUB_DISABLE_TELEMETRY"] = "1" # Disable Hugging Face telemetry
os.environ["HF_HUB_ETAG_TIMEOUT"] = "10" # Metadata request timeout (seconds)
os.environ["HF_HUB_DOWNLOAD_TIMEOUT"] = "10" # File download timeout (seconds)
os.environ["HF_HUB_DISABLE_SYMLINKS_WARNING"] = "1" # Disable symlinks warning
# Configure Hugging Face Hub behaviour (mirrors legacy GUI defaults).
os.environ.setdefault("HF_HUB_DISABLE_TELEMETRY", "1")
os.environ.setdefault("HF_HUB_ETAG_TIMEOUT", "10")
os.environ.setdefault("HF_HUB_DOWNLOAD_TIMEOUT", "10")
os.environ.setdefault("HF_HUB_DISABLE_SYMLINKS_WARNING", "1")
if load_config().get("disable_kokoro_internet", False):
print("INFO: Kokoro's internet access is disabled.")
os.environ["HF_HUB_OFFLINE"] = "1" # Disable Hugging Face Hub internet access
os.environ["HF_HUB_OFFLINE"] = "1"
from abogen.gui import abogen
from abogen.constants import PROGRAM_NAME, VERSION
# Prefer faster ROCm tuning defaults when available.
os.environ.setdefault("MIOPEN_FIND_MODE", "FAST")
os.environ.setdefault("MIOPEN_CONV_PRECISE_ROCM_TUNING", "0")
# Set environment variables for AMD ROCm
os.environ["MIOPEN_FIND_MODE"] = "FAST"
os.environ["MIOPEN_CONV_PRECISE_ROCM_TUNING"] = "0"
# Enable MPS GPU acceleration on Apple Silicon.
if platform.system() == "Darwin" and platform.processor() == "arm":
os.environ.setdefault("PYTORCH_ENABLE_MPS_FALLBACK", "1")
# Reset sleep states
atexit.register(prevent_sleep_end)
# Also handle signals (Ctrl+C, kill, etc.)
def _cleanup_sleep(signum, frame):
def _cleanup_sleep(signum, _frame):
prevent_sleep_end()
sys.exit(0)
@@ -118,70 +38,12 @@ def _cleanup_sleep(signum, frame):
signal.signal(signal.SIGINT, _cleanup_sleep)
signal.signal(signal.SIGTERM, _cleanup_sleep)
# Ensure sys.stdout and sys.stderr are valid in GUI mode
if sys.stdout is None:
sys.stdout = open(os.devnull, "w")
if sys.stderr is None:
sys.stderr = open(os.devnull, "w")
# Enable MPS GPU acceleration on Mac Apple Silicon
if platform.system() == "Darwin" and platform.processor() == "arm":
os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"
def main() -> None:
"""Launch the Flask-based web UI."""
_run_web_ui()
# Custom message handler to filter out specific Qt warnings
def qt_message_handler(mode, context, message):
# In PyQt6, the mode is an enum, so we compare with the enum members
if "Wayland does not support QWindow::requestActivate()" in message:
return # Suppress this specific message
if "setGrabPopup called with a parent, QtWaylandClient" in message:
return
if mode == QtMsgType.QtWarningMsg:
print(f"Qt Warning: {message}")
elif mode == QtMsgType.QtCriticalMsg:
print(f"Qt Critical: {message}")
elif mode == QtMsgType.QtFatalMsg:
print(f"Qt Fatal: {message}")
elif mode == QtMsgType.QtInfoMsg:
print(f"Qt Info: {message}")
# Install the custom message handler
qInstallMessageHandler(qt_message_handler)
# Handle Wayland on Linux GNOME
if platform.system() == "Linux":
xdg_session = os.environ.get("XDG_SESSION_TYPE", "").lower()
desktop = os.environ.get("XDG_CURRENT_DESKTOP", "").lower()
if (
"gnome" in desktop
and xdg_session == "wayland"
and "QT_QPA_PLATFORM" not in os.environ
):
os.environ["QT_QPA_PLATFORM"] = "wayland"
def main():
"""Main entry point for console usage."""
app = QApplication(sys.argv)
# Set application icon using get_resource_path from utils
icon_path = get_resource_path("abogen.assets", "icon.ico")
if icon_path:
app.setWindowIcon(QIcon(icon_path))
# Set the .desktop name on Linux
if platform.system() == "Linux":
try:
app.setDesktopFileName("abogen")
except AttributeError:
pass
ex = abogen()
ex.show()
sys.exit(app.exec())
if __name__ == "__main__":
if __name__ == "__main__": # pragma: no cover - manual execution hook
main()
+246
View File
@@ -0,0 +1,246 @@
from __future__ import annotations
import os
from dataclasses import replace
from functools import lru_cache
from typing import Any, Dict, Mapping, Optional
from abogen.kokoro_text_normalization import (
ApostropheConfig,
CONTRACTION_CATEGORY_DEFAULTS,
)
from abogen.llm_client import LLMConfiguration
from abogen.utils import load_config
DEFAULT_LLM_PROMPT = (
"You are assisting with audiobook preparation. Analyze the sentence and identify any apostrophes or "
"contractions that should be expanded for clarity. Call the apply_regex_replacements tool with precise "
"regex substitutions for only the words that need adjustment. If no changes are required, return an empty list.\n"
"Sentence: {{ sentence }}"
)
_LEGACY_REWRITE_ONLY_PROMPT = (
"You are assisting with audiobook preparation. Rewrite the provided sentence so apostrophes and "
"contractions are unambiguous for text-to-speech. Respond with only the rewritten sentence.\n"
"Sentence: {{ sentence }}\n"
"Context: {{ paragraph }}"
)
_SETTINGS_DEFAULTS: Dict[str, Any] = {
"llm_base_url": "",
"llm_api_key": "",
"llm_model": "",
"llm_timeout": 30.0,
"llm_prompt": DEFAULT_LLM_PROMPT,
"llm_context_mode": "sentence",
"normalization_numbers": True,
"normalization_numbers_year_style": "american",
"normalization_currency": True,
"normalization_footnotes": True,
"normalization_titles": True,
"normalization_terminal": True,
"normalization_phoneme_hints": True,
"normalization_caps_quotes": True,
"normalization_internet_slang": False,
"normalization_apostrophes_contractions": True,
"normalization_apostrophes_plural_possessives": True,
"normalization_apostrophes_sibilant_possessives": True,
"normalization_apostrophes_decades": True,
"normalization_apostrophes_leading_elisions": True,
"normalization_apostrophe_mode": "spacy",
"normalization_contraction_aux_be": True,
"normalization_contraction_aux_have": True,
"normalization_contraction_modal_will": True,
"normalization_contraction_modal_would": True,
"normalization_contraction_negation_not": True,
"normalization_contraction_let_us": True,
}
_CONTRACTION_SETTING_MAP: Dict[str, str] = {
"normalization_contraction_aux_be": "contraction_aux_be",
"normalization_contraction_aux_have": "contraction_aux_have",
"normalization_contraction_modal_will": "contraction_modal_will",
"normalization_contraction_modal_would": "contraction_modal_would",
"normalization_contraction_negation_not": "contraction_negation_not",
"normalization_contraction_let_us": "contraction_let_us",
}
_ENVIRONMENT_KEYS: Dict[str, str] = {
"llm_base_url": "ABOGEN_LLM_BASE_URL",
"llm_api_key": "ABOGEN_LLM_API_KEY",
"llm_model": "ABOGEN_LLM_MODEL",
"llm_timeout": "ABOGEN_LLM_TIMEOUT",
"llm_prompt": "ABOGEN_LLM_PROMPT",
"llm_context_mode": "ABOGEN_LLM_CONTEXT_MODE",
}
NORMALIZATION_SAMPLE_TEXTS: Dict[str, str] = {
"apostrophes": "I've heard the captain'll arrive by dusk, but they'd said the same yesterday.",
"numbers": "The ledger listed 1,204 outstanding debts totaling $57,890.",
"titles": "Dr. Smith met Mr. O'Leary outside St. John's Church on Jan. 4th.",
"punctuation": "Meet me at the docks tonight We'll decide then", # missing punctuation
}
@lru_cache(maxsize=1)
def _environment_defaults() -> Dict[str, Any]:
overrides: Dict[str, Any] = {}
for key, env_var in _ENVIRONMENT_KEYS.items():
default = _SETTINGS_DEFAULTS.get(key)
if default is None:
continue
value = os.environ.get(env_var)
if value is None or value == "":
continue
if isinstance(default, bool):
overrides[key] = _coerce_bool(value, default)
elif isinstance(default, float):
overrides[key] = _coerce_float(value, float(default))
else:
overrides[key] = value
return overrides
def environment_llm_defaults() -> Dict[str, Any]:
defaults = dict(_environment_defaults())
if defaults:
_apply_llm_migrations(defaults)
return defaults
def _coerce_bool(value: Any, default: bool) -> bool:
if isinstance(value, bool):
return value
if isinstance(value, str):
lowered = value.strip().lower()
if lowered in {"1", "true", "yes", "on"}:
return True
if lowered in {"0", "false", "no", "off"}:
return False
return default
def _coerce_float(value: Any, default: float) -> float:
try:
return float(value)
except (TypeError, ValueError):
return default
def _apply_llm_migrations(settings: Dict[str, Any]) -> None:
prompt_value = str(settings.get("llm_prompt") or "")
if prompt_value.strip() == _LEGACY_REWRITE_ONLY_PROMPT.strip():
settings["llm_prompt"] = DEFAULT_LLM_PROMPT
context_mode = str(settings.get("llm_context_mode") or "").strip().lower()
if context_mode != "sentence":
settings["llm_context_mode"] = "sentence"
def _extract_settings(source: Mapping[str, Any]) -> Dict[str, Any]:
env_defaults = _environment_defaults()
extracted: Dict[str, Any] = {}
for key, default in _SETTINGS_DEFAULTS.items():
if key in source:
raw_value = source.get(key)
elif key in env_defaults:
raw_value = env_defaults[key]
else:
raw_value = default
if isinstance(default, bool):
extracted[key] = _coerce_bool(raw_value, default)
elif isinstance(default, float):
extracted[key] = _coerce_float(raw_value, default)
else:
extracted[key] = (
str(raw_value or "") if isinstance(default, str) else raw_value
)
_apply_llm_migrations(extracted)
return extracted
@lru_cache(maxsize=1)
def _cached_settings() -> Dict[str, Any]:
config = load_config() or {}
return _extract_settings(config)
def get_runtime_settings() -> Dict[str, Any]:
return dict(_cached_settings())
def clear_cached_settings() -> None:
_cached_settings.cache_clear()
def build_apostrophe_config(
*,
settings: Mapping[str, Any],
base: Optional[ApostropheConfig] = None,
) -> ApostropheConfig:
config = replace(base or ApostropheConfig())
config.convert_numbers = bool(settings.get("normalization_numbers", True))
config.convert_currency = bool(settings.get("normalization_currency", True))
config.remove_footnotes = bool(settings.get("normalization_footnotes", True))
config.year_pronunciation_mode = (
str(settings.get("normalization_numbers_year_style", "american") or "")
.strip()
.lower()
)
config.add_phoneme_hints = bool(settings.get("normalization_phoneme_hints", True))
config.contraction_mode = (
"expand"
if settings.get("normalization_apostrophes_contractions", True)
else "keep"
)
config.plural_possessive_mode = (
"collapse"
if settings.get("normalization_apostrophes_plural_possessives", True)
else "keep"
)
config.sibilant_possessive_mode = (
"mark"
if settings.get("normalization_apostrophes_sibilant_possessives", True)
else "keep"
)
config.decades_mode = (
"expand" if settings.get("normalization_apostrophes_decades", True) else "keep"
)
config.leading_elision_mode = (
"expand"
if settings.get("normalization_apostrophes_leading_elisions", True)
else "keep"
)
config.ambiguous_past_modal_mode = (
"contextual" if config.contraction_mode == "expand" else "keep"
)
category_flags = dict(CONTRACTION_CATEGORY_DEFAULTS)
for setting_key, category in _CONTRACTION_SETTING_MAP.items():
default_value = bool(_SETTINGS_DEFAULTS.get(setting_key, True))
raw_value = settings.get(setting_key, default_value)
category_flags[category] = _coerce_bool(raw_value, default_value)
config.contraction_categories = category_flags
return config
def build_llm_configuration(settings: Mapping[str, Any]) -> LLMConfiguration:
return LLMConfiguration(
base_url=str(settings.get("llm_base_url") or ""),
api_key=str(settings.get("llm_api_key") or ""),
model=str(settings.get("llm_model") or ""),
timeout=_coerce_float(
settings.get("llm_timeout"), float(_SETTINGS_DEFAULTS["llm_timeout"])
),
)
def apply_overrides(
base: Mapping[str, Any], overrides: Mapping[str, Any]
) -> Dict[str, Any]:
merged: Dict[str, Any] = dict(base)
for key, value in overrides.items():
if key not in _SETTINGS_DEFAULTS:
continue
merged[key] = value
_apply_llm_migrations(merged)
return merged
+256
View File
@@ -0,0 +1,256 @@
from __future__ import annotations
import json
import sqlite3
import shutil
import threading
import time
import uuid
from pathlib import Path
from typing import Any, Dict, Iterable, List, Mapping, Optional
from .entity_analysis import normalize_token
from .utils import get_internal_cache_path, get_user_settings_dir
_DB_LOCK = threading.RLock()
_SCHEMA_VERSION = 1
def _store_path() -> Path:
try:
base_dir = Path(get_user_settings_dir())
except ModuleNotFoundError:
base_dir = Path(get_internal_cache_path("pronunciations"))
target = base_dir / "overrides.json"
target.parent.mkdir(parents=True, exist_ok=True)
return target
def _migrate_legacy_sqlite(target_json_path: Path) -> None:
try:
base_dir = Path(get_user_settings_dir())
except ModuleNotFoundError:
base_dir = Path(get_internal_cache_path("pronunciations"))
sqlite_path = base_dir / "pronunciations.db"
if not sqlite_path.exists():
return
try:
conn = sqlite3.connect(sqlite_path)
conn.row_factory = sqlite3.Row
# Check if table exists
cursor = conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name='overrides'"
)
if not cursor.fetchone():
conn.close()
return
cursor = conn.execute("SELECT * FROM overrides")
rows = cursor.fetchall()
data = {"version": _SCHEMA_VERSION, "overrides": {}}
for row in rows:
lang = row["language"]
if lang not in data["overrides"]:
data["overrides"][lang] = {}
entry = {
"id": str(row["id"]),
"normalized": row["normalized"],
"token": row["token"],
"language": row["language"],
"pronunciation": row["pronunciation"],
"voice": row["voice"],
"notes": row["notes"],
"context": row["context"],
"usage_count": row["usage_count"],
"created_at": row["created_at"],
"updated_at": row["updated_at"],
}
data["overrides"][lang][row["normalized"]] = entry
conn.close()
# Save to JSON
with open(target_json_path, "w", encoding="utf-8") as f:
json.dump(data, f, indent=2, ensure_ascii=False)
# Rename old DB
sqlite_path.rename(sqlite_path.with_suffix(".db.bak"))
except Exception:
pass
def _load_db() -> Dict[str, Any]:
path = _store_path()
if not path.exists():
_migrate_legacy_sqlite(path)
if not path.exists():
return {"version": _SCHEMA_VERSION, "overrides": {}}
try:
with open(path, "r", encoding="utf-8") as f:
return json.load(f)
except (json.JSONDecodeError, OSError):
return {"version": _SCHEMA_VERSION, "overrides": {}}
def _save_db(data: Dict[str, Any]) -> None:
path = _store_path()
# Atomic write
temp_path = path.with_suffix(".tmp")
with open(temp_path, "w", encoding="utf-8") as f:
json.dump(data, f, indent=2, ensure_ascii=False)
shutil.move(str(temp_path), str(path))
def load_overrides(language: str, tokens: Iterable[str]) -> Dict[str, Dict[str, Any]]:
normalized_tokens = {normalize_token(token) for token in tokens if token}
if not normalized_tokens:
return {}
with _DB_LOCK:
db = _load_db()
lang_overrides = db.get("overrides", {}).get(language, {})
results: Dict[str, Dict[str, Any]] = {}
for normalized in normalized_tokens:
if normalized in lang_overrides:
results[normalized] = lang_overrides[normalized]
return results
def search_overrides(
language: str, query: str, *, limit: int = 15
) -> List[Dict[str, Any]]:
if not query:
return []
query = query.lower()
with _DB_LOCK:
db = _load_db()
lang_overrides = db.get("overrides", {}).get(language, {})
matches = []
for entry in lang_overrides.values():
if query in entry["normalized"] or query in entry["token"].lower():
matches.append(entry)
# Sort by usage count desc, then updated_at desc
matches.sort(
key=lambda x: (x.get("usage_count", 0), x.get("updated_at", 0)),
reverse=True,
)
return matches[:limit]
def save_override(
*,
language: str,
token: str,
pronunciation: Optional[str] = None,
voice: Optional[str] = None,
notes: Optional[str] = None,
context: Optional[str] = None,
) -> Dict[str, Any]:
normalized = normalize_token(token)
if not normalized:
raise ValueError("Provide a token to override")
timestamp = time.time()
with _DB_LOCK:
db = _load_db()
overrides = db.setdefault("overrides", {})
lang_overrides = overrides.setdefault(language, {})
existing = lang_overrides.get(normalized)
if existing:
entry = existing
entry["token"] = token
entry["pronunciation"] = pronunciation
entry["voice"] = voice
entry["notes"] = notes
entry["context"] = context
entry["updated_at"] = timestamp
else:
entry = {
"id": str(uuid.uuid4()),
"normalized": normalized,
"token": token,
"language": language,
"pronunciation": pronunciation,
"voice": voice,
"notes": notes,
"context": context,
"usage_count": 0,
"created_at": timestamp,
"updated_at": timestamp,
}
lang_overrides[normalized] = entry
_save_db(db)
return entry
def delete_override(*, language: str, token: str) -> None:
normalized = normalize_token(token)
if not normalized:
return
with _DB_LOCK:
db = _load_db()
lang_overrides = db.get("overrides", {}).get(language, {})
if normalized in lang_overrides:
del lang_overrides[normalized]
_save_db(db)
def all_overrides(language: str) -> List[Dict[str, Any]]:
with _DB_LOCK:
db = _load_db()
lang_overrides = db.get("overrides", {}).get(language, {})
results = list(lang_overrides.values())
results.sort(key=lambda x: x.get("updated_at", 0), reverse=True)
return results
def increment_usage(*, language: str, token: str, amount: int = 1) -> None:
normalized = normalize_token(token)
if not normalized:
return
with _DB_LOCK:
db = _load_db()
lang_overrides = db.get("overrides", {}).get(language, {})
if normalized in lang_overrides:
entry = lang_overrides[normalized]
entry["usage_count"] = entry.get("usage_count", 0) + amount
entry["updated_at"] = time.time()
_save_db(db)
def get_override_stats(language: str) -> Dict[str, int]:
with _DB_LOCK:
db = _load_db()
lang_overrides = db.get("overrides", {}).get(language, {})
total = len(lang_overrides)
with_pronunciation = sum(
1 for x in lang_overrides.values() if x.get("pronunciation")
)
with_voice = sum(1 for x in lang_overrides.values() if x.get("voice"))
return {
"total": total,
"filtered": total,
"with_pronunciation": with_pronunciation,
"with_voice": with_voice,
}
+7
View File
@@ -0,0 +1,7 @@
"""PyQt6 Desktop GUI for abogen.
This package contains the traditional PyQt6-based desktop interface.
For the web-based interface, see abogen.webui.
"""
from __future__ import annotations
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+4509
View File
File diff suppressed because it is too large Load Diff
+187
View File
@@ -0,0 +1,187 @@
import os
import sys
import platform
import atexit
import signal
from abogen.utils import get_resource_path, load_config, prevent_sleep_end
# Fix PyTorch DLL loading issue ([WinError 1114]) on Windows before importing PyQt6
if platform.system() == "Windows":
import ctypes
from importlib.util import find_spec
try:
if (
(spec := find_spec("torch"))
and spec.origin
and os.path.exists(
dll_path := os.path.join(os.path.dirname(spec.origin), "lib", "c10.dll")
)
):
ctypes.CDLL(os.path.normpath(dll_path))
except Exception:
pass
# Qt platform plugin detection (fixes #59)
try:
from PyQt6.QtCore import QLibraryInfo
# Get the path to the plugins directory
plugins = QLibraryInfo.path(QLibraryInfo.LibraryPath.PluginsPath)
# Normalize path to use the OS-native separators and absolute path
platform_dir = os.path.normpath(os.path.join(plugins, "platforms"))
# Ensure we work with an absolute path for clarity
platform_dir = os.path.abspath(platform_dir)
if os.path.isdir(platform_dir):
os.environ["QT_QPA_PLATFORM_PLUGIN_PATH"] = platform_dir
print("QT_QPA_PLATFORM_PLUGIN_PATH set to:", platform_dir)
else:
print("PyQt6 platform plugins not found at", platform_dir)
except ImportError:
print("PyQt6 not installed.")
# Pre-load "libxcb-cursor" on Linux (fixes #101)
if platform.system() == "Linux":
arch = platform.machine().lower()
lib_filename = {"x86_64": "libxcb-cursor-amd64.so.0", "amd64": "libxcb-cursor-amd64.so.0", "aarch64": "libxcb-cursor-arm64.so.0", "arm64": "libxcb-cursor-arm64.so.0"}.get(arch)
if lib_filename:
import ctypes
try:
# Try to load the system libxcb-cursor.so.0 first
ctypes.CDLL('libxcb-cursor.so.0', mode=ctypes.RTLD_GLOBAL)
except OSError:
# System lib not available, load the bundled version
lib_path = get_resource_path('abogen.libs', lib_filename)
if lib_path:
try:
ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
except OSError:
# If it fails (e.g. wrong glibc version on very old systems),
# we simply ignore it and hope the system has the library.
pass
# Set application ID for Windows taskbar icon
if platform.system() == "Windows":
try:
from abogen.constants import PROGRAM_NAME, VERSION
import ctypes
app_id = f"{PROGRAM_NAME}.{VERSION}"
ctypes.windll.shell32.SetCurrentProcessExplicitAppUserModelID(app_id)
except Exception as e:
print("Warning: failed to set AppUserModelID:", e)
from PyQt6.QtWidgets import QApplication
from PyQt6.QtGui import QIcon
from PyQt6.QtCore import (
QLibraryInfo,
qInstallMessageHandler,
QtMsgType,
)
# Add the directory to Python path
sys.path.insert(0, os.path.join(os.path.dirname(__file__)))
# Set Hugging Face Hub environment variables
os.environ["HF_HUB_DISABLE_TELEMETRY"] = "1" # Disable Hugging Face telemetry
os.environ["HF_HUB_ETAG_TIMEOUT"] = "10" # Metadata request timeout (seconds)
os.environ["HF_HUB_DOWNLOAD_TIMEOUT"] = "10" # File download timeout (seconds)
os.environ["HF_HUB_DISABLE_SYMLINKS_WARNING"] = "1" # Disable symlinks warning
if load_config().get("disable_kokoro_internet", False):
print("INFO: Kokoro's internet access is disabled.")
os.environ["HF_HUB_OFFLINE"] = "1" # Disable Hugging Face Hub internet access
from abogen.pyqt.gui import abogen
from abogen.constants import PROGRAM_NAME, VERSION
# Set environment variables for AMD ROCm
os.environ["MIOPEN_FIND_MODE"] = "FAST"
os.environ["MIOPEN_CONV_PRECISE_ROCM_TUNING"] = "0"
# Reset sleep states
atexit.register(prevent_sleep_end)
# Also handle signals (Ctrl+C, kill, etc.)
def _cleanup_sleep(signum, frame):
prevent_sleep_end()
sys.exit(0)
signal.signal(signal.SIGINT, _cleanup_sleep)
signal.signal(signal.SIGTERM, _cleanup_sleep)
# Ensure sys.stdout and sys.stderr are valid in GUI mode
if sys.stdout is None:
sys.stdout = open(os.devnull, "w")
if sys.stderr is None:
sys.stderr = open(os.devnull, "w")
# Enable MPS GPU acceleration on Mac Apple Silicon
if platform.system() == "Darwin" and platform.processor() == "arm":
os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"
# Custom message handler to filter out specific Qt warnings
def qt_message_handler(mode, context, message):
# In PyQt6, the mode is an enum, so we compare with the enum members
if "Wayland does not support QWindow::requestActivate()" in message:
return # Suppress this specific message
if "setGrabPopup called with a parent, QtWaylandClient" in message:
return
if mode == QtMsgType.QtWarningMsg:
print(f"Qt Warning: {message}")
elif mode == QtMsgType.QtCriticalMsg:
print(f"Qt Critical: {message}")
elif mode == QtMsgType.QtFatalMsg:
print(f"Qt Fatal: {message}")
elif mode == QtMsgType.QtInfoMsg:
print(f"Qt Info: {message}")
# Install the custom message handler
qInstallMessageHandler(qt_message_handler)
# Handle Wayland on Linux GNOME
if platform.system() == "Linux":
xdg_session = os.environ.get("XDG_SESSION_TYPE", "").lower()
desktop = os.environ.get("XDG_CURRENT_DESKTOP", "").lower()
if (
"gnome" in desktop
and xdg_session == "wayland"
and "QT_QPA_PLATFORM" not in os.environ
):
os.environ["QT_QPA_PLATFORM"] = "wayland"
def main():
"""Main entry point for console usage."""
app = QApplication(sys.argv)
# Set application icon using get_resource_path from utils
icon_path = get_resource_path("abogen.assets", "icon.ico")
if icon_path:
app.setWindowIcon(QIcon(icon_path))
# Set the .desktop name on Linux
if platform.system() == "Linux":
try:
app.setDesktopFileName("abogen")
except AttributeError:
pass
ex = abogen()
ex.show()
sys.exit(app.exec())
if __name__ == "__main__":
main()
+590
View File
@@ -0,0 +1,590 @@
"""
Pre-download dialog and worker for Abogen
This module consolidates pre-download logic for Kokoro voices and model
and spaCy language models. The code favors clarity, avoids duplication,
and handles optional dependencies gracefully.
"""
from typing import List, Optional, Tuple
import importlib
import importlib.util
from PyQt6.QtWidgets import (
QDialog,
QVBoxLayout,
QHBoxLayout,
QLabel,
QPushButton,
QSpacerItem,
QSizePolicy,
)
from PyQt6.QtCore import QThread, pyqtSignal
from abogen.constants import COLORS, VOICES_INTERNAL
from abogen.spacy_utils import SPACY_MODELS
import abogen.hf_tracker
# Helpers
def _unique_sorted_models() -> List[str]:
"""Return a sorted list of unique spaCy model package names."""
return sorted(set(SPACY_MODELS.values()))
def _is_package_installed(pkg_name: str) -> bool:
"""Return True if a package with the given name can be imported (site-packages)."""
try:
return importlib.util.find_spec(pkg_name) is not None
except Exception:
return False
# NOTE: explicit HF cache helper removed; we use try_to_load_from_cache in-scope where needed
class PreDownloadWorker(QThread):
"""Worker thread to download required models/voices.
Emits human-readable messages via `progress`. Uses `category_done` to indicate
a category (voices/model/spacy) finished successfully. Emits `error` on exception
and `finished` after all work completes.
"""
# Emit (category, status, message)
progress = pyqtSignal(str, str, str)
category_done = pyqtSignal(str)
finished = pyqtSignal()
error = pyqtSignal(str)
def __init__(self, parent=None):
super().__init__(parent)
self._cancelled = False
# repo and filenames used for Kokoro model
self._repo_id = "hexgrad/Kokoro-82M"
self._model_files = ["kokoro-v1_0.pth", "config.json"]
# Track download success per category
self._voices_success = False
self._model_success = False
self._spacy_success = False
# Suppress HF tracker warnings during downloads
self._original_emitter = abogen.hf_tracker.show_warning_signal_emitter
def cancel(self) -> None:
self._cancelled = True
def run(self) -> None:
# Suppress HF tracker warnings during downloads
abogen.hf_tracker.show_warning_signal_emitter = None
try:
self._download_kokoro_voices()
if self._cancelled:
return
if self._voices_success:
self.category_done.emit("voices")
self._download_kokoro_model()
if self._cancelled:
return
if self._model_success:
self.category_done.emit("model")
self._download_spacy_models()
if self._cancelled:
return
if self._spacy_success:
self.category_done.emit("spacy")
self.finished.emit()
except Exception as exc: # pragma: no cover - best-effort reporting
self.error.emit(str(exc))
finally:
# Restore original emitter
abogen.hf_tracker.show_warning_signal_emitter = self._original_emitter
# Kokoro voices
def _download_kokoro_voices(self) -> None:
self._voices_success = True
try:
from huggingface_hub import hf_hub_download, try_to_load_from_cache
except Exception:
self.progress.emit(
"voice", "warning", "huggingface_hub not installed, skipping voices..."
)
self._voices_success = False
return
voice_list = VOICES_INTERNAL
for idx, voice in enumerate(voice_list, start=1):
if self._cancelled:
self._voices_success = False
return
filename = f"voices/{voice}.pt"
if try_to_load_from_cache(repo_id=self._repo_id, filename=filename):
self.progress.emit(
"voice",
"installed",
f"{idx}/{len(voice_list)}: {voice} already present",
)
continue
self.progress.emit(
"voice", "downloading", f"{idx}/{len(voice_list)}: {voice}..."
)
try:
hf_hub_download(repo_id=self._repo_id, filename=filename)
self.progress.emit("voice", "downloaded", f"{voice} downloaded")
except Exception as exc:
self.progress.emit(
"voice", "warning", f"could not download {voice}: {exc}"
)
self._voices_success = False
# Kokoro model
def _download_kokoro_model(self) -> None:
self._model_success = True
try:
from huggingface_hub import hf_hub_download, try_to_load_from_cache
except Exception:
self.progress.emit(
"model", "warning", "huggingface_hub not installed, skipping model..."
)
self._model_success = False
return
for fname in self._model_files:
if self._cancelled:
self._model_success = False
return
category = "config" if fname == "config.json" else "model"
if try_to_load_from_cache(repo_id=self._repo_id, filename=fname):
self.progress.emit(
category, "installed", f"file {fname} already present"
)
continue
self.progress.emit(category, "downloading", f"file {fname}...")
try:
hf_hub_download(repo_id=self._repo_id, filename=fname)
self.progress.emit(category, "downloaded", f"file {fname} downloaded")
except Exception as exc:
self.progress.emit(
category, "warning", f"could not download file {fname}: {exc}"
)
self._model_success = False
# spaCy models
def _download_spacy_models(self) -> None:
"""Download spaCy models. Prefer missing models provided by parent.
Parent dialog will populate _spacy_models_missing during checking.
"""
self._spacy_success = True
# Determine which models to process: prefer parent-provided missing list to avoid
# re-checking everything; otherwise use the full unique list.
parent = self.parent()
models_to_process: List[str] = _unique_sorted_models()
try:
if (
parent is not None
and hasattr(parent, "_spacy_models_missing")
and parent._spacy_models_missing
):
models_to_process = list(dict.fromkeys(parent._spacy_models_missing))
except Exception:
pass
# If spaCy is not available to run the CLI, skip gracefully
try:
import spacy.cli as _spacy_cli
except Exception:
self.progress.emit(
"spacy", "warning", "spaCy not available, skipping spaCy models..."
)
self._spacy_success = False
return
for idx, model_name in enumerate(models_to_process, start=1):
if self._cancelled:
self._spacy_success = False
return
if _is_package_installed(model_name):
self.progress.emit(
"spacy",
"installed",
f"{idx}/{len(models_to_process)}: {model_name} already installed",
)
continue
self.progress.emit(
"spacy",
"downloading",
f"{idx}/{len(models_to_process)}: {model_name}...",
)
try:
_spacy_cli.download(model_name)
self.progress.emit("spacy", "downloaded", f"{model_name} downloaded")
except Exception as exc:
self.progress.emit(
"spacy", "warning", f"could not download {model_name}: {exc}"
)
self._spacy_success = False
class PreDownloadDialog(QDialog):
"""Dialog to show and control pre-download process."""
VOICE_PREFIX = "Kokoro voices: "
MODEL_PREFIX = "Kokoro model: "
CONFIG_PREFIX = "Kokoro config: "
SPACY_PREFIX = "spaCy models: "
def __init__(self, parent=None):
super().__init__(parent)
self.setWindowTitle("Pre-download Models and Voices")
self.setMinimumWidth(500)
self.worker: Optional[PreDownloadWorker] = None
self.has_missing = False
self._spacy_models_checked: List[tuple] = []
self._spacy_models_missing: List[str] = []
self._status_worker = None
# Map keywords to (label, prefix) - labels filled after UI creation
self.status_map = {
"voice": (None, self.VOICE_PREFIX),
"spacy": (None, self.SPACY_PREFIX),
"model": (None, self.MODEL_PREFIX),
"config": (None, self.CONFIG_PREFIX),
}
self.category_map = {
"voices": ["voice"],
"model": ["model", "config"],
"spacy": ["spacy"],
}
self._setup_ui()
self._start_status_check()
def _setup_ui(self) -> None:
layout = QVBoxLayout(self)
layout.setSpacing(0)
layout.setContentsMargins(15, 0, 15, 15)
desc = QLabel(
"You can pre-download all required models and voices for offline use.\n"
"This includes Kokoro voices, Kokoro model (and config), and spaCy models."
)
desc.setWordWrap(True)
layout.addWidget(desc)
# Status rows
status_layout = QVBoxLayout()
status_title = QLabel("<b>Current Status:</b>")
status_layout.addWidget(status_title)
self.voices_status = QLabel(self.VOICE_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.voices_status)
row.addStretch()
status_layout.addLayout(row)
self.model_status = QLabel(self.MODEL_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.model_status)
row.addStretch()
status_layout.addLayout(row)
self.config_status = QLabel(self.CONFIG_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.config_status)
row.addStretch()
status_layout.addLayout(row)
self.spacy_status = QLabel(self.SPACY_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.spacy_status)
row.addStretch()
status_layout.addLayout(row)
# register labels
self.status_map["voice"] = (self.voices_status, self.VOICE_PREFIX)
self.status_map["model"] = (self.model_status, self.MODEL_PREFIX)
self.status_map["config"] = (self.config_status, self.CONFIG_PREFIX)
self.status_map["spacy"] = (self.spacy_status, self.SPACY_PREFIX)
layout.addLayout(status_layout)
layout.addItem(
QSpacerItem(0, 20, QSizePolicy.Policy.Minimum, QSizePolicy.Policy.Fixed)
)
# Buttons
button_row = QHBoxLayout()
button_row.setSpacing(10)
self.download_btn = QPushButton("Download all")
self.download_btn.setMinimumWidth(100)
self.download_btn.setMinimumHeight(35)
self.download_btn.setEnabled(False)
self.download_btn.clicked.connect(self._start_download)
button_row.addWidget(self.download_btn)
self.close_btn = QPushButton("Close")
self.close_btn.setMinimumWidth(100)
self.close_btn.setMinimumHeight(35)
self.close_btn.clicked.connect(self._handle_close)
button_row.addWidget(self.close_btn)
layout.addLayout(button_row)
self.adjustSize()
# Status checking worker
class StatusCheckWorker(QThread):
voices_checked = pyqtSignal(bool, list)
model_checked = pyqtSignal(bool)
config_checked = pyqtSignal(bool)
spacy_model_checking = pyqtSignal(str)
spacy_model_result = pyqtSignal(str, bool)
spacy_checked = pyqtSignal(bool, list)
def run(self):
parent = self.parent()
if parent is None:
return
voices_ok, missing_voices = parent._check_kokoro_voices()
self.voices_checked.emit(voices_ok, missing_voices)
model_ok = parent._check_kokoro_model()
self.model_checked.emit(model_ok)
config_ok = parent._check_kokoro_config()
self.config_checked.emit(config_ok)
# Check spaCy models by package name to detect site-package installs
unique = _unique_sorted_models()
missing: List[str] = []
for name in unique:
self.spacy_model_checking.emit(name)
ok = _is_package_installed(name)
self.spacy_model_result.emit(name, ok)
if not ok:
missing.append(name)
parent._spacy_models_missing = missing
self.spacy_checked.emit(len(missing) == 0, missing)
def _start_status_check(self) -> None:
self._status_worker = self.StatusCheckWorker(self)
self._status_worker.voices_checked.connect(self._update_voices_status)
self._status_worker.model_checked.connect(self._update_model_status)
self._status_worker.config_checked.connect(self._update_config_status)
self._status_worker.spacy_model_checking.connect(self._spacy_model_checking)
self._status_worker.spacy_model_result.connect(self._spacy_model_result)
self._status_worker.spacy_checked.connect(self._update_spacy_status)
# These are initialized in __init__ to keep consistent object state
# Set checking visual state
for lbl in (
self.voices_status,
self.model_status,
self.config_status,
self.spacy_status,
):
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
self.spacy_status.setText(self.SPACY_PREFIX + "⏳ Checking...")
self._status_worker.start()
# UI update callbacks
def _spacy_model_checking(self, name: str) -> None:
self.spacy_status.setText(f"{self.SPACY_PREFIX}Checking {name}...")
def _spacy_model_result(self, name: str, ok: bool) -> None:
self._spacy_models_checked.append((name, ok))
if not ok and name not in self._spacy_models_missing:
self._spacy_models_missing.append(name)
checked = len(self._spacy_models_checked)
missing_count = len(self._spacy_models_missing)
if missing_count:
self.spacy_status.setText(
f"{self.SPACY_PREFIX}{checked} checked, {missing_count} missing..."
)
else:
self.spacy_status.setText(f"{self.SPACY_PREFIX}{checked} checked...")
def _update_voices_status(self, ok: bool, missing: List[str]) -> None:
if ok:
self._set_status("voice", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
if missing:
self._set_status(
"voice", f"✗ Missing {len(missing)} voices", COLORS["RED"]
)
else:
self._set_status("voice", "✗ Not downloaded", COLORS["RED"])
def _update_model_status(self, ok: bool) -> None:
if ok:
self._set_status("model", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
self._set_status("model", "✗ Not downloaded", COLORS["RED"])
def _update_config_status(self, ok: bool) -> None:
if ok:
self._set_status("config", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
self._set_status("config", "✗ Not downloaded", COLORS["RED"])
def _update_spacy_status(self, ok: bool, missing: List[str]) -> None:
if ok:
self._set_status("spacy", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
if missing:
self._set_status(
"spacy", f"✗ Missing {len(missing)} model(s)", COLORS["RED"]
)
else:
self._set_status("spacy", "✗ Not downloaded", COLORS["RED"])
self.download_btn.setEnabled(self.has_missing)
def _set_status(self, key: str, text: str, color: str) -> None:
lbl, prefix = self.status_map.get(key, (None, ""))
if not lbl:
return
lbl.setText(prefix + text)
lbl.setStyleSheet(f"color: {color};")
# Helper checks
def _check_kokoro_voices(self) -> Tuple[bool, List[str]]:
"""Return (ok, missing_list) for Kokoro voices check."""
missing = []
try:
from huggingface_hub import try_to_load_from_cache
for voice in VOICES_INTERNAL:
if not try_to_load_from_cache(
repo_id="hexgrad/Kokoro-82M", filename=f"voices/{voice}.pt"
):
missing.append(voice)
except Exception:
# If HF missing, report all as missing
return False, list(VOICES_INTERNAL)
return (len(missing) == 0), missing
def _check_kokoro_model(self) -> bool:
try:
from huggingface_hub import try_to_load_from_cache
return (
try_to_load_from_cache(
repo_id="hexgrad/Kokoro-82M", filename="kokoro-v1_0.pth"
)
is not None
)
except Exception:
return False
def _check_kokoro_config(self) -> bool:
try:
from huggingface_hub import try_to_load_from_cache
return (
try_to_load_from_cache(
repo_id="hexgrad/Kokoro-82M", filename="config.json"
)
is not None
)
except Exception:
return False
def _check_spacy_models(self) -> bool:
unique = _unique_sorted_models()
missing = [m for m in unique if not _is_package_installed(m)]
self._spacy_models_missing = missing
return len(missing) == 0
# Download control
def _start_download(self) -> None:
self.download_btn.setEnabled(False)
self.download_btn.setText("Downloading...")
# mark the start of downloads; this triggers the labels
self._on_progress("system", "starting", "Processing, please wait...")
self.worker = PreDownloadWorker(self)
self.worker.progress.connect(self._on_progress)
self.worker.category_done.connect(self._on_category_done)
self.worker.finished.connect(self._on_download_finished)
self.worker.error.connect(self._on_download_error)
self.worker.start()
def _on_progress(self, category: str, status: str, message: str) -> None:
"""Map worker (category, status, message) to UI label updates.
Status is one of: 'downloading', 'installed', 'downloaded', 'warning', 'starting'.
Category is one of: 'voice', 'model', 'spacy', 'config', or 'system'.
"""
try:
# If the category targets a specific label, update directly
if category in self.status_map:
lbl, prefix = self.status_map[category]
if not lbl:
return
# Compose message and set color based on status token
full_text = prefix + message
if len(full_text) > 60:
display_text = full_text[:57] + "..."
lbl.setText(display_text)
lbl.setToolTip(full_text)
else:
lbl.setText(full_text)
lbl.setToolTip("") # Clear tooltip if not needed
if status == "downloading":
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
elif status in ("installed", "downloaded"):
lbl.setStyleSheet(f"color: {COLORS['GREEN']};")
elif status == "warning":
lbl.setStyleSheet(f"color: {COLORS['RED']};")
elif status == "error":
lbl.setStyleSheet(f"color: {COLORS['RED']};")
return
# System-level messages
if category == "system":
if status == "starting":
for k in self.status_map:
lbl, prefix = self.status_map[k]
if lbl:
lbl.setText(prefix + "Processing, please wait...")
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
# other system statuses don't require action
return
except Exception:
# Do not let UI thread crash on unexpected worker message
pass
def _on_category_done(self, category: str) -> None:
for key in self.category_map.get(category, []):
self._set_status(key, "✓ Downloaded", COLORS["GREEN"])
def _on_download_finished(self) -> None:
self.has_missing = False
self.download_btn.setText("Download all")
self.download_btn.setEnabled(False)
def _on_download_error(self, error_msg: str) -> None:
self.download_btn.setText("Download all")
self.download_btn.setEnabled(True)
for key in self.status_map:
self._set_status(key, f"✗ Error - {error_msg}", COLORS["RED"])
def _handle_close(self) -> None:
if self.worker and self.worker.isRunning():
self.worker.cancel()
self.worker.wait(2000)
self.accept()
def closeEvent(self, event) -> None:
if self.worker and self.worker.isRunning():
self.worker.cancel()
self.worker.wait(2000)
super().closeEvent(event)
+881
View File
@@ -0,0 +1,881 @@
# a simple window with a list of items in the queue, no checkboxes
# button to remove an item from the queue
# button to clear the queue
from PyQt6.QtWidgets import (
QDialog,
QVBoxLayout,
QHBoxLayout,
QDialogButtonBox,
QPushButton,
QListWidget,
QListWidgetItem,
QFileIconProvider,
QLabel,
QWidget,
QSizePolicy,
QAbstractItemView,
QCheckBox,
)
from PyQt6.QtCore import QFileInfo, Qt
from abogen.constants import COLORS
from copy import deepcopy
from PyQt6.QtGui import QFontMetrics
from abogen.utils import load_config, save_config
# Define attributes that are safe to override with global settings
OVERRIDE_FIELDS = [
"lang_code",
"speed",
"voice",
"save_option",
"output_folder",
"subtitle_mode",
"output_format",
"replace_single_newlines",
"use_silent_gaps",
"subtitle_speed_method",
"word_substitutions_enabled",
"word_substitutions_list",
"case_sensitive_substitutions",
"replace_all_caps",
"replace_numerals",
"fix_nonstandard_punctuation",
]
class ElidedLabel(QLabel):
def __init__(self, text):
super().__init__(text)
self._full_text = text
self.setSizePolicy(QSizePolicy.Policy.Expanding, QSizePolicy.Policy.Preferred)
self.setTextFormat(Qt.TextFormat.PlainText)
def setText(self, text):
self._full_text = text
super().setText(text)
self.update()
def resizeEvent(self, event):
metrics = QFontMetrics(self.font())
elided = metrics.elidedText(
self._full_text, Qt.TextElideMode.ElideRight, self.width()
)
super().setText(elided)
super().resizeEvent(event)
def fullText(self):
return self._full_text
class QueueListItemWidget(QWidget):
def __init__(self, file_name, char_count):
super().__init__()
layout = QHBoxLayout()
layout.setContentsMargins(12, 0, 6, 0)
layout.setSpacing(0)
import os
name_label = ElidedLabel(os.path.basename(file_name))
char_label = QLabel(f"Chars: {char_count}")
char_label.setStyleSheet(f"color: {COLORS['LIGHT_DISABLED']};")
char_label.setAlignment(
Qt.AlignmentFlag.AlignRight | Qt.AlignmentFlag.AlignVCenter
)
char_label.setSizePolicy(
QSizePolicy.Policy.Minimum, QSizePolicy.Policy.Preferred
)
layout.addWidget(name_label, 1)
layout.addWidget(char_label, 0)
self.setLayout(layout)
class DroppableQueueListWidget(QListWidget):
def __init__(self, parent_dialog):
super().__init__()
self.parent_dialog = parent_dialog
self.setAcceptDrops(True)
# Overlay for drag hover
self.drag_overlay = QLabel("", self)
self.drag_overlay.setAlignment(Qt.AlignmentFlag.AlignCenter)
self.drag_overlay.setStyleSheet(
f"border:2px dashed {COLORS['BLUE_BORDER_HOVER']}; border-radius:5px; padding:20px; background:{COLORS['BLUE_BG_HOVER']};"
)
self.drag_overlay.setVisible(False)
self.drag_overlay.setAttribute(
Qt.WidgetAttribute.WA_TransparentForMouseEvents, True
)
def dragEnterEvent(self, event):
if event.mimeData().hasUrls():
for url in event.mimeData().urls():
file_path = url.toLocalFile().lower()
if url.isLocalFile() and (
file_path.endswith(".txt")
or file_path.endswith((".srt", ".ass", ".vtt"))
):
self.drag_overlay.resize(self.size())
self.drag_overlay.setVisible(True)
event.acceptProposedAction()
return
self.drag_overlay.setVisible(False)
event.ignore()
def dragMoveEvent(self, event):
if event.mimeData().hasUrls():
for url in event.mimeData().urls():
file_path = url.toLocalFile().lower()
if url.isLocalFile() and (
file_path.endswith(".txt")
or file_path.endswith((".srt", ".ass", ".vtt"))
):
event.acceptProposedAction()
return
event.ignore()
def dragLeaveEvent(self, event):
self.drag_overlay.setVisible(False)
event.accept()
def dropEvent(self, event):
self.drag_overlay.setVisible(False)
if event.mimeData().hasUrls():
file_paths = [
url.toLocalFile()
for url in event.mimeData().urls()
if url.isLocalFile()
and (
url.toLocalFile().lower().endswith(".txt")
or url.toLocalFile().lower().endswith((".srt", ".ass", ".vtt"))
)
]
if file_paths:
self.parent_dialog.add_files_from_paths(file_paths)
event.acceptProposedAction()
else:
event.ignore()
else:
event.ignore()
def resizeEvent(self, event):
super().resizeEvent(event)
if hasattr(self, "drag_overlay"):
self.drag_overlay.resize(self.size())
class QueueManager(QDialog):
def __init__(self, parent, queue: list, title="Queue Manager", size=(600, 700)):
super().__init__()
self.queue = queue
self._original_queue = deepcopy(
queue
) # Store a deep copy of the original queue
self.parent = parent
self.config = load_config() # Load config for persistence
layout = QVBoxLayout()
layout.setContentsMargins(15, 15, 15, 15) # set main layout margins
layout.setSpacing(12) # set spacing between widgets in main layout
# list of queued items
self.listwidget = DroppableQueueListWidget(self)
self.listwidget.setSelectionMode(
QAbstractItemView.SelectionMode.ExtendedSelection
)
self.listwidget.setAlternatingRowColors(True)
self.listwidget.setContextMenuPolicy(Qt.ContextMenuPolicy.CustomContextMenu)
self.listwidget.customContextMenuRequested.connect(self.show_context_menu)
# Add informative instructions at the top
instructions = QLabel(
"<h2>How Queue Works?</h2>"
"You can add text and subtitle files (.txt, .srt, .ass, .vtt) directly using the '<b>Add files</b>' button below. "
"To add PDF, EPUB or markdown files, use the input box in the main window and click the <b>'Add to Queue'</b> button. "
"By default, each file in the queue keeps the configuration settings active when they were added. "
"Enabling the <b>'Override item settings with current selection'</b> option below will force all items to use the configuration currently selected in the main window. "
"You can view each file's configuration by hovering over them."
)
instructions.setAlignment(Qt.AlignmentFlag.AlignLeft)
instructions.setWordWrap(True)
layout.addWidget(instructions)
# Override Checkbox
self.override_chk = QCheckBox("Override item settings with current selection")
self.override_chk.setToolTip(
"If checked, all items in the queue will be processed using the \n"
"settings currently selected in the main window, ignoring their saved state."
)
# Load saved state (default to False)
self.override_chk.setChecked(self.config.get("queue_override_settings", False))
# Trigger process_queue to update tooltips immediately when toggled
self.override_chk.stateChanged.connect(self.process_queue)
self.override_chk.setStyleSheet("margin-bottom: 8px;")
layout.addWidget(self.override_chk)
# Overlay label for empty queue
self.empty_overlay = QLabel(
"Drag and drop your text or subtitle files here or use the 'Add files' button.",
self.listwidget,
)
self.empty_overlay.setAlignment(Qt.AlignmentFlag.AlignCenter)
self.empty_overlay.setStyleSheet(
f"color: {COLORS['LIGHT_DISABLED']}; background: transparent; padding: 20px;"
)
self.empty_overlay.setWordWrap(True)
self.empty_overlay.setAttribute(
Qt.WidgetAttribute.WA_TransparentForMouseEvents, True
)
self.empty_overlay.hide()
# add queue items to the list
self.process_queue()
button_row = QHBoxLayout()
button_row.setContentsMargins(0, 0, 0, 0) # optional: no margins for button row
button_row.setSpacing(7) # set spacing between buttons
# Add files button
add_files_button = QPushButton("Add files")
add_files_button.setFixedHeight(40)
add_files_button.clicked.connect(self.add_more_files)
button_row.addWidget(add_files_button)
# Remove button
self.remove_button = QPushButton("Remove selected")
self.remove_button.setFixedHeight(40)
self.remove_button.clicked.connect(self.remove_item)
button_row.addWidget(self.remove_button)
# Clear button
self.clear_button = QPushButton("Clear Queue")
self.clear_button.setFixedHeight(40)
self.clear_button.clicked.connect(self.clear_queue)
button_row.addWidget(self.clear_button)
layout.addLayout(button_row)
layout.addWidget(self.listwidget)
# Connect selection change to update button state
self.listwidget.currentItemChanged.connect(self.update_button_states)
self.listwidget.itemSelectionChanged.connect(self.update_button_states)
buttons = QDialogButtonBox(
QDialogButtonBox.StandardButton.Ok | QDialogButtonBox.StandardButton.Cancel,
self,
)
buttons.accepted.connect(self.accept)
buttons.rejected.connect(self.reject)
layout.addWidget(buttons)
self.setLayout(layout)
self.setWindowTitle(title)
self.resize(*size)
self.update_button_states()
def process_queue(self):
"""Process the queue items."""
import os
self.listwidget.clear()
if not self.queue:
self.empty_overlay.show()
self.update_button_states()
return
else:
self.empty_overlay.hide()
# Get current global settings and checkbox state for overrides
current_global_settings = self.get_current_attributes()
is_override_active = self.override_chk.isChecked()
icon_provider = QFileIconProvider()
for item in self.queue:
# Dynamic Attribute Retrieval Helper
def get_val(attr, default=""):
# If override is ON and attr is overrideable, use global setting
if is_override_active and attr in OVERRIDE_FIELDS:
return current_global_settings.get(attr, default)
# Otherwise return the item's saved attribute
return getattr(item, attr, default)
# Determine display file path (prefer save_base_path for original file)
display_file_path = getattr(item, "save_base_path", None) or item.file_name
processing_file_path = item.file_name
# Normalize paths for consistent display (fixes Windows path separator issues)
display_file_path = (
os.path.normpath(display_file_path)
if display_file_path
else display_file_path
)
processing_file_path = (
os.path.normpath(processing_file_path)
if processing_file_path
else processing_file_path
)
# Only show the file name, not the full path
display_name = display_file_path
if os.path.sep in display_file_path:
display_name = os.path.basename(display_file_path)
# Get icon for the display file
icon = icon_provider.icon(QFileInfo(display_file_path))
list_item = QListWidgetItem()
# Tooltip Generation
tooltip = ""
# If override is active, add the warning header on its own line
if is_override_active:
tooltip += "<b style='color: #ff9900;'>(Global Override Active)</b><br>"
output_folder = get_val("output_folder")
# For plain .txt inputs we don't need to show a separate processing file
show_processing = True
try:
if isinstance(
display_file_path, str
) and display_file_path.lower().endswith(".txt"):
show_processing = False
except Exception:
show_processing = True
tooltip += f"<b>Input File:</b> {display_file_path}<br>"
if (
show_processing
and processing_file_path
and processing_file_path != display_file_path
):
tooltip += f"<b>Processing File:</b> {processing_file_path}<br>"
tooltip += (
f"<b>Language:</b> {get_val('lang_code')}<br>"
f"<b>Speed:</b> {get_val('speed')}<br>"
f"<b>Voice:</b> {get_val('voice')}<br>"
f"<b>Save Option:</b> {get_val('save_option')}<br>"
)
if output_folder not in (None, "", "None"):
tooltip += f"<b>Output Folder:</b> {output_folder}<br>"
tooltip += (
f"<b>Subtitle Mode:</b> {get_val('subtitle_mode')}<br>"
f"<b>Output Format:</b> {get_val('output_format')}<br>"
f"<b>Characters:</b> {getattr(item, 'total_char_count', '')}<br>"
f"<b>Replace Single Newlines:</b> {get_val('replace_single_newlines', True)}<br>"
f"<b>Use Silent Gaps:</b> {get_val('use_silent_gaps', False)}<br>"
f"<b>Speed Method:</b> {get_val('subtitle_speed_method', 'tts')}"
)
# Add book handler options if present (Preserve logic: specific to file structure)
save_chapters_separately = getattr(item, "save_chapters_separately", None)
merge_chapters_at_end = getattr(item, "merge_chapters_at_end", None)
if save_chapters_separately is not None:
tooltip += f"<br><b>Save chapters separately:</b> {'Yes' if save_chapters_separately else 'No'}"
# Only show merge option if saving chapters separately
if save_chapters_separately and merge_chapters_at_end is not None:
tooltip += f"<br><b>Merge chapters at the end:</b> {'Yes' if merge_chapters_at_end else 'No'}"
list_item.setToolTip(tooltip)
list_item.setIcon(icon)
# Store both paths for context menu
list_item.setData(
Qt.ItemDataRole.UserRole,
{
"display_path": display_file_path,
"processing_path": processing_file_path,
},
)
# Use custom widget for display
char_count = getattr(item, "total_char_count", 0)
widget = QueueListItemWidget(display_file_path, char_count)
self.listwidget.addItem(list_item)
self.listwidget.setItemWidget(list_item, widget)
self.update_button_states()
def remove_item(self):
items = self.listwidget.selectedItems()
if not items:
return
from PyQt6.QtWidgets import QMessageBox
# Remove by index to ensure correct mapping
rows = sorted([self.listwidget.row(item) for item in items], reverse=True)
# Warn user if removing multiple files
if len(rows) > 1:
reply = QMessageBox.question(
self,
"Confirm Remove",
f"Are you sure you want to remove {len(rows)} selected items from the queue?",
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
QMessageBox.StandardButton.No,
)
if reply != QMessageBox.StandardButton.Yes:
return
for row in rows:
if 0 <= row < len(self.queue):
del self.queue[row]
self.process_queue()
self.update_button_states()
def clear_queue(self):
from PyQt6.QtWidgets import QMessageBox
if len(self.queue) > 1:
reply = QMessageBox.question(
self,
"Confirm Clear Queue",
f"Are you sure you want to clear {len(self.queue)} items from the queue?",
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
QMessageBox.StandardButton.No,
)
if reply != QMessageBox.StandardButton.Yes:
return
self.queue.clear()
self.listwidget.clear()
self.empty_overlay.resize(
self.listwidget.size()
) # Ensure overlay is sized correctly
self.empty_overlay.show() # Show the overlay when queue is empty
self.update_button_states()
def get_queue(self):
return self.queue
def get_current_attributes(self):
# Fetch current attribute values from the parent abogen GUI
attrs = {}
parent = self.parent
if parent is not None:
# lang_code: use parent's get_voice_formula and get_selected_lang
if hasattr(parent, "get_voice_formula") and hasattr(
parent, "get_selected_lang"
):
voice_formula = parent.get_voice_formula()
attrs["lang_code"] = parent.get_selected_lang(voice_formula)
attrs["voice"] = voice_formula
else:
attrs["lang_code"] = getattr(parent, "selected_lang", "")
attrs["voice"] = getattr(parent, "selected_voice", "")
# speed
if hasattr(parent, "speed_slider"):
attrs["speed"] = parent.speed_slider.value() / 100.0
else:
attrs["speed"] = getattr(parent, "speed", 1.0)
# save_option
attrs["save_option"] = getattr(parent, "save_option", "")
# output_folder
attrs["output_folder"] = getattr(parent, "selected_output_folder", "")
# subtitle_mode
if hasattr(parent, "get_actual_subtitle_mode"):
attrs["subtitle_mode"] = parent.get_actual_subtitle_mode()
else:
attrs["subtitle_mode"] = getattr(parent, "subtitle_mode", "")
# output_format
attrs["output_format"] = getattr(parent, "selected_format", "")
# total_char_count
attrs["total_char_count"] = getattr(parent, "char_count", "")
# replace_single_newlines
attrs["replace_single_newlines"] = getattr(
parent, "replace_single_newlines", True
)
# use_silent_gaps
attrs["use_silent_gaps"] = getattr(parent, "use_silent_gaps", False)
# subtitle_speed_method
attrs["subtitle_speed_method"] = getattr(
parent, "subtitle_speed_method", "tts"
)
# word substitutions
attrs["word_substitutions_enabled"] = getattr(
parent, "word_substitutions_enabled", False
)
attrs["word_substitutions_list"] = getattr(
parent, "word_substitutions_list", ""
)
attrs["case_sensitive_substitutions"] = getattr(
parent, "case_sensitive_substitutions", False
)
attrs["replace_all_caps"] = getattr(parent, "replace_all_caps", False)
attrs["replace_numerals"] = getattr(parent, "replace_numerals", False)
attrs["fix_nonstandard_punctuation"] = getattr(
parent, "fix_nonstandard_punctuation", False
)
# book handler options
attrs["save_chapters_separately"] = getattr(
parent, "save_chapters_separately", None
)
attrs["merge_chapters_at_end"] = getattr(
parent, "merge_chapters_at_end", None
)
else:
# fallback: empty values
attrs = {
k: ""
for k in [
"lang_code",
"speed",
"voice",
"save_option",
"output_folder",
"subtitle_mode",
"output_format",
"total_char_count",
"replace_single_newlines",
]
}
attrs["save_chapters_separately"] = None
attrs["merge_chapters_at_end"] = None
return attrs
def add_files_from_paths(self, file_paths):
from abogen.subtitle_utils import calculate_text_length
from PyQt6.QtWidgets import QMessageBox
import os
current_attrs = self.get_current_attributes()
duplicates = []
for file_path in file_paths:
class QueueItem:
pass
item = QueueItem()
item.file_name = file_path
item.save_base_path = (
file_path # For .txt files, processing and save paths are the same
)
for attr, value in current_attrs.items():
setattr(item, attr, value)
# Override subtitle_mode to "Disabled" for subtitle files
if file_path.lower().endswith((".srt", ".ass", ".vtt")):
item.subtitle_mode = "Disabled"
# Read file content and calculate total_char_count using calculate_text_length
try:
with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
file_content = f.read()
item.total_char_count = calculate_text_length(file_content)
except Exception:
item.total_char_count = 0
# Prevent adding duplicate items to the queue (check all attributes)
is_duplicate = False
for queued_item in self.queue:
if (
getattr(queued_item, "file_name", None)
== getattr(item, "file_name", None)
and getattr(queued_item, "lang_code", None)
== getattr(item, "lang_code", None)
and getattr(queued_item, "speed", None)
== getattr(item, "speed", None)
and getattr(queued_item, "voice", None)
== getattr(item, "voice", None)
and getattr(queued_item, "save_option", None)
== getattr(item, "save_option", None)
and getattr(queued_item, "output_folder", None)
== getattr(item, "output_folder", None)
and getattr(queued_item, "subtitle_mode", None)
== getattr(item, "subtitle_mode", None)
and getattr(queued_item, "output_format", None)
== getattr(item, "output_format", None)
and getattr(queued_item, "total_char_count", None)
== getattr(item, "total_char_count", None)
and getattr(queued_item, "replace_single_newlines", True)
== getattr(item, "replace_single_newlines", True)
and getattr(queued_item, "use_silent_gaps", False)
== getattr(item, "use_silent_gaps", False)
and getattr(queued_item, "subtitle_speed_method", "tts")
== getattr(item, "subtitle_speed_method", "tts")
and getattr(queued_item, "save_base_path", None)
== getattr(item, "save_base_path", None)
and getattr(queued_item, "save_chapters_separately", None)
== getattr(item, "save_chapters_separately", None)
and getattr(queued_item, "merge_chapters_at_end", None)
== getattr(item, "merge_chapters_at_end", None)
):
is_duplicate = True
break
if is_duplicate:
duplicates.append(os.path.basename(file_path))
continue
self.queue.append(item)
if duplicates:
QMessageBox.warning(
self,
"Duplicate Item(s)",
f"Skipping {len(duplicates)} file(s) with the same attributes, already in the queue.",
)
self.process_queue()
self.update_button_states()
def add_more_files(self):
from PyQt6.QtWidgets import QFileDialog
# Allow .txt, .srt, .ass, and .vtt files
files, _ = QFileDialog.getOpenFileNames(
self,
"Select text or subtitle files",
"",
"Supported Files (*.txt *.srt *.ass *.vtt)",
)
if not files:
return
self.add_files_from_paths(files)
def resizeEvent(self, event):
super().resizeEvent(event)
if hasattr(self, "empty_overlay"):
self.empty_overlay.resize(self.listwidget.size())
def update_button_states(self):
# Enable Remove if at least one item is selected, else disable
if hasattr(self, "remove_button"):
selected_count = len(self.listwidget.selectedItems())
self.remove_button.setEnabled(selected_count > 0)
if selected_count > 1:
self.remove_button.setText(f"Remove selected ({selected_count})")
else:
self.remove_button.setText("Remove selected")
# Disable Clear if queue is empty
if hasattr(self, "clear_button"):
self.clear_button.setEnabled(bool(self.queue))
def show_context_menu(self, pos):
from PyQt6.QtWidgets import QMenu
from PyQt6.QtGui import QAction, QDesktopServices
from PyQt6.QtCore import QUrl
import os
global_pos = self.listwidget.viewport().mapToGlobal(pos)
selected_items = self.listwidget.selectedItems()
menu = QMenu(self)
if len(selected_items) == 1:
# Add Remove action
remove_action = QAction("Remove this item", self)
remove_action.triggered.connect(self.remove_item)
menu.addAction(remove_action)
# Get paths for determining if it's a document input
item = selected_items[0]
paths = item.data(Qt.ItemDataRole.UserRole)
if isinstance(paths, dict):
display_path = paths.get("display_path", "")
processing_path = paths.get("processing_path", "")
else:
display_path = paths
processing_path = paths
doc_exts = (".md", ".markdown", ".pdf", ".epub")
is_document_input = (
isinstance(display_path, str)
and display_path.lower().endswith(doc_exts)
) or (
isinstance(processing_path, str)
and processing_path.lower().endswith(doc_exts)
)
# Add Open file action(s)
def open_file_by_path(path_label: str):
from PyQt6.QtWidgets import QMessageBox
p = display_path if path_label == "display" else processing_path
if not p:
QMessageBox.warning(
self, "File Not Found", "Path is not available."
)
return
# Find the queue item and resolve the target path
target_path = None
for q in self.queue:
if (
getattr(q, "save_base_path", None) == display_path
or q.file_name == display_path
):
if path_label == "display":
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
else:
target_path = q.file_name
break
if (
getattr(q, "save_base_path", None) == processing_path
or q.file_name == processing_path
):
if path_label == "display":
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
else:
target_path = q.file_name
break
# Fallback to the raw path if resolution failed
if not target_path:
target_path = p
if not os.path.exists(target_path):
QMessageBox.warning(
self, "File Not Found", f"The file does not exist."
)
return
QDesktopServices.openUrl(QUrl.fromLocalFile(target_path))
if is_document_input:
# For documents, show two open options
open_processed_action = QAction("Open processed file", self)
open_processed_action.triggered.connect(
lambda: open_file_by_path("processing")
)
menu.addAction(open_processed_action)
open_input_action = QAction("Open input file", self)
open_input_action.triggered.connect(
lambda: open_file_by_path("display")
)
menu.addAction(open_input_action)
else:
# For plain text files, show single open option
open_file_action = QAction("Open file", self)
open_file_action.triggered.connect(lambda: open_file_by_path("display"))
menu.addAction(open_file_action)
# Add Go to folder action
# If the queued item represents a converted document (markdown, pdf, epub)
# show two actions: Go to processed file (the cached .txt) and Go to input file (original source)
from PyQt6.QtWidgets import QMessageBox
def open_folder_for(path_label: str):
# path_label should be either 'display' or 'processing'
p = display_path if path_label == "display" else processing_path
if not p:
QMessageBox.warning(
self, "File Not Found", "Path is not available."
)
return
# If the stored path is the display path (original) but the actual file may be
# stored on the queue object differently, try to resolve via the queue entry.
target_path = None
for q in self.queue:
if (
getattr(q, "save_base_path", None) == display_path
or q.file_name == display_path
):
if path_label == "display":
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
else:
target_path = q.file_name
break
if (
getattr(q, "save_base_path", None) == processing_path
or q.file_name == processing_path
):
if path_label == "display":
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
else:
target_path = q.file_name
break
# Fallback to the raw path if resolution failed
if not target_path:
target_path = p
if not os.path.exists(target_path):
QMessageBox.warning(
self,
"File Not Found",
f"The file does not exist: {target_path}",
)
return
folder = os.path.dirname(target_path)
if os.path.exists(folder):
QDesktopServices.openUrl(QUrl.fromLocalFile(folder))
if is_document_input:
processed_action = QAction("Go to processed file", self)
processed_action.triggered.connect(
lambda: open_folder_for("processing")
)
menu.addAction(processed_action)
input_action = QAction("Go to input file", self)
input_action.triggered.connect(lambda: open_folder_for("display"))
menu.addAction(input_action)
else:
# Default behavior for non-document inputs: single "Go to folder" action
go_to_folder_action = QAction("Go to folder", self)
def go_to_folder():
item = selected_items[0]
paths = item.data(Qt.ItemDataRole.UserRole)
if isinstance(paths, dict):
file_path = paths.get(
"display_path", paths.get("processing_path", "")
)
else:
file_path = paths # Fallback for old format
# Find the queue item
for q in self.queue:
if (
getattr(q, "save_base_path", None) == file_path
or q.file_name == file_path
):
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
if not os.path.exists(target_path):
QMessageBox.warning(
self, "File Not Found", f"The file does not exist."
)
return
folder = os.path.dirname(target_path)
if os.path.exists(folder):
QDesktopServices.openUrl(QUrl.fromLocalFile(folder))
break
go_to_folder_action.triggered.connect(go_to_folder)
menu.addAction(go_to_folder_action)
elif len(selected_items) > 1:
remove_action = QAction(f"Remove selected ({len(selected_items)})", self)
remove_action.triggered.connect(self.remove_item)
menu.addAction(remove_action)
# Always add Clear Queue
clear_action = QAction("Clear Queue", self)
clear_action.triggered.connect(self.clear_queue)
menu.addAction(clear_action)
menu.exec(global_pos)
def accept(self):
# Save the override state to config so it persists globally
self.config["queue_override_settings"] = self.override_chk.isChecked()
save_config(self.config)
super().accept()
def reject(self):
# Cancel: restore original queue
from PyQt6.QtWidgets import QMessageBox
# Warn if user changed a lot (e.g., more than 1 items difference)
original_count = len(self._original_queue)
current_count = len(self.queue)
if abs(original_count - current_count) > 1:
reply = QMessageBox.question(
self,
"Confirm Cancel",
f"Are you sure you want to cancel and discard all changes?",
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
QMessageBox.StandardButton.No,
)
if reply != QMessageBox.StandardButton.Yes:
return
self.queue.clear()
self.queue.extend(deepcopy(self._original_queue))
super().reject()
def keyPressEvent(self, event):
from PyQt6.QtCore import Qt
if event.key() == Qt.Key.Key_Delete:
self.remove_item()
else:
super().keyPressEvent(event)
+32
View File
@@ -0,0 +1,32 @@
# represents a queued item - book, chapters, voice, etc.
from dataclasses import dataclass
@dataclass
class QueuedItem:
file_name: str
lang_code: str
speed: float
voice: str
save_option: str
output_folder: str
subtitle_mode: str
output_format: str
total_char_count: int
replace_single_newlines: bool = True
use_silent_gaps: bool = False
subtitle_speed_method: str = "tts"
save_base_path: str = None
save_chapters_separately: bool = None
merge_chapters_at_end: bool = None
# Word Substitution fields
word_substitutions_enabled: bool = False
word_substitutions_list: str = ""
case_sensitive_substitutions: bool = False
replace_all_caps: bool = False
replace_numerals: bool = False
fix_nonstandard_punctuation: bool = False
# TTS Provider fields
tts_provider: str = "kokoro"
supertonic_language: str = "en"
supertonic_total_steps: int = 8
File diff suppressed because it is too large Load Diff
+7 -856
View File
@@ -1,860 +1,11 @@
# a simple window with a list of items in the queue, no checkboxes
# button to remove an item from the queue
# button to clear the queue
"""Backwards-compatible re-export of the PyQt queue manager.
from PyQt6.QtWidgets import (
QDialog,
QVBoxLayout,
QHBoxLayout,
QDialogButtonBox,
QPushButton,
QListWidget,
QListWidgetItem,
QFileIconProvider,
QLabel,
QWidget,
QSizePolicy,
QAbstractItemView,
QCheckBox,
)
from PyQt6.QtCore import QFileInfo, Qt
from abogen.constants import COLORS
from copy import deepcopy
from PyQt6.QtGui import QFontMetrics
from abogen.utils import load_config, save_config
The actual implementation lives in abogen.pyqt.queue_manager_gui.
"""
# Define attributes that are safe to override with global settings
OVERRIDE_FIELDS = [
"lang_code",
"speed",
"voice",
"save_option",
"output_folder",
"subtitle_mode",
"output_format",
"replace_single_newlines",
"use_silent_gaps",
"subtitle_speed_method",
]
from __future__ import annotations
from abogen.pyqt.queue_manager_gui import * # noqa: F401, F403
from abogen.pyqt.queue_manager_gui import QueueManager
class ElidedLabel(QLabel):
def __init__(self, text):
super().__init__(text)
self._full_text = text
self.setSizePolicy(QSizePolicy.Policy.Expanding, QSizePolicy.Policy.Preferred)
self.setTextFormat(Qt.TextFormat.PlainText)
def setText(self, text):
self._full_text = text
super().setText(text)
self.update()
def resizeEvent(self, event):
metrics = QFontMetrics(self.font())
elided = metrics.elidedText(
self._full_text, Qt.TextElideMode.ElideRight, self.width()
)
super().setText(elided)
super().resizeEvent(event)
def fullText(self):
return self._full_text
class QueueListItemWidget(QWidget):
def __init__(self, file_name, char_count):
super().__init__()
layout = QHBoxLayout()
layout.setContentsMargins(12, 0, 6, 0)
layout.setSpacing(0)
import os
name_label = ElidedLabel(os.path.basename(file_name))
char_label = QLabel(f"Chars: {char_count}")
char_label.setStyleSheet(f"color: {COLORS['LIGHT_DISABLED']};")
char_label.setAlignment(
Qt.AlignmentFlag.AlignRight | Qt.AlignmentFlag.AlignVCenter
)
char_label.setSizePolicy(
QSizePolicy.Policy.Minimum, QSizePolicy.Policy.Preferred
)
layout.addWidget(name_label, 1)
layout.addWidget(char_label, 0)
self.setLayout(layout)
class DroppableQueueListWidget(QListWidget):
def __init__(self, parent_dialog):
super().__init__()
self.parent_dialog = parent_dialog
self.setAcceptDrops(True)
# Overlay for drag hover
self.drag_overlay = QLabel("", self)
self.drag_overlay.setAlignment(Qt.AlignmentFlag.AlignCenter)
self.drag_overlay.setStyleSheet(
f"border:2px dashed {COLORS['BLUE_BORDER_HOVER']}; border-radius:5px; padding:20px; background:{COLORS['BLUE_BG_HOVER']};"
)
self.drag_overlay.setVisible(False)
self.drag_overlay.setAttribute(
Qt.WidgetAttribute.WA_TransparentForMouseEvents, True
)
def dragEnterEvent(self, event):
if event.mimeData().hasUrls():
for url in event.mimeData().urls():
file_path = url.toLocalFile().lower()
if url.isLocalFile() and (
file_path.endswith(".txt")
or file_path.endswith((".srt", ".ass", ".vtt"))
):
self.drag_overlay.resize(self.size())
self.drag_overlay.setVisible(True)
event.acceptProposedAction()
return
self.drag_overlay.setVisible(False)
event.ignore()
def dragMoveEvent(self, event):
if event.mimeData().hasUrls():
for url in event.mimeData().urls():
file_path = url.toLocalFile().lower()
if url.isLocalFile() and (
file_path.endswith(".txt")
or file_path.endswith((".srt", ".ass", ".vtt"))
):
event.acceptProposedAction()
return
event.ignore()
def dragLeaveEvent(self, event):
self.drag_overlay.setVisible(False)
event.accept()
def dropEvent(self, event):
self.drag_overlay.setVisible(False)
if event.mimeData().hasUrls():
file_paths = [
url.toLocalFile()
for url in event.mimeData().urls()
if url.isLocalFile()
and (
url.toLocalFile().lower().endswith(".txt")
or url.toLocalFile().lower().endswith((".srt", ".ass", ".vtt"))
)
]
if file_paths:
self.parent_dialog.add_files_from_paths(file_paths)
event.acceptProposedAction()
else:
event.ignore()
else:
event.ignore()
def resizeEvent(self, event):
super().resizeEvent(event)
if hasattr(self, "drag_overlay"):
self.drag_overlay.resize(self.size())
class QueueManager(QDialog):
def __init__(self, parent, queue: list, title="Queue Manager", size=(600, 700)):
super().__init__()
self.queue = queue
self._original_queue = deepcopy(
queue
) # Store a deep copy of the original queue
self.parent = parent
self.config = load_config() # Load config for persistence
layout = QVBoxLayout()
layout.setContentsMargins(15, 15, 15, 15) # set main layout margins
layout.setSpacing(12) # set spacing between widgets in main layout
# list of queued items
self.listwidget = DroppableQueueListWidget(self)
self.listwidget.setSelectionMode(
QAbstractItemView.SelectionMode.ExtendedSelection
)
self.listwidget.setAlternatingRowColors(True)
self.listwidget.setContextMenuPolicy(Qt.ContextMenuPolicy.CustomContextMenu)
self.listwidget.customContextMenuRequested.connect(self.show_context_menu)
# Add informative instructions at the top
instructions = QLabel(
"<h2>How Queue Works?</h2>"
"You can add text and subtitle files (.txt, .srt, .ass, .vtt) directly using the '<b>Add files</b>' button below. "
"To add PDF, EPUB or markdown files, use the input box in the main window and click the <b>'Add to Queue'</b> button. "
"By default, each file in the queue keeps the configuration settings active when they were added. "
"Enabling the <b>'Override item settings with current selection'</b> option below will force all items to use the configuration currently selected in the main window. "
"You can view each file's configuration by hovering over them."
)
instructions.setAlignment(Qt.AlignmentFlag.AlignLeft)
instructions.setWordWrap(True)
layout.addWidget(instructions)
# Override Checkbox
self.override_chk = QCheckBox("Override item settings with current selection")
self.override_chk.setToolTip(
"If checked, all items in the queue will be processed using the \n"
"settings currently selected in the main window, ignoring their saved state."
)
# Load saved state (default to False)
self.override_chk.setChecked(self.config.get("queue_override_settings", False))
# Trigger process_queue to update tooltips immediately when toggled
self.override_chk.stateChanged.connect(self.process_queue)
self.override_chk.setStyleSheet("margin-bottom: 8px;")
layout.addWidget(self.override_chk)
# Overlay label for empty queue
self.empty_overlay = QLabel(
"Drag and drop your text or subtitle files here or use the 'Add files' button.",
self.listwidget,
)
self.empty_overlay.setAlignment(Qt.AlignmentFlag.AlignCenter)
self.empty_overlay.setStyleSheet(
f"color: {COLORS['LIGHT_DISABLED']}; background: transparent; padding: 20px;"
)
self.empty_overlay.setWordWrap(True)
self.empty_overlay.setAttribute(
Qt.WidgetAttribute.WA_TransparentForMouseEvents, True
)
self.empty_overlay.hide()
# add queue items to the list
self.process_queue()
button_row = QHBoxLayout()
button_row.setContentsMargins(0, 0, 0, 0) # optional: no margins for button row
button_row.setSpacing(7) # set spacing between buttons
# Add files button
add_files_button = QPushButton("Add files")
add_files_button.setFixedHeight(40)
add_files_button.clicked.connect(self.add_more_files)
button_row.addWidget(add_files_button)
# Remove button
self.remove_button = QPushButton("Remove selected")
self.remove_button.setFixedHeight(40)
self.remove_button.clicked.connect(self.remove_item)
button_row.addWidget(self.remove_button)
# Clear button
self.clear_button = QPushButton("Clear Queue")
self.clear_button.setFixedHeight(40)
self.clear_button.clicked.connect(self.clear_queue)
button_row.addWidget(self.clear_button)
layout.addLayout(button_row)
layout.addWidget(self.listwidget)
# Connect selection change to update button state
self.listwidget.currentItemChanged.connect(self.update_button_states)
self.listwidget.itemSelectionChanged.connect(self.update_button_states)
buttons = QDialogButtonBox(
QDialogButtonBox.StandardButton.Ok | QDialogButtonBox.StandardButton.Cancel,
self,
)
buttons.accepted.connect(self.accept)
buttons.rejected.connect(self.reject)
layout.addWidget(buttons)
self.setLayout(layout)
self.setWindowTitle(title)
self.resize(*size)
self.update_button_states()
def process_queue(self):
"""Process the queue items."""
import os
self.listwidget.clear()
if not self.queue:
self.empty_overlay.show()
self.update_button_states()
return
else:
self.empty_overlay.hide()
# Get current global settings and checkbox state for overrides
current_global_settings = self.get_current_attributes()
is_override_active = self.override_chk.isChecked()
icon_provider = QFileIconProvider()
for item in self.queue:
# Dynamic Attribute Retrieval Helper
def get_val(attr, default=""):
# If override is ON and attr is overrideable, use global setting
if is_override_active and attr in OVERRIDE_FIELDS:
return current_global_settings.get(attr, default)
# Otherwise return the item's saved attribute
return getattr(item, attr, default)
# Determine display file path (prefer save_base_path for original file)
display_file_path = getattr(item, "save_base_path", None) or item.file_name
processing_file_path = item.file_name
# Normalize paths for consistent display (fixes Windows path separator issues)
display_file_path = (
os.path.normpath(display_file_path)
if display_file_path
else display_file_path
)
processing_file_path = (
os.path.normpath(processing_file_path)
if processing_file_path
else processing_file_path
)
# Only show the file name, not the full path
display_name = display_file_path
if os.path.sep in display_file_path:
display_name = os.path.basename(display_file_path)
# Get icon for the display file
icon = icon_provider.icon(QFileInfo(display_file_path))
list_item = QListWidgetItem()
# Tooltip Generation
tooltip = ""
# If override is active, add the warning header on its own line
if is_override_active:
tooltip += "<b style='color: #ff9900;'>(Global Override Active)</b><br>"
output_folder = get_val("output_folder")
# For plain .txt inputs we don't need to show a separate processing file
show_processing = True
try:
if isinstance(
display_file_path, str
) and display_file_path.lower().endswith(".txt"):
show_processing = False
except Exception:
show_processing = True
tooltip += f"<b>Input File:</b> {display_file_path}<br>"
if (
show_processing
and processing_file_path
and processing_file_path != display_file_path
):
tooltip += f"<b>Processing File:</b> {processing_file_path}<br>"
tooltip += (
f"<b>Language:</b> {get_val('lang_code')}<br>"
f"<b>Speed:</b> {get_val('speed')}<br>"
f"<b>Voice:</b> {get_val('voice')}<br>"
f"<b>Save Option:</b> {get_val('save_option')}<br>"
)
if output_folder not in (None, "", "None"):
tooltip += f"<b>Output Folder:</b> {output_folder}<br>"
tooltip += (
f"<b>Subtitle Mode:</b> {get_val('subtitle_mode')}<br>"
f"<b>Output Format:</b> {get_val('output_format')}<br>"
f"<b>Characters:</b> {getattr(item, 'total_char_count', '')}<br>"
f"<b>Replace Single Newlines:</b> {get_val('replace_single_newlines', True)}<br>"
f"<b>Use Silent Gaps:</b> {get_val('use_silent_gaps', False)}<br>"
f"<b>Speed Method:</b> {get_val('subtitle_speed_method', 'tts')}"
)
# Add book handler options if present (Preserve logic: specific to file structure)
save_chapters_separately = getattr(item, "save_chapters_separately", None)
merge_chapters_at_end = getattr(item, "merge_chapters_at_end", None)
if save_chapters_separately is not None:
tooltip += f"<br><b>Save chapters separately:</b> {'Yes' if save_chapters_separately else 'No'}"
# Only show merge option if saving chapters separately
if save_chapters_separately and merge_chapters_at_end is not None:
tooltip += f"<br><b>Merge chapters at the end:</b> {'Yes' if merge_chapters_at_end else 'No'}"
list_item.setToolTip(tooltip)
list_item.setIcon(icon)
# Store both paths for context menu
list_item.setData(
Qt.ItemDataRole.UserRole,
{
"display_path": display_file_path,
"processing_path": processing_file_path,
},
)
# Use custom widget for display
char_count = getattr(item, "total_char_count", 0)
widget = QueueListItemWidget(display_file_path, char_count)
self.listwidget.addItem(list_item)
self.listwidget.setItemWidget(list_item, widget)
self.update_button_states()
def remove_item(self):
items = self.listwidget.selectedItems()
if not items:
return
from PyQt6.QtWidgets import QMessageBox
# Remove by index to ensure correct mapping
rows = sorted([self.listwidget.row(item) for item in items], reverse=True)
# Warn user if removing multiple files
if len(rows) > 1:
reply = QMessageBox.question(
self,
"Confirm Remove",
f"Are you sure you want to remove {len(rows)} selected items from the queue?",
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
QMessageBox.StandardButton.No,
)
if reply != QMessageBox.StandardButton.Yes:
return
for row in rows:
if 0 <= row < len(self.queue):
del self.queue[row]
self.process_queue()
self.update_button_states()
def clear_queue(self):
from PyQt6.QtWidgets import QMessageBox
if len(self.queue) > 1:
reply = QMessageBox.question(
self,
"Confirm Clear Queue",
f"Are you sure you want to clear {len(self.queue)} items from the queue?",
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
QMessageBox.StandardButton.No,
)
if reply != QMessageBox.StandardButton.Yes:
return
self.queue.clear()
self.listwidget.clear()
self.empty_overlay.resize(
self.listwidget.size()
) # Ensure overlay is sized correctly
self.empty_overlay.show() # Show the overlay when queue is empty
self.update_button_states()
def get_queue(self):
return self.queue
def get_current_attributes(self):
# Fetch current attribute values from the parent abogen GUI
attrs = {}
parent = self.parent
if parent is not None:
# lang_code: use parent's get_voice_formula and get_selected_lang
if hasattr(parent, "get_voice_formula") and hasattr(
parent, "get_selected_lang"
):
voice_formula = parent.get_voice_formula()
attrs["lang_code"] = parent.get_selected_lang(voice_formula)
attrs["voice"] = voice_formula
else:
attrs["lang_code"] = getattr(parent, "selected_lang", "")
attrs["voice"] = getattr(parent, "selected_voice", "")
# speed
if hasattr(parent, "speed_slider"):
attrs["speed"] = parent.speed_slider.value() / 100.0
else:
attrs["speed"] = getattr(parent, "speed", 1.0)
# save_option
attrs["save_option"] = getattr(parent, "save_option", "")
# output_folder
attrs["output_folder"] = getattr(parent, "selected_output_folder", "")
# subtitle_mode
if hasattr(parent, "get_actual_subtitle_mode"):
attrs["subtitle_mode"] = parent.get_actual_subtitle_mode()
else:
attrs["subtitle_mode"] = getattr(parent, "subtitle_mode", "")
# output_format
attrs["output_format"] = getattr(parent, "selected_format", "")
# total_char_count
attrs["total_char_count"] = getattr(parent, "char_count", "")
# replace_single_newlines
attrs["replace_single_newlines"] = getattr(
parent, "replace_single_newlines", True
)
# use_silent_gaps
attrs["use_silent_gaps"] = getattr(parent, "use_silent_gaps", False)
# subtitle_speed_method
attrs["subtitle_speed_method"] = getattr(
parent, "subtitle_speed_method", "tts"
)
# book handler options
attrs["save_chapters_separately"] = getattr(
parent, "save_chapters_separately", None
)
attrs["merge_chapters_at_end"] = getattr(
parent, "merge_chapters_at_end", None
)
else:
# fallback: empty values
attrs = {
k: ""
for k in [
"lang_code",
"speed",
"voice",
"save_option",
"output_folder",
"subtitle_mode",
"output_format",
"total_char_count",
"replace_single_newlines",
]
}
attrs["save_chapters_separately"] = None
attrs["merge_chapters_at_end"] = None
return attrs
def add_files_from_paths(self, file_paths):
from abogen.utils import calculate_text_length
from PyQt6.QtWidgets import QMessageBox
import os
current_attrs = self.get_current_attributes()
duplicates = []
for file_path in file_paths:
class QueueItem:
pass
item = QueueItem()
item.file_name = file_path
item.save_base_path = (
file_path # For .txt files, processing and save paths are the same
)
for attr, value in current_attrs.items():
setattr(item, attr, value)
# Override subtitle_mode to "Disabled" for subtitle files
if file_path.lower().endswith((".srt", ".ass", ".vtt")):
item.subtitle_mode = "Disabled"
# Read file content and calculate total_char_count using calculate_text_length
try:
with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
file_content = f.read()
item.total_char_count = calculate_text_length(file_content)
except Exception:
item.total_char_count = 0
# Prevent adding duplicate items to the queue (check all attributes)
is_duplicate = False
for queued_item in self.queue:
if (
getattr(queued_item, "file_name", None)
== getattr(item, "file_name", None)
and getattr(queued_item, "lang_code", None)
== getattr(item, "lang_code", None)
and getattr(queued_item, "speed", None)
== getattr(item, "speed", None)
and getattr(queued_item, "voice", None)
== getattr(item, "voice", None)
and getattr(queued_item, "save_option", None)
== getattr(item, "save_option", None)
and getattr(queued_item, "output_folder", None)
== getattr(item, "output_folder", None)
and getattr(queued_item, "subtitle_mode", None)
== getattr(item, "subtitle_mode", None)
and getattr(queued_item, "output_format", None)
== getattr(item, "output_format", None)
and getattr(queued_item, "total_char_count", None)
== getattr(item, "total_char_count", None)
and getattr(queued_item, "replace_single_newlines", True)
== getattr(item, "replace_single_newlines", True)
and getattr(queued_item, "use_silent_gaps", False)
== getattr(item, "use_silent_gaps", False)
and getattr(queued_item, "subtitle_speed_method", "tts")
== getattr(item, "subtitle_speed_method", "tts")
and getattr(queued_item, "save_base_path", None)
== getattr(item, "save_base_path", None)
and getattr(queued_item, "save_chapters_separately", None)
== getattr(item, "save_chapters_separately", None)
and getattr(queued_item, "merge_chapters_at_end", None)
== getattr(item, "merge_chapters_at_end", None)
):
is_duplicate = True
break
if is_duplicate:
duplicates.append(os.path.basename(file_path))
continue
self.queue.append(item)
if duplicates:
QMessageBox.warning(
self,
"Duplicate Item(s)",
f"Skipping {len(duplicates)} file(s) with the same attributes, already in the queue.",
)
self.process_queue()
self.update_button_states()
def add_more_files(self):
from PyQt6.QtWidgets import QFileDialog
# Allow .txt, .srt, .ass, and .vtt files
files, _ = QFileDialog.getOpenFileNames(
self,
"Select text or subtitle files",
"",
"Supported Files (*.txt *.srt *.ass *.vtt)",
)
if not files:
return
self.add_files_from_paths(files)
def resizeEvent(self, event):
super().resizeEvent(event)
if hasattr(self, "empty_overlay"):
self.empty_overlay.resize(self.listwidget.size())
def update_button_states(self):
# Enable Remove if at least one item is selected, else disable
if hasattr(self, "remove_button"):
selected_count = len(self.listwidget.selectedItems())
self.remove_button.setEnabled(selected_count > 0)
if selected_count > 1:
self.remove_button.setText(f"Remove selected ({selected_count})")
else:
self.remove_button.setText("Remove selected")
# Disable Clear if queue is empty
if hasattr(self, "clear_button"):
self.clear_button.setEnabled(bool(self.queue))
def show_context_menu(self, pos):
from PyQt6.QtWidgets import QMenu
from PyQt6.QtGui import QAction, QDesktopServices
from PyQt6.QtCore import QUrl
import os
global_pos = self.listwidget.viewport().mapToGlobal(pos)
selected_items = self.listwidget.selectedItems()
menu = QMenu(self)
if len(selected_items) == 1:
# Add Remove action
remove_action = QAction("Remove this item", self)
remove_action.triggered.connect(self.remove_item)
menu.addAction(remove_action)
# Get paths for determining if it's a document input
item = selected_items[0]
paths = item.data(Qt.ItemDataRole.UserRole)
if isinstance(paths, dict):
display_path = paths.get("display_path", "")
processing_path = paths.get("processing_path", "")
else:
display_path = paths
processing_path = paths
doc_exts = (".md", ".markdown", ".pdf", ".epub")
is_document_input = (
isinstance(display_path, str)
and display_path.lower().endswith(doc_exts)
) or (
isinstance(processing_path, str)
and processing_path.lower().endswith(doc_exts)
)
# Add Open file action(s)
def open_file_by_path(path_label: str):
from PyQt6.QtWidgets import QMessageBox
p = display_path if path_label == "display" else processing_path
if not p:
QMessageBox.warning(
self, "File Not Found", "Path is not available."
)
return
# Find the queue item and resolve the target path
target_path = None
for q in self.queue:
if (
getattr(q, "save_base_path", None) == display_path
or q.file_name == display_path
):
if path_label == "display":
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
else:
target_path = q.file_name
break
if (
getattr(q, "save_base_path", None) == processing_path
or q.file_name == processing_path
):
if path_label == "display":
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
else:
target_path = q.file_name
break
# Fallback to the raw path if resolution failed
if not target_path:
target_path = p
if not os.path.exists(target_path):
QMessageBox.warning(
self, "File Not Found", f"The file does not exist."
)
return
QDesktopServices.openUrl(QUrl.fromLocalFile(target_path))
if is_document_input:
# For documents, show two open options
open_processed_action = QAction("Open processed file", self)
open_processed_action.triggered.connect(
lambda: open_file_by_path("processing")
)
menu.addAction(open_processed_action)
open_input_action = QAction("Open input file", self)
open_input_action.triggered.connect(
lambda: open_file_by_path("display")
)
menu.addAction(open_input_action)
else:
# For plain text files, show single open option
open_file_action = QAction("Open file", self)
open_file_action.triggered.connect(lambda: open_file_by_path("display"))
menu.addAction(open_file_action)
# Add Go to folder action
# If the queued item represents a converted document (markdown, pdf, epub)
# show two actions: Go to processed file (the cached .txt) and Go to input file (original source)
from PyQt6.QtWidgets import QMessageBox
def open_folder_for(path_label: str):
# path_label should be either 'display' or 'processing'
p = display_path if path_label == "display" else processing_path
if not p:
QMessageBox.warning(
self, "File Not Found", "Path is not available."
)
return
# If the stored path is the display path (original) but the actual file may be
# stored on the queue object differently, try to resolve via the queue entry.
target_path = None
for q in self.queue:
if (
getattr(q, "save_base_path", None) == display_path
or q.file_name == display_path
):
if path_label == "display":
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
else:
target_path = q.file_name
break
if (
getattr(q, "save_base_path", None) == processing_path
or q.file_name == processing_path
):
if path_label == "display":
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
else:
target_path = q.file_name
break
# Fallback to the raw path if resolution failed
if not target_path:
target_path = p
if not os.path.exists(target_path):
QMessageBox.warning(
self,
"File Not Found",
f"The file does not exist: {target_path}",
)
return
folder = os.path.dirname(target_path)
if os.path.exists(folder):
QDesktopServices.openUrl(QUrl.fromLocalFile(folder))
if is_document_input:
processed_action = QAction("Go to processed file", self)
processed_action.triggered.connect(
lambda: open_folder_for("processing")
)
menu.addAction(processed_action)
input_action = QAction("Go to input file", self)
input_action.triggered.connect(lambda: open_folder_for("display"))
menu.addAction(input_action)
else:
# Default behavior for non-document inputs: single "Go to folder" action
go_to_folder_action = QAction("Go to folder", self)
def go_to_folder():
item = selected_items[0]
paths = item.data(Qt.ItemDataRole.UserRole)
if isinstance(paths, dict):
file_path = paths.get(
"display_path", paths.get("processing_path", "")
)
else:
file_path = paths # Fallback for old format
# Find the queue item
for q in self.queue:
if (
getattr(q, "save_base_path", None) == file_path
or q.file_name == file_path
):
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
if not os.path.exists(target_path):
QMessageBox.warning(
self, "File Not Found", f"The file does not exist."
)
return
folder = os.path.dirname(target_path)
if os.path.exists(folder):
QDesktopServices.openUrl(QUrl.fromLocalFile(folder))
break
go_to_folder_action.triggered.connect(go_to_folder)
menu.addAction(go_to_folder_action)
elif len(selected_items) > 1:
remove_action = QAction(f"Remove selected ({len(selected_items)})", self)
remove_action.triggered.connect(self.remove_item)
menu.addAction(remove_action)
# Always add Clear Queue
clear_action = QAction("Clear Queue", self)
clear_action.triggered.connect(self.clear_queue)
menu.addAction(clear_action)
menu.exec(global_pos)
def accept(self):
# Save the override state to config so it persists globally
self.config["queue_override_settings"] = self.override_chk.isChecked()
save_config(self.config)
super().accept()
def reject(self):
# Cancel: restore original queue
from PyQt6.QtWidgets import QMessageBox
# Warn if user changed a lot (e.g., more than 1 items difference)
original_count = len(self._original_queue)
current_count = len(self.queue)
if abs(original_count - current_count) > 1:
reply = QMessageBox.question(
self,
"Confirm Cancel",
f"Are you sure you want to cancel and discard all changes?",
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
QMessageBox.StandardButton.No,
)
if reply != QMessageBox.StandardButton.Yes:
return
self.queue.clear()
self.queue.extend(deepcopy(self._original_queue))
super().reject()
def keyPressEvent(self, event):
from PyQt6.QtCore import Qt
if event.key() == Qt.Key.Key_Delete:
self.remove_item()
else:
super().keyPressEvent(event)
__all__ = ["QueueManager"]
+265
View File
@@ -0,0 +1,265 @@
from __future__ import annotations
import os
import logging
from dataclasses import dataclass
from functools import lru_cache
from typing import Any, Dict, Optional, Tuple
try: # pragma: no cover - optional dependency
import spacy
except Exception: # pragma: no cover - spaCy unavailable at runtime
spacy = None
# Lazy spaCy type hints to avoid a hard dependency at import time.
Language = Any # type: ignore[assignment]
Token = Any # type: ignore[assignment]
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class ContractionResolution:
start: int
end: int
surface: str
expansion: str
category: str
lemma: str
@property
def span(self) -> Tuple[int, int]:
return self.start, self.end
_DEFAULT_MODEL = os.environ.get("ABOGEN_SPACY_MODEL", "en_core_web_sm")
@lru_cache(maxsize=1)
def _load_spacy_model(model: str = _DEFAULT_MODEL) -> Optional[Language]:
if spacy is None:
logger.debug("spaCy is not installed; skipping contraction disambiguation")
return None
try:
nlp = spacy.load(model)
except Exception as exc: # pragma: no cover - depends on environment
logger.warning("Failed to load spaCy model '%s': %s", model, exc)
return None
return nlp
def resolve_ambiguous_contractions(
text: str, *, model: Optional[str] = None
) -> Dict[Tuple[int, int], ContractionResolution]:
"""Use spaCy to disambiguate ambiguous contractions in *text*.
Returns a mapping from (start, end) spans to their resolved expansion.
Only ambiguous `'s` and `'d` contractions are considered.
"""
if not text:
return {}
nlp = _load_spacy_model(model or _DEFAULT_MODEL)
if nlp is None:
return {}
doc = nlp(text)
resolutions: Dict[Tuple[int, int], ContractionResolution] = {}
for token in doc:
if token.text == "'s":
resolution = _resolve_apostrophe_s(token)
elif token.text == "'d":
resolution = _resolve_apostrophe_d(token)
else:
resolution = None
if resolution is None:
continue
if resolution.span not in resolutions:
resolutions[resolution.span] = resolution
return resolutions
def _resolution(
prev: Token, token: Token, expansion_word: str, category: str, lemma_hint: str
) -> Optional[ContractionResolution]:
if token is None or prev is None:
return None
if prev.idx + len(prev.text) != token.idx:
# Not a contiguous contraction (whitespace or punctuation in between)
return None
surface_start = prev.idx
surface_end = token.idx + len(token.text)
surface_text = token.doc.text[surface_start:surface_end]
expansion = _assemble_expansion(prev.text, surface_text, expansion_word)
return ContractionResolution(
start=surface_start,
end=surface_end,
surface=surface_text,
expansion=expansion,
category=category,
lemma=lemma_hint,
)
def _assemble_expansion(base_text: str, surface_text: str, expansion_word: str) -> str:
"""Combine *base_text* with *expansion_word*, preserving coarse casing."""
if not expansion_word:
return base_text
if surface_text.isupper() and expansion_word.isalpha():
adjusted = expansion_word.upper()
elif len(surface_text) > 2 and surface_text[:-2].istitle() and expansion_word:
# Surface like "It's" -> keep appended word lowercase
adjusted = expansion_word.lower()
else:
adjusted = expansion_word
return f"{base_text} {adjusted}".strip()
def _resolve_apostrophe_s(token: Token) -> Optional[ContractionResolution]:
prev = token.nbor(-1) if token.i > 0 else None
if prev is None:
return None
# Possessive marker e.g., dog's
if token.tag_ == "POS" or token.lemma_ == "'s":
return None
prev_lower = prev.lemma_.lower()
surface = token.doc.text[prev.idx : token.idx + len(token.text)]
if prev_lower == "let":
return _resolution(prev, token, "us", "contraction_let_us", "us")
# Special check for 's been -> has been, overriding lemma
next_content = _next_content_token(token)
if next_content and next_content.text.lower() == "been":
return _resolution(prev, token, "has", "contraction_aux_have", "have")
lemma = token.lemma_.lower()
if not lemma:
lemma = "be" if _favors_be(token) else "have" if _favors_have(token) else "be"
if lemma == "be":
return _resolution(prev, token, "is", "contraction_aux_be", "be")
if lemma == "have":
return _resolution(prev, token, "has", "contraction_aux_have", "have")
if _favors_have(token):
return _resolution(prev, token, "has", "contraction_aux_have", "have")
if _favors_be(token):
return _resolution(prev, token, "is", "contraction_aux_be", "be")
# Default to copula expansion.
return _resolution(prev, token, "is", "contraction_aux_be", lemma or "be")
def _resolve_apostrophe_d(token: Token) -> Optional[ContractionResolution]:
prev = token.nbor(-1) if token.i > 0 else None
if prev is None:
return None
if token.morph.get("VerbForm") == ["Part"]:
# spaCy sometimes tags possessives oddly; guard anyway
return None
lemma = token.lemma_.lower()
tense = set(token.morph.get("Tense"))
next_content = _next_content_token(token)
prefers_had = _context_prefers_had(token)
if prefers_had:
return _resolution(prev, token, "had", "contraction_aux_have", "have")
if "Past" in tense and lemma in {"have", "had"}:
return _resolution(prev, token, "had", "contraction_aux_have", "have")
if next_content is not None:
next_tag = next_content.tag_
next_lemma = next_content.lemma_.lower()
else:
next_tag = ""
next_lemma = ""
if next_tag == "VB":
return _resolution(
prev, token, "would", "contraction_modal_would", lemma or "will"
)
if token.tag_ == "MD" or lemma in {"will", "would", "shall"}:
return _resolution(
prev, token, "would", "contraction_modal_would", lemma or "will"
)
if next_lemma in {"been", "gone", "had", "better"} or next_tag in {"VBN", "VBD"}:
return _resolution(prev, token, "had", "contraction_aux_have", "have")
if lemma in {"have", "had"}:
return _resolution(prev, token, "had", "contraction_aux_have", lemma)
return _resolution(prev, token, "would", "contraction_modal_would", lemma or "will")
def _next_content_token(token: Token) -> Optional[Token]:
doc = token.doc
for candidate in doc[token.i + 1 :]:
if candidate.is_space:
continue
if candidate.is_punct and candidate.text not in {"-"}:
break
if candidate.text in {"'", ""}:
continue
return candidate
return None
def _favors_have(token: Token) -> bool:
next_content = _next_content_token(token)
if next_content is None:
return False
if next_content.tag_ in {"VBN"}:
return True
if next_content.lemma_.lower() in {"been", "gone", "had"}:
return True
return False
def _favors_be(token: Token) -> bool:
next_content = _next_content_token(token)
if next_content is None:
return True
if next_content.tag_ in {"VBG", "JJ", "RB", "DT", "IN"}:
return True
return False
def _context_prefers_had(token: Token) -> bool:
head = token.head if token.head is not None else None
if head is not None and head.i > token.i:
head_tag = head.tag_
head_lemma = head.lemma_.lower()
if head_tag in {"VBN", "VBD"} or head_lemma in {"gone", "been", "had"}:
return True
if head_lemma == "better":
return True
next_content = _next_content_token(token)
if next_content is None:
return False
next_tag = next_content.tag_
next_lemma = next_content.lemma_.lower()
if next_tag in {"VBN", "VBD"}:
return True
if next_lemma in {"been", "gone", "had"}:
return True
if next_lemma == "better":
return True
return False
+3 -3
View File
@@ -83,12 +83,12 @@ def get_spacy_model(lang_code, log_callback=None):
model_name,
disable=["ner", "tagger", "lemmatizer", "attribute_ruler"],
)
# Ensure a sentence segmentation strategy is in place
# The parser provides sents, but if it's missing (unlikely for core models), fallback to sentencizer
if "parser" not in nlp.pipe_names and "sentencizer" not in nlp.pipe_names:
nlp.add_pipe("sentencizer")
_nlp_cache[lang_code] = nlp
return nlp
except OSError:
@@ -105,7 +105,7 @@ def get_spacy_model(lang_code, log_callback=None):
)
if "parser" not in nlp.pipe_names and "sentencizer" not in nlp.pipe_names:
nlp.add_pipe("sentencizer")
_nlp_cache[lang_code] = nlp
log(f"spaCy model '{model_name}' downloaded and loaded")
return nlp
+762
View File
@@ -0,0 +1,762 @@
from __future__ import annotations
import re
from dataclasses import dataclass, field
from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple
import unicodedata
_DIALOGUE_VERBS = (
"said",
"asked",
"replied",
"whispered",
"shouted",
"cried",
"muttered",
"answered",
"hissed",
"called",
"added",
"continued",
"insisted",
"remarked",
"yelled",
"breathed",
"murmured",
"exclaimed",
"explained",
"noted",
)
_VERB_PATTERN = "(?:" + "|".join(_DIALOGUE_VERBS) + ")"
_NAME_FRAGMENT = r"[A-ZÀ-ÖØ-Þ][\w'\-]*"
_NAME_PATTERN = rf"{_NAME_FRAGMENT}(?:\s+{_NAME_FRAGMENT})*"
_COLON_PATTERN = re.compile(rf"^\s*({_NAME_PATTERN})\s*:\s*(.+)$")
_NAME_BEFORE_VERB = re.compile(rf"({_NAME_PATTERN})\s+{_VERB_PATTERN}\b", re.IGNORECASE)
_VERB_BEFORE_NAME = re.compile(rf"{_VERB_PATTERN}\s+({_NAME_PATTERN})", re.IGNORECASE)
_PRONOUN_PATTERN = re.compile(r"\b(?:he|she|they)\b", re.IGNORECASE)
_QUOTE_PATTERN = re.compile(r'["“”]([^"“”\\]*(?:\\.[^"“”\\]*)*)["”]')
_MALE_PRONOUN_PATTERN = re.compile(r"\b(?:he|him|his|himself)\b", re.IGNORECASE)
_FEMALE_PRONOUN_PATTERN = re.compile(r"\b(?:she|her|hers|herself)\b", re.IGNORECASE)
_PRONOUN_LABELS = {
"he",
"she",
"they",
"them",
"theirs",
"their",
"themselves",
"him",
"his",
"himself",
"her",
"hers",
"herself",
"we",
"us",
"our",
"ours",
"ourselves",
"i",
"me",
"my",
"mine",
"myself",
"you",
"your",
"yours",
"yourself",
"yourselves",
}
_CONFIDENCE_RANK = {"low": 1, "medium": 2, "high": 3}
_FEMALE_TITLE_HINTS = (
"madame",
"mme",
"madam",
"mrs",
"miss",
"ms",
"lady",
"countess",
"baroness",
"princess",
"queen",
"mademoiselle",
)
_MALE_TITLE_HINTS = (
"monsieur",
"m.",
"mr",
"sir",
"lord",
"count",
"baron",
"prince",
"king",
"abbé",
"abbe",
)
_MALE_TOKEN_WEIGHTS = {
"he": 1.0,
"him": 0.6,
"his": 0.75,
"himself": 1.0,
}
_FEMALE_TOKEN_WEIGHTS = {
"she": 1.0,
"her": 0.4,
"hers": 0.75,
"herself": 1.0,
}
_STOP_LABELS = {
"and",
"but",
"then",
"though",
"meanwhile",
"therefore",
"after",
"before",
"when",
"while",
"because",
"as",
"yet",
"nor",
"so",
"thus",
"suddenly",
"eventually",
"finally",
"until",
"unless",
}
@dataclass(slots=True)
class SpeakerGuess:
speaker_id: str
label: str
count: int = 0
confidence: str = "low"
sample_quotes: List[Dict[str, str]] = field(default_factory=list)
suppressed: bool = False
gender: str = "unknown"
detected_gender: str = "unknown"
male_votes: int = 0
female_votes: int = 0
def register_occurrence(
self,
confidence: str,
text: str,
quote: Optional[str],
male_votes: int,
female_votes: int,
sample_excerpt: Optional[str] = None,
) -> None:
self.count += 1
if _CONFIDENCE_RANK.get(confidence, 0) > _CONFIDENCE_RANK.get(
self.confidence, 0
):
self.confidence = confidence
excerpt = (
sample_excerpt
if sample_excerpt is not None
else _build_excerpt(text, quote)
)
gender_hint = _format_gender_hint(male_votes, female_votes)
if excerpt:
payload = {"excerpt": excerpt, "gender_hint": gender_hint}
if payload not in self.sample_quotes:
self.sample_quotes.append(payload)
if len(self.sample_quotes) > 3:
self.sample_quotes = self.sample_quotes[:3]
if male_votes:
self.male_votes += male_votes
if female_votes:
self.female_votes += female_votes
self.detected_gender = _derive_gender(
self.male_votes, self.female_votes, self.detected_gender
)
if self.gender in {"unknown", "male", "female"}:
self.gender = _derive_gender(
self.male_votes, self.female_votes, self.gender
)
def as_dict(self) -> Dict[str, Any]:
return {
"id": self.speaker_id,
"label": self.label,
"count": self.count,
"confidence": self.confidence,
"sample_quotes": [dict(sample) for sample in self.sample_quotes],
"suppressed": self.suppressed,
"gender": self.gender,
"detected_gender": self.detected_gender,
}
@dataclass(slots=True)
class SpeakerAnalysis:
assignments: Dict[str, str]
speakers: Dict[str, SpeakerGuess]
suppressed: List[str]
narrator: str = "narrator"
version: str = "1.0"
stats: Dict[str, Any] = field(default_factory=dict)
def to_dict(self) -> Dict[str, Any]:
return {
"version": self.version,
"narrator": self.narrator,
"assignments": dict(self.assignments),
"speakers": {
speaker_id: guess.as_dict()
for speaker_id, guess in self.speakers.items()
},
"suppressed": list(self.suppressed),
"stats": dict(self.stats),
}
def analyze_speakers(
chapters: Sequence[Dict[str, Any]] | Iterable[Dict[str, Any]],
chunks: Sequence[Dict[str, Any]] | Iterable[Dict[str, Any]],
*,
threshold: int = 3,
max_speakers: int = 8,
) -> SpeakerAnalysis:
narrator_id = "narrator"
speaker_guesses: Dict[str, SpeakerGuess] = {
narrator_id: SpeakerGuess(
speaker_id=narrator_id, label="Narrator", confidence="low"
)
}
label_index: Dict[str, str] = {"Narrator": narrator_id}
assignments: Dict[str, str] = {}
suppressed: List[str] = []
ordered_chunks = sorted(
(dict(chunk) for chunk in chunks),
key=lambda entry: (
_safe_int(entry.get("chapter_index")),
_safe_int(entry.get("chunk_index")),
),
)
last_explicit: Optional[str] = None
explicit_assignments = 0
unique_speakers: set[str] = set()
for index, chunk in enumerate(ordered_chunks):
chunk_id = str(chunk.get("id") or "")
text = _get_chunk_text(chunk)
speaker_id, confidence, quote = _infer_chunk_speaker(text, last_explicit)
if speaker_id is None:
speaker_id = last_explicit or narrator_id
confidence = "medium" if last_explicit else "low"
quote = quote or _extract_quote(text)
if speaker_id != narrator_id:
last_explicit = speaker_id
explicit_assignments += 1
if speaker_id in speaker_guesses:
record_id = speaker_id
guess = speaker_guesses[record_id]
label = guess.label
else:
label = _normalize_label(speaker_id)
record_id = label_index.get(label)
if record_id is None:
record_id = _dedupe_slug(_slugify(label), speaker_guesses)
label_index[label] = record_id
speaker_guesses[record_id] = SpeakerGuess(
speaker_id=record_id, label=label
)
guess = speaker_guesses[record_id]
assignments[chunk_id] = record_id
unique_speakers.add(record_id)
if (
record_id != narrator_id
and record_id != speaker_id
and speaker_id == last_explicit
):
last_explicit = record_id
sample_excerpt = None
if record_id != narrator_id:
sample_excerpt = _select_sample_excerpt(
ordered_chunks, index, guess.label, quote, confidence
)
male_votes, female_votes = _count_gender_votes(text, guess.label)
guess.register_occurrence(
confidence, text, quote, male_votes, female_votes, sample_excerpt
)
active_speakers = [sid for sid in speaker_guesses if sid != narrator_id]
# Apply minimum occurrence threshold.
for speaker_id in list(active_speakers):
guess = speaker_guesses[speaker_id]
if guess.count < max(1, threshold):
guess.suppressed = True
suppressed.append(speaker_id)
_reassign(assignments, speaker_id, narrator_id)
active_speakers.remove(speaker_id)
# Apply maximum active speaker cap.
if max_speakers and len(active_speakers) > max_speakers:
active_speakers.sort(key=lambda sid: (-speaker_guesses[sid].count, sid))
for speaker_id in active_speakers[max_speakers:]:
guess = speaker_guesses[speaker_id]
guess.suppressed = True
suppressed.append(speaker_id)
_reassign(assignments, speaker_id, narrator_id)
active_speakers = active_speakers[:max_speakers]
narrator_guess = speaker_guesses[narrator_id]
narrator_guess.count = sum(
1 for value in assignments.values() if value == narrator_id
)
narrator_guess.confidence = "low"
stats = {
"total_chunks": len(ordered_chunks),
"explicit_chunks": explicit_assignments,
"active_speakers": len(active_speakers),
"unique_speakers": len(unique_speakers),
"suppressed": len(suppressed),
}
return SpeakerAnalysis(
assignments=assignments,
speakers=speaker_guesses,
suppressed=suppressed,
narrator=narrator_id,
stats=stats,
)
def _infer_chunk_speaker(
text: str, last_explicit: Optional[str]
) -> Tuple[Optional[str], str, Optional[str]]:
normalized = text.strip()
if not normalized:
return None, "low", None
colon_match = _COLON_PATTERN.match(normalized)
if colon_match:
raw_label = colon_match.group(1)
cleaned = _normalize_candidate_name(raw_label)
if cleaned is None:
return None, "low", colon_match.group(2).strip()
quote = colon_match.group(2).strip()
return cleaned, "high", quote
quote = _extract_quote(normalized)
if not quote:
return None, "low", None
before, after = _split_around_quote(normalized, quote)
candidate = _match_name_near_quote(before, after)
if candidate:
cleaned = _normalize_candidate_name(candidate)
if cleaned:
return cleaned, "high", quote
if last_explicit:
pronoun_after = _PRONOUN_PATTERN.search(after)
pronoun_before = _PRONOUN_PATTERN.search(before)
if pronoun_after or pronoun_before:
return last_explicit, "medium", quote
return None, "low", quote
def _split_around_quote(text: str, quote: str) -> Tuple[str, str]:
quote_index = text.find(quote)
if quote_index == -1:
return text, ""
before = text[:quote_index]
after = text[quote_index + len(quote) :]
return before, after
def _match_name_near_quote(before: str, after: str) -> Optional[str]:
trailing = before[-120:]
leading = after[:120]
match = _NAME_BEFORE_VERB.search(trailing)
if match:
name = match.group(1)
if _looks_like_name(name):
return name
match = re.search(
rf"({_NAME_PATTERN})\s*,?\s*{_VERB_PATTERN}", leading, flags=re.IGNORECASE
)
if match:
name = match.group(1)
if _looks_like_name(name):
return name
match = _VERB_BEFORE_NAME.search(leading)
if match:
name = match.group(1)
if _looks_like_name(name):
return name
return None
def _looks_like_name(value: str) -> bool:
normalized = _normalize_candidate_name(value)
if not normalized:
return False
parts = normalized.split()
if not parts:
return False
return all(part and part[0].isupper() for part in parts)
def _extract_quote(text: str) -> Optional[str]:
match = _QUOTE_PATTERN.search(text)
if not match:
return None
return match.group(0)
def _slugify(label: str) -> str:
slug = re.sub(r"[^a-z0-9]+", "_", label.lower()).strip("_")
return slug or "speaker"
def _dedupe_slug(slug: str, existing: Dict[str, SpeakerGuess]) -> str:
candidate = slug
index = 2
while candidate in existing:
candidate = f"{slug}_{index}"
index += 1
return candidate
def _normalize_label(label: str) -> str:
words = re.split(r"\s+", label.strip())
return " ".join(word.capitalize() for word in words if word)
def _safe_int(value: Any, default: int = 0) -> int:
try:
return int(value)
except (TypeError, ValueError):
return default
def _reassign(assignments: Dict[str, str], old: str, new: str) -> None:
for key, value in list(assignments.items()):
if value == old:
assignments[key] = new
def _strip_diacritics(value: str) -> str:
normalized = unicodedata.normalize("NFKD", value)
return "".join(char for char in normalized if not unicodedata.combining(char))
def _count_gender_votes(text: str, label: Optional[str]) -> Tuple[int, int]:
if not text:
return 0, 0
search_text = text
windows: List[Tuple[int, int]] = []
degrade_factor = 1.0
if label:
pattern = re.compile(re.escape(label), re.IGNORECASE)
matches = list(pattern.finditer(search_text))
if not matches:
alt_label = _strip_diacritics(label)
if alt_label and alt_label != label:
ascii_text = _strip_diacritics(search_text)
pattern_alt = re.compile(re.escape(alt_label), re.IGNORECASE)
windows = [match.span() for match in pattern_alt.finditer(ascii_text)]
# Map spans back roughly using proportional index
if windows:
mapped: List[Tuple[int, int]] = []
for start, end in windows:
start_idx = min(
len(search_text) - 1,
int(start * len(search_text) / max(len(ascii_text), 1)),
)
end_idx = min(
len(search_text),
int(end * len(search_text) / max(len(ascii_text), 1)),
)
mapped.append((start_idx, end_idx))
windows = mapped
else:
windows = [match.span() for match in matches]
if not windows:
windows = [(0, len(search_text))]
degrade_factor = 0.25
radius = 60
quote_spans: List[Tuple[int, int, str]] = []
for match in _QUOTE_PATTERN.finditer(search_text):
try:
content_start, content_end = match.span(1)
except IndexError:
content_start, content_end = match.span()
if content_start < content_end:
quote_spans.append(
(content_start, content_end, search_text[content_start:content_end])
)
normalized_label = _normalize_candidate_name(label) if label else None
normalized_label_lower = normalized_label.lower() if normalized_label else None
def _window_weight(position: int) -> float:
for start, end in windows:
if position < start - radius or position > end + radius:
continue
if position >= end:
return 1.0
if position <= start:
return 0.2
return 1.0
return 0.0
def _quote_weight(position: int) -> float:
for start, end, content in quote_spans:
if position < start or position >= end:
continue
local_index = position - start
prefix = content[:local_index]
tail = prefix[-80:]
name_matches = list(re.finditer(_NAME_PATTERN, tail))
if name_matches:
last_name = _normalize_candidate_name(name_matches[-1].group(0))
if (
normalized_label_lower
and last_name
and last_name.lower() == normalized_label_lower
):
return 0.6
return 0.05
if re.search(r"[.!?]\s", prefix):
return 0.2
if prefix.strip():
return 0.15
return 0.1
return 1.0
male_score = 0.0
for match in _MALE_PRONOUN_PATTERN.finditer(search_text):
base_weight = _window_weight(match.start())
if not base_weight:
continue
quote_modifier = _quote_weight(match.start())
weight = base_weight * quote_modifier
if not weight:
continue
token = match.group(0).lower()
male_score += _MALE_TOKEN_WEIGHTS.get(token, 0.6) * weight
female_score = 0.0
for match in _FEMALE_PRONOUN_PATTERN.finditer(search_text):
base_weight = _window_weight(match.start())
if not base_weight:
continue
quote_modifier = _quote_weight(match.start())
weight = base_weight * quote_modifier
if not weight:
continue
if quote_modifier >= 0.95:
weight = max(weight, 0.4)
token = match.group(0).lower()
female_score += _FEMALE_TOKEN_WEIGHTS.get(token, 0.4) * weight
for start, end in windows:
span_start = max(0, start - 40)
span_end = min(len(search_text), end + 40)
span_text = search_text[span_start:span_end].lower()
if any(title in span_text for title in _FEMALE_TITLE_HINTS):
female_score += 2.5
if any(title in span_text for title in _MALE_TITLE_HINTS):
male_score += 2.5
male_votes = int(round(male_score * degrade_factor))
female_votes = int(round(female_score * degrade_factor))
return male_votes, female_votes
def _derive_gender(male_votes: int, female_votes: int, current: str) -> str:
if male_votes == 0 and female_votes == 0:
return current if current != "unknown" else "unknown"
male_threshold = max(2, female_votes + 1)
female_threshold = max(2, male_votes + 1)
if male_votes >= male_threshold:
return "male"
if female_votes >= female_threshold:
return "female"
if current in {"male", "female"}:
return current
return "unknown"
def _get_chunk_text(chunk: Dict[str, Any]) -> str:
if not isinstance(chunk, dict):
return ""
value = chunk.get("normalized_text") or chunk.get("text") or ""
return str(value)
def _trim_paragraph(paragraph: str, limit: int = 600) -> str:
normalized = (paragraph or "").strip()
if not normalized:
return ""
if len(normalized) <= limit:
return normalized
return normalized[: limit - 1].rstrip() + ""
def _compose_context_excerpt(before: str, current: str, after: str) -> str:
segments = []
for value in (before, current, after):
trimmed = _trim_paragraph(value)
if trimmed:
segments.append(trimmed)
return "\n\n".join(segments)
def _contains_dialogue_attribution(label: str, text: str, quote: Optional[str]) -> bool:
if not label or not text:
return False
escaped_label = re.escape(label)
direct_pattern = re.compile(
rf"\b{escaped_label}\b\s+(?:{_VERB_PATTERN})\b", re.IGNORECASE
)
reverse_pattern = re.compile(
rf"(?:{_VERB_PATTERN})\s+\b{escaped_label}\b", re.IGNORECASE
)
colon_pattern = re.compile(rf"^\s*{escaped_label}\s*:\s*", re.IGNORECASE)
if colon_pattern.search(text):
return True
if direct_pattern.search(text) or reverse_pattern.search(text):
return True
if quote:
before, after = _split_around_quote(text, quote)
if direct_pattern.search(before) or reverse_pattern.search(after):
return True
return False
def _select_sample_excerpt(
chunks: Sequence[Dict[str, Any]],
index: int,
label: str,
quote: Optional[str],
confidence: str,
) -> Optional[str]:
if confidence != "high" or not label:
return None
if index < 0 or index >= len(chunks):
return None
current = _get_chunk_text(chunks[index])
if not current or not _contains_dialogue_attribution(label, current, quote):
return None
previous = _get_chunk_text(chunks[index - 1]) if index > 0 else ""
following = _get_chunk_text(chunks[index + 1]) if index + 1 < len(chunks) else ""
excerpt = _compose_context_excerpt(previous, current, following)
return excerpt or None
def _build_excerpt(text: str, quote: Optional[str]) -> str:
normalized = (text or "").strip()
if not normalized:
return ""
if quote:
location = normalized.find(quote)
if location != -1:
start = max(0, location - 120)
end = min(len(normalized), location + len(quote) + 120)
snippet = normalized[start:end].strip()
if start > 0:
snippet = "" + snippet
if end < len(normalized):
snippet = snippet + ""
return snippet
if len(normalized) > 240:
return normalized[:240].rstrip() + ""
return normalized
def _format_gender_hint(male_votes: int, female_votes: int) -> str:
if male_votes and female_votes:
return "Context mentions both male and female pronouns."
if male_votes:
if male_votes >= 3:
return "Multiple male pronouns detected nearby."
return "Some male pronouns detected in the surrounding text."
if female_votes:
if female_votes >= 3:
return "Multiple female pronouns detected nearby."
return "Some female pronouns detected in the surrounding text."
return "No clear pronoun signal detected."
def _normalize_candidate_name(raw: str) -> Optional[str]:
if not raw:
return None
cleaned = raw.strip().strip("\"“”'.,:;!")
cleaned = re.sub(r"\s+", " ", cleaned).strip()
if not cleaned:
return None
parts = cleaned.split()
filtered: List[str] = []
for part in parts:
if not part:
continue
if not filtered and part.lower() in _STOP_LABELS:
continue
filtered.append(part)
while filtered and filtered[-1].lower() in _STOP_LABELS:
filtered.pop()
if not filtered:
return None
if all(part.lower() in _STOP_LABELS for part in filtered):
return None
contiguous: List[str] = []
for part in filtered:
if part and part[0].isupper():
contiguous.append(part)
else:
break
if contiguous:
candidate = " ".join(contiguous)
else:
candidate = ""
if not candidate:
return None
lowered = candidate.lower()
if lowered in _PRONOUN_LABELS or lowered in _STOP_LABELS:
return None
return candidate
+166
View File
@@ -0,0 +1,166 @@
from __future__ import annotations
import json
import os
from typing import Any, Dict, List, Optional
from abogen.constants import LANGUAGE_DESCRIPTIONS
from abogen.utils import get_user_config_path
_CONFIG_WRAPPER_KEY = "abogen_speaker_configs"
def _config_path() -> str:
config_path = get_user_config_path()
config_dir = os.path.dirname(config_path)
os.makedirs(config_dir, exist_ok=True)
return os.path.join(config_dir, "speaker_configs.json")
def load_configs() -> Dict[str, Dict[str, Any]]:
path = _config_path()
if not os.path.exists(path):
return {}
try:
with open(path, "r", encoding="utf-8") as handle:
payload = json.load(handle)
except Exception:
return {}
if isinstance(payload, dict) and _CONFIG_WRAPPER_KEY in payload:
payload = payload[_CONFIG_WRAPPER_KEY]
if not isinstance(payload, dict):
return {}
sanitized: Dict[str, Dict[str, Any]] = {}
for name, entry in payload.items():
if not isinstance(name, str) or not isinstance(entry, dict):
continue
sanitized[name] = _sanitize_config(entry)
return sanitized
def save_configs(configs: Dict[str, Dict[str, Any]]) -> None:
path = _config_path()
sanitized: Dict[str, Dict[str, Any]] = {}
for name, entry in configs.items():
if not isinstance(name, str) or not name.strip():
continue
sanitized[name] = _sanitize_config(entry)
with open(path, "w", encoding="utf-8") as handle:
json.dump({_CONFIG_WRAPPER_KEY: sanitized}, handle, indent=2, sort_keys=True)
def get_config(name: str) -> Optional[Dict[str, Any]]:
name = (name or "").strip()
if not name:
return None
configs = load_configs()
data = configs.get(name)
return dict(data) if isinstance(data, dict) else None
def upsert_config(name: str, payload: Dict[str, Any]) -> Dict[str, Any]:
name = (name or "").strip()
if not name:
raise ValueError("Configuration name is required")
configs = load_configs()
configs[name] = _sanitize_config(payload or {})
save_configs(configs)
return configs[name]
def delete_config(name: str) -> None:
name = (name or "").strip()
if not name:
return
configs = load_configs()
if name in configs:
del configs[name]
save_configs(configs)
def _sanitize_config(entry: Dict[str, Any]) -> Dict[str, Any]:
language = str(entry.get("language") or "a").strip() or "a"
speakers_raw = entry.get("speakers")
if not isinstance(speakers_raw, dict):
speakers_raw = {}
speakers: Dict[str, Any] = {}
for speaker_id, payload in speakers_raw.items():
if not isinstance(speaker_id, str) or not isinstance(payload, dict):
continue
record = _sanitize_speaker({"id": speaker_id, **payload})
speakers[record["id"]] = record
allowed_languages = entry.get("languages") or entry.get("allowed_languages") or []
if not isinstance(allowed_languages, list):
allowed_languages = []
normalized_langs = []
for code in allowed_languages:
if isinstance(code, str) and code:
normalized_langs.append(code.lower())
default_voice = entry.get("default_voice")
if not isinstance(default_voice, str):
default_voice = ""
return {
"language": language.lower(),
"languages": normalized_langs,
"default_voice": default_voice,
"speakers": speakers,
"version": int(entry.get("version", 1)),
"notes": entry.get("notes") if isinstance(entry.get("notes"), str) else "",
}
def slugify_label(label: str) -> str:
normalized = (label or "").strip().lower()
if not normalized:
return "speaker"
slug = "".join(ch if ch.isalnum() else "_" for ch in normalized)
slug = "_".join(filter(None, slug.split("_")))
return slug or "speaker"
def _sanitize_speaker(entry: Dict[str, Any]) -> Dict[str, Any]:
label = (entry.get("label") or entry.get("name") or "").strip()
gender = (entry.get("gender") or "unknown").strip().lower()
if gender not in {"male", "female", "unknown"}:
gender = "unknown"
voice = entry.get("voice")
voice_profile = entry.get("voice_profile")
voice_formula = entry.get("voice_formula")
voice_languages = entry.get("languages") or []
if not isinstance(voice_languages, list):
voice_languages = []
normalized_langs = []
for code in voice_languages:
if isinstance(code, str) and code:
normalized_langs.append(code.lower())
resolved_voice = entry.get("resolved_voice") or voice_formula or voice
resolved_label = label or entry.get("id") or ""
slug = (
entry.get("id")
if isinstance(entry.get("id"), str)
else slugify_label(resolved_label)
)
return {
"id": slug,
"label": resolved_label,
"gender": gender,
"voice": voice if isinstance(voice, str) else "",
"voice_profile": voice_profile if isinstance(voice_profile, str) else "",
"voice_formula": voice_formula if isinstance(voice_formula, str) else "",
"resolved_voice": resolved_voice if isinstance(resolved_voice, str) else "",
"languages": normalized_langs,
}
def list_configs() -> List[Dict[str, Any]]:
configs = load_configs()
ordered = []
for name in sorted(configs):
entry = configs[name]
ordered.append({"name": name, **entry})
return ordered
def describe_language(code: str) -> str:
code = (code or "a").lower()
return LANGUAGE_DESCRIPTIONS.get(code, code.upper())
+584
View File
@@ -0,0 +1,584 @@
import re
import platform
from abogen.utils import detect_encoding, load_config
from abogen.constants import SAMPLE_VOICE_TEXTS
# Pre-compile frequently used regex patterns for better performance
_METADATA_TAG_PATTERN = re.compile(r"<<METADATA_[^:]+:[^>]*>>")
_WHITESPACE_PATTERN = re.compile(r"[^\S\n]+")
_MULTIPLE_NEWLINES_PATTERN = re.compile(r"\n{3,}")
_SINGLE_NEWLINE_PATTERN = re.compile(r"(?<!\n)\n(?!\n)")
_CHAPTER_MARKER_PATTERN = re.compile(r"<<CHAPTER_MARKER:[^>]*>>")
_HTML_TAG_PATTERN = re.compile(r"<[^>]+>")
_VOICE_TAG_PATTERN = re.compile(r"{[^}]+}")
_ASS_STYLING_PATTERN = re.compile(r"\{[^}]+\}")
_ASS_NEWLINE_N_PATTERN = re.compile(r"\\N")
_ASS_NEWLINE_LOWER_N_PATTERN = re.compile(r"\\n")
_CHAPTER_MARKER_SEARCH_PATTERN = re.compile(r"<<CHAPTER_MARKER:(.*?)>>")
_VOICE_MARKER_PATTERN = re.compile(r"<<VOICE:[^>]*>>")
_VOICE_MARKER_SEARCH_PATTERN = re.compile(r"<<VOICE:(.*?)>>")
_WEBVTT_HEADER_PATTERN = re.compile(r"^WEBVTT.*?\n", re.MULTILINE)
_VTT_STYLE_PATTERN = re.compile(r"STYLE\s*\n.*?(?=\n\n|$)", re.DOTALL)
_VTT_NOTE_PATTERN = re.compile(r"NOTE\s*\n.*?(?=\n\n|$)", re.DOTALL)
_DOUBLE_NEWLINE_SPLIT_PATTERN = re.compile(r"\n\s*\n")
_VTT_TIMESTAMP_PATTERN = re.compile(r"([\d:.]+)\s*-->\s*([\d:.]+)")
_TIMESTAMP_ONLY_PATTERN = re.compile(r"^(\d{1,2}:\d{2}:\d{2}(?:[.,]\d{1,3})?)$")
_WINDOWS_ILLEGAL_CHARS_PATTERN = re.compile(r'[<>:"/\\|?*]')
_CONTROL_CHARS_PATTERN = re.compile(r"[\x00-\x1f]")
_LINUX_CONTROL_CHARS_PATTERN = re.compile(
r"[\x01-\x1f]"
) # Linux: exclude \x00 for separate handling
_MACOS_ILLEGAL_CHARS_PATTERN = re.compile(r"[:]")
_LINUX_ILLEGAL_CHARS_PATTERN = re.compile(r"[/\x00]")
def clean_subtitle_text(text):
"""Remove chapter markers, voice markers, and metadata tags from subtitle text."""
# Use pre-compiled patterns for better performance
text = _METADATA_TAG_PATTERN.sub("", text)
text = _CHAPTER_MARKER_PATTERN.sub("", text)
text = _VOICE_MARKER_PATTERN.sub("", text)
return text.strip()
def calculate_text_length(text):
# Use pre-compiled patterns for better performance
# Ignore chapter markers, voice markers, and metadata patterns in a single pass
text = _CHAPTER_MARKER_PATTERN.sub("", text)
text = _VOICE_MARKER_PATTERN.sub("", text)
text = _METADATA_TAG_PATTERN.sub("", text)
# Ignore newlines and leading/trailing spaces
text = text.replace("\n", "").strip()
# Calculate character count
char_count = len(text)
return char_count
def clean_text(text, *args, **kwargs):
# Remove metadata tags first
text = _METADATA_TAG_PATTERN.sub("", text)
# Load replace_single_newlines from config
cfg = load_config()
replace_single_newlines = cfg.get("replace_single_newlines", True)
# Collapse all whitespace (excluding newlines) into single spaces per line and trim edges
# Use pre-compiled pattern for better performance
lines = [_WHITESPACE_PATTERN.sub(" ", line).strip() for line in text.splitlines()]
text = "\n".join(lines)
# Standardize paragraph breaks (multiple newlines become exactly two) and trim overall whitespace
# Use pre-compiled pattern for better performance
text = _MULTIPLE_NEWLINES_PATTERN.sub("\n\n", text).strip()
# Optionally replace single newlines with spaces, but preserve double newlines
if replace_single_newlines:
# Use pre-compiled pattern for better performance
text = _SINGLE_NEWLINE_PATTERN.sub(" ", text)
return text
def parse_srt_file(file_path):
"""
Parse an SRT subtitle file and return a list of subtitle entries.
Args:
file_path: Path to the SRT file
Returns:
List of tuples: [(start_time_seconds, end_time_seconds, text), ...]
"""
encoding = detect_encoding(file_path)
with open(file_path, "r", encoding=encoding, errors="replace") as f:
content = f.read()
# Split by double newlines to get individual subtitle blocks
blocks = re.split(r"\n\s*\n", content.strip())
subtitles = []
for block in blocks:
if not block.strip():
continue
lines = block.strip().split("\n")
if len(lines) < 3:
continue
# First line is index, second line is timestamp, rest is text
try:
timestamp_line = lines[1]
match = re.match(
r"(\d{2}:\d{2}:\d{2},\d{3})\s*-->\s*(\d{2}:\d{2}:\d{2},\d{3})",
timestamp_line,
)
if not match:
continue
start_str = match.group(1)
end_str = match.group(2)
text = "\n".join(lines[2:])
# Convert timestamp to seconds
def time_to_seconds(t):
h, m, s_ms = t.split(":")
s, ms = s_ms.split(",")
return int(h) * 3600 + int(m) * 60 + int(s) + int(ms) / 1000.0
start_sec = time_to_seconds(start_str)
end_sec = time_to_seconds(end_str)
# Clean text of any styling tags using pre-compiled pattern
text = _HTML_TAG_PATTERN.sub("", text)
# Remove chapter markers and metadata tags
text = clean_subtitle_text(text)
if text: # Only add non-empty subtitles
subtitles.append((start_sec, end_sec, text))
except (ValueError, IndexError):
continue
return subtitles
def parse_vtt_file(file_path):
"""
Parse a VTT (WebVTT) subtitle file and return a list of subtitle entries.
Args:
file_path: Path to the VTT file
Returns:
List of tuples: [(start_time_seconds, end_time_seconds, text), ...]
"""
encoding = detect_encoding(file_path)
with open(file_path, "r", encoding=encoding, errors="replace") as f:
content = f.read()
# Remove WEBVTT header and any style/note blocks using pre-compiled patterns
content = _WEBVTT_HEADER_PATTERN.sub("", content)
content = _VTT_STYLE_PATTERN.sub("", content)
content = _VTT_NOTE_PATTERN.sub("", content)
# Split by double newlines to get individual subtitle blocks using pre-compiled pattern
blocks = _DOUBLE_NEWLINE_SPLIT_PATTERN.split(content.strip())
subtitles = []
for block in blocks:
if not block.strip():
continue
lines = block.strip().split("\n")
if len(lines) < 2:
continue
# VTT can have optional identifier on first line, timestamp on second or first
timestamp_line = None
text_start_idx = 0
# Check if first line is timestamp
if "-->" in lines[0]:
timestamp_line = lines[0]
text_start_idx = 1
elif len(lines) > 1 and "-->" in lines[1]:
timestamp_line = lines[1]
text_start_idx = 2
else:
continue
try:
# VTT format: 00:00:00.000 --> 00:00:05.000 or 00:00.000 --> 00:05.000
# Use pre-compiled pattern
match = _VTT_TIMESTAMP_PATTERN.match(timestamp_line)
if not match:
continue
start_str = match.group(1)
end_str = match.group(2)
text = "\n".join(lines[text_start_idx:])
# Convert timestamp to seconds
def time_to_seconds(t):
parts = t.split(":")
if len(parts) == 3: # HH:MM:SS.mmm
h, m, s = parts
s, ms = s.split(".")
return int(h) * 3600 + int(m) * 60 + int(s) + int(ms) / 1000.0
elif len(parts) == 2: # MM:SS.mmm
m, s = parts
s, ms = s.split(".")
return int(m) * 60 + int(s) + int(ms) / 1000.0
return 0
start_sec = time_to_seconds(start_str)
end_sec = time_to_seconds(end_str)
# Clean text of any styling tags and cue settings using pre-compiled patterns
text = _HTML_TAG_PATTERN.sub("", text)
text = _VOICE_TAG_PATTERN.sub("", text) # Remove voice tags
# Remove chapter markers and metadata tags
text = clean_subtitle_text(text)
if text: # Only add non-empty subtitles
subtitles.append((start_sec, end_sec, text))
except (ValueError, IndexError, AttributeError):
continue
return subtitles
def detect_timestamps_in_text(file_path):
"""Detect if text file contains timestamp markers (HH:MM:SS or HH:MM:SS,ms format) on separate lines."""
try:
encoding = detect_encoding(file_path)
with open(file_path, "r", encoding=encoding, errors="replace") as f:
lines = [
line.strip() for line in f.readlines()[:50] if line.strip()
] # Check first 50 non-empty lines
# Count lines that are ONLY timestamps (no other text)
# Supports HH:MM:SS or HH:MM:SS,ms format
# Use pre-compiled pattern for better performance
timestamp_lines = sum(
1 for line in lines if _TIMESTAMP_ONLY_PATTERN.match(line)
)
# Must have at least 2 timestamp-only lines and they should be >5% of total lines
return timestamp_lines >= 2 and (timestamp_lines / max(len(lines), 1)) > 0.05
except Exception:
return False
def parse_timestamp_text_file(file_path):
"""Parse text file with timestamps. Returns list of (start_time, end_time, text) tuples.
Supports HH:MM:SS or HH:MM:SS,ms format. Returns time in seconds as float."""
encoding = detect_encoding(file_path)
with open(file_path, "r", encoding=encoding, errors="replace") as f:
content = f.read()
# Split by timestamp pattern (supports HH:MM:SS or HH:MM:SS,ms)
pattern = r"^(\d{1,2}:\d{2}:\d{2}(?:[.,]\d{1,3})?)$"
lines = content.split("\n")
def parse_time(time_str):
"""Convert HH:MM:SS or HH:MM:SS,ms to seconds as float."""
time_str = time_str.replace(",", ".")
parts = time_str.split(":")
return float(int(parts[0]) * 3600 + int(parts[1]) * 60 + float(parts[2]))
entries = []
current_time = None
current_text = []
pre_timestamp_text = [] # Text before first timestamp
for line in lines:
match = re.match(pattern, line.strip())
if match:
# Save previous entry
if current_time is not None and current_text:
text = "\n".join(current_text).strip()
if text:
entries.append((current_time, text))
elif current_time is None and pre_timestamp_text:
# First timestamp found, save pre-timestamp text with time 0
text = "\n".join(pre_timestamp_text).strip()
if text:
entries.append((0.0, text))
pre_timestamp_text = []
# Start new entry
time_str = match.group(1)
current_time = parse_time(time_str)
current_text = []
elif current_time is not None:
current_text.append(line)
else:
# Text before first timestamp
pre_timestamp_text.append(line)
# Save last entry
if current_time is not None and current_text:
text = "\n".join(current_text).strip()
if text:
entries.append((current_time, text))
elif not entries and pre_timestamp_text:
# No timestamps found at all, treat entire file as starting at 0
text = "\n".join(pre_timestamp_text).strip()
if text:
entries.append((0.0, text))
# Convert to subtitle format with end times
subtitles = []
for i, (start_time, text) in enumerate(entries):
end_time = entries[i + 1][0] if i + 1 < len(entries) else None
# Remove chapter markers and metadata tags
text = clean_subtitle_text(text)
if text: # Only add non-empty entries
subtitles.append((start_time, end_time, text))
return subtitles
def parse_ass_file(file_path):
"""
Parse an ASS/SSA subtitle file and return a list of subtitle entries.
Args:
file_path: Path to the ASS/SSA file
Returns:
List of tuples: [(start_time_seconds, end_time_seconds, text), ...]
"""
encoding = detect_encoding(file_path)
with open(file_path, "r", encoding=encoding, errors="replace") as f:
lines = f.readlines()
subtitles = []
in_events = False
format_indices = {}
for line in lines:
line = line.strip()
if line.startswith("[Events]"):
in_events = True
continue
if line.startswith("[") and in_events:
# New section, stop processing
break
if in_events and line.startswith("Format:"):
# Parse format line to know column positions
parts = line.split(":", 1)[1].strip().split(",")
for i, part in enumerate(parts):
format_indices[part.strip().lower()] = i
continue
if in_events and (line.startswith("Dialogue:") or line.startswith("Comment:")):
if line.startswith("Comment:"):
continue # Skip comments
parts = line.split(":", 1)[1].strip().split(",", len(format_indices) - 1)
if (
"start" in format_indices
and "end" in format_indices
and "text" in format_indices
):
start_str = parts[format_indices["start"]].strip()
end_str = parts[format_indices["end"]].strip()
text = parts[format_indices["text"]].strip()
# Convert timestamp to seconds (ASS format: H:MM:SS.CS where CS is centiseconds)
def ass_time_to_seconds(t):
parts = t.split(":")
if len(parts) == 3:
h, m, s = parts
s_parts = s.split(".")
seconds = float(s_parts[0])
centiseconds = float(s_parts[1]) if len(s_parts) > 1 else 0
return (
int(h) * 3600 + int(m) * 60 + seconds + centiseconds / 100.0
)
return 0
start_sec = ass_time_to_seconds(start_str)
end_sec = ass_time_to_seconds(end_str)
# Clean text of ASS styling tags using pre-compiled patterns
text = _ASS_STYLING_PATTERN.sub("", text) # Remove {tags}
text = _ASS_NEWLINE_N_PATTERN.sub("\n", text) # Convert \N to newline
text = _ASS_NEWLINE_LOWER_N_PATTERN.sub(
"\n", text
) # Convert \n to newline
# Remove chapter markers and metadata tags
text = clean_subtitle_text(text)
if text: # Only add non-empty subtitles
subtitles.append((start_sec, end_sec, text))
return subtitles
def get_sample_voice_text(lang_code):
return SAMPLE_VOICE_TEXTS.get(lang_code, SAMPLE_VOICE_TEXTS["a"])
def sanitize_name_for_os(name, is_folder=True):
"""
Sanitize a filename or folder name based on the operating system.
Args:
name: The name to sanitize
is_folder: Whether this is a folder name (default: True)
Returns:
Sanitized name safe for the current OS
"""
if not name:
return "audiobook"
system = platform.system()
if system == "Windows":
# Windows illegal characters: < > : " / \ | ? *
# Also can't end with space or dot
# Use pre-compiled pattern for better performance
sanitized = _WINDOWS_ILLEGAL_CHARS_PATTERN.sub("_", name)
# Remove control characters (0-31)
sanitized = _CONTROL_CHARS_PATTERN.sub("_", sanitized)
# Remove trailing spaces and dots
sanitized = sanitized.rstrip(". ")
# Windows reserved names (CON, PRN, AUX, NUL, COM1-9, LPT1-9)
reserved = (
["CON", "PRN", "AUX", "NUL"]
+ [f"COM{i}" for i in range(1, 10)]
+ [f"LPT{i}" for i in range(1, 10)]
)
if sanitized.upper() in reserved or sanitized.upper().split(".")[0] in reserved:
sanitized = f"_{sanitized}"
elif system == "Darwin": # macOS
# macOS illegal characters: : (colon is converted to / by the system)
# Also can't start with dot (hidden file) for folders typically
# Use pre-compiled pattern for better performance
sanitized = _MACOS_ILLEGAL_CHARS_PATTERN.sub("_", name)
# Remove control characters
sanitized = _CONTROL_CHARS_PATTERN.sub("_", sanitized)
# Avoid leading dot for folders (creates hidden folders)
if is_folder and sanitized.startswith("."):
sanitized = "_" + sanitized[1:]
else: # Linux and others
# Linux illegal characters: / and null character
# Though / is illegal, most other chars are technically allowed
# Use pre-compiled pattern for better performance
sanitized = _LINUX_ILLEGAL_CHARS_PATTERN.sub("_", name)
# Remove other control characters for safety (excluding \x00 which is already handled)
sanitized = _LINUX_CONTROL_CHARS_PATTERN.sub("_", sanitized)
# Avoid leading dot for folders (creates hidden folders)
if is_folder and sanitized.startswith("."):
sanitized = "_" + sanitized[1:]
# Ensure the name is not empty after sanitization
if not sanitized or sanitized.strip() == "":
sanitized = "audiobook"
# Limit length to 255 characters (common limit across filesystems)
if len(sanitized) > 255:
sanitized = sanitized[:255].rstrip(". ")
return sanitized
def validate_voice_name(voice_name):
"""Validate voice name against VOICES_INTERNAL list (case-insensitive).
Handles both single voices and formulas like 'af_heart*0.5 + am_echo*0.5'.
Args:
voice_name: Voice name or formula string to validate
Returns:
Tuple of (is_valid, invalid_voice_name):
- is_valid: True if all voices in the name/formula are valid
- invalid_voice_name: The first invalid voice found, or None if all valid
"""
from abogen.constants import VOICES_INTERNAL
# Create case-insensitive lookup set (done once per call)
voice_lookup_lower = {v.lower() for v in VOICES_INTERNAL}
voice_name = voice_name.strip()
# Check if it's a formula (contains *)
if "*" in voice_name:
# Extract voice names from formula
voices = voice_name.split("+")
for term in voices:
if "*" in term:
base_voice = term.split("*")[0].strip()
# Case-insensitive comparison
if base_voice.lower() not in voice_lookup_lower:
return False, base_voice
return True, None
else:
# Single voice - case-insensitive comparison
if voice_name.lower() not in voice_lookup_lower:
return False, voice_name
return True, None
def split_text_by_voice_markers(text, default_voice):
"""Split text by voice markers, returning list of (voice, text) tuples.
IMPORTANT: Returns the last voice used so it can persist across chapters.
Voice names are normalized to lowercase to match VOICES_INTERNAL.
Args:
text: Text potentially containing <<VOICE:name>> markers
default_voice: Voice to use if no markers found or before first marker
Returns:
Tuple of (segments_list, last_voice_used, valid_count, invalid_count):
- segments_list: List of (voice_name, segment_text) tuples
- last_voice_used: The voice that should continue into next chapter
- valid_count: Number of valid voice markers processed
- invalid_count: Number of invalid voice markers skipped
"""
from abogen.constants import VOICES_INTERNAL
voice_splits = list(_VOICE_MARKER_SEARCH_PATTERN.finditer(text))
if not voice_splits:
# No voice markers, return entire text with default voice
return [(default_voice, text)], default_voice, 0, 0
segments = []
current_voice = default_voice
valid_markers = 0
invalid_markers = 0
# Text before first marker uses default voice
first_start = voice_splits[0].start()
if first_start > 0:
intro_text = text[:first_start].strip()
if intro_text:
segments.append((current_voice, intro_text))
# Process each voice marker
for idx, match in enumerate(voice_splits):
voice_name = match.group(1).strip()
start = match.end()
end = voice_splits[idx + 1].start() if idx + 1 < len(voice_splits) else len(text)
segment_text = text[start:end].strip()
# Validate voice name
is_valid, invalid_voice = validate_voice_name(voice_name)
if is_valid:
# Normalize to lowercase to match canonical form
# Handle both single voices and formulas
if "*" in voice_name:
# Normalize each voice in the formula
normalized_parts = []
for part in voice_name.split("+"):
part = part.strip()
if "*" in part:
voice_part, weight = part.split("*", 1)
# Find the canonical (lowercase) voice name
voice_part_lower = voice_part.strip().lower()
canonical_voice = next(
(v for v in VOICES_INTERNAL if v.lower() == voice_part_lower),
voice_part.strip()
)
normalized_parts.append(f"{canonical_voice}*{weight.strip()}")
current_voice = " + ".join(normalized_parts)
else:
# Find the canonical (lowercase) voice name
voice_name_lower = voice_name.lower()
current_voice = next(
(v for v in VOICES_INTERNAL if v.lower() == voice_name_lower),
voice_name
)
valid_markers += 1
else:
# Invalid voice - stay with previous voice
invalid_markers += 1
if segment_text:
segments.append((current_voice, segment_text))
# Return segments, last voice, and counts
return segments, current_voice, valid_markers, invalid_markers
File diff suppressed because it is too large Load Diff
+291
View File
@@ -0,0 +1,291 @@
from __future__ import annotations
import ast
from dataclasses import dataclass
import logging
import math
import os
import re
from typing import Any, Iterable, Iterator, Optional
import numpy as np
logger = logging.getLogger(__name__)
DEFAULT_SUPERTONIC_VOICES = ("M1", "M2", "M3", "M4", "M5", "F1", "F2", "F3", "F4", "F5")
SUPERTONIC_AVAILABLE_LANGS = [
"en", "ko", "ja", "ar", "bg", "cs", "da", "de", "el",
"es", "et", "fi", "fr", "hi", "hr", "hu", "id", "it",
"lt", "lv", "nl", "pl", "pt", "ro", "ru", "sk", "sl",
"sv", "tr", "uk", "vi", "na",
]
@dataclass
class SupertonicSegment:
graphemes: str
audio: np.ndarray
def _ensure_float32_mono(wav: Any) -> np.ndarray:
arr = np.asarray(wav, dtype="float32")
if arr.ndim == 2:
# (n, 1) or (1, n) or (n, channels)
if arr.shape[0] == 1 and arr.shape[1] > 1:
arr = arr.reshape(-1)
else:
arr = arr[:, 0]
return arr.reshape(-1)
def _resample_linear(audio: np.ndarray, src_rate: int, dst_rate: int) -> np.ndarray:
if src_rate == dst_rate:
return audio
if audio.size == 0:
return audio
ratio = dst_rate / float(src_rate)
new_len = int(round(audio.size * ratio))
if new_len <= 1:
return np.zeros(0, dtype="float32")
x_old = np.linspace(0.0, 1.0, num=audio.size, endpoint=False)
x_new = np.linspace(0.0, 1.0, num=new_len, endpoint=False)
return np.interp(x_new, x_old, audio).astype("float32", copy=False)
def _split_text(
text: str, *, split_pattern: Optional[str], max_chunk_length: int
) -> list[str]:
stripped = (text or "").strip()
if not stripped:
return []
parts: list[str]
if split_pattern:
try:
parts = [p.strip() for p in re.split(split_pattern, stripped) if p.strip()]
except re.error:
parts = [stripped]
else:
parts = [stripped]
# Enforce max length by hard-splitting long parts.
result: list[str] = []
for part in parts:
if len(part) <= max_chunk_length:
result.append(part)
continue
start = 0
while start < len(part):
end = min(len(part), start + max_chunk_length)
# Try to split at whitespace.
if end < len(part):
ws = part.rfind(" ", start, end)
if ws > start + 40:
end = ws
chunk = part[start:end].strip()
if chunk:
result.append(chunk)
start = end
return result
_UNSUPPORTED_CHARS_RE = re.compile(
r"unsupported character\(s\):\s*(\[[^\]]*\])", re.IGNORECASE
)
def _parse_unsupported_characters(error: BaseException) -> list[str]:
"""Best-effort extraction of unsupported characters from Supertonic errors."""
message = " ".join(
str(part) for part in getattr(error, "args", ()) if part is not None
) or str(error)
match = _UNSUPPORTED_CHARS_RE.search(message)
if not match:
return []
raw = match.group(1)
try:
value = ast.literal_eval(raw)
except Exception:
return []
if isinstance(value, (list, tuple)):
out: list[str] = []
for item in value:
if item is None:
continue
s = str(item)
if s:
out.append(s)
return out
if isinstance(value, str) and value:
return [value]
return []
def _remove_unsupported_characters(text: str, unsupported: Iterable[str]) -> str:
result = text
for item in unsupported:
if not item:
continue
result = result.replace(item, "")
return result
def _configure_supertonic_gpu() -> None:
"""Patch supertonic's config to enable GPU acceleration if available."""
try:
import onnxruntime as ort
available = ort.get_available_providers()
# Use CUDA if available, skip TensorRT (requires extra libs not always present)
# TensorrtExecutionProvider may be listed as available but fail at runtime
# if TensorRT libraries (libnvinfer.so) are not installed
providers = []
if "CUDAExecutionProvider" in available:
providers.append("CUDAExecutionProvider")
providers.append("CPUExecutionProvider")
# Patch supertonic's config and loader before TTS import
# We must patch both because loader imports the value at module load time
import supertonic.config as supertonic_config
import supertonic.loader as supertonic_loader
supertonic_config.DEFAULT_ONNX_PROVIDERS = providers
supertonic_loader.DEFAULT_ONNX_PROVIDERS = providers
logger.info("Supertonic ONNX providers configured: %s", providers)
except Exception as exc:
logger.warning("Could not configure supertonic GPU providers: %s", exc)
SUPERTONIC_MAX_CHUNK_LENGTH = 500
class SupertonicPipeline:
"""Minimal adapter that mimics Kokoro's pipeline iteration interface."""
def __init__(
self,
*,
sample_rate: int,
auto_download: bool = True,
total_steps: int = 5,
max_chunk_length: int = SUPERTONIC_MAX_CHUNK_LENGTH,
lang: str = "en",
intra_op_num_threads: Optional[int] = None,
) -> None:
self.sample_rate = int(sample_rate)
self.total_steps = int(total_steps)
self.max_chunk_length = int(max_chunk_length)
self.lang = str(lang or "en")
# Configure GPU providers before importing TTS
_configure_supertonic_gpu()
try:
from supertonic import TTS # type: ignore[import-not-found]
except Exception as exc: # pragma: no cover
raise RuntimeError(
"Supertonic is not installed. Install it with `pip install supertonic`."
) from exc
threads = intra_op_num_threads if intra_op_num_threads is not None else os.cpu_count()
self._tts = TTS(auto_download=auto_download, intra_op_num_threads=threads)
def __call__(
self,
text: str,
*,
voice: str,
speed: float,
split_pattern: Optional[str] = None,
total_steps: Optional[int] = None,
lang: Optional[str] = None,
) -> Iterator[SupertonicSegment]:
voice_name = (voice or "").strip() or "M1"
steps = int(total_steps) if total_steps is not None else self.total_steps
steps = max(2, min(15, steps))
speed_value = float(speed) if speed is not None else 1.0
speed_value = max(0.7, min(2.0, speed_value))
language = str(lang or self.lang or "en")
style = self._tts.get_voice_style(voice_name=voice_name)
chunks = _split_text(
text, split_pattern=split_pattern, max_chunk_length=self.max_chunk_length
)
for chunk in chunks:
chunk_to_speak = chunk
removed: set[str] = set()
last_exc: Exception | None = None
# Supertonic can raise ValueError for unsupported characters; strip and retry.
for attempt in range(3):
try:
wav, duration = self._tts.synthesize(
text=chunk_to_speak,
voice_style=style,
lang=language,
total_steps=steps,
speed=speed_value,
max_chunk_length=self.max_chunk_length,
silence_duration=0.0,
verbose=False,
)
break
except ValueError as exc:
last_exc = exc
unsupported = _parse_unsupported_characters(exc)
if not unsupported:
raise
removed.update(unsupported)
sanitized = _remove_unsupported_characters(
chunk_to_speak, unsupported
).strip()
# If we didn't change anything, don't loop forever.
if sanitized == chunk_to_speak.strip():
raise
chunk_to_speak = sanitized
if not chunk_to_speak:
logger.warning(
"Supertonic: dropped a chunk after removing unsupported characters: %s",
sorted(removed),
)
break
if attempt == 0:
logger.warning(
"Supertonic: removed unsupported characters %s and retried.",
sorted(removed),
)
else:
# Exhausted retries.
assert last_exc is not None
raise last_exc
if not chunk_to_speak:
continue
audio = _ensure_float32_mono(wav)
# If duration is present, infer the source sample rate and resample if needed.
src_rate = self.sample_rate
try:
dur = float(duration)
if dur > 0 and audio.size > 0:
inferred = int(round(audio.size / dur))
if 8000 <= inferred <= 96000:
src_rate = inferred
except Exception:
pass
if src_rate != self.sample_rate:
audio = _resample_linear(audio, src_rate, self.sample_rate)
yield SupertonicSegment(graphemes=chunk_to_speak, audio=audio)
+248 -73
View File
@@ -1,31 +1,52 @@
import os
import sys
import json
import warnings
import logging
import os
import platform
import re
import shutil
import subprocess
import re
import sys
import warnings
from threading import Thread
from typing import Dict, Optional
from functools import lru_cache
from dotenv import load_dotenv, find_dotenv
def _load_environment() -> None:
explicit_path = os.environ.get("ABOGEN_ENV_FILE")
if explicit_path:
load_dotenv(explicit_path, override=False)
return
dotenv_path = find_dotenv(usecwd=True)
if dotenv_path:
load_dotenv(dotenv_path, override=False)
_load_environment()
warnings.filterwarnings("ignore")
# Pre-compile frequently used regex patterns for better performance
_WHITESPACE_PATTERN = re.compile(r"[^\S\n]+")
_MULTIPLE_NEWLINES_PATTERN = re.compile(r"\n{3,}")
_SINGLE_NEWLINE_PATTERN = re.compile(r"(?<!\n)\n(?!\n)")
_CHAPTER_MARKER_PATTERN = re.compile(r"<<CHAPTER_MARKER:.*?>>")
_METADATA_PATTERN = re.compile(r"<<METADATA_[^:]+:[^>]*>>")
def detect_encoding(file_path):
import chardet
import charset_normalizer
try:
import chardet # type: ignore[import-not-found]
except ImportError: # pragma: no cover - optional dependency
chardet = None # type: ignore[assignment]
try:
import charset_normalizer # type: ignore[import-not-found]
except ImportError: # pragma: no cover - optional dependency
charset_normalizer = None # type: ignore[assignment]
with open(file_path, "rb") as f:
raw_data = f.read()
detected_encoding = None
for detectors in (charset_normalizer, chardet):
if detectors is None:
continue
try:
result = detectors.detect(raw_data)["encoding"]
except Exception:
@@ -84,67 +105,214 @@ def get_resource_path(package, resource):
def get_version():
"""Return the current version of the application."""
try:
with open(get_resource_path("/", "VERSION"), "r") as f:
version_path = get_resource_path("/", "VERSION")
if not version_path:
raise FileNotFoundError("VERSION resource missing")
with open(version_path, "r") as f:
return f.read().strip()
except Exception:
return "Unknown"
# Define config path
def get_user_config_path():
def ensure_directory(path):
resolved = os.path.abspath(os.path.expanduser(str(path)))
os.makedirs(resolved, exist_ok=True)
return resolved
@lru_cache(maxsize=1)
def get_user_settings_dir():
override = os.environ.get("ABOGEN_SETTINGS_DIR")
if override:
return ensure_directory(override)
data_root = os.environ.get("ABOGEN_DATA") or os.environ.get("ABOGEN_DATA_DIR")
if data_root:
try:
return ensure_directory(os.path.join(data_root, "settings"))
except OSError:
pass
data_mount = "/data"
if os.path.isdir(data_mount):
try:
return ensure_directory(os.path.join(data_mount, "settings"))
except OSError:
pass
from platformdirs import user_config_dir
# TODO Config directory is changed for Linux and MacOS. But if old config exists, it will be used.
# On nonWindows, prefer ~/.config/abogen if it already exists
if platform.system() != "Windows":
custom_dir = os.path.join(os.path.expanduser("~"), ".config", "abogen")
if os.path.exists(custom_dir):
config_dir = custom_dir
else:
config_dir = user_config_dir(
"abogen", appauthor=False, roaming=True, ensure_exists=True
)
else:
# Windows and fallback case
config_dir = user_config_dir(
"abogen", appauthor=False, roaming=True, ensure_exists=True
)
legacy_dir = os.path.join(os.path.expanduser("~"), ".config", "abogen")
if os.path.exists(legacy_dir):
return ensure_directory(legacy_dir)
return os.path.join(config_dir, "config.json")
config_dir = user_config_dir(
"abogen", appauthor=False, roaming=True, ensure_exists=True
)
return ensure_directory(config_dir)
def get_user_config_path():
return os.path.join(get_user_settings_dir(), "config.json")
# Define cache path
def get_user_cache_path(folder=None):
from platformdirs import user_cache_dir
@lru_cache(maxsize=1)
def get_user_cache_root():
logger = logging.getLogger(__name__)
cache_dir = user_cache_dir(
"abogen", appauthor=False, opinion=True, ensure_exists=True
def _try_paths(*paths):
last_error = None
for candidate in paths:
if not candidate:
continue
try:
return ensure_directory(candidate)
except OSError as exc:
last_error = exc
logger.debug("Unable to use cache directory %s: %s", candidate, exc)
if last_error is not None:
raise last_error
def _configure_cache_env(root: Optional[str]) -> None:
temp_root = None
if root:
try:
temp_root = ensure_directory(root)
except OSError:
temp_root = None
home_dir = os.environ.get("HOME")
if not home_dir:
home_dir = ensure_directory(os.path.join("/tmp", "abogen-home"))
os.environ["HOME"] = home_dir
else:
home_dir = ensure_directory(home_dir)
cache_base = os.environ.get("XDG_CACHE_HOME")
if cache_base:
cache_base = ensure_directory(cache_base)
elif temp_root:
cache_base = temp_root
os.environ["XDG_CACHE_HOME"] = cache_base
else:
cache_base = ensure_directory(os.path.join(home_dir, ".cache"))
os.environ["XDG_CACHE_HOME"] = cache_base
hf_cache = os.environ.get("HF_HOME")
if hf_cache:
hf_cache = ensure_directory(hf_cache)
elif temp_root:
hf_cache = ensure_directory(os.path.join(temp_root, "huggingface"))
os.environ["HF_HOME"] = hf_cache
else:
hf_cache = ensure_directory(os.path.join(cache_base, "huggingface"))
os.environ["HF_HOME"] = hf_cache
for env_var in ("HUGGINGFACE_HUB_CACHE", "TRANSFORMERS_CACHE"):
os.environ.setdefault(env_var, hf_cache)
os.environ.setdefault("ABOGEN_INTERNAL_CACHE_ROOT", cache_base)
cache_root: Optional[str] = None
override = os.environ.get("ABOGEN_TEMP_DIR")
if override:
try:
cache_root = ensure_directory(override)
except OSError as exc:
logger.warning("ABOGEN_TEMP_DIR=%s is not writable: %s", override, exc)
if cache_root is None:
from platformdirs import user_cache_dir
default_cache = user_cache_dir("abogen", appauthor=False, opinion=True)
data_root = os.environ.get("ABOGEN_DATA") or os.environ.get("ABOGEN_DATA_DIR")
fallback_paths = [
default_cache,
os.path.join(data_root, "cache") if data_root else None,
"/data/cache",
"/tmp/abogen-cache",
]
try:
cache_root = _try_paths(*fallback_paths)
except OSError:
# Final safety net attempt a tmp directory unique to this process.
tmp_candidate = os.path.join("/tmp", f"abogen-cache-{os.getpid()}")
logger.warning("Falling back to temp cache directory %s", tmp_candidate)
cache_root = ensure_directory(tmp_candidate)
if cache_root is None:
raise RuntimeError("Unable to determine cache directory")
_configure_cache_env(cache_root)
return cache_root
def get_internal_cache_root():
root = os.environ.get("ABOGEN_INTERNAL_CACHE_ROOT") or os.environ.get(
"XDG_CACHE_HOME"
)
if root:
return ensure_directory(root)
home_dir = os.environ.get("HOME") or os.path.join("/tmp", "abogen-home")
home_dir = ensure_directory(home_dir)
return ensure_directory(os.path.join(home_dir, ".cache"))
def get_internal_cache_path(folder=None):
base = get_internal_cache_root()
if folder:
cache_dir = os.path.join(cache_dir, folder)
# Ensure the directory exists
os.makedirs(cache_dir, exist_ok=True)
return cache_dir
return ensure_directory(os.path.join(base, folder))
return base
_sleep_procs = {"Darwin": None, "Linux": None} # Store sleep prevention processes
def get_user_cache_path(folder=None):
base = get_user_cache_root()
if folder:
return ensure_directory(os.path.join(base, folder))
return base
@lru_cache(maxsize=1)
def get_user_output_root():
override = os.environ.get("ABOGEN_OUTPUT_DIR") or os.environ.get(
"ABOGEN_OUTPUT_ROOT"
)
if override:
return ensure_directory(override)
return ensure_directory(os.path.join(get_user_cache_root(), "outputs"))
def get_user_output_path(folder=None):
base = get_user_output_root()
if folder:
return ensure_directory(os.path.join(base, folder))
return base
_sleep_procs: Dict[str, Optional[subprocess.Popen[str]]] = {
"Darwin": None,
"Linux": None,
} # Store sleep prevention processes
def clean_text(text, *args, **kwargs):
# Load replace_single_newlines from config
cfg = load_config()
replace_single_newlines = cfg.get("replace_single_newlines", True)
replace_single_newlines = cfg.get("replace_single_newlines", False)
# Collapse all whitespace (excluding newlines) into single spaces per line and trim edges
# Use pre-compiled pattern for better performance
lines = [_WHITESPACE_PATTERN.sub(" ", line).strip() for line in text.splitlines()]
lines = [re.sub(r"[^\S\n]+", " ", line).strip() for line in text.splitlines()]
text = "\n".join(lines)
# Standardize paragraph breaks (multiple newlines become exactly two) and trim overall whitespace
# Use pre-compiled pattern for better performance
text = _MULTIPLE_NEWLINES_PATTERN.sub("\n\n", text).strip()
text = re.sub(r"\n{3,}", "\n\n", text).strip()
# Optionally replace single newlines with spaces, but preserve double newlines
if replace_single_newlines:
# Use pre-compiled pattern for better performance
text = _SINGLE_NEWLINE_PATTERN.sub(" ", text)
text = re.sub(r"(?<!\n)\n(?!\n)", " ", text)
return text
@@ -167,6 +335,11 @@ def create_process(cmd, stdin=None, text=True, capture_output=False):
# Determine shell usage: use shell only for string commands
use_shell = isinstance(cmd, str)
if use_shell:
logger.warning(
"Security Warning: create_process called with string command. Prefer using a list of arguments to avoid shell injection risks."
)
kwargs = {
"shell": use_shell,
"stdout": subprocess.PIPE,
@@ -189,11 +362,14 @@ def create_process(cmd, stdin=None, text=True, capture_output=False):
kwargs["stdin"] = stdin
if platform.system() == "Windows":
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = subprocess.SW_HIDE
startupinfo = subprocess.STARTUPINFO() # type: ignore[attr-defined]
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW # type: ignore[attr-defined]
startupinfo.wShowWindow = subprocess.SW_HIDE # type: ignore[attr-defined]
kwargs.update(
{"startupinfo": startupinfo, "creationflags": subprocess.CREATE_NO_WINDOW}
{
"startupinfo": startupinfo,
"creationflags": subprocess.CREATE_NO_WINDOW, # type: ignore[attr-defined]
}
)
# Print the command being executed
@@ -253,27 +429,23 @@ def save_config(config):
def calculate_text_length(text):
# Use pre-compiled patterns for better performance
# Ignore chapter markers and metadata patterns in a single pass
text = _CHAPTER_MARKER_PATTERN.sub("", text)
text = _METADATA_PATTERN.sub("", text)
# Ignore newlines and leading/trailing spaces
text = text.replace("\n", "").strip()
# Ignore chapter markers
text = re.sub(r"<<CHAPTER_MARKER:.*?>>", "", text)
# Ignore metadata patterns
text = re.sub(r"<<METADATA_[^:]+:[^>]*>>", "", text)
# Ignore newlines
text = text.replace("\n", "")
# Ignore leading/trailing spaces
text = text.strip()
# Calculate character count
char_count = len(text)
return char_count
def get_gpu_acceleration(enabled):
"""
Check GPU acceleration availability.
Note: On Windows, torch DLLs must be pre-loaded in main.py before PyQt6
to avoid DLL initialization errors.
"""
try:
import torch
from torch.cuda import is_available as cuda_available
import torch # type: ignore[import-not-found]
from torch.cuda import is_available as cuda_available # type: ignore[import-not-found]
if not enabled:
return "GPU available but using CPU.", False
@@ -311,7 +483,7 @@ def prevent_sleep_start():
if system == "Windows":
import ctypes
ctypes.windll.kernel32.SetThreadExecutionState(
ctypes.windll.kernel32.SetThreadExecutionState( # type: ignore[attr-defined]
0x80000000 | 0x00000001 | 0x00000040
)
elif system == "Darwin":
@@ -345,18 +517,21 @@ def prevent_sleep_end():
if system == "Windows":
import ctypes
ctypes.windll.kernel32.SetThreadExecutionState(0x80000000) # ES_CONTINUOUS
elif system in ("Darwin", "Linux") and _sleep_procs[system]:
try:
_sleep_procs[system].terminate()
_sleep_procs[system] = None
except Exception:
pass
ctypes.windll.kernel32.SetThreadExecutionState(0x80000000) # type: ignore[attr-defined]
elif system in ("Darwin", "Linux"):
proc = _sleep_procs.get(system)
if proc:
try:
proc.terminate()
except Exception:
pass
finally:
_sleep_procs[system] = None
def load_numpy_kpipeline():
import numpy as np
from kokoro import KPipeline
from kokoro import KPipeline # type: ignore[import-not-found]
return np, KPipeline
+145
View File
@@ -0,0 +1,145 @@
from __future__ import annotations
import os
import threading
from typing import Callable, Dict, Iterable, Optional, Set, Tuple
try: # pragma: no cover - optional dependency guard
from huggingface_hub import hf_hub_download # type: ignore
from huggingface_hub.utils import LocalEntryNotFoundError # type: ignore
except Exception: # pragma: no cover - import fallback
hf_hub_download = None # type: ignore[assignment]
LocalEntryNotFoundError = None # type: ignore[assignment]
if LocalEntryNotFoundError is None: # pragma: no cover - fallback for tests
class LocalEntryNotFoundError(Exception):
pass
from abogen.constants import VOICES_INTERNAL
_CACHE_LOCK = threading.Lock()
_CACHED_VOICES: Set[str] = set()
_BOOTSTRAP_LOCK = threading.Lock()
_BOOTSTRAPPED = False
def _normalize_targets(voices: Optional[Iterable[str]]) -> Set[str]:
if not voices:
return set(VOICES_INTERNAL)
normalized: Set[str] = set()
for voice in voices:
if not voice:
continue
voice_id = str(voice).strip()
if not voice_id:
continue
if voice_id in VOICES_INTERNAL:
normalized.add(voice_id)
return normalized
def ensure_voice_assets(
voices: Optional[Iterable[str]] = None,
*,
repo_id: str = "hexgrad/Kokoro-82M",
cache_dir: Optional[str] = None,
on_progress: Optional[Callable[[str], None]] = None,
) -> Tuple[Set[str], Dict[str, str]]:
"""Ensure Kokoro voice weight files are present locally.
Returns a tuple of (downloaded voices, errors) where errors maps the
voice id to the underlying exception message.
"""
if hf_hub_download is None:
raise RuntimeError("huggingface_hub is required to cache voices")
effective_cache_dir = cache_dir
if effective_cache_dir is None:
env_cache_dir = os.environ.get("ABOGEN_VOICE_CACHE_DIR", "").strip()
effective_cache_dir = env_cache_dir or None
targets = _normalize_targets(voices)
if not targets:
return set(), {}
with _CACHE_LOCK:
missing = [voice for voice in targets if voice not in _CACHED_VOICES]
downloaded: Set[str] = set()
errors: Dict[str, str] = {}
for voice_id in missing:
if on_progress:
on_progress(f"Fetching voice asset '{voice_id}'")
try:
downloaded_flag = _ensure_single_voice_asset(
voice_id,
repo_id=repo_id,
cache_dir=effective_cache_dir,
)
except Exception as exc: # pragma: no cover - network variance
errors[voice_id] = str(exc)
continue
if downloaded_flag:
downloaded.add(voice_id)
with _CACHE_LOCK:
_CACHED_VOICES.add(voice_id)
return downloaded, errors
def bootstrap_voice_cache(
voices: Optional[Iterable[str]] = None,
*,
repo_id: str = "hexgrad/Kokoro-82M",
cache_dir: Optional[str] = None,
on_progress: Optional[Callable[[str], None]] = None,
) -> Tuple[Set[str], Dict[str, str]]:
"""Ensure voices are cached once per process.
Subsequent calls are no-ops and return empty structures.
"""
global _BOOTSTRAPPED
with _BOOTSTRAP_LOCK:
if _BOOTSTRAPPED:
return set(), {}
downloaded, errors = ensure_voice_assets(
voices,
repo_id=repo_id,
cache_dir=cache_dir,
on_progress=on_progress,
)
_BOOTSTRAPPED = True
return downloaded, errors
def _ensure_single_voice_asset(
voice_id: str,
*,
repo_id: str,
cache_dir: Optional[str],
) -> bool:
if hf_hub_download is None:
raise RuntimeError("huggingface_hub is required to cache voices")
filename = f"voices/{voice_id}.pt"
common_kwargs = {
"repo_id": repo_id,
"filename": filename,
}
if cache_dir is not None:
common_kwargs["cache_dir"] = cache_dir
try:
hf_hub_download(local_files_only=True, **common_kwargs)
return False
except LocalEntryNotFoundError:
pass
hf_hub_download(resume_download=True, **common_kwargs)
return True
File diff suppressed because it is too large Load Diff
+46 -22
View File
@@ -1,4 +1,6 @@
import re
from typing import List, Tuple
from abogen.constants import VOICES_INTERNAL
@@ -15,38 +17,56 @@ def get_new_voice(pipeline, formula, use_gpu):
raise ValueError(f"Failed to create voice: {str(e)}")
# Parse the formula and get the combined voice tensor
def parse_voice_formula(pipeline, formula):
if not formula.strip():
def parse_formula_terms(formula: str) -> List[Tuple[str, float]]:
if not formula or not formula.strip():
raise ValueError("Empty voice formula")
# Initialize the weighted sum
weighted_sum = None
total_weight = calculate_sum_from_formula(formula)
# Split the formula into terms
voices = formula.split("+")
for term in voices:
# Parse each term (format: "voice_name*0.333")
voice_name, weight = term.strip().split("*")
weight = float(weight.strip())
# normalize the weight
weight /= total_weight if total_weight > 0 else 1.0
terms: List[Tuple[str, float]] = []
for segment in formula.split("+"):
part = segment.strip()
if not part:
continue
if "*" not in part:
raise ValueError("Each component must be in the form voice*weight")
voice_name, raw_weight = part.split("*", 1)
voice_name = voice_name.strip()
# Get the voice tensor
if voice_name not in VOICES_INTERNAL:
raise ValueError(f"Unknown voice: {voice_name}")
try:
weight = float(raw_weight.strip())
except ValueError as exc:
raise ValueError(f"Invalid weight for {voice_name}") from exc
if weight <= 0:
raise ValueError(f"Weight for {voice_name} must be positive")
terms.append((voice_name, weight))
if not terms:
raise ValueError("Voice weights must sum to a positive value")
return terms
def parse_voice_formula(pipeline, formula):
terms = parse_formula_terms(formula)
total_weight = sum(weight for _, weight in terms)
if total_weight <= 0:
raise ValueError("Voice weights must sum to a positive value")
weighted_sum = None
for voice_name, weight in terms:
normalized_weight = weight / total_weight if total_weight > 0 else weight
voice_tensor = pipeline.load_single_voice(voice_name)
# Add to weighted sum
if weighted_sum is None:
weighted_sum = weight * voice_tensor
weighted_sum = normalized_weight * voice_tensor
else:
weighted_sum += weight * voice_tensor
weighted_sum += normalized_weight * voice_tensor
if weighted_sum is None:
raise ValueError("Voice formula produced no components")
return weighted_sum
@@ -55,3 +75,7 @@ def calculate_sum_from_formula(formula):
weights = re.findall(r"\* *([\d.]+)", formula)
total_sum = sum(float(weight) for weight in weights)
return total_sum
def extract_voice_ids(formula: str) -> List[str]:
return [voice for voice, _ in parse_formula_terms(formula)]
+171 -1
View File
@@ -1,5 +1,9 @@
import os
import json
import os
from typing import Any, Dict, Iterable, List, Tuple
from abogen.constants import VOICES_INTERNAL
from abogen.tts_supertonic import DEFAULT_SUPERTONIC_VOICES
from abogen.utils import get_user_config_path
@@ -57,3 +61,169 @@ def export_profiles(export_path):
profiles = load_profiles()
with open(export_path, "w", encoding="utf-8") as f:
json.dump({"abogen_voice_profiles": profiles}, f, indent=2)
def serialize_profiles() -> Dict[str, Dict[str, Iterable[Tuple[str, float]]]]:
"""Return profiles in canonical dictionary form."""
return load_profiles()
def _normalize_supertonic_voice(value: Any) -> str:
raw = str(value or "").strip().upper()
return raw if raw in DEFAULT_SUPERTONIC_VOICES else "M1"
def _coerce_supertonic_steps(value: Any) -> int:
try:
steps = int(value)
except (TypeError, ValueError):
return 5
return max(2, min(15, steps))
def _coerce_supertonic_speed(value: Any) -> float:
try:
speed = float(value)
except (TypeError, ValueError):
return 1.0
return max(0.7, min(2.0, speed))
def normalize_profile_entry(entry: Any) -> Dict[str, Any]:
"""Normalize a stored profile entry.
Backwards compatible:
- Legacy Kokoro-only entries: {language, voices}
- New entries: include provider.
"""
if not isinstance(entry, dict):
return {}
provider = str(entry.get("provider") or "kokoro").strip().lower()
if provider not in {"kokoro", "supertonic"}:
provider = "kokoro"
language = str(entry.get("language") or "a").strip().lower() or "a"
if provider == "supertonic":
return {
"provider": "supertonic",
"language": language,
"voice": _normalize_supertonic_voice(
entry.get("voice") or entry.get("voice_name") or entry.get("name")
),
"total_steps": _coerce_supertonic_steps(
entry.get("total_steps")
or entry.get("supertonic_total_steps")
or entry.get("quality")
),
"speed": _coerce_supertonic_speed(
entry.get("speed") or entry.get("supertonic_speed")
),
}
voices = _normalize_voice_entries(entry.get("voices", []))
if not voices:
return {}
return {
"provider": "kokoro",
"language": language,
"voices": voices,
}
def _normalize_voice_entries(entries: Iterable) -> List[Tuple[str, float]]:
normalized: List[Tuple[str, float]] = []
for item in entries or []:
if isinstance(item, dict):
voice = item.get("id") or item.get("voice")
weight = item.get("weight")
elif isinstance(item, (list, tuple)) and len(item) >= 2:
voice, weight = item[0], item[1]
else:
continue
if voice not in VOICES_INTERNAL:
continue
if weight is None:
continue
try:
weight_val = float(weight)
except (TypeError, ValueError):
continue
if weight_val <= 0:
continue
normalized.append((voice, weight_val))
return normalized
def normalize_voice_entries(entries: Iterable) -> List[Tuple[str, float]]:
"""Public helper to normalize voice-weight pairs from arbitrary payloads."""
return _normalize_voice_entries(entries)
def save_profile(name: str, *, language: str, voices: Iterable) -> None:
"""Persist a single profile after validating its data."""
name = (name or "").strip()
if not name:
raise ValueError("Profile name is required")
normalized = _normalize_voice_entries(voices)
if not normalized:
raise ValueError("At least one voice with a weight above zero is required")
if not language:
language = "a"
profiles = load_profiles()
profiles[name] = {"provider": "kokoro", "language": language, "voices": normalized}
save_profiles(profiles)
def remove_profile(name: str) -> None:
delete_profile(name)
def import_profiles_data(data: Dict, *, replace_existing: bool = False) -> List[str]:
"""Merge profiles from a dictionary structure and persist them.
Returns the list of profile names that were added or updated.
"""
if not isinstance(data, dict):
raise ValueError("Invalid profile payload")
if "abogen_voice_profiles" in data:
data = data["abogen_voice_profiles"]
if not isinstance(data, dict):
raise ValueError("Invalid profile payload")
current = load_profiles()
updated: List[str] = []
for name, entry in data.items():
normalized = normalize_profile_entry(entry)
if not normalized:
continue
if name in current and not replace_existing:
# skip duplicates unless explicit replacement is requested
continue
current[name] = normalized
updated.append(name)
if updated:
save_profiles(current)
return updated
def export_profiles_payload(names: Iterable[str] | None = None) -> Dict[str, Dict]:
"""Return profiles limited to the provided names for download/export."""
profiles = load_profiles()
if names is None:
subset = profiles
else:
subset = {name: profiles[name] for name in names if name in profiles}
return {"abogen_voice_profiles": subset}
+74
View File
@@ -0,0 +1,74 @@
FROM nvidia/cuda:12.6.3-cudnn-runtime-ubuntu22.04
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
VIRTUAL_ENV=/opt/venv \
PATH=/opt/venv/bin:$PATH
ARG TORCH_INDEX_URL=https://download.pytorch.org/whl/cu126
ARG TORCH_VERSION=
ARG USE_GPU=true
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
python3 \
python3-venv \
python3-pip \
ffmpeg \
libsndfile1 \
libgl1 \
libglib2.0-0 \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN python3 -m venv "$VIRTUAL_ENV"
WORKDIR /app
COPY pyproject.toml README.md ./
COPY abogen ./abogen
RUN pip install --upgrade pip \
&& if [ -n "$TORCH_VERSION" ]; then \
pip install torch=="$TORCH_VERSION" torchvision=="$TORCH_VERSION" torchaudio=="$TORCH_VERSION" --index-url "$TORCH_INDEX_URL"; \
else \
pip install torch torchvision torchaudio --index-url "$TORCH_INDEX_URL"; \
fi \
&& pip install --no-cache-dir . \
https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl \
&& pip install --no-cache-dir "mutagen>=1.47.0"
# Install onnxruntime-gpu for CUDA acceleration (supertonic uses ONNX Runtime)
# Set USE_GPU=false to skip this for CPU-only deployments
RUN if [ "$USE_GPU" = "true" ]; then \
pip install --no-cache-dir onnxruntime-gpu; \
fi
ENV ABOGEN_HOST=0.0.0.0 \
ABOGEN_PORT=8808
EXPOSE 8808
VOLUME ["/data"]
ENV ABOGEN_UPLOAD_ROOT=/data/uploads \
ABOGEN_OUTPUT_ROOT=/data/outputs \
ABOGEN_TEMP_DIR=/data/cache \
ABOGEN_VOICE_CACHE_DIR=/data/voice-cache \
HF_HOME=/data/huggingface \
HUGGINGFACE_HUB_CACHE=/data/huggingface/hub
# Copy and setup entrypoint script
COPY abogen/webui/entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
# Create non-root user and setup permissions
RUN useradd -m -u 1000 abogen \
&& mkdir -p /data/uploads /data/outputs /data/cache /data/voice-cache /data/huggingface \
&& chown -R abogen:abogen /data /app
USER abogen
ENTRYPOINT ["/entrypoint.sh"]
CMD ["abogen-web"]
+9
View File
@@ -0,0 +1,9 @@
__all__ = ["create_app"]
def __getattr__(name: str):
if name == "create_app":
from .app import create_app
return create_app
raise AttributeError(name)
+135
View File
@@ -0,0 +1,135 @@
from __future__ import annotations
import atexit
import logging
import os
from pathlib import Path
from typing import Any, Optional
from flask import Flask
from abogen.utils import get_user_cache_path, get_user_output_path, get_user_settings_dir
from .conversion_runner import run_conversion_job
from .service import build_service
class _SuppressSuccessfulAccessFilter(logging.Filter):
"""Filter out successful (HTTP 200) werkzeug access logs."""
def filter(self, record: logging.LogRecord) -> bool: # pragma: no cover - small utility
try:
message = record.getMessage()
except Exception: # pragma: no cover - defensive
return True
# Werkzeug access logs include the status code near the end, e.g.
# "GET /path HTTP/1.1" 200 -
# Treat any 2xx response as success to suppress.
return " 200 " not in message and " 201 " not in message and " 204 " not in message
_access_log_filter_attached = False
def _default_dirs() -> tuple[Path, Path]:
uploads_override = os.environ.get("ABOGEN_UPLOAD_ROOT")
outputs_override = os.environ.get("ABOGEN_OUTPUT_ROOT")
if uploads_override:
uploads = Path(os.path.expanduser(uploads_override)).resolve()
else:
uploads = Path(get_user_cache_path("web/uploads"))
if outputs_override:
outputs = Path(os.path.expanduser(outputs_override)).resolve()
else:
outputs = Path(get_user_output_path("web"))
uploads.mkdir(parents=True, exist_ok=True)
outputs.mkdir(parents=True, exist_ok=True)
return uploads, outputs
def _get_secret_key() -> str:
env_key = os.environ.get("ABOGEN_SECRET_KEY")
if env_key:
return env_key
try:
settings_dir = Path(get_user_settings_dir())
settings_dir.mkdir(parents=True, exist_ok=True)
secret_file = settings_dir / ".secret_key"
if secret_file.exists():
return secret_file.read_text(encoding="utf-8").strip()
key = os.urandom(24).hex()
secret_file.write_text(key, encoding="utf-8")
return key
except Exception:
# Fallback if we can't write to settings dir
return os.urandom(24).hex()
def create_app(config: Optional[dict[str, Any]] = None) -> Flask:
uploads_dir, outputs_dir = _default_dirs()
app = Flask(
__name__,
static_folder="static",
template_folder="templates",
)
base_config = {
"SECRET_KEY": _get_secret_key(),
"UPLOAD_FOLDER": str(uploads_dir),
"OUTPUT_FOLDER": str(outputs_dir),
"MAX_CONTENT_LENGTH": 1024 * 1024 * 400, # 400 MB uploads
}
if config:
base_config.update(config)
app.config.update(base_config)
service = build_service(
runner=run_conversion_job,
output_root=Path(app.config["OUTPUT_FOLDER"]),
uploads_root=Path(app.config["UPLOAD_FOLDER"]),
)
app.extensions["conversion_service"] = service
from abogen.webui.routes import (
main_bp,
jobs_bp,
settings_bp,
voices_bp,
entities_bp,
books_bp,
api_bp,
)
app.register_blueprint(main_bp)
app.register_blueprint(jobs_bp, url_prefix="/jobs")
app.register_blueprint(settings_bp, url_prefix="/settings")
app.register_blueprint(voices_bp, url_prefix="/voices")
app.register_blueprint(entities_bp, url_prefix="/overrides")
app.register_blueprint(books_bp, url_prefix="/find-books")
app.register_blueprint(api_bp, url_prefix="/api")
atexit.register(service.shutdown)
global _access_log_filter_attached
if not _access_log_filter_attached:
logging.getLogger("werkzeug").addFilter(_SuppressSuccessfulAccessFilter())
_access_log_filter_attached = True
return app
def main() -> None:
app = create_app()
host = os.environ.get("ABOGEN_HOST", "0.0.0.0")
port = int(os.environ.get("ABOGEN_PORT", "8808"))
debug = os.environ.get("ABOGEN_DEBUG", "false").lower() == "true"
app.run(host=host, port=port, debug=debug)
if __name__ == "__main__": # pragma: no cover
main()
File diff suppressed because it is too large Load Diff
+251
View File
@@ -0,0 +1,251 @@
from __future__ import annotations
import json
import re
import uuid
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple
import numpy as np
from abogen.debug_tts_samples import MARKER_PREFIX, MARKER_SUFFIX, build_debug_epub, iter_expected_codes
from abogen.kokoro_text_normalization import normalize_for_pipeline
from abogen.normalization_settings import build_apostrophe_config
from abogen.text_extractor import extract_from_path
from abogen.voice_cache import ensure_voice_assets
from abogen.webui.conversion_runner import SAMPLE_RATE, SPLIT_PATTERN, _select_device, _to_float32, _resolve_voice, _spec_to_voice_ids
from abogen.utils import load_numpy_kpipeline
_MARKER_RE = re.compile(re.escape(MARKER_PREFIX) + r"(?P<code>[A-Z0-9_]+)" + re.escape(MARKER_SUFFIX))
@dataclass(frozen=True)
class DebugWavArtifact:
label: str
filename: str
code: Optional[str] = None
text: Optional[str] = None
def _resolve_voice_setting(value: str) -> tuple[str, Optional[str], Optional[str]]:
"""Resolve settings voice strings into a pipeline-ready voice spec.
Supports "profile:<name>" by converting it into a concrete voice formula.
Returns (resolved_voice_spec, profile_name, profile_language).
"""
from abogen.webui.routes.utils.voice import resolve_voice_setting
return resolve_voice_setting(value)
def _load_pipeline(language: str, use_gpu: bool) -> Any:
device = "cpu"
if use_gpu:
device = _select_device()
_np, KPipeline = load_numpy_kpipeline()
return KPipeline(lang_code=language, repo_id="hexgrad/Kokoro-82M", device=device)
def _extract_cases_from_text(text: str) -> List[Tuple[str, str]]:
raw = str(text or "")
matches = list(_MARKER_RE.finditer(raw))
cases: List[Tuple[str, str]] = []
if not matches:
return cases
for idx, match in enumerate(matches):
code = match.group("code")
start = match.end()
end = matches[idx + 1].start() if idx + 1 < len(matches) else len(raw)
snippet = raw[start:end]
# Keep it small and predictable: collapse whitespace.
snippet = " ".join(snippet.strip().split())
cases.append((code, snippet))
return cases
def _spoken_id(code: str) -> str:
# Make IDs pronounceable and stable (avoid reading as a word).
out: List[str] = []
for ch in str(code or ""):
if ch == "_":
out.append(" ")
elif ch.isalnum():
out.append(ch)
else:
out.append(" ")
# Add spaces between alnum to encourage letter-by-letter reading.
spaced = " ".join("".join(out).split())
return spaced
def run_debug_tts_wavs(
*,
output_root: Path,
settings: Mapping[str, Any],
epub_path: Optional[Path] = None,
) -> Dict[str, Any]:
"""Generate WAV artifacts for the debug EPUB samples.
Writes:
- overall.wav: concatenation of all samples
- case_<CODE>.wav: each sample rendered separately
- manifest.json: metadata + file list
"""
output_root = Path(output_root)
output_root.mkdir(parents=True, exist_ok=True)
run_id = uuid.uuid4().hex
run_dir = output_root / "debug" / run_id
run_dir.mkdir(parents=True, exist_ok=True)
if epub_path is None:
epub_path = run_dir / "abogen_debug_samples.epub"
build_debug_epub(epub_path)
else:
epub_path = Path(epub_path)
extraction = extract_from_path(epub_path)
combined_text = extraction.combined_text or "\n\n".join((c.text or "") for c in extraction.chapters)
cases = _extract_cases_from_text(combined_text)
# Prefer the canonical sample catalog for text (EPUB extraction may include headings).
try:
from abogen.debug_tts_samples import DEBUG_TTS_SAMPLES
sample_text_by_code = {sample.code: sample.text for sample in DEBUG_TTS_SAMPLES}
except Exception:
sample_text_by_code = {}
expected = list(iter_expected_codes())
found_codes = {code for code, _ in cases}
missing = [code for code in expected if code not in found_codes]
if missing:
raise RuntimeError(f"Debug EPUB missing expected codes: {', '.join(missing)}")
language = str(settings.get("language") or "a").strip() or "a"
# Kokoro's KPipeline expects short language codes like "a" (American English),
# but older settings may store ISO-like values such as "en".
language_aliases = {
"en": "a",
"en-us": "a",
"en_us": "a",
"en-gb": "b",
"en_gb": "b",
"es": "e",
"es-es": "e",
"fr": "f",
"fr-fr": "f",
"hi": "h",
"it": "i",
"pt": "p",
"pt-br": "p",
"ja": "j",
"jp": "j",
"zh": "z",
"zh-cn": "z",
}
language = language_aliases.get(language.lower(), language)
voice_spec = str(settings.get("default_voice") or "").strip()
use_gpu = bool(settings.get("use_gpu", False))
speed = float(settings.get("default_speed", 1.0) or 1.0)
# Settings may store "profile:<name>" which is not a Kokoro voice ID.
# Resolve it to a concrete voice formula (e.g. "af_heart*0.5+...") so Kokoro
# doesn't attempt to download a non-existent "voices/profile:<name>.pt".
try:
resolved_voice, _profile_name, profile_language = _resolve_voice_setting(voice_spec)
if resolved_voice:
voice_spec = resolved_voice
if profile_language:
language = str(profile_language).strip() or language
except Exception:
# Voice profile resolution is best-effort; fall back to raw voice_spec.
pass
# Best-effort voice caching (only for known Kokoro internal voices).
voice_ids = _spec_to_voice_ids(voice_spec)
if voice_ids:
try:
ensure_voice_assets(voice_ids)
except Exception:
# Network / optional dependency variance; debug runner can still proceed.
pass
pipeline = _load_pipeline(language, use_gpu)
voice_choice = _resolve_voice(pipeline, voice_spec, use_gpu)
apostrophe_config = build_apostrophe_config(settings=settings)
normalization_settings = dict(settings)
artifacts: List[DebugWavArtifact] = []
overall_path = run_dir / "overall.wav"
overall_audio: List[np.ndarray] = []
def synth(text: str, *, apply_normalization: bool = True) -> np.ndarray:
normalized = (
normalize_for_pipeline(
text,
config=apostrophe_config,
settings=normalization_settings,
)
if apply_normalization
else str(text or "")
)
parts: List[np.ndarray] = []
for segment in pipeline(
normalized,
voice=voice_choice,
speed=speed,
split_pattern=SPLIT_PATTERN,
):
audio = _to_float32(getattr(segment, "audio", None))
if audio.size:
parts.append(audio)
if not parts:
return np.zeros(0, dtype="float32")
return np.concatenate(parts).astype("float32", copy=False)
pause_1s = np.zeros(int(1.0 * SAMPLE_RATE), dtype="float32")
between_cases = np.zeros(int(0.35 * SAMPLE_RATE), dtype="float32")
# Per sample
for code, snippet in cases:
snippet = sample_text_by_code.get(code, snippet)
if not snippet:
continue
id_audio = synth(_spoken_id(code), apply_normalization=False)
text_audio = synth(snippet, apply_normalization=True)
audio = np.concatenate([id_audio, pause_1s, text_audio]).astype("float32", copy=False)
filename = f"case_{code}.wav"
path = run_dir / filename
# Write float32 PCM WAV.
import soundfile as sf
sf.write(path, audio, SAMPLE_RATE, subtype="FLOAT")
artifacts.append(DebugWavArtifact(label=f"{code}", filename=filename, code=code, text=snippet))
overall_audio.append(audio)
overall_audio.append(between_cases)
# Overall
if overall_audio:
combined = np.concatenate(overall_audio).astype("float32", copy=False)
else:
combined = np.zeros(0, dtype="float32")
import soundfile as sf
sf.write(overall_path, combined, SAMPLE_RATE, subtype="FLOAT")
artifacts.insert(0, DebugWavArtifact(label="Overall", filename="overall.wav", code=None, text=None))
manifest = {
"run_id": run_id,
"epub": str(epub_path),
"artifacts": [artifact.__dict__ for artifact in artifacts],
"sample_rate": SAMPLE_RATE,
}
(run_dir / "manifest.json").write_text(json.dumps(manifest, indent=2), encoding="utf-8")
return manifest
+36
View File
@@ -0,0 +1,36 @@
#!/bin/bash
# Entrypoint script for abogen container
# Performs CUDA diagnostics and starts the web server
set -e
echo "=== Abogen Container Starting ==="
# Check CUDA availability
if command -v nvidia-smi &> /dev/null; then
echo "NVIDIA Driver detected:"
nvidia-smi --query-gpu=name,driver_version,memory.total,memory.free --format=csv,noheader 2>/dev/null || echo " (nvidia-smi query failed)"
# Check PyTorch CUDA support
python3 -c "
import torch
print(f'PyTorch version: {torch.__version__}')
print(f'CUDA available: {torch.cuda.is_available()}')
if torch.cuda.is_available():
print(f'CUDA version (PyTorch): {torch.version.cuda}')
print(f'GPU count: {torch.cuda.device_count()}')
for i in range(torch.cuda.device_count()):
props = torch.cuda.get_device_properties(i)
print(f' GPU {i}: {props.name} ({props.total_memory // 1024**2} MB)')
else:
print('WARNING: PyTorch cannot access CUDA. Running on CPU.')
" 2>&1 || echo "PyTorch CUDA check failed"
else
echo "No NVIDIA driver detected. Running on CPU."
fi
echo "================================="
echo ""
# Start the application
exec "$@"
+18
View File
@@ -0,0 +1,18 @@
from abogen.webui.routes.main import main_bp
from abogen.webui.routes.jobs import jobs_bp
from abogen.webui.routes.settings import settings_bp
from abogen.webui.routes.voices import voices_bp
from abogen.webui.routes.entities import entities_bp
from abogen.webui.routes.books import books_bp
from abogen.webui.routes.api import api_bp
__all__ = [
"main_bp",
"jobs_bp",
"settings_bp",
"voices_bp",
"entities_bp",
"books_bp",
"api_bp",
]
+680
View File
@@ -0,0 +1,680 @@
from typing import Any, Dict, Mapping, List, Optional
import base64
import uuid
from pathlib import Path
from flask import Blueprint, request, jsonify, send_file, url_for, current_app
from flask.typing import ResponseReturnValue
from abogen.webui.routes.utils.settings import (
load_settings,
load_integration_settings,
coerce_float,
coerce_bool,
audiobookshelf_settings_from_payload,
calibre_settings_from_payload,
)
from abogen.voice_profiles import (
load_profiles,
save_profiles,
delete_profile,
duplicate_profile,
serialize_profiles,
import_profiles_data,
export_profiles_payload,
normalize_profile_entry,
)
from abogen.webui.routes.utils.common import split_profile_spec
from abogen.webui.routes.utils.preview import synthesize_preview, generate_preview_audio
from abogen.webui.routes.utils.voice import formula_from_profile
from abogen.normalization_settings import (
build_llm_configuration,
build_apostrophe_config,
apply_overrides,
)
from abogen.llm_client import list_models, LLMClientError
from abogen.kokoro_text_normalization import normalize_for_pipeline
from abogen.integrations.audiobookshelf import AudiobookshelfClient, AudiobookshelfConfig
from abogen.integrations.calibre_opds import (
CalibreOPDSClient,
CalibreOPDSError,
)
from abogen.webui.routes.utils.service import get_service
from abogen.webui.routes.utils.form import build_pending_job_from_extraction
from abogen.text_extractor import extract_from_path
from werkzeug.utils import secure_filename
api_bp = Blueprint("api", __name__)
# --- Voice Profile Routes ---
@api_bp.get("/voice-profiles")
def api_get_voice_profiles() -> ResponseReturnValue:
profiles = load_profiles()
return jsonify(profiles)
@api_bp.post("/voice-profiles")
def api_save_voice_profile() -> ResponseReturnValue:
payload = request.get_json(force=True, silent=True) or {}
name = str(payload.get("name") or "").strip()
original_name = str(payload.get("originalName") or "").strip() or None
profile = payload.get("profile")
if profile is None:
# Speaker Studio payload format
provider = str(payload.get("provider") or "kokoro").strip().lower()
if provider not in {"kokoro", "supertonic"}:
provider = "kokoro"
if provider == "supertonic":
profile = {
"provider": "supertonic",
"language": str(payload.get("language") or "a").strip().lower() or "a",
"voice": payload.get("voice"),
"total_steps": payload.get("total_steps") or payload.get("supertonic_total_steps"),
"speed": payload.get("speed") or payload.get("supertonic_speed"),
}
else:
profile = {
"provider": "kokoro",
"language": str(payload.get("language") or "a").strip().lower() or "a",
"voices": payload.get("voices") or [],
}
if not name or not profile:
return jsonify({"error": "Name and profile are required"}), 400
profiles = load_profiles()
normalized = normalize_profile_entry(profile)
if not normalized:
return jsonify({"error": "Invalid profile payload"}), 400
if original_name and original_name in profiles and original_name != name:
del profiles[original_name]
profiles[name] = normalized
save_profiles(profiles)
return jsonify({"success": True, "profile": name, "profiles": serialize_profiles()})
@api_bp.delete("/voice-profiles/<path:name>")
def api_delete_voice_profile(name: str) -> ResponseReturnValue:
delete_profile(name)
return jsonify({"success": True, "profiles": serialize_profiles()})
@api_bp.post("/voice-profiles/<path:name>/duplicate")
def api_duplicate_voice_profile(name: str) -> ResponseReturnValue:
payload = request.get_json(force=True, silent=True) or {}
new_name = str(payload.get("name") or "").strip()
if not new_name:
return jsonify({"error": "Name is required"}), 400
duplicate_profile(name, new_name)
return jsonify({"success": True, "profile": new_name, "profiles": serialize_profiles()})
@api_bp.post("/voice-profiles/import")
def api_import_voice_profiles() -> ResponseReturnValue:
payload = request.get_json(force=True, silent=True) or {}
data = payload.get("data")
replace_existing = bool(payload.get("replace_existing"))
if not isinstance(data, dict):
return jsonify({"error": "Invalid profile payload"}), 400
try:
imported = import_profiles_data(data, replace_existing=replace_existing)
except Exception as exc:
return jsonify({"error": str(exc)}), 400
return jsonify({"success": True, "imported": imported, "profiles": serialize_profiles()})
@api_bp.get("/voice-profiles/export")
def api_export_voice_profiles() -> ResponseReturnValue:
names_param = request.args.get("names")
names = None
if names_param:
names = [item.strip() for item in names_param.split(",") if item.strip()]
payload = export_profiles_payload(names)
import io
import json
data = json.dumps(payload, indent=2).encode("utf-8")
filename = "voice_profiles.json" if not names else "voice_profiles_export.json"
return send_file(
io.BytesIO(data),
mimetype="application/json",
as_attachment=True,
download_name=filename,
)
@api_bp.post("/voice-profiles/preview")
def api_voice_profiles_preview() -> ResponseReturnValue:
payload = request.get_json(force=True, silent=True) or {}
text = str(payload.get("text") or "").strip() or "Hello world"
language = str(payload.get("language") or "a").strip().lower() or "a"
speed = coerce_float(payload.get("speed"), 1.0)
max_seconds = coerce_float(payload.get("max_seconds"), 8.0)
settings = load_settings()
use_gpu = settings.get("use_gpu", False)
# Accept a direct formula string or a full profile entry.
formula = str(payload.get("formula") or "").strip()
profile_name = str(payload.get("profile") or "").strip()
provider = str(payload.get("tts_provider") or payload.get("provider") or "").strip().lower() or None
supertonic_total_steps = int(payload.get("supertonic_total_steps") or payload.get("total_steps") or settings.get("supertonic_total_steps") or 8)
voice_spec = ""
resolved_provider = provider or "kokoro"
profiles = load_profiles()
if resolved_provider == "supertonic" and not profile_name:
voice_spec = str(payload.get("voice") or payload.get("supertonic_voice") or "M1").strip() or "M1"
# Allow per-speaker overrides via payload.
supertonic_total_steps = int(payload.get("supertonic_total_steps") or payload.get("total_steps") or supertonic_total_steps)
speed = coerce_float(payload.get("supertonic_speed") or payload.get("speed"), speed)
elif profile_name:
entry = profiles.get(profile_name)
normalized_entry = normalize_profile_entry(entry)
if not normalized_entry:
return jsonify({"error": "Unknown profile"}), 404
resolved_provider = str(normalized_entry.get("provider") or "kokoro")
if resolved_provider == "supertonic":
voice_spec = str(normalized_entry.get("voice") or "M1")
supertonic_total_steps = int(normalized_entry.get("total_steps") or supertonic_total_steps)
speed = float(normalized_entry.get("speed") or speed)
else:
voice_spec = formula_from_profile(normalized_entry) or ""
language = str(normalized_entry.get("language") or language)
elif formula:
voice_spec = formula
resolved_provider = "kokoro"
else:
# Raw voices payload -> Kokoro mix.
voices = payload.get("voices") or []
pseudo = {"provider": "kokoro", "language": language, "voices": voices}
normalized_entry = normalize_profile_entry(pseudo)
voice_spec = formula_from_profile(normalized_entry) or ""
resolved_provider = "kokoro"
if not voice_spec:
return jsonify({"error": "Unable to resolve preview voice"}), 400
try:
return synthesize_preview(
text=text,
voice_spec=voice_spec,
language=language,
speed=speed,
use_gpu=use_gpu,
tts_provider=resolved_provider,
supertonic_total_steps=supertonic_total_steps,
max_seconds=max_seconds,
)
except Exception as exc:
return jsonify({"error": str(exc)}), 500
@api_bp.post("/speaker-preview")
def api_speaker_preview() -> ResponseReturnValue:
payload = request.get_json(force=True, silent=True) or {}
pending_id = str(payload.get("pending_id") or "").strip()
text = payload.get("text", "Hello world")
voice = payload.get("voice", "af_heart")
language = payload.get("language", "a")
speed_value = payload.get("speed")
speed = coerce_float(speed_value, 1.0)
tts_provider = str(payload.get("tts_provider") or "").strip().lower()
supertonic_total_steps = int(payload.get("supertonic_total_steps") or 8)
settings = load_settings()
use_gpu = settings.get("use_gpu", False)
base_spec, speaker_name = split_profile_spec(voice)
resolved_provider = tts_provider if tts_provider in {"kokoro", "supertonic"} else ""
if speaker_name:
entry = normalize_profile_entry(load_profiles().get(speaker_name))
if entry:
resolved_provider = str(entry.get("provider") or resolved_provider or "")
if resolved_provider == "supertonic":
voice = str(entry.get("voice") or "M1")
supertonic_total_steps = int(entry.get("total_steps") or supertonic_total_steps)
if speed_value is None:
speed = coerce_float(entry.get("speed"), speed)
elif resolved_provider == "kokoro":
voice = formula_from_profile(entry) or (base_spec or voice)
if not resolved_provider:
resolved_provider = "supertonic" if str(base_spec or "").strip() in {"M1","M2","M3","M4","M5","F1","F2","F3","F4","F5"} else "kokoro"
pronunciation_overrides = None
manual_overrides = None
speakers = None
if pending_id:
try:
pending = get_service().get_pending_job(pending_id)
except Exception:
pending = None
if pending is not None:
manual_overrides = getattr(pending, "manual_overrides", None)
pronunciation_overrides = getattr(pending, "pronunciation_overrides", None)
speakers = getattr(pending, "speakers", None)
try:
return synthesize_preview(
text=text,
voice_spec=voice,
language=language,
speed=speed,
use_gpu=use_gpu
,
tts_provider=resolved_provider,
supertonic_total_steps=supertonic_total_steps or int(settings.get("supertonic_total_steps") or 8),
pronunciation_overrides=pronunciation_overrides,
manual_overrides=manual_overrides,
speakers=speakers,
)
except Exception as e:
return jsonify({"error": str(e)}), 500
# --- Integration Routes ---
def _opds_metadata_overrides(metadata_payload: Mapping[str, Any]) -> Dict[str, Any]:
metadata_overrides: Dict[str, Any] = {}
def _stringify_metadata_value(value: Any) -> str:
if value is None:
return ""
if isinstance(value, (list, tuple, set)):
parts = [str(item).strip() for item in value if item is not None]
parts = [part for part in parts if part]
return ", ".join(parts)
return str(value).strip()
raw_series = metadata_payload.get("series") or metadata_payload.get("series_name")
series_name = str(raw_series or "").strip()
if series_name:
metadata_overrides["series"] = series_name
metadata_overrides.setdefault("series_name", series_name)
series_index_value = (
metadata_payload.get("series_index")
or metadata_payload.get("series_position")
or metadata_payload.get("series_sequence")
or metadata_payload.get("book_number")
)
if series_index_value is not None:
series_index_text = str(series_index_value).strip()
if series_index_text:
metadata_overrides.setdefault("series_index", series_index_text)
metadata_overrides.setdefault("series_position", series_index_text)
metadata_overrides.setdefault("series_sequence", series_index_text)
metadata_overrides.setdefault("book_number", series_index_text)
tags_value = metadata_payload.get("tags") or metadata_payload.get("keywords")
if tags_value:
tags_text = _stringify_metadata_value(tags_value)
if tags_text:
metadata_overrides.setdefault("tags", tags_text)
metadata_overrides.setdefault("keywords", tags_text)
metadata_overrides.setdefault("genre", tags_text)
description_value = metadata_payload.get("description") or metadata_payload.get("summary")
if description_value:
description_text = _stringify_metadata_value(description_value)
if description_text:
metadata_overrides.setdefault("description", description_text)
metadata_overrides.setdefault("summary", description_text)
subtitle_value = (
metadata_payload.get("subtitle")
or metadata_payload.get("sub_title")
or metadata_payload.get("calibre_subtitle")
)
if subtitle_value:
subtitle_text = _stringify_metadata_value(subtitle_value)
if subtitle_text:
metadata_overrides.setdefault("subtitle", subtitle_text)
publisher_value = metadata_payload.get("publisher")
if publisher_value:
publisher_text = _stringify_metadata_value(publisher_value)
if publisher_text:
metadata_overrides.setdefault("publisher", publisher_text)
# Author mapping: Abogen templates look for either 'authors' or 'author'.
authors_value = (
metadata_payload.get("authors")
or metadata_payload.get("author")
or metadata_payload.get("creator")
or metadata_payload.get("dc_creator")
)
if authors_value:
authors_text = _stringify_metadata_value(authors_value)
if authors_text:
metadata_overrides.setdefault("authors", authors_text)
metadata_overrides.setdefault("author", authors_text)
return metadata_overrides
@api_bp.get("/integrations/calibre-opds/feed")
def api_calibre_opds_feed() -> ResponseReturnValue:
integrations = load_integration_settings()
calibre_settings = integrations.get("calibre_opds", {})
payload = {
"base_url": calibre_settings.get("base_url"),
"username": calibre_settings.get("username"),
"password": calibre_settings.get("password"),
"verify_ssl": calibre_settings.get("verify_ssl", True),
}
if not payload.get("base_url"):
return jsonify({"error": "Calibre OPDS base URL is not configured."}), 400
try:
client = CalibreOPDSClient(
base_url=payload.get("base_url") or "",
username=payload.get("username"),
password=payload.get("password"),
verify=bool(payload.get("verify_ssl", True)),
)
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
href = request.args.get("href", type=str)
query = request.args.get("q", type=str)
letter = request.args.get("letter", type=str)
try:
if letter:
feed = client.browse_letter(letter, start_href=href)
elif query:
feed = client.search(query, start_href=href)
else:
feed = client.fetch_feed(href)
except CalibreOPDSError as exc:
return jsonify({"error": str(exc)}), 502
except Exception as exc:
return jsonify({"error": f"Unexpected error: {str(exc)}"}), 500
return jsonify({
"feed": feed.to_dict(),
"href": href or "",
"query": query or "",
})
@api_bp.post("/integrations/audiobookshelf/folders")
def api_abs_folders() -> ResponseReturnValue:
payload = request.get_json(force=True, silent=True) or {}
# Use the helper to resolve saved tokens when use_saved_token is set
settings = audiobookshelf_settings_from_payload(payload)
host = settings.get("base_url")
token = settings.get("api_token")
library_id = settings.get("library_id")
if not host or not token:
return jsonify({"error": "Base URL and API token are required"}), 400
if not library_id:
return jsonify({"error": "Library ID is required to list folders"}), 400
try:
config = AudiobookshelfConfig(base_url=host, api_token=token, library_id=library_id)
client = AudiobookshelfClient(config)
folders = client.list_folders()
return jsonify({"folders": folders})
except Exception as e:
return jsonify({"error": str(e)}), 400
@api_bp.post("/integrations/audiobookshelf/test")
def api_abs_test() -> ResponseReturnValue:
payload = request.get_json(force=True, silent=True) or {}
# Use the helper to resolve saved tokens when use_saved_token is set
settings = audiobookshelf_settings_from_payload(payload)
host = settings.get("base_url")
token = settings.get("api_token")
if not host or not token:
return jsonify({"error": "Base URL and API token are required"}), 400
try:
config = AudiobookshelfConfig(base_url=host, api_token=token)
client = AudiobookshelfClient(config)
# Just getting libraries is a good enough test
client.get_libraries()
return jsonify({"success": True, "message": "Connection successful."})
except Exception as e:
return jsonify({"error": str(e)}), 400
@api_bp.post("/integrations/calibre-opds/test")
def api_calibre_opds_test() -> ResponseReturnValue:
payload = request.get_json(force=True, silent=True) or {}
# Use the helper to resolve saved passwords when use_saved_password is set
settings = calibre_settings_from_payload(payload)
base_url = settings.get("base_url")
username = settings.get("username")
password = settings.get("password")
verify_ssl = settings.get("verify_ssl", False)
if not base_url:
return jsonify({"error": "Base URL is required"}), 400
try:
client = CalibreOPDSClient(
base_url=base_url,
username=username,
password=password,
verify=verify_ssl,
timeout=10.0
)
client.fetch_feed()
return jsonify({"success": True, "message": "Connection successful."})
except Exception as e:
return jsonify({"error": str(e)}), 400
@api_bp.post("/integrations/calibre-opds/import")
def api_calibre_opds_import() -> ResponseReturnValue:
if not request.is_json:
return jsonify({"error": "Expected JSON payload."}), 400
data = request.get_json(force=True, silent=True) or {}
href = str(data.get("href") or "").strip()
if not href:
return jsonify({"error": "Download URL (href) is required."}), 400
metadata_payload = data.get("metadata") if isinstance(data, Mapping) else None
metadata_overrides: Dict[str, Any] = {}
if isinstance(metadata_payload, Mapping):
metadata_overrides = _opds_metadata_overrides(metadata_payload)
settings = load_settings()
integrations = load_integration_settings()
calibre_settings = integrations.get("calibre_opds", {})
try:
client = CalibreOPDSClient(
base_url=calibre_settings.get("base_url") or "",
username=calibre_settings.get("username"),
password=calibre_settings.get("password"),
verify=bool(calibre_settings.get("verify_ssl", True)),
)
temp_dir = Path(current_app.config.get("UPLOAD_FOLDER", "uploads"))
temp_dir.mkdir(exist_ok=True)
resource = client.download(href)
filename = resource.filename
content = resource.content
if not filename:
filename = f"{uuid.uuid4().hex}.epub"
file_path = temp_dir / f"{uuid.uuid4().hex}_{filename}"
file_path.write_bytes(content)
extraction = extract_from_path(file_path)
if metadata_overrides:
extraction.metadata.update(metadata_overrides)
result = build_pending_job_from_extraction(
stored_path=file_path,
original_name=filename,
extraction=extraction,
form={},
settings=settings,
profiles=serialize_profiles(),
metadata_overrides=metadata_overrides,
)
get_service().store_pending_job(result.pending)
return jsonify({
"success": True,
"status": "imported",
"pending_id": result.pending.id,
"redirect_url": url_for("main.wizard_step", step="book", pending_id=result.pending.id)
})
except Exception as e:
return jsonify({"error": str(e)}), 500
# --- LLM Routes ---
@api_bp.post("/llm/models")
def api_llm_models() -> ResponseReturnValue:
payload = request.get_json(force=True, silent=False) or {}
current_settings = load_settings()
base_url = str(payload.get("base_url") or payload.get("llm_base_url") or current_settings.get("llm_base_url") or "").strip()
if not base_url:
return jsonify({"error": "LLM base URL is required."}), 400
api_key = str(payload.get("api_key") or payload.get("llm_api_key") or current_settings.get("llm_api_key") or "")
timeout = coerce_float(payload.get("timeout"), current_settings.get("llm_timeout", 30.0))
overrides = {
"llm_base_url": base_url,
"llm_api_key": api_key,
"llm_timeout": timeout,
}
merged = apply_overrides(current_settings, overrides)
configuration = build_llm_configuration(merged)
try:
models = list_models(configuration)
except LLMClientError as exc:
return jsonify({"error": str(exc)}), 400
return jsonify({"models": models})
@api_bp.post("/llm/preview")
def api_llm_preview() -> ResponseReturnValue:
payload = request.get_json(force=True, silent=False) or {}
sample_text = str(payload.get("text") or "").strip()
if not sample_text:
return jsonify({"error": "Text is required."}), 400
base_settings = load_settings()
overrides: Dict[str, Any] = {
"llm_base_url": str(
payload.get("base_url")
or payload.get("llm_base_url")
or base_settings.get("llm_base_url")
or ""
).strip(),
"llm_api_key": str(
payload.get("api_key")
or payload.get("llm_api_key")
or base_settings.get("llm_api_key")
or ""
),
"llm_model": str(
payload.get("model")
or payload.get("llm_model")
or base_settings.get("llm_model")
or ""
),
"llm_prompt": payload.get("prompt") or payload.get("llm_prompt") or base_settings.get("llm_prompt"),
"llm_context_mode": payload.get("context_mode") or base_settings.get("llm_context_mode"),
"llm_timeout": coerce_float(payload.get("timeout"), base_settings.get("llm_timeout", 30.0)),
"normalization_apostrophe_mode": "llm",
}
merged = apply_overrides(base_settings, overrides)
if not merged.get("llm_base_url"):
return jsonify({"error": "LLM base URL is required."}), 400
if not merged.get("llm_model"):
return jsonify({"error": "Select an LLM model before previewing."}), 400
apostrophe_config = build_apostrophe_config(settings=merged)
try:
normalized_text = normalize_for_pipeline(sample_text, config=apostrophe_config, settings=merged)
except LLMClientError as exc:
return jsonify({"error": str(exc)}), 400
context = {
"text": sample_text,
"normalized_text": normalized_text,
}
return jsonify(context)
# --- Normalization Routes ---
@api_bp.post("/normalization/preview")
def api_normalization_preview() -> ResponseReturnValue:
payload = request.get_json(force=True, silent=False) or {}
sample_text = str(payload.get("text") or "").strip()
if not sample_text:
return jsonify({"error": "Sample text is required."}), 400
base_settings = load_settings()
# We might want to apply overrides from payload if any normalization settings are passed
# For now, just use base settings as in original code (presumably)
apostrophe_config = build_apostrophe_config(settings=base_settings)
try:
normalized_text = normalize_for_pipeline(sample_text, config=apostrophe_config, settings=base_settings)
except Exception as exc:
return jsonify({"error": str(exc)}), 400
return jsonify({
"text": sample_text,
"normalized_text": normalized_text,
})
@api_bp.post("/entity-pronunciation/preview")
def api_entity_pronunciation_preview() -> ResponseReturnValue:
payload = request.get_json(force=True, silent=True) or {}
token = payload.get("token", "").strip()
pronunciation = payload.get("pronunciation", "").strip()
voice = payload.get("voice", "").strip()
language = payload.get("language", "a").strip()
if not token and not pronunciation:
return jsonify({"error": "Token or pronunciation required"}), 400
text_to_speak = pronunciation if pronunciation else token
if not voice:
settings = load_settings()
voice = settings.get("default_voice", "af_heart")
try:
# Check GPU setting
settings = load_settings()
use_gpu = coerce_bool(settings.get("use_gpu"), False)
audio_bytes = generate_preview_audio(
text=text_to_speak,
voice_spec=voice,
language=language,
speed=1.0,
use_gpu=use_gpu,
)
audio_base64 = base64.b64encode(audio_bytes).decode("utf-8")
return jsonify({"audio_base64": audio_base64})
except Exception as e:
return jsonify({"error": str(e)}), 400

Some files were not shown because too many files have changed in this diff Show More