27 Commits
Author SHA1 Message Date
Deniz Şafak 55a4f958ee Update changelog 2025-11-28 05:17:31 +03:00
Deniz Şafak 6c633e9167 v1.2.4 2025-11-28 05:16:39 +03:00
Deniz Şafak 566158c132 Reformat using black 2025-11-28 05:07:26 +03:00
Deniz Şafak bf43d1799d Update readme 2025-11-28 05:06:51 +03:00
Deniz Şafak 7293b4b826 New option: **Pre-download models and voices for offline use** 2025-11-28 04:58:13 +03:00
Deniz Şafak 34f2d712b3 Update readme 2025-11-28 00:56:44 +03:00
Deniz Şafak 5ef2612df6 Add spaCy support for improved sentence segmentation, possible fix for #91 2025-11-28 00:49:36 +03:00
Deniz Şafak 290c265d5e Improve cleanup_preview_threads function 2025-11-24 00:46:13 +03:00
Deniz Şafak ed418ac11d Added support for . separator in timestamps 2025-11-23 17:30:05 +03:00
Deniz Şafak e6abf2e541 Change audiobook to audio 2025-11-23 16:07:28 +03:00
Deniz Şafak f5d95547c5 Added subtitle generation support for non-English languages, improvements in code and documentation 2025-11-23 16:03:41 +03:00
Deniz Şafak 795bd8f1aa Fix WINDWOS_INSTALL.bat 2025-11-22 15:41:40 +03:00
Deniz ŞafakandGitHub c4e2a5ef0d Merge pull request #103 from denizsafak/copilot/improve-slow-code-efficiency
Optimize regex compilation and eliminate busy-wait loops
2025-11-22 15:22:15 +03:00
Deniz Şafak f57d1994cf Update CHANGELOG.md for pre-release 1.2.4: optimize regex compilation and eliminate busy-wait loops 2025-11-22 15:20:17 +03:00
Deniz ŞafakandGitHub 6482a56479 Remove unused import of QEventLoop 2025-11-22 15:14:41 +03:00
Deniz ŞafakandGitHub 8ddfd01dff Delete test_performance.py 2025-11-22 15:13:50 +03:00
Deniz ŞafakandGitHub ac551abd55 Delete PERFORMANCE_OPTIMIZATIONS.md 2025-11-22 15:13:36 +03:00
copilot-swe-agent[bot]anddenizsafak 711858ce2c Address final code review nitpicks: improve code organization
Co-authored-by: denizsafak <39929354+denizsafak@users.noreply.github.com>
2025-11-20 23:29:23 +00:00
copilot-swe-agent[bot]anddenizsafak 5cca6235e1 Fix potential division by zero in performance tests
Co-authored-by: denizsafak <39929354+denizsafak@users.noreply.github.com>
2025-11-20 23:26:36 +00:00
copilot-swe-agent[bot]anddenizsafak 115ab2a0f2 Add comprehensive performance optimization documentation
Co-authored-by: denizsafak <39929354+denizsafak@users.noreply.github.com>
2025-11-20 23:24:11 +00:00
copilot-swe-agent[bot]anddenizsafak bf5dfddee6 Improve cancellation handling and use generator expressions for memory efficiency
Co-authored-by: denizsafak <39929354+denizsafak@users.noreply.github.com>
2025-11-20 23:23:25 +00:00
copilot-swe-agent[bot]anddenizsafak 23a89a618b Address code review feedback: Fix Linux control chars and improve readability
Co-authored-by: denizsafak <39929354+denizsafak@users.noreply.github.com>
2025-11-20 23:20:51 +00:00
copilot-swe-agent[bot]anddenizsafak 4ad384dc9f Add performance tests and optimize repeated path operations
Co-authored-by: denizsafak <39929354+denizsafak@users.noreply.github.com>
2025-11-20 23:18:11 +00:00
copilot-swe-agent[bot]anddenizsafak 5ec06e7d49 Replace busy-wait loop with threading.Event for better efficiency
Co-authored-by: denizsafak <39929354+denizsafak@users.noreply.github.com>
2025-11-20 23:13:22 +00:00
copilot-swe-agent[bot]anddenizsafak 7f75aa8209 Optimize regex patterns by pre-compiling frequently used patterns
Co-authored-by: denizsafak <39929354+denizsafak@users.noreply.github.com>
2025-11-20 23:11:06 +00:00
copilot-swe-agent[bot] cd2e5c9150 Initial plan 2025-11-20 22:59:44 +00:00
Deniz Şafak f477703b1a v1.2.3 2025-11-19 02:24:42 +03:00
12 changed files with 1489 additions and 341 deletions
+18
View File
@@ -1,3 +1,21 @@
# 1.2.4
- **Subtitle generation is now available for all languages!** Abogen now supports subtitle generation for non-English languages using audio duration-based timing. Available modes include `Line`, `Sentence`, and `Sentence + Comma`. (Note: Word-level subtitle modes remain English-only due to Kokoro's timestamp token limitations.)
- New option: **"Use spaCy for sentence segmentation"** You can now use [spaCy](https://spacy.io/) to automatically detect sentence boundaries and produce cleaner, more readable subtitles. Quick summary:
- **What it does:** Splits text into natural sentences so subtitle entries read better and align more naturally with speech.
- **Why this helps:** The previous punctuation-based splitting could break sentences incorrectly at common abbreviations (e.g. "Mr.", "Dr.", "Prof.") or initials, producing wrong subtitle breaks. spaCy avoids those false splits by using linguistic rules to detect real sentence boundaries.
- **For Non-English:** spaCy runs **before** audio generation to create better sentence chunks for TTS.
- **For English:** spaCy runs **during** subtitle generation to find accurate sentence breaks after TTS.
- **Note:** spaCy segmentation is only applied when subtitle mode is `Sentence` or `Sentence + Comma`. When turned off, it falls back to simple punctuation-based splitting.
- New option: **Pre-download models and voices for offline use** You can now pre-download all required Kokoro models, voices, and spaCy language models using this option in the settings menu. Allowing you to use Abogen completely offline without any internet connection.
- Added support for `.` separator in timestamps (e.g. `HH:MM:SS.ms`) for timestamp-based text files.
- Optimized regex compilation and eliminated busy-wait loops.
- Possibly fixed `Silent truncation of long paragraphs` issue mentioned in [#91](https://github.com/denizsafak/abogen/issues/91) by [@xklzlxr](https://github.com/xklzlxr)
- Fixed unused regex patterns and variable naming conventions.
- Improvements in code and documentation.
# 1.2.3
- Same as 1.2.2, re-released to fix an issue with subtitle timing when using timestamp-based text files.
# 1.2.2 # 1.2.2
- **You can now voice your subtitle files!** Simply add `.srt`, `.ass` or `.vtt` files to generate timed audio. Alternatively, add a text file with timestamps in `HH:MM:SS` or `HH:MM:SS,ms` format to generate audio that matches the timestamps. See [here](https://github.com/denizsafak/abogen?tab=readme-ov-file#about-timestamp-based-text-files) for detailed instructions. - **You can now voice your subtitle files!** Simply add `.srt`, `.ass` or `.vtt` files to generate timed audio. Alternatively, add a text file with timestamps in `HH:MM:SS` or `HH:MM:SS,ms` format to generate audio that matches the timestamps. See [here](https://github.com/denizsafak/abogen?tab=readme-ov-file#about-timestamp-based-text-files) for detailed instructions.
- New option: **"Use silent gaps between subtitles"**: Prevents unnecessary audio speed-up by letting speech continue into the silent gaps between subtitles. - New option: **"Use silent gaps between subtitles"**: Prevents unnecessary audio speed-up by letting speech continue into the silent gaps between subtitles.
+8 -5
View File
@@ -171,8 +171,10 @@ Heres Abogen in action: in this demo, it processes 3,000 characters of tex
| **Clear cache files** | Deletes cache files created during the conversion or preview. | | **Clear cache files** | Deletes cache files created during the conversion or preview. |
| **Use silent gaps between subtitles** | Prevents unnecessary audio speed-up by letting speech continue into the silent gaps between subtitle etries. In short, it ignores the end times in subtitle entries and uses the silent space until the beginning of the next subtitle entry. When disabled, it speeds up the audio to fit the exact time interval specified in the subtitle. (for subtitle files). | | **Use silent gaps between subtitles** | Prevents unnecessary audio speed-up by letting speech continue into the silent gaps between subtitle etries. In short, it ignores the end times in subtitle entries and uses the silent space until the beginning of the next subtitle entry. When disabled, it speeds up the audio to fit the exact time interval specified in the subtitle. (for subtitle files). |
| **Subtitle speed adjustment method** | Choose how to speed up audio when needed: `TTS Regeneration (better quality)` re-generates the audio at a faster speed, while `FFmpeg Time-stretch (better speed)` quickly speeds up the generated audio. (for subtitle files). | | **Subtitle speed adjustment method** | Choose how to speed up audio when needed: `TTS Regeneration (better quality)` re-generates the audio at a faster speed, while `FFmpeg Time-stretch (better speed)` quickly speeds up the generated audio. (for subtitle files). |
| **Check for updates at startup** | Automatically checks for updates when the program starts. | | **Use spaCy for sentence segmentation** | When this option is enabled, Abogen uses [spaCy](https://spacy.io/) to detect sentence boundaries more accurately, instead of using punctuation marks (like periods, question marks, etc.) to split sentences, which could incorrectly cut off phrases like "Mr." or "Dr.". With spaCy, sentences are divided more accurately. For non-English text, spaCy runs **before** audio generation to create sentence chunks. For English text, spaCy runs **during** subtitle generation to improve timing and readability. spaCy is only used when subtitle mode is `Sentence` or `Sentence + Comma`. If you prefer the old punctuation splitting method, you can turn this option off. |
| **Pre-download models and voices for offline use** | Opens a window that displays the available models and voices. Click `Download all` button to download all required models and voices, allowing you to use Abogen completely offline without any internet connection. |
| **Disable Kokoro's internet access** | Prevents Kokoro from downloading models or voices from HuggingFace Hub, useful for offline use. | | **Disable Kokoro's internet access** | Prevents Kokoro from downloading models or voices from HuggingFace Hub, useful for offline use. |
| **Check for updates at startup** | Automatically checks for updates when the program starts. |
| **Reset to default settings** | Resets all settings to their default values. | | **Reset to default settings** | Resets all settings to their default values. |
> Special thanks to [@robmckinnon](https://github.com/robmckinnon) for adding Sentence + Highlighting feature in PR [#65](https://github.com/denizsafak/abogen/pull/65) > Special thanks to [@robmckinnon](https://github.com/robmckinnon) for adding Sentence + Highlighting feature in PR [#65](https://github.com/denizsafak/abogen/pull/65)
@@ -235,7 +237,7 @@ Similar to chapter markers, it is possible to add metadata tags for `M4B` files.
> Note: `METADATA_COVER_PATH` is used to embed a cover image into the generated M4B file. Abogen automatically extracts the cover from EPUB and PDF files and adds this tag for you. > Note: `METADATA_COVER_PATH` is used to embed a cover image into the generated M4B file. Abogen automatically extracts the cover from EPUB and PDF files and adds this tag for you.
## `About Timestamp-based Text Files` ## `About Timestamp-based Text Files`
Similar to converting subtitle files to audio, Abogen can automatically detect text files that contain timestamps in `HH:MM:SS` or `HH:MM:SS,ms` format. When timestamps are found inside your text file, Abogen will ask if you want to use them for audio timing. This is useful for creating timed narrations, scripts, or transcripts where you need exact control over when each segment is spoken. Similar to converting subtitle files to audio, Abogen can automatically detect text files that contain timestamps in `HH:MM:SS`, `HH:MM:SS,ms` or `HH:MM:SS.ms` format. When timestamps are found inside your text file, Abogen will ask if you want to use them for audio timing. This is useful for creating timed narrations, scripts, or transcripts where you need exact control over when each segment is spoken.
Format your text file like this: Format your text file like this:
``` ```
@@ -250,7 +252,7 @@ And this is the third segment, starting at 45 seconds.
``` ```
**Important notes:** **Important notes:**
- Timestamps must be in `HH:MM:SS` or `HH:MM:SS,ms` format (e.g., `00:05:30` for 5 minutes 30 seconds, or `00:05:30,500` for 5 minutes 30.5 seconds) - Timestamps must be in `HH:MM:SS`, `HH:MM:SS,ms` or `HH:MM:SS.ms` format (e.g., `00:05:30` for 5 minutes 30 seconds, or `00:05:30.500` for 5 minutes 30.5 seconds)
- Milliseconds are optional and provide precision up to 1/1000th of a second - Milliseconds are optional and provide precision up to 1/1000th of a second
- Text before the first timestamp (if any) will automatically start at `00:00:00` - Text before the first timestamp (if any) will automatically start at `00:00:00`
- When using timestamps, the subtitle generation mode setting is ignored - When using timestamps, the subtitle generation mode setting is ignored
@@ -476,6 +478,7 @@ Feel free to explore the code and make any changes you like.
## `Credits` ## `Credits`
- Abogen uses [Kokoro](https://github.com/hexgrad/kokoro) for its high-quality, natural-sounding text-to-speech synthesis. Huge thanks to the Kokoro team for making this possible. - Abogen uses [Kokoro](https://github.com/hexgrad/kokoro) for its high-quality, natural-sounding text-to-speech synthesis. Huge thanks to the Kokoro team for making this possible.
- Thanks to the [spaCy](https://spacy.io/) project for its sentence-segmentation tools, which help Abogen produce cleaner, more natural sentence segmentation.
- Thanks to [@wojiushixiaobai](https://github.com/wojiushixiaobai) for [Embedded Python](https://github.com/wojiushixiaobai/Python-Embed-Win64) packages. These modified packages include pip pre-installed, enabling Abogen to function as a standalone application without requiring users to separately install Python in Windows. - Thanks to [@wojiushixiaobai](https://github.com/wojiushixiaobai) for [Embedded Python](https://github.com/wojiushixiaobai/Python-Embed-Win64) packages. These modified packages include pip pre-installed, enabling Abogen to function as a standalone application without requiring users to separately install Python in Windows.
- Thanks to creators of [EbookLib](https://github.com/aerkalov/ebooklib), a Python library for reading and writing ePub files, which is used for extracting text from ePub files. - Thanks to creators of [EbookLib](https://github.com/aerkalov/ebooklib), a Python library for reading and writing ePub files, which is used for extracting text from ePub files.
- Special thanks to the [PyQt](https://www.riverbankcomputing.com/software/pyqt/) team for providing the cross-platform GUI toolkit that powers Abogen's interface. - Special thanks to the [PyQt](https://www.riverbankcomputing.com/software/pyqt/) team for providing the cross-platform GUI toolkit that powers Abogen's interface.
@@ -488,7 +491,7 @@ This project is available under the MIT License - see the [LICENSE](https://gith
## `Star History` ## `Star History`
[![Star History Chart](https://api.star-history.com/svg?repos=denizsafak/abogen&type=Date)](https://www.star-history.com/#denizsafak/abogen&Date) [![Star History Chart](https://api.star-history.com/svg?repos=denizsafak/abogen&type=Date)](https://www.star-history.com/#denizsafak/abogen&Date)
> [!IMPORTANT] > [!NOTE]
> Subtitle generation currently works only for English. This is because Kokoro provides timestamp tokens only for English text. If you want subtitles in other languages, please request this feature in the [Kokoro project](https://github.com/hexgrad/kokoro). For more technical details, see [this line](https://github.com/hexgrad/kokoro/blob/6d87f4ae7abc2d14dbc4b3ef2e5f19852e861ac2/kokoro/pipeline.py#L383) in the Kokoro's code. > Abogen supports subtitle generation for all languages. However, word-level subtitle modes (e.g., "1 word", "2 words", "3 words", etc.) are only available for English because [Kokoro provides timestamp tokens only for English text](https://github.com/hexgrad/kokoro/blob/6d87f4ae7abc2d14dbc4b3ef2e5f19852e861ac2/kokoro/pipeline.py#L383). For non-English languages, Abogen uses a duration-based fallback that supports sentence-level and comma-based subtitle modes ("Line", "Sentence", "Sentence + Comma"). If you need word-level subtitles for other languages, please request that feature in the [Kokoro project](https://github.com/hexgrad/kokoro).
> Tags: audiobook, kokoro, text-to-speech, TTS, audiobook generator, audiobooks, text to speech, audiobook maker, audiobook creator, audiobook generator, voice-synthesis, text to audio, text to audio converter, text to speech converter, text to speech generator, text to speech software, text to speech app, epub to audio, pdf to audio, markdown to audio, subtitle to audio, srt to audio, ass to audio, vtt to audio, webvtt to audio, content-creation, media-generation > Tags: audiobook, kokoro, text-to-speech, TTS, audiobook generator, audiobooks, text to speech, audiobook maker, audiobook creator, audiobook generator, voice-synthesis, text to audio, text to audio converter, text to speech converter, text to speech generator, text to speech software, text to speech app, epub to audio, pdf to audio, markdown to audio, subtitle to audio, srt to audio, ass to audio, vtt to audio, webvtt to audio, content-creation, media-generation
+2 -2
View File
@@ -278,7 +278,7 @@ if /I "%IS_NVIDIA%"=="true" (
for /f %%i in ('%PYTHON_CONSOLE_PATH% -c "from torch.cuda import is_available; print(is_available())"') do set cuda_available=%%i for /f %%i in ('%PYTHON_CONSOLE_PATH% -c "from torch.cuda import is_available; print(is_available())"') do set cuda_available=%%i
if "%cuda_available%"=="False" ( if "%cuda_available%"=="False" (
echo Installing PyTorch with CUDA (12.8) support... echo "Installing PyTorch with CUDA (12.8) support..."
:: We need to use an older version of PyTorch (2.8.0) until this issue is fixed: https://github.com/pytorch/pytorch/issues/166628 :: We need to use an older version of PyTorch (2.8.0) until this issue is fixed: https://github.com/pytorch/pytorch/issues/166628
:: Solution mentioned by @mazenemam19 in #99: :: Solution mentioned by @mazenemam19 in #99:
%PYTHON_CONSOLE_PATH% -m pip install torch==2.8.0+cu128 torchvision==0.23.0+cu128 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128 --no-warn-script-location %PYTHON_CONSOLE_PATH% -m pip install torch==2.8.0+cu128 torchvision==0.23.0+cu128 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128 --no-warn-script-location
@@ -304,7 +304,7 @@ if /I "%IS_NVIDIA%"=="true" (
if errorlevel 2 ( if errorlevel 2 (
echo Skipping PyTorch installation. echo Skipping PyTorch installation.
) else ( ) else (
echo Installing PyTorch with CUDA (12.8) support... echo "Installing PyTorch with CUDA (12.8) support..."
:: We need to use an older version of PyTorch (2.8.0) until this issue is fixed: https://github.com/pytorch/pytorch/issues/166628 :: We need to use an older version of PyTorch (2.8.0) until this issue is fixed: https://github.com/pytorch/pytorch/issues/166628
:: Solution mentioned by @mazenemam19 in #99: :: Solution mentioned by @mazenemam19 in #99:
%PYTHON_CONSOLE_PATH% -m pip install torch==2.8.0+cu128 torchvision==0.23.0+cu128 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128 --no-warn-script-location %PYTHON_CONSOLE_PATH% -m pip install torch==2.8.0+cu128 torchvision==0.23.0+cu128 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128 --no-warn-script-location
+1 -1
View File
@@ -1 +1 @@
1.2.2 1.2.4
+33 -22
View File
@@ -44,6 +44,17 @@ logging.basicConfig(
level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s" level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s"
) )
# Pre-compile frequently used regex patterns for better performance
_BRACKETED_NUMBERS_PATTERN = re.compile(r"\[\s*\d+\s*\]")
_STANDALONE_PAGE_NUMBERS_PATTERN = re.compile(r"^\s*\d+\s*$", re.MULTILINE)
_PAGE_NUMBERS_AT_END_PATTERN = re.compile(r"\s+\d+\s*$", re.MULTILINE)
_PAGE_NUMBERS_WITH_DASH_PATTERN = re.compile(
r"\s+[-–—]\s*\d+\s*[-–—]?\s*$", re.MULTILINE
)
_HTML_TAG_PATTERN = re.compile(r"<[^>]+>")
_LEADING_DASH_PATTERN = re.compile(r"^\s*[-–—]\s*")
_LEADING_SIMPLE_DASH_PATTERN = re.compile(r"^\s*-\s*")
class HandlerDialog(QDialog): class HandlerDialog(QDialog):
# Class variables to remember checkbox states between dialog instances # Class variables to remember checkbox states between dialog instances
@@ -428,19 +439,12 @@ class HandlerDialog(QDialog):
"""Pre-process all page contents from PDF document""" """Pre-process all page contents from PDF document"""
for page_num in range(len(self.pdf_doc)): for page_num in range(len(self.pdf_doc)):
text = clean_text(self.pdf_doc[page_num].get_text()) text = clean_text(self.pdf_doc[page_num].get_text())
# Remove bracketed numbers (citations, footnotes) # Remove bracketed numbers, page numbers, etc. using pre-compiled patterns
text = re.sub(r"\[\s*\d+\s*\]", "", text) # Combine all regex operations for better performance
text = _BRACKETED_NUMBERS_PATTERN.sub("", text)
# Remove standalone page numbers (numbers alone on a line) text = _STANDALONE_PAGE_NUMBERS_PATTERN.sub("", text)
text = re.sub(r"^\s*\d+\s*$", "", text, flags=re.MULTILINE) text = _PAGE_NUMBERS_AT_END_PATTERN.sub("", text)
text = _PAGE_NUMBERS_WITH_DASH_PATTERN.sub("", text)
# Remove page numbers at the end of paragraphs
# This pattern looks for digits surrounded by whitespace at the end of paragraphs
text = re.sub(r"\s+\d+\s*$", "", text, flags=re.MULTILINE)
# Also remove page numbers followed by a hyphen or dash at paragraph end
# (common in headers/footers like "- 42 -")
text = re.sub(r"\s+[-–—]\s*\d+\s*[-–—]?\s*$", "", text, flags=re.MULTILINE)
page_id = f"page_{page_num + 1}" page_id = f"page_{page_num + 1}"
self.content_texts[page_id] = text self.content_texts[page_id] = text
@@ -569,7 +573,7 @@ class HandlerDialog(QDialog):
text = clean_text(soup.get_text()).strip() text = clean_text(soup.get_text()).strip()
if text: if text:
self.content_texts[doc_href] = text self.content_texts[doc_href] = text
self.content_lengths[doc_href] = len(text) self.content_lengths[doc_href] = calculate_text_length(text)
title = None title = None
if soup.title and soup.title.string: if soup.title and soup.title.string:
@@ -892,7 +896,7 @@ class HandlerDialog(QDialog):
text = clean_text(slice_soup.get_text()).strip() text = clean_text(slice_soup.get_text()).strip()
if text: if text:
self.content_texts[current_src] = text self.content_texts[current_src] = text
self.content_lengths[current_src] = len(text) self.content_lengths[current_src] = calculate_text_length(text)
else: else:
self.content_texts[current_src] = "" self.content_texts[current_src] = ""
self.content_lengths[current_src] = 0 self.content_lengths[current_src] = 0
@@ -2015,7 +2019,8 @@ class HandlerDialog(QDialog):
html_content += "<hr/>" html_content += "<hr/>"
if self.book_metadata["description"]: if self.book_metadata["description"]:
desc = re.sub(r"<[^>]+>", "", self.book_metadata["description"]) # Use pre-compiled pattern for better performance
desc = _HTML_TAG_PATTERN.sub("", self.book_metadata["description"])
html_content += f"<h3>Description:</h3><p>{desc}</p>" html_content += f"<h3>Description:</h3><p>{desc}</p>"
if self.file_type == "pdf": if self.file_type == "pdf":
@@ -2296,8 +2301,8 @@ class HandlerDialog(QDialog):
text = self.content_texts.get(identifier) text = self.content_texts.get(identifier)
if text and text.strip(): if text and text.strip():
title = item.text(0) title = item.text(0)
# Remove leading dashes from title # Remove leading dashes from title using pre-compiled pattern
title = re.sub(r"^\s*[-–—]\s*", "", title).strip() title = _LEADING_DASH_PATTERN.sub("", title).strip()
marker = f"<<CHAPTER_MARKER:{title}>>" marker = f"<<CHAPTER_MARKER:{title}>>"
chapter_texts.append(marker + "\n" + text) chapter_texts.append(marker + "\n" + text)
@@ -2331,7 +2336,8 @@ class HandlerDialog(QDialog):
text = self.content_texts.get(identifier) text = self.content_texts.get(identifier)
if text and text.strip(): if text and text.strip():
title = item.text(0) title = item.text(0)
title = re.sub(r"^\s*[-–—]\s*", "", title).strip() # Use pre-compiled pattern for better performance
title = _LEADING_DASH_PATTERN.sub("", title).strip()
marker = f"<<CHAPTER_MARKER:{title}>>" marker = f"<<CHAPTER_MARKER:{title}>>"
chapter_texts.append(marker + "\n" + text) chapter_texts.append(marker + "\n" + text)
@@ -2401,12 +2407,16 @@ class HandlerDialog(QDialog):
combined_text += "\n\n" + child_text combined_text += "\n\n" + child_text
included_text_ids.add(child_id) included_text_ids.add(child_id)
if combined_text.strip(): if combined_text.strip():
title = re.sub(r"^\s*-\s*", "", parent_title).strip() # Use pre-compiled pattern for better performance
title = _LEADING_SIMPLE_DASH_PATTERN.sub(
"", parent_title
).strip()
marker = f"<<CHAPTER_MARKER:{title}>>" marker = f"<<CHAPTER_MARKER:{title}>>"
section_titles.append((title, marker + "\n" + combined_text)) section_titles.append((title, marker + "\n" + combined_text))
included_text_ids.add(parent_id) included_text_ids.add(parent_id)
elif not parent_checked and checked_children: elif not parent_checked and checked_children:
title = re.sub(r"^\s*-\s*", "", parent_title).strip() # Use pre-compiled pattern for better performance
title = _LEADING_SIMPLE_DASH_PATTERN.sub("", parent_title).strip()
marker = f"<<CHAPTER_MARKER:{title}>>" marker = f"<<CHAPTER_MARKER:{title}>>"
for idx, (child, child_id) in enumerate(checked_children): for idx, (child, child_id) in enumerate(checked_children):
text = self.content_texts.get(child_id, "") text = self.content_texts.get(child_id, "")
@@ -2426,7 +2436,8 @@ class HandlerDialog(QDialog):
text = self.content_texts.get(identifier, "") text = self.content_texts.get(identifier, "")
if text: if text:
title = item.text(0) title = item.text(0)
title = re.sub(r"^\s*-\s*", "", title).strip() # Use pre-compiled pattern for better performance
title = _LEADING_SIMPLE_DASH_PATTERN.sub("", title).strip()
marker = f"<<CHAPTER_MARKER:{title}>>" marker = f"<<CHAPTER_MARKER:{title}>>"
section_titles.append((title, marker + "\n" + text)) section_titles.append((title, marker + "\n" + text))
included_text_ids.add(identifier) included_text_ids.add(identifier)
+1 -4
View File
@@ -61,10 +61,7 @@ SUPPORTED_INPUT_FORMATS = [
# Please refer to: https://github.com/hexgrad/kokoro/blob/6d87f4ae7abc2d14dbc4b3ef2e5f19852e861ac2/kokoro/pipeline.py # Please refer to: https://github.com/hexgrad/kokoro/blob/6d87f4ae7abc2d14dbc4b3ef2e5f19852e861ac2/kokoro/pipeline.py
# 383 English processing (unchanged) # 383 English processing (unchanged)
# 384 if self.lang_code in 'ab': # 384 if self.lang_code in 'ab':
SUPPORTED_LANGUAGES_FOR_SUBTITLE_GENERATION = [ SUPPORTED_LANGUAGES_FOR_SUBTITLE_GENERATION = list(LANGUAGE_DESCRIPTIONS.keys())
"a",
"b",
]
# Voice and sample text constants # Voice and sample text constants
VOICES_INTERNAL = [ VOICES_INTERNAL = [
+373 -108
View File
@@ -28,13 +28,40 @@ import threading # for efficient waiting
import subprocess import subprocess
import platform import platform
# Configuration constants
_USER_RESPONSE_TIMEOUT = (
0.1 # Timeout in seconds for checking user response/cancellation
)
# Pre-compile frequently used regex patterns for better performance
_METADATA_TAG_PATTERN = re.compile(r"<<METADATA_[^:]+:[^>]*>>")
_CHAPTER_MARKER_PATTERN = re.compile(r"<<CHAPTER_MARKER:[^>]*>>")
_HTML_TAG_PATTERN = re.compile(r"<[^>]+>")
_VOICE_TAG_PATTERN = re.compile(r"{[^}]+}")
_ASS_STYLING_PATTERN = re.compile(r"\{[^}]+\}")
_ASS_NEWLINE_N_PATTERN = re.compile(r"\\N")
_ASS_NEWLINE_LOWER_N_PATTERN = re.compile(r"\\n")
_CHAPTER_MARKER_SEARCH_PATTERN = re.compile(r"<<CHAPTER_MARKER:(.*?)>>")
_WEBVTT_HEADER_PATTERN = re.compile(r"^WEBVTT.*?\n", re.MULTILINE)
_VTT_STYLE_PATTERN = re.compile(r"STYLE\s*\n.*?(?=\n\n|$)", re.DOTALL)
_VTT_NOTE_PATTERN = re.compile(r"NOTE\s*\n.*?(?=\n\n|$)", re.DOTALL)
_DOUBLE_NEWLINE_SPLIT_PATTERN = re.compile(r"\n\s*\n")
_VTT_TIMESTAMP_PATTERN = re.compile(r"([\d:.]+)\s*-->\s*([\d:.]+)")
_TIMESTAMP_ONLY_PATTERN = re.compile(r"^(\d{1,2}:\d{2}:\d{2}(?:[.,]\d{1,3})?)$")
_WINDOWS_ILLEGAL_CHARS_PATTERN = re.compile(r'[<>:"/\\|?*]')
_CONTROL_CHARS_PATTERN = re.compile(r"[\x00-\x1f]")
_LINUX_CONTROL_CHARS_PATTERN = re.compile(
r"[\x01-\x1f]"
) # Linux: exclude \x00 for separate handling
_MACOS_ILLEGAL_CHARS_PATTERN = re.compile(r"[:]")
_LINUX_ILLEGAL_CHARS_PATTERN = re.compile(r"[/\x00]")
def clean_subtitle_text(text): def clean_subtitle_text(text):
"""Remove chapter markers and metadata tags from subtitle text.""" """Remove chapter markers and metadata tags from subtitle text."""
# Remove metadata tags # Use pre-compiled patterns for better performance
text = re.sub(r"<<METADATA_[^:]+:[^>]*>>", "", text) text = _METADATA_TAG_PATTERN.sub("", text)
# Remove chapter markers text = _CHAPTER_MARKER_PATTERN.sub("", text)
text = re.sub(r"<<CHAPTER_MARKER:[^>]*>>", "", text)
return text.strip() return text.strip()
@@ -87,8 +114,8 @@ def parse_srt_file(file_path):
start_sec = time_to_seconds(start_str) start_sec = time_to_seconds(start_str)
end_sec = time_to_seconds(end_str) end_sec = time_to_seconds(end_str)
# Clean text of any styling tags # Clean text of any styling tags using pre-compiled pattern
text = re.sub(r"<[^>]+>", "", text) text = _HTML_TAG_PATTERN.sub("", text)
# Remove chapter markers and metadata tags # Remove chapter markers and metadata tags
text = clean_subtitle_text(text) text = clean_subtitle_text(text)
@@ -114,13 +141,13 @@ def parse_vtt_file(file_path):
with open(file_path, "r", encoding=encoding, errors="replace") as f: with open(file_path, "r", encoding=encoding, errors="replace") as f:
content = f.read() content = f.read()
# Remove WEBVTT header and any style/note blocks # Remove WEBVTT header and any style/note blocks using pre-compiled patterns
content = re.sub(r"^WEBVTT.*?\n", "", content, flags=re.MULTILINE) content = _WEBVTT_HEADER_PATTERN.sub("", content)
content = re.sub(r"STYLE\s*\n.*?(?=\n\n|$)", "", content, flags=re.DOTALL) content = _VTT_STYLE_PATTERN.sub("", content)
content = re.sub(r"NOTE\s*\n.*?(?=\n\n|$)", "", content, flags=re.DOTALL) content = _VTT_NOTE_PATTERN.sub("", content)
# Split by double newlines to get individual subtitle blocks # Split by double newlines to get individual subtitle blocks using pre-compiled pattern
blocks = re.split(r"\n\s*\n", content.strip()) blocks = _DOUBLE_NEWLINE_SPLIT_PATTERN.split(content.strip())
subtitles = [] subtitles = []
for block in blocks: for block in blocks:
@@ -147,7 +174,8 @@ def parse_vtt_file(file_path):
try: try:
# VTT format: 00:00:00.000 --> 00:00:05.000 or 00:00.000 --> 00:05.000 # VTT format: 00:00:00.000 --> 00:00:05.000 or 00:00.000 --> 00:05.000
match = re.match(r"([\d:.]+)\s*-->\s*([\d:.]+)", timestamp_line) # Use pre-compiled pattern
match = _VTT_TIMESTAMP_PATTERN.match(timestamp_line)
if not match: if not match:
continue continue
@@ -171,9 +199,9 @@ def parse_vtt_file(file_path):
start_sec = time_to_seconds(start_str) start_sec = time_to_seconds(start_str)
end_sec = time_to_seconds(end_str) end_sec = time_to_seconds(end_str)
# Clean text of any styling tags and cue settings # Clean text of any styling tags and cue settings using pre-compiled patterns
text = re.sub(r"<[^>]+>", "", text) text = _HTML_TAG_PATTERN.sub("", text)
text = re.sub(r"{[^}]+}", "", text) # Remove voice tags text = _VOICE_TAG_PATTERN.sub("", text) # Remove voice tags
# Remove chapter markers and metadata tags # Remove chapter markers and metadata tags
text = clean_subtitle_text(text) text = clean_subtitle_text(text)
@@ -196,8 +224,10 @@ def detect_timestamps_in_text(file_path):
# Count lines that are ONLY timestamps (no other text) # Count lines that are ONLY timestamps (no other text)
# Supports HH:MM:SS or HH:MM:SS,ms format # Supports HH:MM:SS or HH:MM:SS,ms format
timestamp_pattern = r"^(\d{1,2}:\d{2}:\d{2}(?:,\d{1,3})?)$" # Use pre-compiled pattern for better performance
timestamp_lines = sum(1 for line in lines if re.match(timestamp_pattern, line)) timestamp_lines = sum(
1 for line in lines if _TIMESTAMP_ONLY_PATTERN.match(line)
)
# Must have at least 2 timestamp-only lines and they should be >5% of total lines # Must have at least 2 timestamp-only lines and they should be >5% of total lines
return timestamp_lines >= 2 and (timestamp_lines / max(len(lines), 1)) > 0.05 return timestamp_lines >= 2 and (timestamp_lines / max(len(lines), 1)) > 0.05
@@ -213,20 +243,14 @@ def parse_timestamp_text_file(file_path):
content = f.read() content = f.read()
# Split by timestamp pattern (supports HH:MM:SS or HH:MM:SS,ms) # Split by timestamp pattern (supports HH:MM:SS or HH:MM:SS,ms)
pattern = r"^(\d{1,2}:\d{2}:\d{2}(?:,\d{1,3})?)$" pattern = r"^(\d{1,2}:\d{2}:\d{2}(?:[.,]\d{1,3})?)$"
lines = content.split("\n") lines = content.split("\n")
def parse_time(time_str): def parse_time(time_str):
"""Convert HH:MM:SS or HH:MM:SS,ms to seconds as float.""" """Convert HH:MM:SS or HH:MM:SS,ms to seconds as float."""
if "," in time_str: time_str = time_str.replace(",", ".")
time_part, ms_part = time_str.split(",")
parts = time_part.split(":")
seconds = int(parts[0]) * 3600 + int(parts[1]) * 60 + int(parts[2])
milliseconds = int(ms_part.ljust(3, "0")) # Pad to 3 digits
return seconds + milliseconds / 1000.0
else:
parts = time_str.split(":") parts = time_str.split(":")
return float(int(parts[0]) * 3600 + int(parts[1]) * 60 + int(parts[2])) return float(int(parts[0]) * 3600 + int(parts[1]) * 60 + float(parts[2]))
entries = [] entries = []
current_time = None current_time = None
@@ -348,10 +372,12 @@ def parse_ass_file(file_path):
start_sec = ass_time_to_seconds(start_str) start_sec = ass_time_to_seconds(start_str)
end_sec = ass_time_to_seconds(end_str) end_sec = ass_time_to_seconds(end_str)
# Clean text of ASS styling tags # Clean text of ASS styling tags using pre-compiled patterns
text = re.sub(r"\{[^}]+\}", "", text) # Remove {tags} text = _ASS_STYLING_PATTERN.sub("", text) # Remove {tags}
text = re.sub(r"\\N", "\n", text) # Convert \N to newline text = _ASS_NEWLINE_N_PATTERN.sub("\n", text) # Convert \N to newline
text = re.sub(r"\\n", "\n", text) # Convert \n to newline text = _ASS_NEWLINE_LOWER_N_PATTERN.sub(
"\n", text
) # Convert \n to newline
# Remove chapter markers and metadata tags # Remove chapter markers and metadata tags
text = clean_subtitle_text(text) text = clean_subtitle_text(text)
@@ -384,9 +410,10 @@ def sanitize_name_for_os(name, is_folder=True):
if system == "Windows": if system == "Windows":
# Windows illegal characters: < > : " / \ | ? * # Windows illegal characters: < > : " / \ | ? *
# Also can't end with space or dot # Also can't end with space or dot
sanitized = re.sub(r'[<>:"/\\|?*]', "_", name) # Use pre-compiled pattern for better performance
sanitized = _WINDOWS_ILLEGAL_CHARS_PATTERN.sub("_", name)
# Remove control characters (0-31) # Remove control characters (0-31)
sanitized = re.sub(r"[\x00-\x1f]", "_", sanitized) sanitized = _CONTROL_CHARS_PATTERN.sub("_", sanitized)
# Remove trailing spaces and dots # Remove trailing spaces and dots
sanitized = sanitized.rstrip(". ") sanitized = sanitized.rstrip(". ")
# Windows reserved names (CON, PRN, AUX, NUL, COM1-9, LPT1-9) # Windows reserved names (CON, PRN, AUX, NUL, COM1-9, LPT1-9)
@@ -400,18 +427,20 @@ def sanitize_name_for_os(name, is_folder=True):
elif system == "Darwin": # macOS elif system == "Darwin": # macOS
# macOS illegal characters: : (colon is converted to / by the system) # macOS illegal characters: : (colon is converted to / by the system)
# Also can't start with dot (hidden file) for folders typically # Also can't start with dot (hidden file) for folders typically
sanitized = re.sub(r"[:]", "_", name) # Use pre-compiled pattern for better performance
sanitized = _MACOS_ILLEGAL_CHARS_PATTERN.sub("_", name)
# Remove control characters # Remove control characters
sanitized = re.sub(r"[\x00-\x1f]", "_", sanitized) sanitized = _CONTROL_CHARS_PATTERN.sub("_", sanitized)
# Avoid leading dot for folders (creates hidden folders) # Avoid leading dot for folders (creates hidden folders)
if is_folder and sanitized.startswith("."): if is_folder and sanitized.startswith("."):
sanitized = "_" + sanitized[1:] sanitized = "_" + sanitized[1:]
else: # Linux and others else: # Linux and others
# Linux illegal characters: / and null character # Linux illegal characters: / and null character
# Though / is illegal, most other chars are technically allowed # Though / is illegal, most other chars are technically allowed
sanitized = re.sub(r"[/\x00]", "_", name) # Use pre-compiled pattern for better performance
# Remove other control characters for safety sanitized = _LINUX_ILLEGAL_CHARS_PATTERN.sub("_", name)
sanitized = re.sub(r"[\x01-\x1f]", "_", sanitized) # Remove other control characters for safety (excluding \x00 which is already handled)
sanitized = _LINUX_CONTROL_CHARS_PATTERN.sub("_", sanitized)
# Avoid leading dot for folders (creates hidden folders) # Avoid leading dot for folders (creates hidden folders)
if is_folder and sanitized.startswith("."): if is_folder and sanitized.startswith("."):
sanitized = "_" + sanitized[1:] sanitized = "_" + sanitized[1:]
@@ -589,24 +618,44 @@ class ConversionThread(QThread):
log_updated = pyqtSignal(object) # Updated signal for log updates log_updated = pyqtSignal(object) # Updated signal for log updates
chapters_detected = pyqtSignal(int) # Signal for chapter detection chapters_detected = pyqtSignal(int) # Signal for chapter detection
# Default split pattern for TTS processing # Punctuation constants for unified handling across languages
DEFAULT_SPLIT_PATTERN = r"\n+" PUNCTUATION_SENTENCE = ".!?।。!?"
PUNCTUATION_SENTENCE_COMMA = ".!?,।。!?、,"
PUNCTUATION_COMMAS = ",,、"
# Languages that should not use split pattern (better handled by Kokoro internally) def _get_split_pattern(self, lang_code, subtitle_mode):
# These languages have different text segmentation rules (no spaces, character-based, etc.) """
NO_SPLIT_LANGUAGES = {"z", "j"} # Chinese, Japanese Get the appropriate split pattern based on language and subtitle mode.
# Language-specific punctuation patterns for subtitle splitting Args:
LANGUAGE_PUNCTUATION = { lang_code: Language code (a, b, e, f, etc.)
"z": { subtitle_mode: Subtitle mode ("Sentence", "Sentence + Comma", "Line", etc.)
"sentence": r"[。!?]", # Chinese: period, exclamation, question
"comma": r"[。!?、,]", # Chinese: includes enumeration comma and comma Returns:
}, Split pattern string
"j": { """
"sentence": r"[。!?]", # Japanese: period, exclamation, question # For English, always use newline splitting only
"comma": r"[。!?、,]", # Japanese: includes enumeration comma and comma if lang_code in ["a", "b"]:
}, return "\n"
}
# Determine spacing pattern based on language
spacing_pattern = r"\s*" if lang_code in ["z", "j"] else r"\s+"
# For Chinese/Japanese, when subtitle mode is Disabled or Line, prefer
# punctuation-based splitting instead of plain newline splitting.
if subtitle_mode in ("Disabled", "Line") and lang_code in ["z", "j"]:
return r"(?<=[{}]){}|\n+".format(self.PUNCTUATION_SENTENCE, spacing_pattern)
if subtitle_mode == "Line":
return "\n"
elif subtitle_mode == "Sentence":
return r"(?<=[{}]){}|\n+".format(self.PUNCTUATION_SENTENCE, spacing_pattern)
elif subtitle_mode == "Sentence + Comma":
return r"(?<=[{}]){}|\n+".format(
self.PUNCTUATION_SENTENCE_COMMA, spacing_pattern
)
else:
return r"\n+" # Default to line breaks
def __init__( def __init__(
self, self,
@@ -628,6 +677,7 @@ class ConversionThread(QThread):
): # Add use_gpu parameter ): # Add use_gpu parameter
super().__init__() super().__init__()
self._chapter_options_event = threading.Event() self._chapter_options_event = threading.Event()
self._timestamp_response_event = threading.Event()
self.np = np_module self.np = np_module
self.KPipeline = kpipeline_class self.KPipeline = kpipeline_class
self.file_name = file_name self.file_name = file_name
@@ -655,10 +705,9 @@ class ConversionThread(QThread):
self.use_gpu = use_gpu # Store the GPU setting self.use_gpu = use_gpu # Store the GPU setting
self.max_subtitle_words = 50 # Default value, will be overridden from GUI self.max_subtitle_words = 50 # Default value, will be overridden from GUI
self.silence_duration = 2.0 # Default value, will be overridden from GUI self.silence_duration = 2.0 # Default value, will be overridden from GUI
# Set split pattern based on language - some languages handle splitting better internally self.use_spacy_segmentation = True # Default, will be overridden from GUI
self.split_pattern = ( # Set split pattern based on language and subtitle mode
None if lang_code in self.NO_SPLIT_LANGUAGES else self.DEFAULT_SPLIT_PATTERN self.split_pattern = self._get_split_pattern(lang_code, subtitle_mode)
)
def _stream_audio_in_chunks( def _stream_audio_in_chunks(
self, segments, process_func, progress_prefix="Processing" self, segments, process_func, progress_prefix="Processing"
@@ -678,7 +727,7 @@ class ConversionThread(QThread):
total_samples = sum(len(segment) for segment in segments) total_samples = sum(len(segment) for segment in segments)
samples_processed = 0 samples_processed = 0
self.log_updated.emit(f"\n{progress_prefix} segments...") self.log_updated.emit((f"\n{progress_prefix} segments...", "grey"))
# Stream each segment individually # Stream each segment individually
for i, segment in enumerate(segments): for i, segment in enumerate(segments):
@@ -706,7 +755,9 @@ class ConversionThread(QThread):
# Clear segment bytes from memory # Clear segment bytes from memory
del segment_bytes del segment_bytes
except Exception as e: except Exception as e:
self.log_updated.emit(f"Error processing segment {i}: {str(e)}") self.log_updated.emit(
(f"Error processing segment {i}: {str(e)}", "red")
)
raise raise
return samples_processed return samples_processed
@@ -764,6 +815,9 @@ class ConversionThread(QThread):
self.log_updated.emit( self.log_updated.emit(
f"- Subtitle format: {next((label for value, label in SUBTITLE_FORMATS if value == getattr(self, 'subtitle_format', 'srt')), getattr(self, 'subtitle_format', 'srt'))}" f"- Subtitle format: {next((label for value, label in SUBTITLE_FORMATS if value == getattr(self, 'subtitle_format', 'srt')), getattr(self, 'subtitle_format', 'srt'))}"
) )
self.log_updated.emit(
f"- Use spaCy for sentence segmentation: {'Yes' if getattr(self, 'use_spacy_segmentation', False) else 'No'}"
)
self.log_updated.emit(f"- Save option: {self.save_option}") self.log_updated.emit(f"- Save option: {self.save_option}")
if self.replace_single_newlines: if self.replace_single_newlines:
self.log_updated.emit(f"- Replace single newlines: Yes") self.log_updated.emit(f"- Replace single newlines: Yes")
@@ -817,7 +871,7 @@ class ConversionThread(QThread):
f"- Output folder: {self.output_folder or os.getcwd()}" f"- Output folder: {self.output_folder or os.getcwd()}"
) )
self.log_updated.emit("\nInitializing TTS pipeline...") self.log_updated.emit(("\nInitializing TTS pipeline...", "grey"))
# Set device based on use_gpu setting and platform # Set device based on use_gpu setting and platform
if self.use_gpu: if self.use_gpu:
@@ -844,18 +898,26 @@ class ConversionThread(QThread):
) )
elif file_ext == ".txt" and detect_timestamps_in_text(self.file_name): elif file_ext == ".txt" and detect_timestamps_in_text(self.file_name):
is_timestamp_text = True is_timestamp_text = True
self.log_updated.emit("\nDetected timestamps in text file") self.log_updated.emit(
("\nDetected timestamps in text file", "grey")
)
# Signal to ask user (-1 indicates timestamp detection) # Signal to ask user (-1 indicates timestamp detection)
self.chapters_detected.emit(-1) self.chapters_detected.emit(-1)
# Wait for user response # Wait for user response using event with timeout for responsive cancellation
while not hasattr(self, "_timestamp_response"): while not self._timestamp_response_event.wait(
timeout=_USER_RESPONSE_TIMEOUT
):
if self.cancel_requested:
self.conversion_finished.emit("Cancelled", None)
return
# Check cancellation one more time after event is set
if self.cancel_requested: if self.cancel_requested:
self.conversion_finished.emit("Cancelled", None) self.conversion_finished.emit("Cancelled", None)
return return
time.sleep(0.1)
if not self._timestamp_response: if not self._timestamp_response:
is_timestamp_text = False is_timestamp_text = False
delattr(self, "_timestamp_response") delattr(self, "_timestamp_response")
self._timestamp_response_event.clear()
# Process subtitle files separately # Process subtitle files separately
if is_subtitle_file or is_timestamp_text: if is_subtitle_file or is_timestamp_text:
@@ -875,12 +937,12 @@ class ConversionThread(QThread):
text = clean_text(text) text = clean_text(text)
# Remove metadata markers from the text to be processed # Remove metadata markers from the text to be processed
metadata_pattern = r"<<METADATA_[^:]+:[^>]*>>" # Use pre-compiled pattern for better performance
text = re.sub(metadata_pattern, "", text) text = _METADATA_TAG_PATTERN.sub("", text)
# --- Chapter splitting logic --- # --- Chapter splitting logic ---
chapter_pattern = r"<<CHAPTER_MARKER:(.*?)>>" # Use pre-compiled pattern for better performance
chapter_splits = list(re.finditer(chapter_pattern, text)) chapter_splits = list(_CHAPTER_MARKER_SEARCH_PATTERN.finditer(text))
chapters = [] chapters = []
if chapter_splits: if chapter_splits:
# prepend Introduction for content before first marker # prepend Introduction for content before first marker
@@ -936,7 +998,7 @@ class ConversionThread(QThread):
(f"\nDetected chapters ({total_chapters}):\n" + chapter_list) (f"\nDetected chapters ({total_chapters}):\n" + chapter_list)
) )
else: else:
self.log_updated.emit((f"\nProcessing {chapters[0][0]}...")) self.log_updated.emit((f"\nProcessing {chapters[0][0]}...", "grey"))
# If save_chapters_separately is enabled, find a unique suffix ONCE and use for both folder and merged file # If save_chapters_separately is enabled, find a unique suffix ONCE and use for both folder and merged file
save_chapters_separately = getattr(self, "save_chapters_separately", False) save_chapters_separately = getattr(self, "save_chapters_separately", False)
@@ -984,10 +1046,14 @@ class ConversionThread(QThread):
parent_dir, f"{sanitized_base_name}{suffix}_chapters" parent_dir, f"{sanitized_base_name}{suffix}_chapters"
) )
# Only check for files with allowed extensions (extension without dot, case-insensitive) # Only check for files with allowed extensions (extension without dot, case-insensitive)
# Use generator expression to avoid processing all files upfront
file_parts = (
os.path.splitext(fname) for fname in os.listdir(parent_dir)
)
clash = any( clash = any(
os.path.splitext(fname)[0] == f"{sanitized_base_name}{suffix}" name == f"{sanitized_base_name}{suffix}"
and os.path.splitext(fname)[1][1:].lower() in allowed_exts and ext[1:].lower() in allowed_exts
for fname in os.listdir(parent_dir) for name, ext in file_parts
) )
if not os.path.exists(chapters_out_dir_candidate) and not clash: if not os.path.exists(chapters_out_dir_candidate) and not clash:
break break
@@ -1334,11 +1400,96 @@ class ConversionThread(QThread):
else: else:
chapter_subtitle_path = None chapter_subtitle_path = None
chapter_subtitle_file = None chapter_subtitle_file = None
for result in tts(
# Determine if spaCy segmentation should be used for PRE-TTS segmentation
# Only non-English languages use spaCy for pre-segmentation
# English uses spaCy only for subtitle generation (post-TTS)
# spaCy is disabled when subtitle mode is "Disabled" or "Line"
# spaCy is also disabled when input is a subtitle file
is_subtitle_input = (
not self.is_direct_text
and self.file_name
and os.path.splitext(self.file_name)[1].lower()
in [".srt", ".ass", ".vtt"]
)
use_spacy = (
getattr(self, "use_spacy_segmentation", False)
and self.subtitle_mode not in ["Disabled", "Line"]
and not is_subtitle_input
)
spacy_sentences = None
active_split_pattern = self.split_pattern
spacing_pattern = r"\s*" if self.lang_code in ["z", "j"] else r"\s+"
# Pre-load spaCy model for English if it will be needed for subtitle generation
if (
use_spacy
and self.lang_code in ["a", "b"]
and self.subtitle_mode in ["Sentence", "Sentence + Comma"]
):
from abogen.spacy_utils import get_spacy_model
nlp = get_spacy_model(
self.lang_code,
log_callback=lambda msg: self.log_updated.emit(msg),
)
if nlp:
self.log_updated.emit(
(
"\nUsing spaCy for sentence segmentation (only for subtitles)...",
"grey",
)
)
if use_spacy and self.lang_code not in ["a", "b"]:
# Non-English: use spaCy for pre-TTS segmentation
self.log_updated.emit(
("\nUsing spaCy for sentence segmentation (pre-TTS)...", "grey")
)
from abogen.spacy_utils import segment_sentences
spacy_sentences = segment_sentences(
chapter_text, chapter_text,
self.lang_code,
log_callback=lambda msg: self.log_updated.emit(msg),
)
if spacy_sentences:
self.log_updated.emit(
(
f"\nspaCy: Text segmented into {len(spacy_sentences)} sentences...",
"grey",
)
)
# For Sentence + Comma mode, still split on commas within spaCy sentences
if self.subtitle_mode == "Sentence + Comma":
active_split_pattern = r"(?<=[{}]){}|\n+".format(
self.PUNCTUATION_COMMAS, spacing_pattern
)
else:
active_split_pattern = (
"\n" # Use newline splitting for Sentence mode
)
else:
self.log_updated.emit(
("\nspaCy: Fallback to default segmentation...", "grey")
)
# Process text - either as spaCy sentences or as single text
text_segments = spacy_sentences if spacy_sentences else [chapter_text]
# Print active split pattern used by the TTS engine once for this batch
try:
print(f"Using split pattern: {active_split_pattern!r}")
except Exception:
# Print must never break processing
print("Using split pattern: (unprintable)")
for text_segment in text_segments:
for result in tts(
text_segment,
voice=loaded_voice, voice=loaded_voice,
speed=self.speed, speed=self.speed,
split_pattern=self.split_pattern, split_pattern=active_split_pattern,
): ):
# Print the result for debugging # Print the result for debugging
# print(f"Result: {result}") # print(f"Result: {result}")
@@ -1383,6 +1534,22 @@ class ConversionThread(QThread):
# Subtitle logic # Subtitle logic
if self.subtitle_mode != "Disabled": if self.subtitle_mode != "Disabled":
tokens_list = getattr(result, "tokens", []) tokens_list = getattr(result, "tokens", [])
# Fallback for languages without token support (non-English)
# Create a single token representing the entire segment duration
if not tokens_list and result.graphemes:
class FakeToken:
def __init__(self, text, start, end):
self.text = text
self.start_ts = start
self.end_ts = end
self.whitespace = ""
tokens_list = [
FakeToken(result.graphemes, 0, chunk_dur)
]
tokens_with_timestamps = [] tokens_with_timestamps = []
chapter_tokens_with_timestamps = [] chapter_tokens_with_timestamps = []
@@ -1401,7 +1568,8 @@ class ConversionThread(QThread):
{ {
"start": chapter_current_time "start": chapter_current_time
+ (tok.start_ts or 0), + (tok.start_ts or 0),
"end": chapter_current_time + (tok.end_ts or 0), "end": chapter_current_time
+ (tok.end_ts or 0),
"text": tok.text, "text": tok.text,
"whitespace": tok.whitespace, "whitespace": tok.whitespace,
} }
@@ -1485,7 +1653,10 @@ class ConversionThread(QThread):
chapter_current_time += chunk_dur chapter_current_time += chunk_dur
# Calculate percentage based on characters processed # Calculate percentage based on characters processed
percent = min( percent = min(
int(self.processed_char_count / self.total_char_count * 100), 99 int(
self.processed_char_count / self.total_char_count * 100
),
99,
) )
# Calculate ETR based on characters processed # Calculate ETR based on characters processed
@@ -1498,7 +1669,9 @@ class ConversionThread(QThread):
chars_done > 0 and elapsed > 0.5 chars_done > 0 and elapsed > 0.5
): # Check elapsed > 0.5 to avoid instability ): # Check elapsed > 0.5 to avoid instability
avg_time_per_char = elapsed / chars_done avg_time_per_char = elapsed / chars_done
remaining = self.total_char_count - self.processed_char_count remaining = (
self.total_char_count - self.processed_char_count
)
if remaining > 0: if remaining > 0:
secs = avg_time_per_char * remaining secs = avg_time_per_char * remaining
h = int(secs // 3600) h = int(secs // 3600)
@@ -1642,14 +1815,14 @@ class ConversionThread(QThread):
if self.subtitle_mode != "Disabled": if self.subtitle_mode != "Disabled":
self.conversion_finished.emit( self.conversion_finished.emit(
( (
f"\nAudiobook saved to: {merged_out_path}\n\nSubtitle saved to: {merged_subtitle_path}", f"\nAudio saved to: {merged_out_path}\n\nSubtitle saved to: {merged_subtitle_path}",
"green", "green",
), ),
merged_out_path, merged_out_path,
) )
else: else:
self.conversion_finished.emit( self.conversion_finished.emit(
(f"\nAudiobook saved to: {merged_out_path}", "green"), (f"\nAudio saved to: {merged_out_path}", "green"),
merged_out_path, merged_out_path,
) )
else: else:
@@ -1701,7 +1874,9 @@ class ConversionThread(QThread):
) )
return return
self.log_updated.emit(f"\nFound {len(subtitles)} subtitle entries") self.log_updated.emit(
(f"\nFound {len(subtitles)} subtitle entries", "grey")
)
# Setup output paths # Setup output paths
base_name = os.path.splitext(os.path.basename(base_path))[0] base_name = os.path.splitext(os.path.basename(base_path))[0]
@@ -1727,10 +1902,12 @@ class ConversionThread(QThread):
allowed_exts = set(SUPPORTED_SOUND_FORMATS + SUPPORTED_SUBTITLE_FORMATS) allowed_exts = set(SUPPORTED_SOUND_FORMATS + SUPPORTED_SUBTITLE_FORMATS)
while True: while True:
suffix = f"_{counter}" if counter > 1 else "" suffix = f"_{counter}" if counter > 1 else ""
# Use generator expression to avoid processing all files upfront
file_parts = (os.path.splitext(f) for f in os.listdir(parent_dir))
if not any( if not any(
os.path.splitext(f)[0] == f"{sanitized_base_name}{suffix}" name == f"{sanitized_base_name}{suffix}"
and os.path.splitext(f)[1][1:].lower() in allowed_exts and ext[1:].lower() in allowed_exts
for f in os.listdir(parent_dir) for name, ext in file_parts
): ):
break break
counter += 1 counter += 1
@@ -1881,7 +2058,11 @@ class ConversionThread(QThread):
int(start_time % 60), int(start_time % 60),
) )
ms1 = int((start_time - int(start_time)) * 1000) ms1 = int((start_time - int(start_time)) * 1000)
is_last = use_gaps and idx == len(subtitles) is_last = (
is_timestamp_text
or (use_gaps and idx == len(subtitles))
or end_time is None
)
if is_last: if is_last:
time_str = ( time_str = (
f"{h1:02d}:{m1:02d}:{s1:02d}" f"{h1:02d}:{m1:02d}:{s1:02d}"
@@ -1938,7 +2119,10 @@ class ConversionThread(QThread):
audio_duration = len(full_audio) / rate audio_duration = len(full_audio) / rate
# Use actual audio length for timing # Use actual audio length for timing
if use_gaps: if is_timestamp_text:
end_time = start_time + audio_duration
subtitle_duration = audio_duration
elif use_gaps:
end_time = min(start_time + audio_duration, next_start) end_time = min(start_time + audio_duration, next_start)
subtitle_duration = end_time - start_time subtitle_duration = end_time - start_time
elif subtitle_duration is None: elif subtitle_duration is None:
@@ -2127,7 +2311,7 @@ class ConversionThread(QThread):
subtitle_file.close() subtitle_file.close()
self.progress_updated.emit(100, "00:00:00") self.progress_updated.emit(100, "00:00:00")
result_msg = f"\nAudiobook saved to: {merged_out_path}" + ( result_msg = f"\nAudio saved to: {merged_out_path}" + (
f"\n\nSubtitle saved to: {subtitle_path}" if subtitle_path else "" f"\n\nSubtitle saved to: {subtitle_path}" if subtitle_path else ""
) )
self.conversion_finished.emit((result_msg, "green"), merged_out_path) self.conversion_finished.emit((result_msg, "green"), merged_out_path)
@@ -2155,6 +2339,7 @@ class ConversionThread(QThread):
def set_timestamp_response(self, treat_as_subtitle): def set_timestamp_response(self, treat_as_subtitle):
"""Set whether to treat timestamp text file as subtitle.""" """Set whether to treat timestamp text file as subtitle."""
self._timestamp_response = treat_as_subtitle self._timestamp_response = treat_as_subtitle
self._timestamp_response_event.set()
def _extract_and_add_metadata_tags_to_ffmpeg_cmd(self): def _extract_and_add_metadata_tags_to_ffmpeg_cmd(self):
"""Extract metadata tags from text content and add them to ffmpeg command""" """Extract metadata tags from text content and add them to ffmpeg command"""
@@ -2281,16 +2466,19 @@ class ConversionThread(QThread):
processed_tokens = tokens_with_timestamps # Use tokens directly processed_tokens = tokens_with_timestamps # Use tokens directly
# For English with spaCy enabled and sentence-based modes, use spaCy for sentence boundaries
# spaCy is disabled when subtitle mode is "Disabled" or "Line"
use_spacy_for_english = (
getattr(self, "use_spacy_segmentation", False)
and self.subtitle_mode not in ["Disabled", "Line"]
and self.lang_code in ["a", "b"]
and self.subtitle_mode in ["Sentence", "Sentence + Comma"]
)
# Use processed_tokens instead of tokens_with_timestamps for the rest of the method # Use processed_tokens instead of tokens_with_timestamps for the rest of the method
if self.subtitle_mode == "Sentence + Highlighting": if self.subtitle_mode == "Sentence + Highlighting":
# Sentence-based processing with karaoke highlighting # Sentence-based processing with karaoke highlighting
# Use language-specific punctuation for CJK languages (without comma) # Use punctuation without comma
lang_punct = self.LANGUAGE_PUNCTUATION.get(self.lang_code, {}) separator = r"[{}]".format(self.PUNCTUATION_SENTENCE)
separator = (
lang_punct.get("sentence", r"[.!?]")
if isinstance(lang_punct, dict)
else r"[.!?]"
)
current_sentence = [] current_sentence = []
word_count = 0 word_count = 0
@@ -2347,25 +2535,102 @@ class ConversionThread(QThread):
subtitle_entries[-1] = (start, fallback_end_time, text) subtitle_entries[-1] = (start, fallback_end_time, text)
elif self.subtitle_mode in ["Sentence", "Sentence + Comma", "Line"]: elif self.subtitle_mode in ["Sentence", "Sentence + Comma", "Line"]:
# Check if we should use spaCy for English sentence boundaries
if use_spacy_for_english and self.subtitle_mode != "Line":
# Use spaCy for English sentence boundary detection (model already loaded)
from abogen.spacy_utils import get_spacy_model
nlp = get_spacy_model(
self.lang_code
) # No log_callback since model is already loaded
if nlp:
# Build full text and track character positions to token indices
full_text = ""
char_to_token = [] # Maps character index to token index
for idx, token in enumerate(processed_tokens):
start_char = len(full_text)
text_part = token["text"] + (token.get("whitespace", "") or "")
full_text += text_part
char_to_token.extend([idx] * len(text_part))
# Get sentence boundaries from spaCy
doc = nlp(full_text)
sentence_boundaries = [sent.end_char for sent in doc.sents]
# For "Sentence + Comma" mode, also split on commas
if self.subtitle_mode == "Sentence + Comma":
comma_positions = [
i + 1 for i, c in enumerate(full_text) if c == ","
]
sentence_boundaries = sorted(
set(sentence_boundaries + comma_positions)
)
# Group tokens by sentence boundaries
current_sentence = []
word_count = 0
current_char_pos = 0
boundary_idx = 0
for idx, token in enumerate(processed_tokens):
current_sentence.append(token)
word_count += 1
text_len = len(token["text"]) + len(
token.get("whitespace", "") or ""
)
current_char_pos += text_len
# Check if we've hit a sentence boundary or max words
at_boundary = (
boundary_idx < len(sentence_boundaries)
and current_char_pos >= sentence_boundaries[boundary_idx]
)
if at_boundary or word_count >= max_subtitle_words:
if current_sentence:
start_time = current_sentence[0]["start"]
end_time = current_sentence[-1]["end"]
sentence_text = "".join(
t["text"] + (t.get("whitespace", "") or "")
for t in current_sentence
)
subtitle_entries.append(
(start_time, end_time, sentence_text.strip())
)
current_sentence = []
word_count = 0
if at_boundary:
boundary_idx += 1
# Add remaining tokens
if current_sentence:
start_time = current_sentence[0]["start"]
end_time = current_sentence[-1]["end"]
sentence_text = "".join(
t["text"] + (t.get("whitespace", "") or "")
for t in current_sentence
)
subtitle_entries.append(
(start_time, end_time, sentence_text.strip())
)
# Fallback for last entry
if subtitle_entries and fallback_end_time is not None:
last_entry = subtitle_entries[-1]
start, end, text = last_entry
if end is None or end <= start or end <= 0:
subtitle_entries[-1] = (start, fallback_end_time, text)
return # Exit early, spaCy processing complete
# Default regex-based processing (non-English or spaCy unavailable)
# Define separator pattern based on mode # Define separator pattern based on mode
if self.subtitle_mode == "Line": if self.subtitle_mode == "Line":
separator = r"\n" separator = r"\n"
elif self.subtitle_mode == "Sentence": elif self.subtitle_mode == "Sentence":
# Use language-specific punctuation for CJK languages (without comma) # Use punctuation without comma
lang_punct = self.LANGUAGE_PUNCTUATION.get(self.lang_code, {}) separator = r"[{}]".format(self.PUNCTUATION_SENTENCE)
separator = (
lang_punct.get("sentence", r"[.!?]")
if isinstance(lang_punct, dict)
else r"[.!?]"
)
else: # Sentence + Comma else: # Sentence + Comma
# Use language-specific punctuation for CJK languages (with comma) # Use punctuation with comma
lang_punct = self.LANGUAGE_PUNCTUATION.get(self.lang_code, {}) separator = r"[{}]".format(self.PUNCTUATION_SENTENCE_COMMA)
separator = (
lang_punct.get("comma", r"[.!?,]")
if isinstance(lang_punct, dict)
else r"[.!?,]"
)
current_sentence = [] current_sentence = []
word_count = 0 word_count = 0
+132 -32
View File
@@ -1,5 +1,6 @@
import os import os
import time import time
import sys
import tempfile import tempfile
import platform import platform
import base64 import base64
@@ -390,7 +391,7 @@ class InputBox(QLabel):
window = self.window() window = self.window()
if hasattr(window, "subtitle_combo"): if hasattr(window, "subtitle_combo"):
# Only enable if language supports it # Only enable if language supports it
current_lang = getattr(window, "lang_code", "a") current_lang = getattr(window, "selected_lang", "a")
window.subtitle_combo.setEnabled( window.subtitle_combo.setEnabled(
current_lang in SUPPORTED_LANGUAGES_FOR_SUBTITLE_GENERATION current_lang in SUPPORTED_LANGUAGES_FOR_SUBTITLE_GENERATION
) )
@@ -824,9 +825,10 @@ class abogen(QWidget):
self.use_gpu = self.config.get( self.use_gpu = self.config.get(
"use_gpu", True # Load GPU setting with default True "use_gpu", True # Load GPU setting with default True
) )
self.replace_single_newlines = self.config.get("replace_single_newlines", False) self.replace_single_newlines = self.config.get("replace_single_newlines", True)
self.use_silent_gaps = self.config.get("use_silent_gaps", False) self.use_silent_gaps = self.config.get("use_silent_gaps", True)
self.subtitle_speed_method = self.config.get("subtitle_speed_method", "tts") self.subtitle_speed_method = self.config.get("subtitle_speed_method", "tts")
self.use_spacy_segmentation = self.config.get("use_spacy_segmentation", True)
self._pending_close_event = None self._pending_close_event = None
self.gpu_ok = False # Initialize GPU availability status self.gpu_ok = False # Initialize GPU availability status
@@ -1062,9 +1064,6 @@ class abogen(QWidget):
) )
self.subtitle_combo.setCurrentText(self.subtitle_mode) self.subtitle_combo.setCurrentText(self.subtitle_mode)
self.subtitle_combo.currentTextChanged.connect(self.on_subtitle_mode_changed) self.subtitle_combo.currentTextChanged.connect(self.on_subtitle_mode_changed)
# Enable/disable subtitle options based on selected language (profile or voice)
enable = self.selected_lang in SUPPORTED_LANGUAGES_FOR_SUBTITLE_GENERATION
self.subtitle_combo.setEnabled(enable)
subtitle_layout.addWidget(self.subtitle_combo) subtitle_layout.addWidget(self.subtitle_combo)
controls_layout.addLayout(subtitle_layout) controls_layout.addLayout(subtitle_layout)
@@ -1146,10 +1145,10 @@ class abogen(QWidget):
except Exception: except Exception:
# Fail-safe: don't crash UI if model manipulation isn't supported on some platforms # Fail-safe: don't crash UI if model manipulation isn't supported on some platforms
pass pass
# Enable/disable subtitle format based on selected language
self.subtitle_format_combo.setEnabled( # Enable/disable subtitle options based on selected language (profile or voice)
self.selected_lang in SUPPORTED_LANGUAGES_FOR_SUBTITLE_GENERATION self.update_subtitle_options_availability()
)
controls_layout.addLayout(subtitle_format_layout) controls_layout.addLayout(subtitle_format_layout)
# Replace single newlines dropdown (acts like checkbox) # Replace single newlines dropdown (acts like checkbox)
@@ -1590,19 +1589,69 @@ class abogen(QWidget):
self.config["speed"] = s self.config["speed"] = s
save_config(self.config) save_config(self.config)
def update_subtitle_options_availability(self):
"""
Update the enabled state of subtitle options based on the selected language.
For non-English languages, only sentence-based and line-based modes are supported.
"""
# Check if current file is a subtitle file
is_subtitle_input = False
if self.selected_file and self.selected_file.lower().endswith(
(".srt", ".ass", ".vtt")
):
is_subtitle_input = True
if self.selected_lang not in SUPPORTED_LANGUAGES_FOR_SUBTITLE_GENERATION:
self.subtitle_combo.setEnabled(False)
self.subtitle_format_combo.setEnabled(False)
return
# Only enable subtitle_combo if it's NOT a subtitle input
self.subtitle_combo.setEnabled(not is_subtitle_input)
self.subtitle_format_combo.setEnabled(True)
is_english = self.selected_lang in ["a", "b"]
# Items to keep enabled for non-English
allowed_modes = ["Disabled", "Line", "Sentence", "Sentence + Comma"]
model = self.subtitle_combo.model()
for i in range(self.subtitle_combo.count()):
text = self.subtitle_combo.itemText(i)
item = model.item(i)
if not item:
continue
if is_english:
item.setEnabled(True)
else:
if text in allowed_modes:
item.setEnabled(True)
else:
item.setEnabled(False)
# If current selection is disabled, switch to a valid one
current_text = self.subtitle_combo.currentText()
current_idx = self.subtitle_combo.currentIndex()
current_item = model.item(current_idx)
if current_item and not current_item.isEnabled():
# Switch to "Sentence" if available, else "Disabled"
sentence_idx = self.subtitle_combo.findText("Sentence")
if sentence_idx >= 0:
self.subtitle_combo.setCurrentIndex(sentence_idx)
else:
self.subtitle_combo.setCurrentIndex(0) # Disabled
self.subtitle_mode = self.subtitle_combo.currentText()
def on_voice_changed(self, index): def on_voice_changed(self, index):
voice = self.voice_combo.itemData(index) voice = self.voice_combo.itemData(index)
self.selected_voice, self.selected_lang = voice, voice[0] self.selected_voice, self.selected_lang = voice, voice[0]
self.config["selected_voice"] = voice self.config["selected_voice"] = voice
save_config(self.config) save_config(self.config)
# Enable/disable subtitle options based on language # Enable/disable subtitle options based on language
if self.selected_lang in SUPPORTED_LANGUAGES_FOR_SUBTITLE_GENERATION: self.update_subtitle_options_availability()
self.subtitle_combo.setEnabled(True)
self.subtitle_format_combo.setEnabled(True)
self.subtitle_mode = self.subtitle_combo.currentText()
else:
self.subtitle_combo.setEnabled(False)
self.subtitle_format_combo.setEnabled(False)
def on_voice_combo_changed(self, index): def on_voice_combo_changed(self, index):
data = self.voice_combo.itemData(index) data = self.voice_combo.itemData(index)
@@ -1624,12 +1673,7 @@ class abogen(QWidget):
self.config.pop("selected_voice", None) self.config.pop("selected_voice", None)
save_config(self.config) save_config(self.config)
# enable subtitles based on profile language # enable subtitles based on profile language
self.subtitle_combo.setEnabled( self.update_subtitle_options_availability()
self.selected_lang in SUPPORTED_LANGUAGES_FOR_SUBTITLE_GENERATION
)
self.subtitle_format_combo.setEnabled(
self.selected_lang in SUPPORTED_LANGUAGES_FOR_SUBTITLE_GENERATION
)
else: else:
self.mixed_voice_state = None self.mixed_voice_state = None
self.selected_profile_name = None self.selected_profile_name = None
@@ -1638,13 +1682,7 @@ class abogen(QWidget):
if "selected_profile_name" in self.config: if "selected_profile_name" in self.config:
del self.config["selected_profile_name"] del self.config["selected_profile_name"]
save_config(self.config) save_config(self.config)
if self.selected_lang in SUPPORTED_LANGUAGES_FOR_SUBTITLE_GENERATION: self.update_subtitle_options_availability()
self.subtitle_combo.setEnabled(True)
self.subtitle_format_combo.setEnabled(True)
self.subtitle_mode = self.subtitle_combo.currentText()
else:
self.subtitle_combo.setEnabled(False)
self.subtitle_format_combo.setEnabled(False)
def update_subtitle_combo_for_profile(self, profile_name): def update_subtitle_combo_for_profile(self, profile_name):
from abogen.voice_profiles import load_profiles from abogen.voice_profiles import load_profiles
@@ -2044,7 +2082,7 @@ class abogen(QWidget):
# pipeline_loaded_callback remains unchanged # pipeline_loaded_callback remains unchanged
def pipeline_loaded_callback(np_module, kpipeline_class, error): def pipeline_loaded_callback(np_module, kpipeline_class, error):
if error: if error:
self.update_log((f"Error loading numpy or KPipeline: {error}", False)) self.update_log((f"Error loading numpy or KPipeline: {error}", "red"))
prevent_sleep_end() prevent_sleep_end()
return return
@@ -2091,6 +2129,8 @@ class abogen(QWidget):
self.conversion_thread.use_silent_gaps = self.use_silent_gaps self.conversion_thread.use_silent_gaps = self.use_silent_gaps
# Pass subtitle_speed_method setting # Pass subtitle_speed_method setting
self.conversion_thread.subtitle_speed_method = self.subtitle_speed_method self.conversion_thread.subtitle_speed_method = self.subtitle_speed_method
# Pass use_spacy_segmentation setting
self.conversion_thread.use_spacy_segmentation = self.use_spacy_segmentation
# Pass separate_chapters_format setting # Pass separate_chapters_format setting
self.conversion_thread.separate_chapters_format = ( self.conversion_thread.separate_chapters_format = (
self.separate_chapters_format self.separate_chapters_format
@@ -2227,7 +2267,7 @@ class abogen(QWidget):
self.is_converting = False self.is_converting = False
elapsed = int(time.time() - self.start_time) elapsed = int(time.time() - self.start_time)
h, m, s = elapsed // 3600, (elapsed % 3600) // 60, elapsed % 60 h, m, s = elapsed // 3600, (elapsed % 3600) // 60, elapsed % 60
self.update_log(f"\nTime elapsed: {h:02d}:{m:02d}:{s:02d}") self.update_log((f"\nTime elapsed: {h:02d}:{m:02d}:{s:02d}", "grey"))
# Default to showing the button # Default to showing the button
show_open_file_button = True show_open_file_button = True
@@ -2790,6 +2830,33 @@ class abogen(QWidget):
self.conversion_thread.cancel() self.conversion_thread.cancel()
self.conversion_thread.wait() self.conversion_thread.wait()
def cleanup_preview_threads(self):
# Stop preview generation thread
if (
hasattr(self, "preview_thread")
and self.preview_thread is not None
and self.preview_thread.isRunning()
):
self.preview_thread.terminate()
self.preview_thread.wait()
# Stop audio playback thread
if (
hasattr(self, "play_audio_thread")
and self.play_audio_thread is not None
and self.play_audio_thread.isRunning()
):
self.play_audio_thread.stop()
self.play_audio_thread.wait()
# Cleanup pygame mixer if initialized
try:
pygame = sys.modules.get("pygame")
if pygame and pygame.mixer.get_init():
pygame.mixer.quit()
except Exception:
pass
def closeEvent(self, event): def closeEvent(self, event):
if self.is_converting: if self.is_converting:
box = QMessageBox(self) box = QMessageBox(self)
@@ -2804,11 +2871,13 @@ class abogen(QWidget):
box.setDefaultButton(QMessageBox.StandardButton.No) box.setDefaultButton(QMessageBox.StandardButton.No)
if box.exec() == QMessageBox.StandardButton.Yes: if box.exec() == QMessageBox.StandardButton.Yes:
self.cleanup_conversion_thread() self.cleanup_conversion_thread()
self.cleanup_preview_threads()
event.accept() event.accept()
else: else:
event.ignore() event.ignore()
else: else:
self.cleanup_conversion_thread() self.cleanup_conversion_thread()
self.cleanup_preview_threads()
event.accept() event.accept()
def show_chapter_options_dialog(self, chapter_count): def show_chapter_options_dialog(self, chapter_count):
@@ -3140,6 +3209,25 @@ class abogen(QWidget):
# Add separator # Add separator
menu.addSeparator() menu.addSeparator()
# Add spaCy sentence segmentation option
spacy_action = QAction("Use spaCy for sentence segmentation", self)
spacy_action.setCheckable(True)
spacy_action.setChecked(self.use_spacy_segmentation)
spacy_action.triggered.connect(
lambda checked: self.toggle_spacy_segmentation(checked)
)
menu.addAction(spacy_action)
# Add separator
menu.addSeparator()
# Add "Pre-download models and voices for offline use" option
predownload_action = QAction(
"Pre-download models and voices for offline use", self
)
predownload_action.triggered.connect(self.show_predownload_dialog)
menu.addAction(predownload_action)
# Add "Disable Kokoro's internet access" option # Add "Disable Kokoro's internet access" option
disable_kokoro_action = QAction("Disable Kokoro's internet access", self) disable_kokoro_action = QAction("Disable Kokoro's internet access", self)
disable_kokoro_action.setCheckable(True) disable_kokoro_action.setCheckable(True)
@@ -3204,6 +3292,11 @@ class abogen(QWidget):
self.config["subtitle_speed_method"] = method self.config["subtitle_speed_method"] = method
save_config(self.config) save_config(self.config)
def toggle_spacy_segmentation(self, enabled):
self.use_spacy_segmentation = enabled
self.config["use_spacy_segmentation"] = enabled
save_config(self.config)
def restart_app(self): def restart_app(self):
import sys import sys
@@ -3499,6 +3592,13 @@ Categories=AudioVideo;Audio;Utility;
self.voice_combo.setCurrentIndex(idx) self.voice_combo.setCurrentIndex(idx)
self.mixed_voice_state = dialog.get_selected_voices() self.mixed_voice_state = dialog.get_selected_voices()
def show_predownload_dialog(self):
"""Show the pre-download models and voices dialog."""
from abogen.predownload_gui import PreDownloadDialog
dialog = PreDownloadDialog(self)
dialog.exec()
def show_about_dialog(self): def show_about_dialog(self):
"""Show an About dialog with program information including GitHub link.""" """Show an About dialog with program information including GitHub link."""
# Get application icon for dialog # Get application icon for dialog
+590
View File
@@ -0,0 +1,590 @@
"""
Pre-download dialog and worker for Abogen
This module consolidates pre-download logic for Kokoro voices and model
and spaCy language models. The code favors clarity, avoids duplication,
and handles optional dependencies gracefully.
"""
from typing import List, Optional, Tuple
import importlib
import importlib.util
from PyQt6.QtWidgets import (
QDialog,
QVBoxLayout,
QHBoxLayout,
QLabel,
QPushButton,
QSpacerItem,
QSizePolicy,
)
from PyQt6.QtCore import QThread, pyqtSignal
from abogen.constants import COLORS, VOICES_INTERNAL
from abogen.spacy_utils import SPACY_MODELS
import abogen.hf_tracker
# Helpers
def _unique_sorted_models() -> List[str]:
"""Return a sorted list of unique spaCy model package names."""
return sorted(set(SPACY_MODELS.values()))
def _is_package_installed(pkg_name: str) -> bool:
"""Return True if a package with the given name can be imported (site-packages)."""
try:
return importlib.util.find_spec(pkg_name) is not None
except Exception:
return False
# NOTE: explicit HF cache helper removed; we use try_to_load_from_cache in-scope where needed
class PreDownloadWorker(QThread):
"""Worker thread to download required models/voices.
Emits human-readable messages via `progress`. Uses `category_done` to indicate
a category (voices/model/spacy) finished successfully. Emits `error` on exception
and `finished` after all work completes.
"""
# Emit (category, status, message)
progress = pyqtSignal(str, str, str)
category_done = pyqtSignal(str)
finished = pyqtSignal()
error = pyqtSignal(str)
def __init__(self, parent=None):
super().__init__(parent)
self._cancelled = False
# repo and filenames used for Kokoro model
self._repo_id = "hexgrad/Kokoro-82M"
self._model_files = ["kokoro-v1_0.pth", "config.json"]
# Track download success per category
self._voices_success = False
self._model_success = False
self._spacy_success = False
# Suppress HF tracker warnings during downloads
self._original_emitter = abogen.hf_tracker.show_warning_signal_emitter
def cancel(self) -> None:
self._cancelled = True
def run(self) -> None:
# Suppress HF tracker warnings during downloads
abogen.hf_tracker.show_warning_signal_emitter = None
try:
self._download_kokoro_voices()
if self._cancelled:
return
if self._voices_success:
self.category_done.emit("voices")
self._download_kokoro_model()
if self._cancelled:
return
if self._model_success:
self.category_done.emit("model")
self._download_spacy_models()
if self._cancelled:
return
if self._spacy_success:
self.category_done.emit("spacy")
self.finished.emit()
except Exception as exc: # pragma: no cover - best-effort reporting
self.error.emit(str(exc))
finally:
# Restore original emitter
abogen.hf_tracker.show_warning_signal_emitter = self._original_emitter
# Kokoro voices
def _download_kokoro_voices(self) -> None:
self._voices_success = True
try:
from huggingface_hub import hf_hub_download, try_to_load_from_cache
except Exception:
self.progress.emit(
"voice", "warning", "huggingface_hub not installed, skipping voices..."
)
self._voices_success = False
return
voice_list = VOICES_INTERNAL
for idx, voice in enumerate(voice_list, start=1):
if self._cancelled:
self._voices_success = False
return
filename = f"voices/{voice}.pt"
if try_to_load_from_cache(repo_id=self._repo_id, filename=filename):
self.progress.emit(
"voice",
"installed",
f"{idx}/{len(voice_list)}: {voice} already present",
)
continue
self.progress.emit(
"voice", "downloading", f"{idx}/{len(voice_list)}: {voice}..."
)
try:
hf_hub_download(repo_id=self._repo_id, filename=filename)
self.progress.emit("voice", "downloaded", f"{voice} downloaded")
except Exception as exc:
self.progress.emit(
"voice", "warning", f"could not download {voice}: {exc}"
)
self._voices_success = False
# Kokoro model
def _download_kokoro_model(self) -> None:
self._model_success = True
try:
from huggingface_hub import hf_hub_download, try_to_load_from_cache
except Exception:
self.progress.emit(
"model", "warning", "huggingface_hub not installed, skipping model..."
)
self._model_success = False
return
for fname in self._model_files:
if self._cancelled:
self._model_success = False
return
category = "config" if fname == "config.json" else "model"
if try_to_load_from_cache(repo_id=self._repo_id, filename=fname):
self.progress.emit(
category, "installed", f"file {fname} already present"
)
continue
self.progress.emit(category, "downloading", f"file {fname}...")
try:
hf_hub_download(repo_id=self._repo_id, filename=fname)
self.progress.emit(category, "downloaded", f"file {fname} downloaded")
except Exception as exc:
self.progress.emit(
category, "warning", f"could not download file {fname}: {exc}"
)
self._model_success = False
# spaCy models
def _download_spacy_models(self) -> None:
"""Download spaCy models. Prefer missing models provided by parent.
Parent dialog will populate _spacy_models_missing during checking.
"""
self._spacy_success = True
# Determine which models to process: prefer parent-provided missing list to avoid
# re-checking everything; otherwise use the full unique list.
parent = self.parent()
models_to_process: List[str] = _unique_sorted_models()
try:
if (
parent is not None
and hasattr(parent, "_spacy_models_missing")
and parent._spacy_models_missing
):
models_to_process = list(dict.fromkeys(parent._spacy_models_missing))
except Exception:
pass
# If spaCy is not available to run the CLI, skip gracefully
try:
import spacy.cli as _spacy_cli
except Exception:
self.progress.emit(
"spacy", "warning", "spaCy not available, skipping spaCy models..."
)
self._spacy_success = False
return
for idx, model_name in enumerate(models_to_process, start=1):
if self._cancelled:
self._spacy_success = False
return
if _is_package_installed(model_name):
self.progress.emit(
"spacy",
"installed",
f"{idx}/{len(models_to_process)}: {model_name} already installed",
)
continue
self.progress.emit(
"spacy",
"downloading",
f"{idx}/{len(models_to_process)}: {model_name}...",
)
try:
_spacy_cli.download(model_name)
self.progress.emit("spacy", "downloaded", f"{model_name} downloaded")
except Exception as exc:
self.progress.emit(
"spacy", "warning", f"could not download {model_name}: {exc}"
)
self._spacy_success = False
class PreDownloadDialog(QDialog):
"""Dialog to show and control pre-download process."""
VOICE_PREFIX = "Kokoro voices: "
MODEL_PREFIX = "Kokoro model: "
CONFIG_PREFIX = "Kokoro config: "
SPACY_PREFIX = "spaCy models: "
def __init__(self, parent=None):
super().__init__(parent)
self.setWindowTitle("Pre-download Models and Voices")
self.setMinimumWidth(500)
self.worker: Optional[PreDownloadWorker] = None
self.has_missing = False
self._spacy_models_checked: List[tuple] = []
self._spacy_models_missing: List[str] = []
self._status_worker = None
# Map keywords to (label, prefix) - labels filled after UI creation
self.status_map = {
"voice": (None, self.VOICE_PREFIX),
"spacy": (None, self.SPACY_PREFIX),
"model": (None, self.MODEL_PREFIX),
"config": (None, self.CONFIG_PREFIX),
}
self.category_map = {
"voices": ["voice"],
"model": ["model", "config"],
"spacy": ["spacy"],
}
self._setup_ui()
self._start_status_check()
def _setup_ui(self) -> None:
layout = QVBoxLayout(self)
layout.setSpacing(0)
layout.setContentsMargins(15, 0, 15, 15)
desc = QLabel(
"You can pre-download all required models and voices for offline use.\n"
"This includes Kokoro voices, Kokoro model (and config), and spaCy models."
)
desc.setWordWrap(True)
layout.addWidget(desc)
# Status rows
status_layout = QVBoxLayout()
status_title = QLabel("<b>Current Status:</b>")
status_layout.addWidget(status_title)
self.voices_status = QLabel(self.VOICE_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.voices_status)
row.addStretch()
status_layout.addLayout(row)
self.model_status = QLabel(self.MODEL_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.model_status)
row.addStretch()
status_layout.addLayout(row)
self.config_status = QLabel(self.CONFIG_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.config_status)
row.addStretch()
status_layout.addLayout(row)
self.spacy_status = QLabel(self.SPACY_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.spacy_status)
row.addStretch()
status_layout.addLayout(row)
# register labels
self.status_map["voice"] = (self.voices_status, self.VOICE_PREFIX)
self.status_map["model"] = (self.model_status, self.MODEL_PREFIX)
self.status_map["config"] = (self.config_status, self.CONFIG_PREFIX)
self.status_map["spacy"] = (self.spacy_status, self.SPACY_PREFIX)
layout.addLayout(status_layout)
layout.addItem(
QSpacerItem(0, 20, QSizePolicy.Policy.Minimum, QSizePolicy.Policy.Fixed)
)
# Buttons
button_row = QHBoxLayout()
button_row.setSpacing(10)
self.download_btn = QPushButton("Download all")
self.download_btn.setMinimumWidth(100)
self.download_btn.setMinimumHeight(35)
self.download_btn.setEnabled(False)
self.download_btn.clicked.connect(self._start_download)
button_row.addWidget(self.download_btn)
self.close_btn = QPushButton("Close")
self.close_btn.setMinimumWidth(100)
self.close_btn.setMinimumHeight(35)
self.close_btn.clicked.connect(self._handle_close)
button_row.addWidget(self.close_btn)
layout.addLayout(button_row)
self.adjustSize()
# Status checking worker
class StatusCheckWorker(QThread):
voices_checked = pyqtSignal(bool, list)
model_checked = pyqtSignal(bool)
config_checked = pyqtSignal(bool)
spacy_model_checking = pyqtSignal(str)
spacy_model_result = pyqtSignal(str, bool)
spacy_checked = pyqtSignal(bool, list)
def run(self):
parent = self.parent()
if parent is None:
return
voices_ok, missing_voices = parent._check_kokoro_voices()
self.voices_checked.emit(voices_ok, missing_voices)
model_ok = parent._check_kokoro_model()
self.model_checked.emit(model_ok)
config_ok = parent._check_kokoro_config()
self.config_checked.emit(config_ok)
# Check spaCy models by package name to detect site-package installs
unique = _unique_sorted_models()
missing: List[str] = []
for name in unique:
self.spacy_model_checking.emit(name)
ok = _is_package_installed(name)
self.spacy_model_result.emit(name, ok)
if not ok:
missing.append(name)
parent._spacy_models_missing = missing
self.spacy_checked.emit(len(missing) == 0, missing)
def _start_status_check(self) -> None:
self._status_worker = self.StatusCheckWorker(self)
self._status_worker.voices_checked.connect(self._update_voices_status)
self._status_worker.model_checked.connect(self._update_model_status)
self._status_worker.config_checked.connect(self._update_config_status)
self._status_worker.spacy_model_checking.connect(self._spacy_model_checking)
self._status_worker.spacy_model_result.connect(self._spacy_model_result)
self._status_worker.spacy_checked.connect(self._update_spacy_status)
# These are initialized in __init__ to keep consistent object state
# Set checking visual state
for lbl in (
self.voices_status,
self.model_status,
self.config_status,
self.spacy_status,
):
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
self.spacy_status.setText(self.SPACY_PREFIX + "⏳ Checking...")
self._status_worker.start()
# UI update callbacks
def _spacy_model_checking(self, name: str) -> None:
self.spacy_status.setText(f"{self.SPACY_PREFIX}Checking {name}...")
def _spacy_model_result(self, name: str, ok: bool) -> None:
self._spacy_models_checked.append((name, ok))
if not ok and name not in self._spacy_models_missing:
self._spacy_models_missing.append(name)
checked = len(self._spacy_models_checked)
missing_count = len(self._spacy_models_missing)
if missing_count:
self.spacy_status.setText(
f"{self.SPACY_PREFIX}{checked} checked, {missing_count} missing..."
)
else:
self.spacy_status.setText(f"{self.SPACY_PREFIX}{checked} checked...")
def _update_voices_status(self, ok: bool, missing: List[str]) -> None:
if ok:
self._set_status("voice", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
if missing:
self._set_status(
"voice", f"✗ Missing {len(missing)} voices", COLORS["RED"]
)
else:
self._set_status("voice", "✗ Not downloaded", COLORS["RED"])
def _update_model_status(self, ok: bool) -> None:
if ok:
self._set_status("model", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
self._set_status("model", "✗ Not downloaded", COLORS["RED"])
def _update_config_status(self, ok: bool) -> None:
if ok:
self._set_status("config", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
self._set_status("config", "✗ Not downloaded", COLORS["RED"])
def _update_spacy_status(self, ok: bool, missing: List[str]) -> None:
if ok:
self._set_status("spacy", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
if missing:
self._set_status(
"spacy", f"✗ Missing {len(missing)} model(s)", COLORS["RED"]
)
else:
self._set_status("spacy", "✗ Not downloaded", COLORS["RED"])
self.download_btn.setEnabled(self.has_missing)
def _set_status(self, key: str, text: str, color: str) -> None:
lbl, prefix = self.status_map.get(key, (None, ""))
if not lbl:
return
lbl.setText(prefix + text)
lbl.setStyleSheet(f"color: {color};")
# Helper checks
def _check_kokoro_voices(self) -> Tuple[bool, List[str]]:
"""Return (ok, missing_list) for Kokoro voices check."""
missing = []
try:
from huggingface_hub import try_to_load_from_cache
for voice in VOICES_INTERNAL:
if not try_to_load_from_cache(
repo_id="hexgrad/Kokoro-82M", filename=f"voices/{voice}.pt"
):
missing.append(voice)
except Exception:
# If HF missing, report all as missing
return False, list(VOICES_INTERNAL)
return (len(missing) == 0), missing
def _check_kokoro_model(self) -> bool:
try:
from huggingface_hub import try_to_load_from_cache
return (
try_to_load_from_cache(
repo_id="hexgrad/Kokoro-82M", filename="kokoro-v1_0.pth"
)
is not None
)
except Exception:
return False
def _check_kokoro_config(self) -> bool:
try:
from huggingface_hub import try_to_load_from_cache
return (
try_to_load_from_cache(
repo_id="hexgrad/Kokoro-82M", filename="config.json"
)
is not None
)
except Exception:
return False
def _check_spacy_models(self) -> bool:
unique = _unique_sorted_models()
missing = [m for m in unique if not _is_package_installed(m)]
self._spacy_models_missing = missing
return len(missing) == 0
# Download control
def _start_download(self) -> None:
self.download_btn.setEnabled(False)
self.download_btn.setText("Downloading...")
# mark the start of downloads; this triggers the labels
self._on_progress("system", "starting", "Processing, please wait...")
self.worker = PreDownloadWorker(self)
self.worker.progress.connect(self._on_progress)
self.worker.category_done.connect(self._on_category_done)
self.worker.finished.connect(self._on_download_finished)
self.worker.error.connect(self._on_download_error)
self.worker.start()
def _on_progress(self, category: str, status: str, message: str) -> None:
"""Map worker (category, status, message) to UI label updates.
Status is one of: 'downloading', 'installed', 'downloaded', 'warning', 'starting'.
Category is one of: 'voice', 'model', 'spacy', 'config', or 'system'.
"""
try:
# If the category targets a specific label, update directly
if category in self.status_map:
lbl, prefix = self.status_map[category]
if not lbl:
return
# Compose message and set color based on status token
full_text = prefix + message
if len(full_text) > 60:
display_text = full_text[:57] + "..."
lbl.setText(display_text)
lbl.setToolTip(full_text)
else:
lbl.setText(full_text)
lbl.setToolTip("") # Clear tooltip if not needed
if status == "downloading":
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
elif status in ("installed", "downloaded"):
lbl.setStyleSheet(f"color: {COLORS['GREEN']};")
elif status == "warning":
lbl.setStyleSheet(f"color: {COLORS['RED']};")
elif status == "error":
lbl.setStyleSheet(f"color: {COLORS['RED']};")
return
# System-level messages
if category == "system":
if status == "starting":
for k in self.status_map:
lbl, prefix = self.status_map[k]
if lbl:
lbl.setText(prefix + "Processing, please wait...")
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
# other system statuses don't require action
return
except Exception:
# Do not let UI thread crash on unexpected worker message
pass
def _on_category_done(self, category: str) -> None:
for key in self.category_map.get(category, []):
self._set_status(key, "✓ Downloaded", COLORS["GREEN"])
def _on_download_finished(self) -> None:
self.has_missing = False
self.download_btn.setText("Download all")
self.download_btn.setEnabled(False)
def _on_download_error(self, error_msg: str) -> None:
self.download_btn.setText("Download all")
self.download_btn.setEnabled(True)
for key in self.status_map:
self._set_status(key, f"✗ Error - {error_msg}", COLORS["RED"])
def _handle_close(self) -> None:
if self.worker and self.worker.isRunning():
self.worker.cancel()
self.worker.wait(2000)
self.accept()
def closeEvent(self, event) -> None:
if self.worker and self.worker.isRunning():
self.worker.cancel()
self.worker.wait(2000)
super().closeEvent(event)
+155
View File
@@ -0,0 +1,155 @@
"""
Lazy-loaded spaCy utilities for sentence segmentation.
"""
# Cached spaCy module and models (lazy loaded)
_spacy = None
_nlp_cache = {}
# Language code to spaCy model mapping
SPACY_MODELS = {
"a": "en_core_web_sm", # American English
"b": "en_core_web_sm", # British English
"e": "es_core_news_sm", # Spanish
"f": "fr_core_news_sm", # French
"i": "it_core_news_sm", # Italian
"p": "pt_core_news_sm", # Brazilian Portuguese
"z": "zh_core_web_sm", # Mandarin Chinese
"j": "ja_core_news_sm", # Japanese
"h": "xx_sent_ud_sm", # Hindi (multi-language model)
}
def _load_spacy():
"""Lazy load spaCy module."""
global _spacy
if _spacy is None:
try:
import spacy
_spacy = spacy
except ImportError:
return None
return _spacy
def get_spacy_model(lang_code, log_callback=None):
"""
Get or load a spaCy model for the given language code.
Downloads the model automatically if not available.
Args:
lang_code: Language code (a, b, e, f, etc.)
log_callback: Optional function to log messages
Returns:
Loaded spaCy model or None if unavailable
"""
def log(msg, is_error=False):
# Prefer GUI log callback when provided to avoid spamming stdout.
if log_callback:
color = "red" if is_error else "grey"
try:
log_callback((msg, color))
except Exception:
# Fallback to printing if callback misbehaves
print(msg)
else:
print(msg)
# Check if model is cached
if lang_code in _nlp_cache:
return _nlp_cache[lang_code]
# Check if language is supported
model_name = SPACY_MODELS.get(lang_code)
if not model_name:
log(f"\nspaCy: No model mapping for language '{lang_code}'...")
return None
# Lazy load spaCy
spacy = _load_spacy()
if spacy is None:
log("\nspaCy: Module not installed, falling back to default segmentation...")
return None
# Try to load the model
try:
log(f"\nLoading spaCy model '{model_name}'...")
nlp = spacy.load(
model_name,
disable=["ner", "parser", "tagger", "lemmatizer", "attribute_ruler"],
)
# Enable sentence segmentation only
if "sentencizer" not in nlp.pipe_names:
nlp.add_pipe("sentencizer")
_nlp_cache[lang_code] = nlp
return nlp
except OSError:
# Model not found, attempt download
log(f"\nspaCy: Downloading model '{model_name}'...")
try:
from spacy.cli import download
download(model_name)
# Retry loading
nlp = spacy.load(
model_name,
disable=["ner", "parser", "tagger", "lemmatizer", "attribute_ruler"],
)
if "sentencizer" not in nlp.pipe_names:
nlp.add_pipe("sentencizer")
_nlp_cache[lang_code] = nlp
log(f"spaCy model '{model_name}' downloaded and loaded")
return nlp
except Exception as e:
log(
f"\nspaCy: Failed to download model '{model_name}': {e}...",
is_error=True,
)
return None
except Exception as e:
log(f"\nspaCy: Error loading model '{model_name}': {e}...", is_error=True)
return None
def segment_sentences(text, lang_code, log_callback=None):
"""
Segment text into sentences using spaCy.
Args:
text: Text to segment
lang_code: Language code
log_callback: Optional function to log messages
Returns:
List of sentence strings, or None if spaCy unavailable
"""
nlp = get_spacy_model(lang_code, log_callback)
if nlp is None:
return None
# Ensure spaCy can handle large texts by adjusting max_length if necessary
try:
text_len = len(text or "")
if text_len and hasattr(nlp, "max_length") and text_len > nlp.max_length:
# increase a bit beyond the text length to be safe
nlp.max_length = text_len + 1000
except Exception:
pass
# Process text and extract sentences
doc = nlp(text)
return [sent.text.strip() for sent in doc.sents if sent.text.strip()]
def is_spacy_available():
"""Check if spaCy can be imported."""
return _load_spacy() is not None
def clear_cache():
"""Clear the model cache to free memory."""
global _nlp_cache
_nlp_cache.clear()
+19 -11
View File
@@ -10,6 +10,13 @@ from threading import Thread
warnings.filterwarnings("ignore") warnings.filterwarnings("ignore")
# Pre-compile frequently used regex patterns for better performance
_WHITESPACE_PATTERN = re.compile(r"[^\S\n]+")
_MULTIPLE_NEWLINES_PATTERN = re.compile(r"\n{3,}")
_SINGLE_NEWLINE_PATTERN = re.compile(r"(?<!\n)\n(?!\n)")
_CHAPTER_MARKER_PATTERN = re.compile(r"<<CHAPTER_MARKER:.*?>>")
_METADATA_PATTERN = re.compile(r"<<METADATA_[^:]+:[^>]*>>")
def detect_encoding(file_path): def detect_encoding(file_path):
import chardet import chardet
@@ -128,13 +135,16 @@ def clean_text(text, *args, **kwargs):
cfg = load_config() cfg = load_config()
replace_single_newlines = cfg.get("replace_single_newlines", False) replace_single_newlines = cfg.get("replace_single_newlines", False)
# Collapse all whitespace (excluding newlines) into single spaces per line and trim edges # Collapse all whitespace (excluding newlines) into single spaces per line and trim edges
lines = [re.sub(r"[^\S\n]+", " ", line).strip() for line in text.splitlines()] # Use pre-compiled pattern for better performance
lines = [_WHITESPACE_PATTERN.sub(" ", line).strip() for line in text.splitlines()]
text = "\n".join(lines) text = "\n".join(lines)
# Standardize paragraph breaks (multiple newlines become exactly two) and trim overall whitespace # Standardize paragraph breaks (multiple newlines become exactly two) and trim overall whitespace
text = re.sub(r"\n{3,}", "\n\n", text).strip() # Use pre-compiled pattern for better performance
text = _MULTIPLE_NEWLINES_PATTERN.sub("\n\n", text).strip()
# Optionally replace single newlines with spaces, but preserve double newlines # Optionally replace single newlines with spaces, but preserve double newlines
if replace_single_newlines: if replace_single_newlines:
text = re.sub(r"(?<!\n)\n(?!\n)", " ", text) # Use pre-compiled pattern for better performance
text = _SINGLE_NEWLINE_PATTERN.sub(" ", text)
return text return text
@@ -243,14 +253,12 @@ def save_config(config):
def calculate_text_length(text): def calculate_text_length(text):
# Ignore chapter markers # Use pre-compiled patterns for better performance
text = re.sub(r"<<CHAPTER_MARKER:.*?>>", "", text) # Ignore chapter markers and metadata patterns in a single pass
# Ignore metadata patterns text = _CHAPTER_MARKER_PATTERN.sub("", text)
text = re.sub(r"<<METADATA_[^:]+:[^>]*>>", "", text) text = _METADATA_PATTERN.sub("", text)
# Ignore newlines # Ignore newlines and leading/trailing spaces
text = text.replace("\n", "") text = text.replace("\n", "").strip()
# Ignore leading/trailing spaces
text = text.strip()
# Calculate character count # Calculate character count
char_count = len(text) char_count = len(text)
return char_count return char_count
+2 -1
View File
@@ -25,7 +25,8 @@ dependencies = [
"charset_normalizer>=3.4.1", "charset_normalizer>=3.4.1",
"chardet>=5.2.0", "chardet>=5.2.0",
"static_ffmpeg>=2.13", "static_ffmpeg>=2.13",
"Markdown>=3.9" "Markdown>=3.9",
"spacy>=3.8.7"
] ]
classifiers = [ classifiers = [