mirror of
https://github.com/denizsafak/abogen.git
synced 2026-07-18 13:40:27 +02:00
feat: Additive merge of webui branch with PyQt GUI support
- Add abogen/pyqt/ package with full PyQt6 desktop GUI - Restore PyQt GUI files (gui.py, book_handler.py, queue_manager_gui.py, voice_formula_gui.py, conversion.py) - Add new shared modules from webui: book_parser.py, subtitle_utils.py, spacy_utils.py - Add Linux libraries (libxcb-cursor) for Qt platform plugin support - Add new epub parsing tests from webui branch - Update pyproject.toml with dual entry points: - abogen: PyQt6 desktop GUI - abogen-web: Flask Web UI - Add PyQt6 to dependencies - Re-export PyQt classes from root modules for backwards compatibility - Merge CHANGELOG.md entries (1.2.0-1.2.5 from webui) - Update README.md with dual interface documentation Implements #26 - shared core with separate UI folders
This commit is contained in:
@@ -1,6 +1,63 @@
|
|||||||
# Unreleased
|
# Unreleased
|
||||||
- Added an EPUB 3 packaging pipeline that builds media-overlay EPUBs from generated audio and chunk metadata.
|
- Added an EPUB 3 packaging pipeline that builds media-overlay EPUBs from generated audio and chunk metadata.
|
||||||
- Persisted chunk timing metadata in job artifacts and exercised the exporter with automated tests.
|
- Persisted chunk timing metadata in job artifacts and exercised the exporter with automated tests.
|
||||||
|
- Added Flask-based Web UI (`abogen-web`) for Docker and headless server deployments.
|
||||||
|
- Reorganized codebase to support both PyQt6 desktop GUI and Web UI from a shared core.
|
||||||
|
- Added Supertonic TTS engine support with GPU acceleration.
|
||||||
|
- Added entity analysis and pronunciation override system for proper nouns.
|
||||||
|
- Added speaker/role assignment for multi-voice "theatrical" audiobooks.
|
||||||
|
- Added Calibre OPDS and Audiobookshelf integration.
|
||||||
|
|
||||||
|
# 1.2.5
|
||||||
|
- Added new option: `Override item settings with current selection` in the queue manager. When enabled, all items in the queue will be processed using the current global settings selected in the main GUI, overriding their individual settings. When disabled, each item will retain its own specific settings.
|
||||||
|
- Fixed `Error "Could not load the Qt platform plugin "xcb"` error that occurred in some Linux distributions due to missing `libxcb-cursor0` library by conditionally loading the bundled library when the system version is unavailable, issue mentioned by @bmcgonag in #101.
|
||||||
|
- Fixed the `No module named pip` error that occurred for users who installed Abogen via the [**uv**](https://github.com/astral-sh/uv) installer.
|
||||||
|
- Fixed defaults for `replace_single_newlines` not being applied correctly in some cases.
|
||||||
|
- Fixed `Save chapters separately for queued epubs is ignored`, issue mentioned by @dymas-cz in #109.
|
||||||
|
- Fixed incorrect sentence segmentation when using spaCy, where text would erroneously split after opening parentheses.
|
||||||
|
- Improvements in code and documentation.
|
||||||
|
|
||||||
|
# 1.2.4
|
||||||
|
- **Subtitle generation is now available for all languages!** Abogen now supports subtitle generation for non-English languages using audio duration-based timing. Available modes include `Line`, `Sentence`, and `Sentence + Comma`. (Note: Word-level subtitle modes remain English-only due to Kokoro's timestamp token limitations.)
|
||||||
|
- New option: **"Use spaCy for sentence segmentation"** You can now use [spaCy](https://spacy.io/) to automatically detect sentence boundaries and produce cleaner, more readable subtitles.
|
||||||
|
- New option: **Pre-download models and voices for offline use** You can now pre-download all required Kokoro models, voices, and spaCy language models.
|
||||||
|
- Added support for `.` separator in timestamps (e.g. `HH:MM:SS.ms`) for timestamp-based text files.
|
||||||
|
- Optimized regex compilation and eliminated busy-wait loops.
|
||||||
|
- Possibly fixed `Silent truncation of long paragraphs` issue mentioned in #91 by @xklzlxr.
|
||||||
|
- Fixed unused regex patterns and variable naming conventions.
|
||||||
|
- Improvements in code and documentation.
|
||||||
|
|
||||||
|
# 1.2.3
|
||||||
|
- Same as 1.2.2, re-released to fix an issue with subtitle timing when using timestamp-based text files.
|
||||||
|
|
||||||
|
# 1.2.2
|
||||||
|
- **You can now voice your subtitle files!** Simply add `.srt`, `.ass` or `.vtt` files to generate timed audio.
|
||||||
|
- New option: **"Use silent gaps between subtitles"**: Prevents unnecessary audio speed-up by letting speech continue into the silent gaps between subtitles.
|
||||||
|
- New option: **"Subtitle speed adjustment method"**: Choose how to speed up audio when needed (TTS Regeneration or FFmpeg Time-stretch).
|
||||||
|
- Added support for embedding cover images in M4B files.
|
||||||
|
- Fixed `[WinError 1114] A dynamic link library (DLL) initialization routine failed` error on Windows.
|
||||||
|
- Potential fix for `CUDA GPU is not available` issue.
|
||||||
|
- Improvements in code and documentation.
|
||||||
|
|
||||||
|
# 1.2.1
|
||||||
|
- Upgraded Abogen's interface from PyQt5 to PyQt6 for better compatibility and long-term support.
|
||||||
|
- Added tooltip indicators in queue manager to display book handler options.
|
||||||
|
- Added `Open processed file` and `Open input file` options for items in the queue manager.
|
||||||
|
- Added loading gif animation to book handler window.
|
||||||
|
- Fixed light theme slider colors in voice mixer for better visibility (for non-Windows users).
|
||||||
|
- Fixed subtitle word-count splitting logic for more accurate segmentation.
|
||||||
|
- Improvements in code and documentation.
|
||||||
|
|
||||||
|
# 1.2.0
|
||||||
|
- Added `Line` option to subtitle generation modes, allowing subtitles to be generated based on line breaks in the text, by @mleg in #94.
|
||||||
|
- Added a loading indicator to the book handler window for better user experience during book preprocessing.
|
||||||
|
- Fixed `cannot access local variable 'is_narrow'` error when subtitle format `SRT` was selected, mentioned by @Kinasa0096 in #88.
|
||||||
|
- Fixed folder and filename sanitization to properly handle OS-specific illegal characters.
|
||||||
|
- Fixed `/` and `\` path display by normalizing paths.
|
||||||
|
- Fixed book reprocessing issue where books were being processed every time the chapters window was opened.
|
||||||
|
- Fixed taskbar icon not appearing correctly in Windows.
|
||||||
|
- Fixed "Go to folder" button not opening the chapter output directory when only separate chapters were generated.
|
||||||
|
- Improvements in code and documentation.
|
||||||
|
|
||||||
# 1.1.9
|
# 1.1.9
|
||||||
- Fixed the issue where spaces were deleted before punctuation marks while generating subtitles.
|
- Fixed the issue where spaces were deleted before punctuation marks while generating subtitles.
|
||||||
|
|||||||
@@ -9,6 +9,17 @@ Abogen is a web-first text-to-speech workstation. Drop in an EPUB, PDF, Markdown
|
|||||||
- LLM-assisted text normalization with live previews and configurable prompts
|
- LLM-assisted text normalization with live previews and configurable prompts
|
||||||
- Runs well in Docker, ships a REST-style JSON API, and works across macOS, Linux, and Windows
|
- Runs well in Docker, ships a REST-style JSON API, and works across macOS, Linux, and Windows
|
||||||
|
|
||||||
|
## Interfaces
|
||||||
|
|
||||||
|
Abogen offers **two interfaces** to suit different workflows:
|
||||||
|
|
||||||
|
| Command | Interface | Best for |
|
||||||
|
|---------|-----------|----------|
|
||||||
|
| `abogen` | PyQt6 Desktop GUI | Local desktop use on Windows/macOS/Linux |
|
||||||
|
| `abogen-web` | Flask Web UI | Docker, headless servers, remote access |
|
||||||
|
|
||||||
|
Both interfaces share the same core processing engine and produce identical output.
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
Abogen supports Python 3.10–3.12.
|
Abogen supports Python 3.10–3.12.
|
||||||
|
|
||||||
@@ -19,11 +30,16 @@ source .venv/bin/activate # On Windows use: .venv\Scripts\activate
|
|||||||
pip install abogen
|
pip install abogen
|
||||||
```
|
```
|
||||||
|
|
||||||
### Launch the web app
|
### Launch the desktop app (PyQt6)
|
||||||
```bash
|
```bash
|
||||||
abogen
|
abogen
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Launch the web app (Flask)
|
||||||
|
```bash
|
||||||
|
abogen-web
|
||||||
|
```
|
||||||
|
|
||||||
Then open http://localhost:8808 and drag in your documents. Jobs run in the background worker and the browser updates automatically.
|
Then open http://localhost:8808 and drag in your documents. Jobs run in the background worker and the browser updates automatically.
|
||||||
|
|
||||||
> **Tip:** Keep the terminal open while the server is running. Use `Ctrl+C` to stop it.
|
> **Tip:** Keep the terminal open while the server is running. Use `Ctrl+C` to stop it.
|
||||||
|
|||||||
@@ -1,9 +1,11 @@
|
|||||||
"""Legacy PyQt-based chapter selection dialog has been removed."""
|
"""Backwards-compatible re-export of the PyQt book handler.
|
||||||
|
|
||||||
|
The actual implementation lives in abogen.pyqt.book_handler.
|
||||||
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from abogen.pyqt.book_handler import * # noqa: F401, F403
|
||||||
|
from abogen.pyqt.book_handler import HandlerDialog
|
||||||
|
|
||||||
def __getattr__(name: str):
|
__all__ = ["HandlerDialog"]
|
||||||
raise AttributeError(
|
|
||||||
"The PyQt chapter selection dialog was removed. Use the web interface instead."
|
|
||||||
)
|
|
||||||
|
|||||||
@@ -0,0 +1,917 @@
|
|||||||
|
import os
|
||||||
|
import re
|
||||||
|
import logging
|
||||||
|
import textwrap
|
||||||
|
import urllib.parse
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
|
||||||
|
import ebooklib
|
||||||
|
from ebooklib import epub
|
||||||
|
from bs4 import BeautifulSoup, NavigableString
|
||||||
|
import fitz # PyMuPDF
|
||||||
|
import markdown
|
||||||
|
|
||||||
|
from abogen.utils import detect_encoding
|
||||||
|
from abogen.subtitle_utils import clean_text, calculate_text_length
|
||||||
|
|
||||||
|
# Pre-compile frequently used regex patterns
|
||||||
|
_BRACKETED_NUMBERS_PATTERN = re.compile(r"\[\s*\d+\s*\]")
|
||||||
|
_STANDALONE_PAGE_NUMBERS_PATTERN = re.compile(r"^\s*\d+\s*$", re.MULTILINE)
|
||||||
|
_PAGE_NUMBERS_AT_END_PATTERN = re.compile(r"\s+\d+\s*$", re.MULTILINE)
|
||||||
|
_PAGE_NUMBERS_WITH_DASH_PATTERN = re.compile(r"\s+[-–—]\s*\d+\s*[-–—]?\s*$", re.MULTILINE)
|
||||||
|
|
||||||
|
|
||||||
|
class BaseBookParser(ABC):
|
||||||
|
"""
|
||||||
|
Abstract base class for parsing different book formats.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, book_path):
|
||||||
|
self.book_path = os.path.normpath(os.path.abspath(book_path))
|
||||||
|
self.content_texts = {}
|
||||||
|
self.content_lengths = {}
|
||||||
|
self.book_metadata = {}
|
||||||
|
# Unified structure for navigation: list of dicts
|
||||||
|
# { 'title': str, 'src': str, 'children': [], 'has_content': bool }
|
||||||
|
self.processed_nav_structure = []
|
||||||
|
self.load()
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def load(self):
|
||||||
|
"""Load the book file."""
|
||||||
|
pass
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def process_content(self, replace_single_newlines=True):
|
||||||
|
"""Process the book content to extract text and structure."""
|
||||||
|
pass
|
||||||
|
|
||||||
|
@property
|
||||||
|
@abstractmethod
|
||||||
|
def file_type(self):
|
||||||
|
"""Return the type of the file (pdf, epub, markdown)."""
|
||||||
|
pass
|
||||||
|
|
||||||
|
def get_chapters(self):
|
||||||
|
"""Return a list of chapter IDs and Names."""
|
||||||
|
chapters = []
|
||||||
|
if self.processed_nav_structure:
|
||||||
|
|
||||||
|
def flatten_nav(nodes):
|
||||||
|
for node in nodes:
|
||||||
|
if node.get("has_content"):
|
||||||
|
chapters.append((node["src"], node["title"]))
|
||||||
|
if node.get("children"):
|
||||||
|
flatten_nav(node["children"])
|
||||||
|
|
||||||
|
flatten_nav(self.processed_nav_structure)
|
||||||
|
else:
|
||||||
|
# Fallback for simple content without nav structure
|
||||||
|
for ch_id, content in self.content_texts.items():
|
||||||
|
# This could be improved, but serves as a generic fallback
|
||||||
|
chapters.append((ch_id, ch_id))
|
||||||
|
return chapters
|
||||||
|
|
||||||
|
def get_formatted_text(self):
|
||||||
|
"""
|
||||||
|
Returns the full text of the book formatted with chapter markers.
|
||||||
|
"""
|
||||||
|
chapters = self.get_chapters()
|
||||||
|
full_text = []
|
||||||
|
|
||||||
|
for chapter_id, chapter_name in chapters:
|
||||||
|
text = self.content_texts.get(chapter_id, "")
|
||||||
|
if text:
|
||||||
|
full_text.append(f"\n<<CHAPTER_MARKER:{chapter_name}>>\n")
|
||||||
|
full_text.append(text)
|
||||||
|
|
||||||
|
return "\n".join(full_text)
|
||||||
|
|
||||||
|
def get_metadata(self):
|
||||||
|
"""Return extracted metadata."""
|
||||||
|
return self.book_metadata
|
||||||
|
|
||||||
|
|
||||||
|
class PdfParser(BaseBookParser):
|
||||||
|
def __init__(self, book_path):
|
||||||
|
self.pdf_doc = None
|
||||||
|
super().__init__(book_path)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def file_type(self):
|
||||||
|
return "pdf"
|
||||||
|
|
||||||
|
def load(self):
|
||||||
|
try:
|
||||||
|
self.pdf_doc = fitz.open(self.book_path)
|
||||||
|
except Exception as e:
|
||||||
|
logging.error(f"Error loading PDF {self.book_path}: {e}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
def process_content(self, replace_single_newlines=True):
|
||||||
|
if not self.pdf_doc:
|
||||||
|
self.load()
|
||||||
|
|
||||||
|
for page_num in range(len(self.pdf_doc)):
|
||||||
|
text = clean_text(self.pdf_doc[page_num].get_text())
|
||||||
|
|
||||||
|
# Clean up common PDF artifacts:
|
||||||
|
# - Remove bracketed numbers often used for citations [1]
|
||||||
|
text = _BRACKETED_NUMBERS_PATTERN.sub("", text)
|
||||||
|
# - Remove standalone page numbers often found in headers/footers
|
||||||
|
text = _STANDALONE_PAGE_NUMBERS_PATTERN.sub("", text)
|
||||||
|
# - Remove page numbers at end of lines
|
||||||
|
text = _PAGE_NUMBERS_AT_END_PATTERN.sub("", text)
|
||||||
|
# - Remove page numbers with dashes - 4 -
|
||||||
|
text = _PAGE_NUMBERS_WITH_DASH_PATTERN.sub("", text)
|
||||||
|
|
||||||
|
page_id = f"page_{page_num + 1}"
|
||||||
|
self.content_texts[page_id] = text
|
||||||
|
self.content_lengths[page_id] = calculate_text_length(text)
|
||||||
|
|
||||||
|
return self.content_texts, self.content_lengths
|
||||||
|
|
||||||
|
def _extract_book_metadata(self):
|
||||||
|
# PDF metadata extraction can be added here if needed
|
||||||
|
# For now, base class metadata is empty dict
|
||||||
|
pass
|
||||||
|
|
||||||
|
def get_chapters(self):
|
||||||
|
# PDF specific implementation because it doesn't use nav structure
|
||||||
|
chapters = []
|
||||||
|
if self.pdf_doc:
|
||||||
|
for i in range(len(self.pdf_doc)):
|
||||||
|
chapters.append((f"page_{i+1}", f"Page {i+1}"))
|
||||||
|
return chapters
|
||||||
|
|
||||||
|
|
||||||
|
class MarkdownParser(BaseBookParser):
|
||||||
|
def __init__(self, book_path):
|
||||||
|
self.markdown_text = None
|
||||||
|
super().__init__(book_path)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def file_type(self):
|
||||||
|
return "markdown"
|
||||||
|
|
||||||
|
def load(self):
|
||||||
|
try:
|
||||||
|
encoding = detect_encoding(self.book_path)
|
||||||
|
with open(self.book_path, "r", encoding=encoding, errors="replace") as f:
|
||||||
|
self.markdown_text = f.read()
|
||||||
|
except Exception as e:
|
||||||
|
logging.error(f"Error reading markdown file: {e}")
|
||||||
|
self.markdown_text = ""
|
||||||
|
|
||||||
|
def process_content(self, replace_single_newlines=True):
|
||||||
|
if self.markdown_text is None:
|
||||||
|
self.load()
|
||||||
|
|
||||||
|
self._process_markdown_content()
|
||||||
|
return self.content_texts, self.content_lengths
|
||||||
|
|
||||||
|
def _convert_markdown_toc_to_nav(self, toc_tokens):
|
||||||
|
nav_nodes = []
|
||||||
|
for token in toc_tokens:
|
||||||
|
node = {
|
||||||
|
"title": token["name"],
|
||||||
|
"src": token["id"],
|
||||||
|
"children": self._convert_markdown_toc_to_nav(
|
||||||
|
token.get("children", [])
|
||||||
|
),
|
||||||
|
"has_content": True,
|
||||||
|
}
|
||||||
|
nav_nodes.append(node)
|
||||||
|
return nav_nodes
|
||||||
|
|
||||||
|
def _process_markdown_content(self):
|
||||||
|
if not self.markdown_text:
|
||||||
|
return
|
||||||
|
|
||||||
|
original_text = textwrap.dedent(self.markdown_text)
|
||||||
|
md = markdown.Markdown(extensions=["toc", "fenced_code"])
|
||||||
|
html = md.convert(original_text)
|
||||||
|
markdown_toc = md.toc_tokens
|
||||||
|
|
||||||
|
# Convert markdown TOC tokens to our unified navigation structure
|
||||||
|
self.processed_nav_structure = self._convert_markdown_toc_to_nav(markdown_toc)
|
||||||
|
|
||||||
|
cleaned_full_text = clean_text(original_text)
|
||||||
|
|
||||||
|
# If no TOC found, treat as single chapter
|
||||||
|
if not self.processed_nav_structure:
|
||||||
|
chapter_id = "markdown_content"
|
||||||
|
self.content_texts[chapter_id] = cleaned_full_text
|
||||||
|
self.content_lengths[chapter_id] = calculate_text_length(cleaned_full_text)
|
||||||
|
return
|
||||||
|
|
||||||
|
soup = BeautifulSoup(html, "html.parser")
|
||||||
|
|
||||||
|
all_headers = []
|
||||||
|
|
||||||
|
def flatten_nav_internal(nodes):
|
||||||
|
for node in nodes:
|
||||||
|
all_headers.append(node)
|
||||||
|
if node.get("children"):
|
||||||
|
flatten_nav_internal(node["children"])
|
||||||
|
|
||||||
|
flatten_nav_internal(self.processed_nav_structure)
|
||||||
|
|
||||||
|
header_positions = []
|
||||||
|
for node in all_headers:
|
||||||
|
header_id = node["src"]
|
||||||
|
id_pattern = f'id="{header_id}"'
|
||||||
|
pos = html.find(id_pattern)
|
||||||
|
if pos != -1:
|
||||||
|
tag_start = html.rfind("<", 0, pos)
|
||||||
|
header_positions.append(
|
||||||
|
{"id": header_id, "start": tag_start, "name": node["title"]}
|
||||||
|
)
|
||||||
|
header_positions.sort(key=lambda x: x["start"])
|
||||||
|
|
||||||
|
for i, header_pos in enumerate(header_positions):
|
||||||
|
header_id = header_pos["id"]
|
||||||
|
header_name = header_pos["name"]
|
||||||
|
content_start = header_pos["start"]
|
||||||
|
|
||||||
|
content_end = (
|
||||||
|
header_positions[i + 1]["start"]
|
||||||
|
if i + 1 < len(header_positions)
|
||||||
|
else len(html)
|
||||||
|
)
|
||||||
|
section_html = html[content_start:content_end]
|
||||||
|
section_soup = BeautifulSoup(section_html, "html.parser")
|
||||||
|
|
||||||
|
header_tag = section_soup.find(attrs={"id": header_id})
|
||||||
|
if header_tag:
|
||||||
|
header_tag.decompose()
|
||||||
|
|
||||||
|
section_text = clean_text(section_soup.get_text()).strip()
|
||||||
|
chapter_id = header_id
|
||||||
|
if section_text:
|
||||||
|
full_content = f"{header_name}\n\n{section_text}"
|
||||||
|
self.content_texts[chapter_id] = full_content
|
||||||
|
self.content_lengths[chapter_id] = calculate_text_length(full_content)
|
||||||
|
else:
|
||||||
|
self.content_texts[chapter_id] = header_name
|
||||||
|
self.content_lengths[chapter_id] = calculate_text_length(header_name)
|
||||||
|
|
||||||
|
def get_chapters(self):
|
||||||
|
chapters = super().get_chapters()
|
||||||
|
if not chapters and "markdown_content" in self.content_texts:
|
||||||
|
chapters.append(("markdown_content", "Content"))
|
||||||
|
return chapters
|
||||||
|
|
||||||
|
|
||||||
|
class EpubParser(BaseBookParser):
|
||||||
|
def __init__(self, book_path):
|
||||||
|
self.book = None
|
||||||
|
self.doc_content = {}
|
||||||
|
super().__init__(book_path)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def file_type(self):
|
||||||
|
return "epub"
|
||||||
|
|
||||||
|
def load(self):
|
||||||
|
try:
|
||||||
|
self.book = epub.read_epub(self.book_path)
|
||||||
|
except KeyError as e:
|
||||||
|
# TODO: should we just patch the ebooklib pre-emptively to avoid the need to catch this exception?
|
||||||
|
logging.warning(f"EPUB missing referenced file: {e}. Attempting to patch.")
|
||||||
|
# Patch ebooklib to skip missing files
|
||||||
|
import types
|
||||||
|
from ebooklib import epub as _epub_module
|
||||||
|
|
||||||
|
reader_class = _epub_module.EpubReader
|
||||||
|
orig_read_file = reader_class.read_file
|
||||||
|
|
||||||
|
def safe_read_file(self, name):
|
||||||
|
try:
|
||||||
|
return orig_read_file(self, name)
|
||||||
|
except KeyError:
|
||||||
|
logging.warning(f"Missing file in EPUB: {name}. Returning empty bytes.")
|
||||||
|
return b""
|
||||||
|
|
||||||
|
reader_class.read_file = safe_read_file
|
||||||
|
try:
|
||||||
|
self.book = epub.read_epub(self.book_path)
|
||||||
|
finally:
|
||||||
|
reader_class.read_file = orig_read_file
|
||||||
|
|
||||||
|
def process_content(self, replace_single_newlines=True):
|
||||||
|
if not self.book:
|
||||||
|
self.load()
|
||||||
|
|
||||||
|
self.book_metadata = self._extract_book_metadata()
|
||||||
|
try:
|
||||||
|
nav_item, nav_type = self._identify_nav_item()
|
||||||
|
self._execute_nav_parsing_logic(nav_item, nav_type)
|
||||||
|
except Exception as e:
|
||||||
|
logging.warning(f"EPUB nav processing failed: {e}. Falling back to spine.")
|
||||||
|
self._process_epub_content_spine_fallback()
|
||||||
|
|
||||||
|
return self.content_texts, self.content_lengths
|
||||||
|
|
||||||
|
def _extract_book_metadata(self):
|
||||||
|
metadata = {}
|
||||||
|
if not self.book:
|
||||||
|
return metadata
|
||||||
|
|
||||||
|
try:
|
||||||
|
metadata["title"] = self.book.get_metadata("DC", "title")[0][0]
|
||||||
|
except Exception:
|
||||||
|
metadata["title"] = os.path.splitext(os.path.basename(self.book_path))[0]
|
||||||
|
|
||||||
|
try:
|
||||||
|
metadata["author"] = self.book.get_metadata("DC", "creator")[0][0]
|
||||||
|
except Exception:
|
||||||
|
metadata["author"] = "Unknown Author"
|
||||||
|
|
||||||
|
try:
|
||||||
|
metadata["language"] = self.book.get_metadata("DC", "language")[0][0]
|
||||||
|
except Exception:
|
||||||
|
metadata["language"] = "en"
|
||||||
|
|
||||||
|
return metadata
|
||||||
|
|
||||||
|
def _find_doc_key(self, base_href, doc_order, doc_order_decoded):
|
||||||
|
candidates = [
|
||||||
|
base_href,
|
||||||
|
urllib.parse.unquote(base_href),
|
||||||
|
]
|
||||||
|
base_name = os.path.basename(base_href).lower()
|
||||||
|
for k in list(doc_order.keys()) + list(doc_order_decoded.keys()):
|
||||||
|
if os.path.basename(k).lower() == base_name:
|
||||||
|
candidates.append(k)
|
||||||
|
for candidate in candidates:
|
||||||
|
if candidate in doc_order:
|
||||||
|
return candidate, doc_order[candidate]
|
||||||
|
elif candidate in doc_order_decoded:
|
||||||
|
return candidate, doc_order_decoded[candidate]
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
def _find_position_robust(self, doc_href, fragment_id):
|
||||||
|
if doc_href not in self.doc_content:
|
||||||
|
logging.warning(f"Document '{doc_href}' not found in cached content.")
|
||||||
|
return 0
|
||||||
|
html_content = self.doc_content[doc_href]
|
||||||
|
if not fragment_id:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
try:
|
||||||
|
temp_soup = BeautifulSoup(f"<div>{html_content}</div>", "html.parser")
|
||||||
|
target_element = temp_soup.find(id=fragment_id)
|
||||||
|
if target_element:
|
||||||
|
tag_str = str(target_element)
|
||||||
|
pos = html_content.find(tag_str[: min(len(tag_str), 200)])
|
||||||
|
if pos != -1:
|
||||||
|
return pos
|
||||||
|
except Exception as e:
|
||||||
|
logging.warning(f"BeautifulSoup failed to find id='{fragment_id}': {e}")
|
||||||
|
|
||||||
|
safe_fragment_id = re.escape(fragment_id)
|
||||||
|
id_name_pattern = re.compile(
|
||||||
|
f"<[^>]+(?:id|name)\\s*=\\s*[\"']{safe_fragment_id}[\"']", re.IGNORECASE
|
||||||
|
)
|
||||||
|
match = id_name_pattern.search(html_content)
|
||||||
|
if match:
|
||||||
|
return match.start()
|
||||||
|
|
||||||
|
id_match_str = f'id="{fragment_id}"'
|
||||||
|
name_match_str = f'name="{fragment_id}"'
|
||||||
|
id_pos = html_content.find(id_match_str)
|
||||||
|
name_pos = html_content.find(name_match_str)
|
||||||
|
|
||||||
|
pos = -1
|
||||||
|
if id_pos != -1 and name_pos != -1:
|
||||||
|
pos = min(id_pos, name_pos)
|
||||||
|
elif id_pos != -1:
|
||||||
|
pos = id_pos
|
||||||
|
elif name_pos != -1:
|
||||||
|
pos = name_pos
|
||||||
|
|
||||||
|
if pos != -1:
|
||||||
|
tag_start_pos = html_content.rfind("<", 0, pos)
|
||||||
|
final_pos = tag_start_pos if tag_start_pos != -1 else 0
|
||||||
|
return final_pos
|
||||||
|
|
||||||
|
logging.warning(
|
||||||
|
f"Anchor '{fragment_id}' not found in {doc_href}. Defaulting to position 0."
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def _parse_ncx_navpoint(
|
||||||
|
self,
|
||||||
|
nav_point,
|
||||||
|
ordered_entries,
|
||||||
|
doc_order,
|
||||||
|
doc_order_decoded,
|
||||||
|
tree_structure_list,
|
||||||
|
find_position_func,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Recursive parsing of NCX navigation nodes.
|
||||||
|
|
||||||
|
Logic tested by: tests/test_epub_ncx_parsing.py
|
||||||
|
"""
|
||||||
|
nav_label = nav_point.find("navLabel")
|
||||||
|
content = nav_point.find("content")
|
||||||
|
title = (
|
||||||
|
nav_label.find("text").get_text(strip=True)
|
||||||
|
if nav_label and nav_label.find("text")
|
||||||
|
else "Untitled Section"
|
||||||
|
)
|
||||||
|
src = content["src"] if content and "src" in content.attrs else None
|
||||||
|
|
||||||
|
current_entry_node = {"title": title, "src": src, "children": []}
|
||||||
|
|
||||||
|
if src:
|
||||||
|
base_href, fragment = src.split("#", 1) if "#" in src else (src, None)
|
||||||
|
doc_key, doc_idx = self._find_doc_key(base_href, doc_order, doc_order_decoded)
|
||||||
|
if not doc_key:
|
||||||
|
current_entry_node["has_content"] = False
|
||||||
|
else:
|
||||||
|
position = find_position_func(doc_key, fragment)
|
||||||
|
entry_data = {
|
||||||
|
"src": src,
|
||||||
|
"title": title,
|
||||||
|
"doc_href": doc_key,
|
||||||
|
"position": position,
|
||||||
|
"doc_order": doc_idx,
|
||||||
|
}
|
||||||
|
ordered_entries.append(entry_data)
|
||||||
|
current_entry_node["has_content"] = True
|
||||||
|
else:
|
||||||
|
current_entry_node["has_content"] = False
|
||||||
|
|
||||||
|
child_navpoints = nav_point.find_all("navPoint", recursive=False)
|
||||||
|
if child_navpoints:
|
||||||
|
for child_np in child_navpoints:
|
||||||
|
self._parse_ncx_navpoint(
|
||||||
|
child_np,
|
||||||
|
ordered_entries,
|
||||||
|
doc_order,
|
||||||
|
doc_order_decoded,
|
||||||
|
current_entry_node["children"],
|
||||||
|
find_position_func,
|
||||||
|
)
|
||||||
|
|
||||||
|
if title and (
|
||||||
|
current_entry_node.get("has_content", False) or current_entry_node["children"]
|
||||||
|
):
|
||||||
|
tree_structure_list.append(current_entry_node)
|
||||||
|
|
||||||
|
def _extract_nav_li_title(self, li_element, link_element=None, span_element=None):
|
||||||
|
"""Helper to extract title from a nav <li> element, handling various structures."""
|
||||||
|
title = "Untitled Section"
|
||||||
|
|
||||||
|
if link_element:
|
||||||
|
title = link_element.get_text(strip=True) or title
|
||||||
|
elif span_element:
|
||||||
|
title = span_element.get_text(strip=True) or title
|
||||||
|
|
||||||
|
# Fallback to direct text if title is empty or default
|
||||||
|
# If we used link/span but got empty string, we try fallback.
|
||||||
|
# If we didn't use link/span, we try fallback.
|
||||||
|
if not title.strip() or title == "Untitled Section":
|
||||||
|
li_text = "".join(
|
||||||
|
t for t in li_element.contents if isinstance(t, NavigableString)
|
||||||
|
).strip()
|
||||||
|
if li_text:
|
||||||
|
title = li_text
|
||||||
|
|
||||||
|
# Second fallback: if we have a span but title is still empty, try span text again
|
||||||
|
# (covered by logic above mostly, but mirroring original logic's intense fallback)
|
||||||
|
if (not title.strip() or title == "Untitled Section") and span_text:
|
||||||
|
title = span_text.get_text(strip=True) or title
|
||||||
|
|
||||||
|
return title
|
||||||
|
|
||||||
|
def _parse_html_nav_li(
|
||||||
|
self,
|
||||||
|
li_element,
|
||||||
|
ordered_entries,
|
||||||
|
doc_order,
|
||||||
|
doc_order_decoded,
|
||||||
|
tree_structure_list,
|
||||||
|
find_position_func,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Recursive parsing of HTML5 Navigation (li) nodes.
|
||||||
|
|
||||||
|
Logic tested by: tests/test_epub_html_nav_parsing.py
|
||||||
|
"""
|
||||||
|
link = li_element.find("a", recursive=False)
|
||||||
|
span_text = li_element.find("span", recursive=False)
|
||||||
|
src = None
|
||||||
|
current_entry_node = {"children": []}
|
||||||
|
|
||||||
|
if link and "href" in link.attrs:
|
||||||
|
src = link["href"]
|
||||||
|
|
||||||
|
title = self._extract_nav_li_title(li_element, link, span_text)
|
||||||
|
|
||||||
|
current_entry_node["title"] = title
|
||||||
|
current_entry_node["src"] = src
|
||||||
|
|
||||||
|
doc_key = None
|
||||||
|
doc_idx = None
|
||||||
|
position = 0
|
||||||
|
fragment = None
|
||||||
|
if src:
|
||||||
|
base_href, fragment = src.split("#", 1) if "#" in src else (src, None)
|
||||||
|
doc_key, doc_idx = self._find_doc_key(base_href, doc_order, doc_order_decoded)
|
||||||
|
if doc_key is not None:
|
||||||
|
position = find_position_func(doc_key, fragment)
|
||||||
|
entry_data = {
|
||||||
|
"src": src,
|
||||||
|
"title": title,
|
||||||
|
"doc_href": doc_key,
|
||||||
|
"position": position,
|
||||||
|
"doc_order": doc_idx,
|
||||||
|
}
|
||||||
|
ordered_entries.append(entry_data)
|
||||||
|
current_entry_node["has_content"] = True
|
||||||
|
else:
|
||||||
|
current_entry_node["has_content"] = False
|
||||||
|
else:
|
||||||
|
current_entry_node["has_content"] = False
|
||||||
|
|
||||||
|
for child_ol in li_element.find_all("ol", recursive=False):
|
||||||
|
for child_li in child_ol.find_all("li", recursive=False):
|
||||||
|
self._parse_html_nav_li(
|
||||||
|
child_li,
|
||||||
|
ordered_entries,
|
||||||
|
doc_order,
|
||||||
|
doc_order_decoded,
|
||||||
|
current_entry_node["children"],
|
||||||
|
find_position_func,
|
||||||
|
)
|
||||||
|
tree_structure_list.append(current_entry_node)
|
||||||
|
|
||||||
|
def _identify_nav_item(self):
|
||||||
|
"""Identify the navigation item (HTML Nav or NCX) and its type."""
|
||||||
|
nav_item = None
|
||||||
|
nav_type = None
|
||||||
|
|
||||||
|
# 1. Check ITEM_NAVIGATION
|
||||||
|
nav_items = list(self.book.get_items_of_type(ebooklib.ITEM_NAVIGATION))
|
||||||
|
|
||||||
|
# 1.1 Support for EPUB 3 EpubNav which might be ITEM_DOCUMENT (9) but with properties=['nav']
|
||||||
|
if not nav_items:
|
||||||
|
# Look in ITEM_DOCUMENT for items with 'nav' property
|
||||||
|
for item in self.book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
|
||||||
|
if hasattr(item, 'get_type') and item.get_type() == ebooklib.ITEM_DOCUMENT:
|
||||||
|
# Check properties - ebooklib stores opf properties in list
|
||||||
|
# Some versions use item.properties, some need checking
|
||||||
|
props = getattr(item, 'properties', [])
|
||||||
|
if 'nav' in props:
|
||||||
|
nav_items.append(item)
|
||||||
|
|
||||||
|
if nav_items:
|
||||||
|
nav_item = next(
|
||||||
|
(
|
||||||
|
item
|
||||||
|
for item in nav_items
|
||||||
|
if "nav" in item.get_name().lower()
|
||||||
|
and item.get_name().lower().endswith((".xhtml", ".html"))
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
) or next(
|
||||||
|
(
|
||||||
|
item
|
||||||
|
for item in nav_items
|
||||||
|
if item.get_name().lower().endswith((".xhtml", ".html"))
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if nav_item:
|
||||||
|
nav_type = "html"
|
||||||
|
|
||||||
|
# 2. NCX in NAV
|
||||||
|
if not nav_item and nav_items:
|
||||||
|
ncx_in_nav = next(
|
||||||
|
(item for item in nav_items if item.get_name().lower().endswith(".ncx")),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if ncx_in_nav:
|
||||||
|
nav_item = ncx_in_nav
|
||||||
|
nav_type = "ncx"
|
||||||
|
|
||||||
|
# 3. ITEM_NCX or Fallback
|
||||||
|
# If no explicit navigation item found, try to find a standard NCX file
|
||||||
|
if not nav_item:
|
||||||
|
ncx_constant = getattr(epub, "ITEM_NCX", None)
|
||||||
|
if ncx_constant is not None:
|
||||||
|
ncx_items = list(self.book.get_items_of_type(ncx_constant))
|
||||||
|
if ncx_items:
|
||||||
|
nav_item = ncx_items[0]
|
||||||
|
nav_type = "ncx"
|
||||||
|
|
||||||
|
# 4. Heuristic Search
|
||||||
|
# Scan documents for something that looks like a TOC if standard methods fail
|
||||||
|
if not nav_item:
|
||||||
|
for item in self.book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
|
||||||
|
try:
|
||||||
|
html_content = item.get_content().decode("utf-8", errors="ignore")
|
||||||
|
if "<nav" in html_content and 'epub:type="toc"' in html_content:
|
||||||
|
nav_item = item
|
||||||
|
nav_type = "html"
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if not nav_item or not nav_type:
|
||||||
|
raise ValueError("No navigation document found")
|
||||||
|
|
||||||
|
return nav_item, nav_type
|
||||||
|
|
||||||
|
|
||||||
|
def _execute_nav_parsing_logic(self, nav_item, nav_type):
|
||||||
|
"""Parse the identified navigation item and slice content accordingly."""
|
||||||
|
|
||||||
|
parser_type = "html.parser" if nav_type == "html" else "xml"
|
||||||
|
try:
|
||||||
|
nav_content = nav_item.get_content().decode("utf-8", errors="ignore")
|
||||||
|
nav_soup = BeautifulSoup(nav_content, parser_type)
|
||||||
|
except Exception as e:
|
||||||
|
raise ValueError(f"Failed to parse navigation content: {e}")
|
||||||
|
|
||||||
|
self.doc_content = {}
|
||||||
|
spine_docs = []
|
||||||
|
for spine_item_tuple in self.book.spine:
|
||||||
|
item_id = spine_item_tuple[0]
|
||||||
|
item = self.book.get_item_with_id(item_id)
|
||||||
|
if item:
|
||||||
|
spine_docs.append(item.get_name())
|
||||||
|
doc_order = {href: i for i, href in enumerate(spine_docs)}
|
||||||
|
doc_order_decoded = {
|
||||||
|
urllib.parse.unquote(href): i for href, i in doc_order.items()
|
||||||
|
}
|
||||||
|
|
||||||
|
self.content_texts = {}
|
||||||
|
self.content_lengths = {}
|
||||||
|
|
||||||
|
for item in self.book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
|
||||||
|
href = item.get_name()
|
||||||
|
if href in doc_order or any(
|
||||||
|
href in nav_point.get("src", "")
|
||||||
|
for nav_point in nav_soup.find_all(["content", "a"])
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
self.doc_content[href] = item.get_content().decode(
|
||||||
|
"utf-8", errors="ignore"
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
self.doc_content[href] = ""
|
||||||
|
|
||||||
|
ordered_nav_entries = []
|
||||||
|
parse_successful = False
|
||||||
|
|
||||||
|
if nav_type == "ncx":
|
||||||
|
nav_map = nav_soup.find("navMap")
|
||||||
|
if nav_map:
|
||||||
|
for nav_point in nav_map.find_all("navPoint", recursive=False):
|
||||||
|
self._parse_ncx_navpoint(
|
||||||
|
nav_point,
|
||||||
|
ordered_nav_entries,
|
||||||
|
doc_order,
|
||||||
|
doc_order_decoded,
|
||||||
|
self.processed_nav_structure,
|
||||||
|
self._find_position_robust,
|
||||||
|
)
|
||||||
|
parse_successful = bool(ordered_nav_entries)
|
||||||
|
elif nav_type == "html":
|
||||||
|
toc_nav = nav_soup.find("nav", attrs={"epub:type": "toc"})
|
||||||
|
if not toc_nav:
|
||||||
|
for nav in nav_soup.find_all("nav"):
|
||||||
|
if nav.find("ol"):
|
||||||
|
toc_nav = nav
|
||||||
|
break
|
||||||
|
if toc_nav:
|
||||||
|
top_ol = toc_nav.find("ol", recursive=False)
|
||||||
|
if top_ol:
|
||||||
|
for li in top_ol.find_all("li", recursive=False):
|
||||||
|
self._parse_html_nav_li(
|
||||||
|
li,
|
||||||
|
ordered_nav_entries,
|
||||||
|
doc_order,
|
||||||
|
doc_order_decoded,
|
||||||
|
self.processed_nav_structure,
|
||||||
|
self._find_position_robust,
|
||||||
|
)
|
||||||
|
parse_successful = bool(ordered_nav_entries)
|
||||||
|
|
||||||
|
if not parse_successful:
|
||||||
|
raise ValueError("No valid navigation entries found after parsing")
|
||||||
|
|
||||||
|
ordered_nav_entries.sort(key=lambda x: (x["doc_order"], x["position"]))
|
||||||
|
|
||||||
|
num_entries = len(ordered_nav_entries)
|
||||||
|
for i in range(num_entries):
|
||||||
|
current_entry = ordered_nav_entries[i]
|
||||||
|
current_src = current_entry["src"]
|
||||||
|
current_doc = current_entry["doc_href"]
|
||||||
|
current_pos = current_entry["position"]
|
||||||
|
current_doc_html = self.doc_content.get(current_doc, "")
|
||||||
|
|
||||||
|
start_slice_pos = current_pos
|
||||||
|
slice_html = ""
|
||||||
|
|
||||||
|
next_entry = ordered_nav_entries[i + 1] if (i + 1) < num_entries else None
|
||||||
|
|
||||||
|
if next_entry:
|
||||||
|
next_doc = next_entry["doc_href"]
|
||||||
|
next_pos = next_entry["position"]
|
||||||
|
|
||||||
|
if current_doc == next_doc:
|
||||||
|
slice_html = current_doc_html[start_slice_pos:next_pos]
|
||||||
|
else:
|
||||||
|
slice_html = current_doc_html[start_slice_pos:]
|
||||||
|
docs_between = []
|
||||||
|
try:
|
||||||
|
idx_current = spine_docs.index(current_doc)
|
||||||
|
idx_next = spine_docs.index(next_doc)
|
||||||
|
if idx_current < idx_next:
|
||||||
|
docs_between = [
|
||||||
|
spine_docs[k] for k in range(idx_current + 1, idx_next)
|
||||||
|
]
|
||||||
|
elif idx_current > idx_next:
|
||||||
|
docs_between = [
|
||||||
|
spine_docs[k]
|
||||||
|
for k in range(idx_current + 1, len(spine_docs))
|
||||||
|
]
|
||||||
|
docs_between.extend(
|
||||||
|
[spine_docs[k] for k in range(0, idx_next)]
|
||||||
|
)
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
for doc_href in docs_between:
|
||||||
|
slice_html += self.doc_content.get(doc_href, "")
|
||||||
|
next_doc_html = self.doc_content.get(next_doc, "")
|
||||||
|
slice_html += next_doc_html[:next_pos]
|
||||||
|
else:
|
||||||
|
slice_html = current_doc_html[start_slice_pos:]
|
||||||
|
try:
|
||||||
|
idx_current = spine_docs.index(current_doc)
|
||||||
|
for doc_idx in range(idx_current + 1, len(spine_docs)):
|
||||||
|
slice_html += self.doc_content.get(spine_docs[doc_idx], "")
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if not slice_html.strip() and current_doc_html:
|
||||||
|
slice_html = current_doc_html
|
||||||
|
|
||||||
|
if slice_html.strip():
|
||||||
|
slice_soup = BeautifulSoup(slice_html, "html.parser")
|
||||||
|
for tag in slice_soup.find_all(["p", "div"]):
|
||||||
|
tag.append("\n\n")
|
||||||
|
|
||||||
|
for ol in slice_soup.find_all("ol"):
|
||||||
|
start = int(ol.get("start", 1))
|
||||||
|
for idx, li in enumerate(ol.find_all("li", recursive=False)):
|
||||||
|
number_text = f"{start + idx}) "
|
||||||
|
if li.string:
|
||||||
|
li.string.replace_with(number_text + li.string)
|
||||||
|
else:
|
||||||
|
li.insert(0, NavigableString(number_text))
|
||||||
|
|
||||||
|
for tag in slice_soup.find_all(["sup", "sub"]):
|
||||||
|
tag.decompose()
|
||||||
|
|
||||||
|
text = clean_text(slice_soup.get_text()).strip()
|
||||||
|
if text:
|
||||||
|
self.content_texts[current_src] = text
|
||||||
|
self.content_lengths[current_src] = calculate_text_length(text)
|
||||||
|
else:
|
||||||
|
self.content_texts[current_src] = ""
|
||||||
|
self.content_lengths[current_src] = 0
|
||||||
|
else:
|
||||||
|
self.content_texts[current_src] = ""
|
||||||
|
self.content_lengths[current_src] = 0
|
||||||
|
|
||||||
|
if ordered_nav_entries:
|
||||||
|
first_entry = ordered_nav_entries[0]
|
||||||
|
first_doc_href = first_entry["doc_href"]
|
||||||
|
first_pos = first_entry["position"]
|
||||||
|
first_doc_order = first_entry["doc_order"]
|
||||||
|
prefix_html = ""
|
||||||
|
|
||||||
|
for doc_idx in range(first_doc_order):
|
||||||
|
if doc_idx < len(spine_docs):
|
||||||
|
intermediate_doc_href = spine_docs[doc_idx]
|
||||||
|
prefix_html += self.doc_content.get(intermediate_doc_href, "")
|
||||||
|
|
||||||
|
first_doc_html = self.doc_content.get(first_doc_href, "")
|
||||||
|
prefix_html += first_doc_html[:first_pos]
|
||||||
|
|
||||||
|
if prefix_html.strip():
|
||||||
|
prefix_soup = BeautifulSoup(prefix_html, "html.parser")
|
||||||
|
for tag in prefix_soup.find_all(["sup", "sub"]):
|
||||||
|
tag.decompose()
|
||||||
|
prefix_text = clean_text(prefix_soup.get_text()).strip()
|
||||||
|
|
||||||
|
if prefix_text:
|
||||||
|
prefix_chapter_src = "internal:prefix_content"
|
||||||
|
self.content_texts[prefix_chapter_src] = prefix_text
|
||||||
|
self.content_lengths[prefix_chapter_src] = len(prefix_text)
|
||||||
|
self.processed_nav_structure.insert(
|
||||||
|
0,
|
||||||
|
{
|
||||||
|
"src": prefix_chapter_src,
|
||||||
|
"title": "Introduction",
|
||||||
|
"children": [],
|
||||||
|
"has_content": True,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def _process_epub_content_spine_fallback(self):
|
||||||
|
"""
|
||||||
|
Process EPUB content using the spine (linear reading order)
|
||||||
|
when navigation processing fails.
|
||||||
|
"""
|
||||||
|
logging.info("Using spine fallback for EPUB processing.")
|
||||||
|
self.doc_content = {}
|
||||||
|
spine_docs = []
|
||||||
|
for spine_item_tuple in self.book.spine:
|
||||||
|
item_id = spine_item_tuple[0]
|
||||||
|
item = self.book.get_item_with_id(item_id)
|
||||||
|
if item:
|
||||||
|
spine_docs.append(item.get_name())
|
||||||
|
else:
|
||||||
|
logging.warning(f"Spine item with id '{item_id}' not found.")
|
||||||
|
|
||||||
|
for item in self.book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
|
||||||
|
href = item.get_name()
|
||||||
|
if href in spine_docs:
|
||||||
|
try:
|
||||||
|
html_content = item.get_content().decode("utf-8", errors="ignore")
|
||||||
|
self.doc_content[href] = html_content
|
||||||
|
except Exception:
|
||||||
|
self.doc_content[href] = ""
|
||||||
|
|
||||||
|
self.content_texts = {}
|
||||||
|
self.content_lengths = {}
|
||||||
|
for i, doc_href in enumerate(spine_docs):
|
||||||
|
html_content = self.doc_content.get(doc_href, "")
|
||||||
|
if html_content:
|
||||||
|
soup = BeautifulSoup(html_content, "html.parser")
|
||||||
|
|
||||||
|
# Handle ordered lists
|
||||||
|
for ol in soup.find_all("ol"):
|
||||||
|
start = int(ol.get("start", 1))
|
||||||
|
for idx, li in enumerate(ol.find_all("li", recursive=False)):
|
||||||
|
number_text = f"{start + idx}) "
|
||||||
|
if li.string:
|
||||||
|
li.string.replace_with(number_text + li.string)
|
||||||
|
else:
|
||||||
|
li.insert(0, NavigableString(number_text))
|
||||||
|
|
||||||
|
# Remove sup/sub
|
||||||
|
for tag in soup.find_all(["sup", "sub"]):
|
||||||
|
tag.decompose()
|
||||||
|
|
||||||
|
text = clean_text(soup.get_text()).strip()
|
||||||
|
if text:
|
||||||
|
self.content_texts[doc_href] = text
|
||||||
|
self.content_lengths[doc_href] = calculate_text_length(text)
|
||||||
|
|
||||||
|
def get_chapters(self):
|
||||||
|
chapters = super().get_chapters()
|
||||||
|
if not chapters:
|
||||||
|
# Use spine order fallback if no Nav structure
|
||||||
|
if self.book:
|
||||||
|
for spine_item_tuple in self.book.spine:
|
||||||
|
item_id = spine_item_tuple[0]
|
||||||
|
item = self.book.get_item_with_id(item_id)
|
||||||
|
if item:
|
||||||
|
href = item.get_name()
|
||||||
|
if href in self.content_texts:
|
||||||
|
chapters.append((href, href))
|
||||||
|
return chapters
|
||||||
|
|
||||||
|
|
||||||
|
def get_book_parser(book_path, file_type=None):
|
||||||
|
"""
|
||||||
|
Factory function to get the appropriate parser instance.
|
||||||
|
"""
|
||||||
|
book_path = os.path.normpath(os.path.abspath(book_path))
|
||||||
|
|
||||||
|
if not file_type:
|
||||||
|
if book_path.lower().endswith(".pdf"):
|
||||||
|
file_type = "pdf"
|
||||||
|
elif book_path.lower().endswith((".md", ".markdown")):
|
||||||
|
file_type = "markdown"
|
||||||
|
else:
|
||||||
|
file_type = "epub"
|
||||||
|
|
||||||
|
if file_type == "pdf":
|
||||||
|
return PdfParser(book_path)
|
||||||
|
elif file_type == "markdown":
|
||||||
|
return MarkdownParser(book_path)
|
||||||
|
elif file_type == "epub":
|
||||||
|
return EpubParser(book_path)
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unsupported file type: {file_type}")
|
||||||
+12
-5
@@ -1,9 +1,16 @@
|
|||||||
"""Legacy PyQt conversion helpers removed."""
|
"""Backwards-compatible re-export of conversion module.
|
||||||
|
|
||||||
|
The PyQt-based implementation lives in abogen.pyqt.conversion.
|
||||||
|
The web-based implementation is in abogen.webui.conversion_runner.
|
||||||
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
# Re-export PyQt conversion classes for backwards compatibility
|
||||||
|
from abogen.pyqt.conversion import ( # noqa: F401
|
||||||
|
ConversionThread,
|
||||||
|
VoicePreviewThread,
|
||||||
|
PlayAudioThread,
|
||||||
|
)
|
||||||
|
|
||||||
def __getattr__(name: str): # pragma: no cover - compatibility shim
|
__all__ = ["ConversionThread", "VoicePreviewThread", "PlayAudioThread"]
|
||||||
raise AttributeError(
|
|
||||||
"The PyQt-based conversion helpers were removed. Use the web service pipeline instead."
|
|
||||||
)
|
|
||||||
|
|||||||
+6
-10
@@ -1,15 +1,11 @@
|
|||||||
"""Legacy PyQt GUI module removed in favor of the web interface."""
|
"""Backwards-compatible re-export of the PyQt GUI.
|
||||||
|
|
||||||
|
The actual implementation lives in abogen.pyqt.gui.
|
||||||
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from abogen.pyqt.gui import * # noqa: F401, F403
|
||||||
class abogen: # pragma: no cover - legacy entry point
|
from abogen.pyqt.gui import abogen
|
||||||
"""Placeholder for the removed PyQt GUI class."""
|
|
||||||
|
|
||||||
def __init__(self, *_args, **_kwargs):
|
|
||||||
raise RuntimeError(
|
|
||||||
"The PyQt desktop interface has been removed. Please use the web UI instead."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["abogen"]
|
__all__ = ["abogen"]
|
||||||
|
|||||||
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,590 @@
|
|||||||
|
"""
|
||||||
|
Pre-download dialog and worker for Abogen
|
||||||
|
|
||||||
|
This module consolidates pre-download logic for Kokoro voices and model
|
||||||
|
and spaCy language models. The code favors clarity, avoids duplication,
|
||||||
|
and handles optional dependencies gracefully.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import List, Optional, Tuple
|
||||||
|
import importlib
|
||||||
|
import importlib.util
|
||||||
|
|
||||||
|
from PyQt6.QtWidgets import (
|
||||||
|
QDialog,
|
||||||
|
QVBoxLayout,
|
||||||
|
QHBoxLayout,
|
||||||
|
QLabel,
|
||||||
|
QPushButton,
|
||||||
|
QSpacerItem,
|
||||||
|
QSizePolicy,
|
||||||
|
)
|
||||||
|
from PyQt6.QtCore import QThread, pyqtSignal
|
||||||
|
|
||||||
|
from abogen.constants import COLORS, VOICES_INTERNAL
|
||||||
|
from abogen.spacy_utils import SPACY_MODELS
|
||||||
|
import abogen.hf_tracker
|
||||||
|
|
||||||
|
|
||||||
|
# Helpers
|
||||||
|
def _unique_sorted_models() -> List[str]:
|
||||||
|
"""Return a sorted list of unique spaCy model package names."""
|
||||||
|
return sorted(set(SPACY_MODELS.values()))
|
||||||
|
|
||||||
|
|
||||||
|
def _is_package_installed(pkg_name: str) -> bool:
|
||||||
|
"""Return True if a package with the given name can be imported (site-packages)."""
|
||||||
|
try:
|
||||||
|
return importlib.util.find_spec(pkg_name) is not None
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
# NOTE: explicit HF cache helper removed; we use try_to_load_from_cache in-scope where needed
|
||||||
|
|
||||||
|
|
||||||
|
class PreDownloadWorker(QThread):
|
||||||
|
"""Worker thread to download required models/voices.
|
||||||
|
|
||||||
|
Emits human-readable messages via `progress`. Uses `category_done` to indicate
|
||||||
|
a category (voices/model/spacy) finished successfully. Emits `error` on exception
|
||||||
|
and `finished` after all work completes.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Emit (category, status, message)
|
||||||
|
progress = pyqtSignal(str, str, str)
|
||||||
|
category_done = pyqtSignal(str)
|
||||||
|
finished = pyqtSignal()
|
||||||
|
error = pyqtSignal(str)
|
||||||
|
|
||||||
|
def __init__(self, parent=None):
|
||||||
|
super().__init__(parent)
|
||||||
|
self._cancelled = False
|
||||||
|
# repo and filenames used for Kokoro model
|
||||||
|
self._repo_id = "hexgrad/Kokoro-82M"
|
||||||
|
self._model_files = ["kokoro-v1_0.pth", "config.json"]
|
||||||
|
# Track download success per category
|
||||||
|
self._voices_success = False
|
||||||
|
self._model_success = False
|
||||||
|
self._spacy_success = False
|
||||||
|
# Suppress HF tracker warnings during downloads
|
||||||
|
self._original_emitter = abogen.hf_tracker.show_warning_signal_emitter
|
||||||
|
|
||||||
|
def cancel(self) -> None:
|
||||||
|
self._cancelled = True
|
||||||
|
|
||||||
|
def run(self) -> None:
|
||||||
|
# Suppress HF tracker warnings during downloads
|
||||||
|
abogen.hf_tracker.show_warning_signal_emitter = None
|
||||||
|
try:
|
||||||
|
self._download_kokoro_voices()
|
||||||
|
if self._cancelled:
|
||||||
|
return
|
||||||
|
if self._voices_success:
|
||||||
|
self.category_done.emit("voices")
|
||||||
|
|
||||||
|
self._download_kokoro_model()
|
||||||
|
if self._cancelled:
|
||||||
|
return
|
||||||
|
if self._model_success:
|
||||||
|
self.category_done.emit("model")
|
||||||
|
|
||||||
|
self._download_spacy_models()
|
||||||
|
if self._cancelled:
|
||||||
|
return
|
||||||
|
if self._spacy_success:
|
||||||
|
self.category_done.emit("spacy")
|
||||||
|
|
||||||
|
self.finished.emit()
|
||||||
|
except Exception as exc: # pragma: no cover - best-effort reporting
|
||||||
|
self.error.emit(str(exc))
|
||||||
|
finally:
|
||||||
|
# Restore original emitter
|
||||||
|
abogen.hf_tracker.show_warning_signal_emitter = self._original_emitter
|
||||||
|
|
||||||
|
# Kokoro voices
|
||||||
|
def _download_kokoro_voices(self) -> None:
|
||||||
|
self._voices_success = True
|
||||||
|
try:
|
||||||
|
from huggingface_hub import hf_hub_download, try_to_load_from_cache
|
||||||
|
except Exception:
|
||||||
|
self.progress.emit(
|
||||||
|
"voice", "warning", "huggingface_hub not installed, skipping voices..."
|
||||||
|
)
|
||||||
|
self._voices_success = False
|
||||||
|
return
|
||||||
|
|
||||||
|
voice_list = VOICES_INTERNAL
|
||||||
|
for idx, voice in enumerate(voice_list, start=1):
|
||||||
|
if self._cancelled:
|
||||||
|
self._voices_success = False
|
||||||
|
return
|
||||||
|
filename = f"voices/{voice}.pt"
|
||||||
|
if try_to_load_from_cache(repo_id=self._repo_id, filename=filename):
|
||||||
|
self.progress.emit(
|
||||||
|
"voice",
|
||||||
|
"installed",
|
||||||
|
f"{idx}/{len(voice_list)}: {voice} already present",
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
self.progress.emit(
|
||||||
|
"voice", "downloading", f"{idx}/{len(voice_list)}: {voice}..."
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
hf_hub_download(repo_id=self._repo_id, filename=filename)
|
||||||
|
self.progress.emit("voice", "downloaded", f"{voice} downloaded")
|
||||||
|
except Exception as exc:
|
||||||
|
self.progress.emit(
|
||||||
|
"voice", "warning", f"could not download {voice}: {exc}"
|
||||||
|
)
|
||||||
|
self._voices_success = False
|
||||||
|
|
||||||
|
# Kokoro model
|
||||||
|
def _download_kokoro_model(self) -> None:
|
||||||
|
self._model_success = True
|
||||||
|
try:
|
||||||
|
from huggingface_hub import hf_hub_download, try_to_load_from_cache
|
||||||
|
except Exception:
|
||||||
|
self.progress.emit(
|
||||||
|
"model", "warning", "huggingface_hub not installed, skipping model..."
|
||||||
|
)
|
||||||
|
self._model_success = False
|
||||||
|
return
|
||||||
|
for fname in self._model_files:
|
||||||
|
if self._cancelled:
|
||||||
|
self._model_success = False
|
||||||
|
return
|
||||||
|
category = "config" if fname == "config.json" else "model"
|
||||||
|
if try_to_load_from_cache(repo_id=self._repo_id, filename=fname):
|
||||||
|
self.progress.emit(
|
||||||
|
category, "installed", f"file {fname} already present"
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
self.progress.emit(category, "downloading", f"file {fname}...")
|
||||||
|
try:
|
||||||
|
hf_hub_download(repo_id=self._repo_id, filename=fname)
|
||||||
|
self.progress.emit(category, "downloaded", f"file {fname} downloaded")
|
||||||
|
except Exception as exc:
|
||||||
|
self.progress.emit(
|
||||||
|
category, "warning", f"could not download file {fname}: {exc}"
|
||||||
|
)
|
||||||
|
self._model_success = False
|
||||||
|
|
||||||
|
# spaCy models
|
||||||
|
def _download_spacy_models(self) -> None:
|
||||||
|
"""Download spaCy models. Prefer missing models provided by parent.
|
||||||
|
|
||||||
|
Parent dialog will populate _spacy_models_missing during checking.
|
||||||
|
"""
|
||||||
|
self._spacy_success = True
|
||||||
|
# Determine which models to process: prefer parent-provided missing list to avoid
|
||||||
|
# re-checking everything; otherwise use the full unique list.
|
||||||
|
parent = self.parent()
|
||||||
|
models_to_process: List[str] = _unique_sorted_models()
|
||||||
|
try:
|
||||||
|
if (
|
||||||
|
parent is not None
|
||||||
|
and hasattr(parent, "_spacy_models_missing")
|
||||||
|
and parent._spacy_models_missing
|
||||||
|
):
|
||||||
|
models_to_process = list(dict.fromkeys(parent._spacy_models_missing))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# If spaCy is not available to run the CLI, skip gracefully
|
||||||
|
try:
|
||||||
|
import spacy.cli as _spacy_cli
|
||||||
|
except Exception:
|
||||||
|
self.progress.emit(
|
||||||
|
"spacy", "warning", "spaCy not available, skipping spaCy models..."
|
||||||
|
)
|
||||||
|
self._spacy_success = False
|
||||||
|
return
|
||||||
|
|
||||||
|
for idx, model_name in enumerate(models_to_process, start=1):
|
||||||
|
if self._cancelled:
|
||||||
|
self._spacy_success = False
|
||||||
|
return
|
||||||
|
if _is_package_installed(model_name):
|
||||||
|
self.progress.emit(
|
||||||
|
"spacy",
|
||||||
|
"installed",
|
||||||
|
f"{idx}/{len(models_to_process)}: {model_name} already installed",
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
self.progress.emit(
|
||||||
|
"spacy",
|
||||||
|
"downloading",
|
||||||
|
f"{idx}/{len(models_to_process)}: {model_name}...",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
_spacy_cli.download(model_name)
|
||||||
|
self.progress.emit("spacy", "downloaded", f"{model_name} downloaded")
|
||||||
|
except Exception as exc:
|
||||||
|
self.progress.emit(
|
||||||
|
"spacy", "warning", f"could not download {model_name}: {exc}"
|
||||||
|
)
|
||||||
|
self._spacy_success = False
|
||||||
|
|
||||||
|
|
||||||
|
class PreDownloadDialog(QDialog):
|
||||||
|
"""Dialog to show and control pre-download process."""
|
||||||
|
|
||||||
|
VOICE_PREFIX = "Kokoro voices: "
|
||||||
|
MODEL_PREFIX = "Kokoro model: "
|
||||||
|
CONFIG_PREFIX = "Kokoro config: "
|
||||||
|
SPACY_PREFIX = "spaCy models: "
|
||||||
|
|
||||||
|
def __init__(self, parent=None):
|
||||||
|
super().__init__(parent)
|
||||||
|
self.setWindowTitle("Pre-download Models and Voices")
|
||||||
|
self.setMinimumWidth(500)
|
||||||
|
self.worker: Optional[PreDownloadWorker] = None
|
||||||
|
self.has_missing = False
|
||||||
|
self._spacy_models_checked: List[tuple] = []
|
||||||
|
self._spacy_models_missing: List[str] = []
|
||||||
|
self._status_worker = None
|
||||||
|
|
||||||
|
# Map keywords to (label, prefix) - labels filled after UI creation
|
||||||
|
self.status_map = {
|
||||||
|
"voice": (None, self.VOICE_PREFIX),
|
||||||
|
"spacy": (None, self.SPACY_PREFIX),
|
||||||
|
"model": (None, self.MODEL_PREFIX),
|
||||||
|
"config": (None, self.CONFIG_PREFIX),
|
||||||
|
}
|
||||||
|
|
||||||
|
self.category_map = {
|
||||||
|
"voices": ["voice"],
|
||||||
|
"model": ["model", "config"],
|
||||||
|
"spacy": ["spacy"],
|
||||||
|
}
|
||||||
|
|
||||||
|
self._setup_ui()
|
||||||
|
self._start_status_check()
|
||||||
|
|
||||||
|
def _setup_ui(self) -> None:
|
||||||
|
layout = QVBoxLayout(self)
|
||||||
|
layout.setSpacing(0)
|
||||||
|
layout.setContentsMargins(15, 0, 15, 15)
|
||||||
|
|
||||||
|
desc = QLabel(
|
||||||
|
"You can pre-download all required models and voices for offline use.\n"
|
||||||
|
"This includes Kokoro voices, Kokoro model (and config), and spaCy models."
|
||||||
|
)
|
||||||
|
desc.setWordWrap(True)
|
||||||
|
layout.addWidget(desc)
|
||||||
|
|
||||||
|
# Status rows
|
||||||
|
status_layout = QVBoxLayout()
|
||||||
|
status_title = QLabel("<b>Current Status:</b>")
|
||||||
|
status_layout.addWidget(status_title)
|
||||||
|
|
||||||
|
self.voices_status = QLabel(self.VOICE_PREFIX + "⏳ Checking...")
|
||||||
|
row = QHBoxLayout()
|
||||||
|
row.addWidget(self.voices_status)
|
||||||
|
row.addStretch()
|
||||||
|
status_layout.addLayout(row)
|
||||||
|
|
||||||
|
self.model_status = QLabel(self.MODEL_PREFIX + "⏳ Checking...")
|
||||||
|
row = QHBoxLayout()
|
||||||
|
row.addWidget(self.model_status)
|
||||||
|
row.addStretch()
|
||||||
|
status_layout.addLayout(row)
|
||||||
|
|
||||||
|
self.config_status = QLabel(self.CONFIG_PREFIX + "⏳ Checking...")
|
||||||
|
row = QHBoxLayout()
|
||||||
|
row.addWidget(self.config_status)
|
||||||
|
row.addStretch()
|
||||||
|
status_layout.addLayout(row)
|
||||||
|
|
||||||
|
self.spacy_status = QLabel(self.SPACY_PREFIX + "⏳ Checking...")
|
||||||
|
row = QHBoxLayout()
|
||||||
|
row.addWidget(self.spacy_status)
|
||||||
|
row.addStretch()
|
||||||
|
status_layout.addLayout(row)
|
||||||
|
|
||||||
|
# register labels
|
||||||
|
self.status_map["voice"] = (self.voices_status, self.VOICE_PREFIX)
|
||||||
|
self.status_map["model"] = (self.model_status, self.MODEL_PREFIX)
|
||||||
|
self.status_map["config"] = (self.config_status, self.CONFIG_PREFIX)
|
||||||
|
self.status_map["spacy"] = (self.spacy_status, self.SPACY_PREFIX)
|
||||||
|
|
||||||
|
layout.addLayout(status_layout)
|
||||||
|
|
||||||
|
layout.addItem(
|
||||||
|
QSpacerItem(0, 20, QSizePolicy.Policy.Minimum, QSizePolicy.Policy.Fixed)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Buttons
|
||||||
|
button_row = QHBoxLayout()
|
||||||
|
button_row.setSpacing(10)
|
||||||
|
self.download_btn = QPushButton("Download all")
|
||||||
|
self.download_btn.setMinimumWidth(100)
|
||||||
|
self.download_btn.setMinimumHeight(35)
|
||||||
|
self.download_btn.setEnabled(False)
|
||||||
|
self.download_btn.clicked.connect(self._start_download)
|
||||||
|
button_row.addWidget(self.download_btn)
|
||||||
|
|
||||||
|
self.close_btn = QPushButton("Close")
|
||||||
|
self.close_btn.setMinimumWidth(100)
|
||||||
|
self.close_btn.setMinimumHeight(35)
|
||||||
|
self.close_btn.clicked.connect(self._handle_close)
|
||||||
|
button_row.addWidget(self.close_btn)
|
||||||
|
|
||||||
|
layout.addLayout(button_row)
|
||||||
|
self.adjustSize()
|
||||||
|
|
||||||
|
# Status checking worker
|
||||||
|
class StatusCheckWorker(QThread):
|
||||||
|
voices_checked = pyqtSignal(bool, list)
|
||||||
|
model_checked = pyqtSignal(bool)
|
||||||
|
config_checked = pyqtSignal(bool)
|
||||||
|
spacy_model_checking = pyqtSignal(str)
|
||||||
|
spacy_model_result = pyqtSignal(str, bool)
|
||||||
|
spacy_checked = pyqtSignal(bool, list)
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
parent = self.parent()
|
||||||
|
if parent is None:
|
||||||
|
return
|
||||||
|
|
||||||
|
voices_ok, missing_voices = parent._check_kokoro_voices()
|
||||||
|
self.voices_checked.emit(voices_ok, missing_voices)
|
||||||
|
|
||||||
|
model_ok = parent._check_kokoro_model()
|
||||||
|
self.model_checked.emit(model_ok)
|
||||||
|
|
||||||
|
config_ok = parent._check_kokoro_config()
|
||||||
|
self.config_checked.emit(config_ok)
|
||||||
|
|
||||||
|
# Check spaCy models by package name to detect site-package installs
|
||||||
|
unique = _unique_sorted_models()
|
||||||
|
missing: List[str] = []
|
||||||
|
for name in unique:
|
||||||
|
self.spacy_model_checking.emit(name)
|
||||||
|
ok = _is_package_installed(name)
|
||||||
|
self.spacy_model_result.emit(name, ok)
|
||||||
|
if not ok:
|
||||||
|
missing.append(name)
|
||||||
|
parent._spacy_models_missing = missing
|
||||||
|
self.spacy_checked.emit(len(missing) == 0, missing)
|
||||||
|
|
||||||
|
def _start_status_check(self) -> None:
|
||||||
|
self._status_worker = self.StatusCheckWorker(self)
|
||||||
|
self._status_worker.voices_checked.connect(self._update_voices_status)
|
||||||
|
self._status_worker.model_checked.connect(self._update_model_status)
|
||||||
|
self._status_worker.config_checked.connect(self._update_config_status)
|
||||||
|
self._status_worker.spacy_model_checking.connect(self._spacy_model_checking)
|
||||||
|
self._status_worker.spacy_model_result.connect(self._spacy_model_result)
|
||||||
|
self._status_worker.spacy_checked.connect(self._update_spacy_status)
|
||||||
|
|
||||||
|
# These are initialized in __init__ to keep consistent object state
|
||||||
|
|
||||||
|
# Set checking visual state
|
||||||
|
for lbl in (
|
||||||
|
self.voices_status,
|
||||||
|
self.model_status,
|
||||||
|
self.config_status,
|
||||||
|
self.spacy_status,
|
||||||
|
):
|
||||||
|
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
|
||||||
|
|
||||||
|
self.spacy_status.setText(self.SPACY_PREFIX + "⏳ Checking...")
|
||||||
|
self._status_worker.start()
|
||||||
|
|
||||||
|
# UI update callbacks
|
||||||
|
def _spacy_model_checking(self, name: str) -> None:
|
||||||
|
self.spacy_status.setText(f"{self.SPACY_PREFIX}Checking {name}...")
|
||||||
|
|
||||||
|
def _spacy_model_result(self, name: str, ok: bool) -> None:
|
||||||
|
self._spacy_models_checked.append((name, ok))
|
||||||
|
if not ok and name not in self._spacy_models_missing:
|
||||||
|
self._spacy_models_missing.append(name)
|
||||||
|
checked = len(self._spacy_models_checked)
|
||||||
|
missing_count = len(self._spacy_models_missing)
|
||||||
|
if missing_count:
|
||||||
|
self.spacy_status.setText(
|
||||||
|
f"{self.SPACY_PREFIX}{checked} checked, {missing_count} missing..."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self.spacy_status.setText(f"{self.SPACY_PREFIX}{checked} checked...")
|
||||||
|
|
||||||
|
def _update_voices_status(self, ok: bool, missing: List[str]) -> None:
|
||||||
|
if ok:
|
||||||
|
self._set_status("voice", "✓ Downloaded", COLORS["GREEN"])
|
||||||
|
else:
|
||||||
|
self.has_missing = True
|
||||||
|
if missing:
|
||||||
|
self._set_status(
|
||||||
|
"voice", f"✗ Missing {len(missing)} voices", COLORS["RED"]
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self._set_status("voice", "✗ Not downloaded", COLORS["RED"])
|
||||||
|
|
||||||
|
def _update_model_status(self, ok: bool) -> None:
|
||||||
|
if ok:
|
||||||
|
self._set_status("model", "✓ Downloaded", COLORS["GREEN"])
|
||||||
|
else:
|
||||||
|
self.has_missing = True
|
||||||
|
self._set_status("model", "✗ Not downloaded", COLORS["RED"])
|
||||||
|
|
||||||
|
def _update_config_status(self, ok: bool) -> None:
|
||||||
|
if ok:
|
||||||
|
self._set_status("config", "✓ Downloaded", COLORS["GREEN"])
|
||||||
|
else:
|
||||||
|
self.has_missing = True
|
||||||
|
self._set_status("config", "✗ Not downloaded", COLORS["RED"])
|
||||||
|
|
||||||
|
def _update_spacy_status(self, ok: bool, missing: List[str]) -> None:
|
||||||
|
if ok:
|
||||||
|
self._set_status("spacy", "✓ Downloaded", COLORS["GREEN"])
|
||||||
|
else:
|
||||||
|
self.has_missing = True
|
||||||
|
if missing:
|
||||||
|
self._set_status(
|
||||||
|
"spacy", f"✗ Missing {len(missing)} model(s)", COLORS["RED"]
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self._set_status("spacy", "✗ Not downloaded", COLORS["RED"])
|
||||||
|
self.download_btn.setEnabled(self.has_missing)
|
||||||
|
|
||||||
|
def _set_status(self, key: str, text: str, color: str) -> None:
|
||||||
|
lbl, prefix = self.status_map.get(key, (None, ""))
|
||||||
|
if not lbl:
|
||||||
|
return
|
||||||
|
lbl.setText(prefix + text)
|
||||||
|
lbl.setStyleSheet(f"color: {color};")
|
||||||
|
|
||||||
|
# Helper checks
|
||||||
|
def _check_kokoro_voices(self) -> Tuple[bool, List[str]]:
|
||||||
|
"""Return (ok, missing_list) for Kokoro voices check."""
|
||||||
|
missing = []
|
||||||
|
try:
|
||||||
|
from huggingface_hub import try_to_load_from_cache
|
||||||
|
|
||||||
|
for voice in VOICES_INTERNAL:
|
||||||
|
if not try_to_load_from_cache(
|
||||||
|
repo_id="hexgrad/Kokoro-82M", filename=f"voices/{voice}.pt"
|
||||||
|
):
|
||||||
|
missing.append(voice)
|
||||||
|
except Exception:
|
||||||
|
# If HF missing, report all as missing
|
||||||
|
return False, list(VOICES_INTERNAL)
|
||||||
|
return (len(missing) == 0), missing
|
||||||
|
|
||||||
|
def _check_kokoro_model(self) -> bool:
|
||||||
|
try:
|
||||||
|
from huggingface_hub import try_to_load_from_cache
|
||||||
|
|
||||||
|
return (
|
||||||
|
try_to_load_from_cache(
|
||||||
|
repo_id="hexgrad/Kokoro-82M", filename="kokoro-v1_0.pth"
|
||||||
|
)
|
||||||
|
is not None
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def _check_kokoro_config(self) -> bool:
|
||||||
|
try:
|
||||||
|
from huggingface_hub import try_to_load_from_cache
|
||||||
|
|
||||||
|
return (
|
||||||
|
try_to_load_from_cache(
|
||||||
|
repo_id="hexgrad/Kokoro-82M", filename="config.json"
|
||||||
|
)
|
||||||
|
is not None
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def _check_spacy_models(self) -> bool:
|
||||||
|
unique = _unique_sorted_models()
|
||||||
|
missing = [m for m in unique if not _is_package_installed(m)]
|
||||||
|
self._spacy_models_missing = missing
|
||||||
|
return len(missing) == 0
|
||||||
|
|
||||||
|
# Download control
|
||||||
|
def _start_download(self) -> None:
|
||||||
|
self.download_btn.setEnabled(False)
|
||||||
|
self.download_btn.setText("Downloading...")
|
||||||
|
# mark the start of downloads; this triggers the labels
|
||||||
|
self._on_progress("system", "starting", "Processing, please wait...")
|
||||||
|
self.worker = PreDownloadWorker(self)
|
||||||
|
self.worker.progress.connect(self._on_progress)
|
||||||
|
self.worker.category_done.connect(self._on_category_done)
|
||||||
|
self.worker.finished.connect(self._on_download_finished)
|
||||||
|
self.worker.error.connect(self._on_download_error)
|
||||||
|
self.worker.start()
|
||||||
|
|
||||||
|
def _on_progress(self, category: str, status: str, message: str) -> None:
|
||||||
|
"""Map worker (category, status, message) to UI label updates.
|
||||||
|
|
||||||
|
Status is one of: 'downloading', 'installed', 'downloaded', 'warning', 'starting'.
|
||||||
|
Category is one of: 'voice', 'model', 'spacy', 'config', or 'system'.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# If the category targets a specific label, update directly
|
||||||
|
if category in self.status_map:
|
||||||
|
lbl, prefix = self.status_map[category]
|
||||||
|
if not lbl:
|
||||||
|
return
|
||||||
|
# Compose message and set color based on status token
|
||||||
|
full_text = prefix + message
|
||||||
|
if len(full_text) > 60:
|
||||||
|
display_text = full_text[:57] + "..."
|
||||||
|
lbl.setText(display_text)
|
||||||
|
lbl.setToolTip(full_text)
|
||||||
|
else:
|
||||||
|
lbl.setText(full_text)
|
||||||
|
lbl.setToolTip("") # Clear tooltip if not needed
|
||||||
|
if status == "downloading":
|
||||||
|
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
|
||||||
|
elif status in ("installed", "downloaded"):
|
||||||
|
lbl.setStyleSheet(f"color: {COLORS['GREEN']};")
|
||||||
|
elif status == "warning":
|
||||||
|
lbl.setStyleSheet(f"color: {COLORS['RED']};")
|
||||||
|
elif status == "error":
|
||||||
|
lbl.setStyleSheet(f"color: {COLORS['RED']};")
|
||||||
|
return
|
||||||
|
|
||||||
|
# System-level messages
|
||||||
|
if category == "system":
|
||||||
|
if status == "starting":
|
||||||
|
for k in self.status_map:
|
||||||
|
lbl, prefix = self.status_map[k]
|
||||||
|
if lbl:
|
||||||
|
lbl.setText(prefix + "Processing, please wait...")
|
||||||
|
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
|
||||||
|
# other system statuses don't require action
|
||||||
|
return
|
||||||
|
except Exception:
|
||||||
|
# Do not let UI thread crash on unexpected worker message
|
||||||
|
pass
|
||||||
|
|
||||||
|
def _on_category_done(self, category: str) -> None:
|
||||||
|
for key in self.category_map.get(category, []):
|
||||||
|
self._set_status(key, "✓ Downloaded", COLORS["GREEN"])
|
||||||
|
|
||||||
|
def _on_download_finished(self) -> None:
|
||||||
|
self.has_missing = False
|
||||||
|
self.download_btn.setText("Download all")
|
||||||
|
self.download_btn.setEnabled(False)
|
||||||
|
|
||||||
|
def _on_download_error(self, error_msg: str) -> None:
|
||||||
|
self.download_btn.setText("Download all")
|
||||||
|
self.download_btn.setEnabled(True)
|
||||||
|
for key in self.status_map:
|
||||||
|
self._set_status(key, f"✗ Error - {error_msg}", COLORS["RED"])
|
||||||
|
|
||||||
|
def _handle_close(self) -> None:
|
||||||
|
if self.worker and self.worker.isRunning():
|
||||||
|
self.worker.cancel()
|
||||||
|
self.worker.wait(2000)
|
||||||
|
self.accept()
|
||||||
|
|
||||||
|
def closeEvent(self, event) -> None:
|
||||||
|
if self.worker and self.worker.isRunning():
|
||||||
|
self.worker.cancel()
|
||||||
|
self.worker.wait(2000)
|
||||||
|
super().closeEvent(event)
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
"""PyQt6 Desktop GUI for abogen.
|
||||||
|
|
||||||
|
This package contains the traditional PyQt6-based desktop interface.
|
||||||
|
For the web-based interface, see abogen.webui.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+4049
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,187 @@
|
|||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import platform
|
||||||
|
import atexit
|
||||||
|
import signal
|
||||||
|
from abogen.utils import get_resource_path, load_config, prevent_sleep_end
|
||||||
|
|
||||||
|
|
||||||
|
# Fix PyTorch DLL loading issue ([WinError 1114]) on Windows before importing PyQt6
|
||||||
|
if platform.system() == "Windows":
|
||||||
|
import ctypes
|
||||||
|
from importlib.util import find_spec
|
||||||
|
|
||||||
|
try:
|
||||||
|
if (
|
||||||
|
(spec := find_spec("torch"))
|
||||||
|
and spec.origin
|
||||||
|
and os.path.exists(
|
||||||
|
dll_path := os.path.join(os.path.dirname(spec.origin), "lib", "c10.dll")
|
||||||
|
)
|
||||||
|
):
|
||||||
|
ctypes.CDLL(os.path.normpath(dll_path))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# Qt platform plugin detection (fixes #59)
|
||||||
|
try:
|
||||||
|
from PyQt6.QtCore import QLibraryInfo
|
||||||
|
|
||||||
|
# Get the path to the plugins directory
|
||||||
|
plugins = QLibraryInfo.path(QLibraryInfo.LibraryPath.PluginsPath)
|
||||||
|
|
||||||
|
# Normalize path to use the OS-native separators and absolute path
|
||||||
|
platform_dir = os.path.normpath(os.path.join(plugins, "platforms"))
|
||||||
|
|
||||||
|
# Ensure we work with an absolute path for clarity
|
||||||
|
platform_dir = os.path.abspath(platform_dir)
|
||||||
|
|
||||||
|
if os.path.isdir(platform_dir):
|
||||||
|
os.environ["QT_QPA_PLATFORM_PLUGIN_PATH"] = platform_dir
|
||||||
|
print("QT_QPA_PLATFORM_PLUGIN_PATH set to:", platform_dir)
|
||||||
|
else:
|
||||||
|
print("PyQt6 platform plugins not found at", platform_dir)
|
||||||
|
except ImportError:
|
||||||
|
print("PyQt6 not installed.")
|
||||||
|
|
||||||
|
|
||||||
|
# Pre-load "libxcb-cursor" on Linux (fixes #101)
|
||||||
|
if platform.system() == "Linux":
|
||||||
|
arch = platform.machine().lower()
|
||||||
|
lib_filename = {"x86_64": "libxcb-cursor-amd64.so.0", "amd64": "libxcb-cursor-amd64.so.0", "aarch64": "libxcb-cursor-arm64.so.0", "arm64": "libxcb-cursor-arm64.so.0"}.get(arch)
|
||||||
|
if lib_filename:
|
||||||
|
import ctypes
|
||||||
|
try:
|
||||||
|
# Try to load the system libxcb-cursor.so.0 first
|
||||||
|
ctypes.CDLL('libxcb-cursor.so.0', mode=ctypes.RTLD_GLOBAL)
|
||||||
|
except OSError:
|
||||||
|
# System lib not available, load the bundled version
|
||||||
|
lib_path = get_resource_path('abogen.libs', lib_filename)
|
||||||
|
if lib_path:
|
||||||
|
try:
|
||||||
|
ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
|
||||||
|
except OSError:
|
||||||
|
# If it fails (e.g. wrong glibc version on very old systems),
|
||||||
|
# we simply ignore it and hope the system has the library.
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# Set application ID for Windows taskbar icon
|
||||||
|
if platform.system() == "Windows":
|
||||||
|
try:
|
||||||
|
from abogen.constants import PROGRAM_NAME, VERSION
|
||||||
|
import ctypes
|
||||||
|
|
||||||
|
app_id = f"{PROGRAM_NAME}.{VERSION}"
|
||||||
|
ctypes.windll.shell32.SetCurrentProcessExplicitAppUserModelID(app_id)
|
||||||
|
except Exception as e:
|
||||||
|
print("Warning: failed to set AppUserModelID:", e)
|
||||||
|
|
||||||
|
from PyQt6.QtWidgets import QApplication
|
||||||
|
from PyQt6.QtGui import QIcon
|
||||||
|
from PyQt6.QtCore import (
|
||||||
|
QLibraryInfo,
|
||||||
|
qInstallMessageHandler,
|
||||||
|
QtMsgType,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Add the directory to Python path
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__)))
|
||||||
|
|
||||||
|
# Set Hugging Face Hub environment variables
|
||||||
|
os.environ["HF_HUB_DISABLE_TELEMETRY"] = "1" # Disable Hugging Face telemetry
|
||||||
|
os.environ["HF_HUB_ETAG_TIMEOUT"] = "10" # Metadata request timeout (seconds)
|
||||||
|
os.environ["HF_HUB_DOWNLOAD_TIMEOUT"] = "10" # File download timeout (seconds)
|
||||||
|
os.environ["HF_HUB_DISABLE_SYMLINKS_WARNING"] = "1" # Disable symlinks warning
|
||||||
|
if load_config().get("disable_kokoro_internet", False):
|
||||||
|
print("INFO: Kokoro's internet access is disabled.")
|
||||||
|
os.environ["HF_HUB_OFFLINE"] = "1" # Disable Hugging Face Hub internet access
|
||||||
|
|
||||||
|
from abogen.pyqt.gui import abogen
|
||||||
|
from abogen.constants import PROGRAM_NAME, VERSION
|
||||||
|
|
||||||
|
# Set environment variables for AMD ROCm
|
||||||
|
os.environ["MIOPEN_FIND_MODE"] = "FAST"
|
||||||
|
os.environ["MIOPEN_CONV_PRECISE_ROCM_TUNING"] = "0"
|
||||||
|
|
||||||
|
# Reset sleep states
|
||||||
|
atexit.register(prevent_sleep_end)
|
||||||
|
|
||||||
|
|
||||||
|
# Also handle signals (Ctrl+C, kill, etc.)
|
||||||
|
def _cleanup_sleep(signum, frame):
|
||||||
|
prevent_sleep_end()
|
||||||
|
sys.exit(0)
|
||||||
|
|
||||||
|
|
||||||
|
signal.signal(signal.SIGINT, _cleanup_sleep)
|
||||||
|
signal.signal(signal.SIGTERM, _cleanup_sleep)
|
||||||
|
|
||||||
|
# Ensure sys.stdout and sys.stderr are valid in GUI mode
|
||||||
|
if sys.stdout is None:
|
||||||
|
sys.stdout = open(os.devnull, "w")
|
||||||
|
if sys.stderr is None:
|
||||||
|
sys.stderr = open(os.devnull, "w")
|
||||||
|
|
||||||
|
# Enable MPS GPU acceleration on Mac Apple Silicon
|
||||||
|
if platform.system() == "Darwin" and platform.processor() == "arm":
|
||||||
|
os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"
|
||||||
|
|
||||||
|
|
||||||
|
# Custom message handler to filter out specific Qt warnings
|
||||||
|
def qt_message_handler(mode, context, message):
|
||||||
|
# In PyQt6, the mode is an enum, so we compare with the enum members
|
||||||
|
if "Wayland does not support QWindow::requestActivate()" in message:
|
||||||
|
return # Suppress this specific message
|
||||||
|
if "setGrabPopup called with a parent, QtWaylandClient" in message:
|
||||||
|
return
|
||||||
|
|
||||||
|
if mode == QtMsgType.QtWarningMsg:
|
||||||
|
print(f"Qt Warning: {message}")
|
||||||
|
elif mode == QtMsgType.QtCriticalMsg:
|
||||||
|
print(f"Qt Critical: {message}")
|
||||||
|
elif mode == QtMsgType.QtFatalMsg:
|
||||||
|
print(f"Qt Fatal: {message}")
|
||||||
|
elif mode == QtMsgType.QtInfoMsg:
|
||||||
|
print(f"Qt Info: {message}")
|
||||||
|
|
||||||
|
|
||||||
|
# Install the custom message handler
|
||||||
|
qInstallMessageHandler(qt_message_handler)
|
||||||
|
|
||||||
|
# Handle Wayland on Linux GNOME
|
||||||
|
if platform.system() == "Linux":
|
||||||
|
xdg_session = os.environ.get("XDG_SESSION_TYPE", "").lower()
|
||||||
|
desktop = os.environ.get("XDG_CURRENT_DESKTOP", "").lower()
|
||||||
|
if (
|
||||||
|
"gnome" in desktop
|
||||||
|
and xdg_session == "wayland"
|
||||||
|
and "QT_QPA_PLATFORM" not in os.environ
|
||||||
|
):
|
||||||
|
os.environ["QT_QPA_PLATFORM"] = "wayland"
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Main entry point for console usage."""
|
||||||
|
app = QApplication(sys.argv)
|
||||||
|
|
||||||
|
# Set application icon using get_resource_path from utils
|
||||||
|
icon_path = get_resource_path("abogen.assets", "icon.ico")
|
||||||
|
if icon_path:
|
||||||
|
app.setWindowIcon(QIcon(icon_path))
|
||||||
|
|
||||||
|
# Set the .desktop name on Linux
|
||||||
|
if platform.system() == "Linux":
|
||||||
|
try:
|
||||||
|
app.setDesktopFileName("abogen")
|
||||||
|
except AttributeError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
ex = abogen()
|
||||||
|
ex.show()
|
||||||
|
sys.exit(app.exec())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,590 @@
|
|||||||
|
"""
|
||||||
|
Pre-download dialog and worker for Abogen
|
||||||
|
|
||||||
|
This module consolidates pre-download logic for Kokoro voices and model
|
||||||
|
and spaCy language models. The code favors clarity, avoids duplication,
|
||||||
|
and handles optional dependencies gracefully.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import List, Optional, Tuple
|
||||||
|
import importlib
|
||||||
|
import importlib.util
|
||||||
|
|
||||||
|
from PyQt6.QtWidgets import (
|
||||||
|
QDialog,
|
||||||
|
QVBoxLayout,
|
||||||
|
QHBoxLayout,
|
||||||
|
QLabel,
|
||||||
|
QPushButton,
|
||||||
|
QSpacerItem,
|
||||||
|
QSizePolicy,
|
||||||
|
)
|
||||||
|
from PyQt6.QtCore import QThread, pyqtSignal
|
||||||
|
|
||||||
|
from abogen.constants import COLORS, VOICES_INTERNAL
|
||||||
|
from abogen.spacy_utils import SPACY_MODELS
|
||||||
|
import abogen.hf_tracker
|
||||||
|
|
||||||
|
|
||||||
|
# Helpers
|
||||||
|
def _unique_sorted_models() -> List[str]:
|
||||||
|
"""Return a sorted list of unique spaCy model package names."""
|
||||||
|
return sorted(set(SPACY_MODELS.values()))
|
||||||
|
|
||||||
|
|
||||||
|
def _is_package_installed(pkg_name: str) -> bool:
|
||||||
|
"""Return True if a package with the given name can be imported (site-packages)."""
|
||||||
|
try:
|
||||||
|
return importlib.util.find_spec(pkg_name) is not None
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
# NOTE: explicit HF cache helper removed; we use try_to_load_from_cache in-scope where needed
|
||||||
|
|
||||||
|
|
||||||
|
class PreDownloadWorker(QThread):
|
||||||
|
"""Worker thread to download required models/voices.
|
||||||
|
|
||||||
|
Emits human-readable messages via `progress`. Uses `category_done` to indicate
|
||||||
|
a category (voices/model/spacy) finished successfully. Emits `error` on exception
|
||||||
|
and `finished` after all work completes.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Emit (category, status, message)
|
||||||
|
progress = pyqtSignal(str, str, str)
|
||||||
|
category_done = pyqtSignal(str)
|
||||||
|
finished = pyqtSignal()
|
||||||
|
error = pyqtSignal(str)
|
||||||
|
|
||||||
|
def __init__(self, parent=None):
|
||||||
|
super().__init__(parent)
|
||||||
|
self._cancelled = False
|
||||||
|
# repo and filenames used for Kokoro model
|
||||||
|
self._repo_id = "hexgrad/Kokoro-82M"
|
||||||
|
self._model_files = ["kokoro-v1_0.pth", "config.json"]
|
||||||
|
# Track download success per category
|
||||||
|
self._voices_success = False
|
||||||
|
self._model_success = False
|
||||||
|
self._spacy_success = False
|
||||||
|
# Suppress HF tracker warnings during downloads
|
||||||
|
self._original_emitter = abogen.hf_tracker.show_warning_signal_emitter
|
||||||
|
|
||||||
|
def cancel(self) -> None:
|
||||||
|
self._cancelled = True
|
||||||
|
|
||||||
|
def run(self) -> None:
|
||||||
|
# Suppress HF tracker warnings during downloads
|
||||||
|
abogen.hf_tracker.show_warning_signal_emitter = None
|
||||||
|
try:
|
||||||
|
self._download_kokoro_voices()
|
||||||
|
if self._cancelled:
|
||||||
|
return
|
||||||
|
if self._voices_success:
|
||||||
|
self.category_done.emit("voices")
|
||||||
|
|
||||||
|
self._download_kokoro_model()
|
||||||
|
if self._cancelled:
|
||||||
|
return
|
||||||
|
if self._model_success:
|
||||||
|
self.category_done.emit("model")
|
||||||
|
|
||||||
|
self._download_spacy_models()
|
||||||
|
if self._cancelled:
|
||||||
|
return
|
||||||
|
if self._spacy_success:
|
||||||
|
self.category_done.emit("spacy")
|
||||||
|
|
||||||
|
self.finished.emit()
|
||||||
|
except Exception as exc: # pragma: no cover - best-effort reporting
|
||||||
|
self.error.emit(str(exc))
|
||||||
|
finally:
|
||||||
|
# Restore original emitter
|
||||||
|
abogen.hf_tracker.show_warning_signal_emitter = self._original_emitter
|
||||||
|
|
||||||
|
# Kokoro voices
|
||||||
|
def _download_kokoro_voices(self) -> None:
|
||||||
|
self._voices_success = True
|
||||||
|
try:
|
||||||
|
from huggingface_hub import hf_hub_download, try_to_load_from_cache
|
||||||
|
except Exception:
|
||||||
|
self.progress.emit(
|
||||||
|
"voice", "warning", "huggingface_hub not installed, skipping voices..."
|
||||||
|
)
|
||||||
|
self._voices_success = False
|
||||||
|
return
|
||||||
|
|
||||||
|
voice_list = VOICES_INTERNAL
|
||||||
|
for idx, voice in enumerate(voice_list, start=1):
|
||||||
|
if self._cancelled:
|
||||||
|
self._voices_success = False
|
||||||
|
return
|
||||||
|
filename = f"voices/{voice}.pt"
|
||||||
|
if try_to_load_from_cache(repo_id=self._repo_id, filename=filename):
|
||||||
|
self.progress.emit(
|
||||||
|
"voice",
|
||||||
|
"installed",
|
||||||
|
f"{idx}/{len(voice_list)}: {voice} already present",
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
self.progress.emit(
|
||||||
|
"voice", "downloading", f"{idx}/{len(voice_list)}: {voice}..."
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
hf_hub_download(repo_id=self._repo_id, filename=filename)
|
||||||
|
self.progress.emit("voice", "downloaded", f"{voice} downloaded")
|
||||||
|
except Exception as exc:
|
||||||
|
self.progress.emit(
|
||||||
|
"voice", "warning", f"could not download {voice}: {exc}"
|
||||||
|
)
|
||||||
|
self._voices_success = False
|
||||||
|
|
||||||
|
# Kokoro model
|
||||||
|
def _download_kokoro_model(self) -> None:
|
||||||
|
self._model_success = True
|
||||||
|
try:
|
||||||
|
from huggingface_hub import hf_hub_download, try_to_load_from_cache
|
||||||
|
except Exception:
|
||||||
|
self.progress.emit(
|
||||||
|
"model", "warning", "huggingface_hub not installed, skipping model..."
|
||||||
|
)
|
||||||
|
self._model_success = False
|
||||||
|
return
|
||||||
|
for fname in self._model_files:
|
||||||
|
if self._cancelled:
|
||||||
|
self._model_success = False
|
||||||
|
return
|
||||||
|
category = "config" if fname == "config.json" else "model"
|
||||||
|
if try_to_load_from_cache(repo_id=self._repo_id, filename=fname):
|
||||||
|
self.progress.emit(
|
||||||
|
category, "installed", f"file {fname} already present"
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
self.progress.emit(category, "downloading", f"file {fname}...")
|
||||||
|
try:
|
||||||
|
hf_hub_download(repo_id=self._repo_id, filename=fname)
|
||||||
|
self.progress.emit(category, "downloaded", f"file {fname} downloaded")
|
||||||
|
except Exception as exc:
|
||||||
|
self.progress.emit(
|
||||||
|
category, "warning", f"could not download file {fname}: {exc}"
|
||||||
|
)
|
||||||
|
self._model_success = False
|
||||||
|
|
||||||
|
# spaCy models
|
||||||
|
def _download_spacy_models(self) -> None:
|
||||||
|
"""Download spaCy models. Prefer missing models provided by parent.
|
||||||
|
|
||||||
|
Parent dialog will populate _spacy_models_missing during checking.
|
||||||
|
"""
|
||||||
|
self._spacy_success = True
|
||||||
|
# Determine which models to process: prefer parent-provided missing list to avoid
|
||||||
|
# re-checking everything; otherwise use the full unique list.
|
||||||
|
parent = self.parent()
|
||||||
|
models_to_process: List[str] = _unique_sorted_models()
|
||||||
|
try:
|
||||||
|
if (
|
||||||
|
parent is not None
|
||||||
|
and hasattr(parent, "_spacy_models_missing")
|
||||||
|
and parent._spacy_models_missing
|
||||||
|
):
|
||||||
|
models_to_process = list(dict.fromkeys(parent._spacy_models_missing))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# If spaCy is not available to run the CLI, skip gracefully
|
||||||
|
try:
|
||||||
|
import spacy.cli as _spacy_cli
|
||||||
|
except Exception:
|
||||||
|
self.progress.emit(
|
||||||
|
"spacy", "warning", "spaCy not available, skipping spaCy models..."
|
||||||
|
)
|
||||||
|
self._spacy_success = False
|
||||||
|
return
|
||||||
|
|
||||||
|
for idx, model_name in enumerate(models_to_process, start=1):
|
||||||
|
if self._cancelled:
|
||||||
|
self._spacy_success = False
|
||||||
|
return
|
||||||
|
if _is_package_installed(model_name):
|
||||||
|
self.progress.emit(
|
||||||
|
"spacy",
|
||||||
|
"installed",
|
||||||
|
f"{idx}/{len(models_to_process)}: {model_name} already installed",
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
self.progress.emit(
|
||||||
|
"spacy",
|
||||||
|
"downloading",
|
||||||
|
f"{idx}/{len(models_to_process)}: {model_name}...",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
_spacy_cli.download(model_name)
|
||||||
|
self.progress.emit("spacy", "downloaded", f"{model_name} downloaded")
|
||||||
|
except Exception as exc:
|
||||||
|
self.progress.emit(
|
||||||
|
"spacy", "warning", f"could not download {model_name}: {exc}"
|
||||||
|
)
|
||||||
|
self._spacy_success = False
|
||||||
|
|
||||||
|
|
||||||
|
class PreDownloadDialog(QDialog):
|
||||||
|
"""Dialog to show and control pre-download process."""
|
||||||
|
|
||||||
|
VOICE_PREFIX = "Kokoro voices: "
|
||||||
|
MODEL_PREFIX = "Kokoro model: "
|
||||||
|
CONFIG_PREFIX = "Kokoro config: "
|
||||||
|
SPACY_PREFIX = "spaCy models: "
|
||||||
|
|
||||||
|
def __init__(self, parent=None):
|
||||||
|
super().__init__(parent)
|
||||||
|
self.setWindowTitle("Pre-download Models and Voices")
|
||||||
|
self.setMinimumWidth(500)
|
||||||
|
self.worker: Optional[PreDownloadWorker] = None
|
||||||
|
self.has_missing = False
|
||||||
|
self._spacy_models_checked: List[tuple] = []
|
||||||
|
self._spacy_models_missing: List[str] = []
|
||||||
|
self._status_worker = None
|
||||||
|
|
||||||
|
# Map keywords to (label, prefix) - labels filled after UI creation
|
||||||
|
self.status_map = {
|
||||||
|
"voice": (None, self.VOICE_PREFIX),
|
||||||
|
"spacy": (None, self.SPACY_PREFIX),
|
||||||
|
"model": (None, self.MODEL_PREFIX),
|
||||||
|
"config": (None, self.CONFIG_PREFIX),
|
||||||
|
}
|
||||||
|
|
||||||
|
self.category_map = {
|
||||||
|
"voices": ["voice"],
|
||||||
|
"model": ["model", "config"],
|
||||||
|
"spacy": ["spacy"],
|
||||||
|
}
|
||||||
|
|
||||||
|
self._setup_ui()
|
||||||
|
self._start_status_check()
|
||||||
|
|
||||||
|
def _setup_ui(self) -> None:
|
||||||
|
layout = QVBoxLayout(self)
|
||||||
|
layout.setSpacing(0)
|
||||||
|
layout.setContentsMargins(15, 0, 15, 15)
|
||||||
|
|
||||||
|
desc = QLabel(
|
||||||
|
"You can pre-download all required models and voices for offline use.\n"
|
||||||
|
"This includes Kokoro voices, Kokoro model (and config), and spaCy models."
|
||||||
|
)
|
||||||
|
desc.setWordWrap(True)
|
||||||
|
layout.addWidget(desc)
|
||||||
|
|
||||||
|
# Status rows
|
||||||
|
status_layout = QVBoxLayout()
|
||||||
|
status_title = QLabel("<b>Current Status:</b>")
|
||||||
|
status_layout.addWidget(status_title)
|
||||||
|
|
||||||
|
self.voices_status = QLabel(self.VOICE_PREFIX + "⏳ Checking...")
|
||||||
|
row = QHBoxLayout()
|
||||||
|
row.addWidget(self.voices_status)
|
||||||
|
row.addStretch()
|
||||||
|
status_layout.addLayout(row)
|
||||||
|
|
||||||
|
self.model_status = QLabel(self.MODEL_PREFIX + "⏳ Checking...")
|
||||||
|
row = QHBoxLayout()
|
||||||
|
row.addWidget(self.model_status)
|
||||||
|
row.addStretch()
|
||||||
|
status_layout.addLayout(row)
|
||||||
|
|
||||||
|
self.config_status = QLabel(self.CONFIG_PREFIX + "⏳ Checking...")
|
||||||
|
row = QHBoxLayout()
|
||||||
|
row.addWidget(self.config_status)
|
||||||
|
row.addStretch()
|
||||||
|
status_layout.addLayout(row)
|
||||||
|
|
||||||
|
self.spacy_status = QLabel(self.SPACY_PREFIX + "⏳ Checking...")
|
||||||
|
row = QHBoxLayout()
|
||||||
|
row.addWidget(self.spacy_status)
|
||||||
|
row.addStretch()
|
||||||
|
status_layout.addLayout(row)
|
||||||
|
|
||||||
|
# register labels
|
||||||
|
self.status_map["voice"] = (self.voices_status, self.VOICE_PREFIX)
|
||||||
|
self.status_map["model"] = (self.model_status, self.MODEL_PREFIX)
|
||||||
|
self.status_map["config"] = (self.config_status, self.CONFIG_PREFIX)
|
||||||
|
self.status_map["spacy"] = (self.spacy_status, self.SPACY_PREFIX)
|
||||||
|
|
||||||
|
layout.addLayout(status_layout)
|
||||||
|
|
||||||
|
layout.addItem(
|
||||||
|
QSpacerItem(0, 20, QSizePolicy.Policy.Minimum, QSizePolicy.Policy.Fixed)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Buttons
|
||||||
|
button_row = QHBoxLayout()
|
||||||
|
button_row.setSpacing(10)
|
||||||
|
self.download_btn = QPushButton("Download all")
|
||||||
|
self.download_btn.setMinimumWidth(100)
|
||||||
|
self.download_btn.setMinimumHeight(35)
|
||||||
|
self.download_btn.setEnabled(False)
|
||||||
|
self.download_btn.clicked.connect(self._start_download)
|
||||||
|
button_row.addWidget(self.download_btn)
|
||||||
|
|
||||||
|
self.close_btn = QPushButton("Close")
|
||||||
|
self.close_btn.setMinimumWidth(100)
|
||||||
|
self.close_btn.setMinimumHeight(35)
|
||||||
|
self.close_btn.clicked.connect(self._handle_close)
|
||||||
|
button_row.addWidget(self.close_btn)
|
||||||
|
|
||||||
|
layout.addLayout(button_row)
|
||||||
|
self.adjustSize()
|
||||||
|
|
||||||
|
# Status checking worker
|
||||||
|
class StatusCheckWorker(QThread):
|
||||||
|
voices_checked = pyqtSignal(bool, list)
|
||||||
|
model_checked = pyqtSignal(bool)
|
||||||
|
config_checked = pyqtSignal(bool)
|
||||||
|
spacy_model_checking = pyqtSignal(str)
|
||||||
|
spacy_model_result = pyqtSignal(str, bool)
|
||||||
|
spacy_checked = pyqtSignal(bool, list)
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
parent = self.parent()
|
||||||
|
if parent is None:
|
||||||
|
return
|
||||||
|
|
||||||
|
voices_ok, missing_voices = parent._check_kokoro_voices()
|
||||||
|
self.voices_checked.emit(voices_ok, missing_voices)
|
||||||
|
|
||||||
|
model_ok = parent._check_kokoro_model()
|
||||||
|
self.model_checked.emit(model_ok)
|
||||||
|
|
||||||
|
config_ok = parent._check_kokoro_config()
|
||||||
|
self.config_checked.emit(config_ok)
|
||||||
|
|
||||||
|
# Check spaCy models by package name to detect site-package installs
|
||||||
|
unique = _unique_sorted_models()
|
||||||
|
missing: List[str] = []
|
||||||
|
for name in unique:
|
||||||
|
self.spacy_model_checking.emit(name)
|
||||||
|
ok = _is_package_installed(name)
|
||||||
|
self.spacy_model_result.emit(name, ok)
|
||||||
|
if not ok:
|
||||||
|
missing.append(name)
|
||||||
|
parent._spacy_models_missing = missing
|
||||||
|
self.spacy_checked.emit(len(missing) == 0, missing)
|
||||||
|
|
||||||
|
def _start_status_check(self) -> None:
|
||||||
|
self._status_worker = self.StatusCheckWorker(self)
|
||||||
|
self._status_worker.voices_checked.connect(self._update_voices_status)
|
||||||
|
self._status_worker.model_checked.connect(self._update_model_status)
|
||||||
|
self._status_worker.config_checked.connect(self._update_config_status)
|
||||||
|
self._status_worker.spacy_model_checking.connect(self._spacy_model_checking)
|
||||||
|
self._status_worker.spacy_model_result.connect(self._spacy_model_result)
|
||||||
|
self._status_worker.spacy_checked.connect(self._update_spacy_status)
|
||||||
|
|
||||||
|
# These are initialized in __init__ to keep consistent object state
|
||||||
|
|
||||||
|
# Set checking visual state
|
||||||
|
for lbl in (
|
||||||
|
self.voices_status,
|
||||||
|
self.model_status,
|
||||||
|
self.config_status,
|
||||||
|
self.spacy_status,
|
||||||
|
):
|
||||||
|
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
|
||||||
|
|
||||||
|
self.spacy_status.setText(self.SPACY_PREFIX + "⏳ Checking...")
|
||||||
|
self._status_worker.start()
|
||||||
|
|
||||||
|
# UI update callbacks
|
||||||
|
def _spacy_model_checking(self, name: str) -> None:
|
||||||
|
self.spacy_status.setText(f"{self.SPACY_PREFIX}Checking {name}...")
|
||||||
|
|
||||||
|
def _spacy_model_result(self, name: str, ok: bool) -> None:
|
||||||
|
self._spacy_models_checked.append((name, ok))
|
||||||
|
if not ok and name not in self._spacy_models_missing:
|
||||||
|
self._spacy_models_missing.append(name)
|
||||||
|
checked = len(self._spacy_models_checked)
|
||||||
|
missing_count = len(self._spacy_models_missing)
|
||||||
|
if missing_count:
|
||||||
|
self.spacy_status.setText(
|
||||||
|
f"{self.SPACY_PREFIX}{checked} checked, {missing_count} missing..."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self.spacy_status.setText(f"{self.SPACY_PREFIX}{checked} checked...")
|
||||||
|
|
||||||
|
def _update_voices_status(self, ok: bool, missing: List[str]) -> None:
|
||||||
|
if ok:
|
||||||
|
self._set_status("voice", "✓ Downloaded", COLORS["GREEN"])
|
||||||
|
else:
|
||||||
|
self.has_missing = True
|
||||||
|
if missing:
|
||||||
|
self._set_status(
|
||||||
|
"voice", f"✗ Missing {len(missing)} voices", COLORS["RED"]
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self._set_status("voice", "✗ Not downloaded", COLORS["RED"])
|
||||||
|
|
||||||
|
def _update_model_status(self, ok: bool) -> None:
|
||||||
|
if ok:
|
||||||
|
self._set_status("model", "✓ Downloaded", COLORS["GREEN"])
|
||||||
|
else:
|
||||||
|
self.has_missing = True
|
||||||
|
self._set_status("model", "✗ Not downloaded", COLORS["RED"])
|
||||||
|
|
||||||
|
def _update_config_status(self, ok: bool) -> None:
|
||||||
|
if ok:
|
||||||
|
self._set_status("config", "✓ Downloaded", COLORS["GREEN"])
|
||||||
|
else:
|
||||||
|
self.has_missing = True
|
||||||
|
self._set_status("config", "✗ Not downloaded", COLORS["RED"])
|
||||||
|
|
||||||
|
def _update_spacy_status(self, ok: bool, missing: List[str]) -> None:
|
||||||
|
if ok:
|
||||||
|
self._set_status("spacy", "✓ Downloaded", COLORS["GREEN"])
|
||||||
|
else:
|
||||||
|
self.has_missing = True
|
||||||
|
if missing:
|
||||||
|
self._set_status(
|
||||||
|
"spacy", f"✗ Missing {len(missing)} model(s)", COLORS["RED"]
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self._set_status("spacy", "✗ Not downloaded", COLORS["RED"])
|
||||||
|
self.download_btn.setEnabled(self.has_missing)
|
||||||
|
|
||||||
|
def _set_status(self, key: str, text: str, color: str) -> None:
|
||||||
|
lbl, prefix = self.status_map.get(key, (None, ""))
|
||||||
|
if not lbl:
|
||||||
|
return
|
||||||
|
lbl.setText(prefix + text)
|
||||||
|
lbl.setStyleSheet(f"color: {color};")
|
||||||
|
|
||||||
|
# Helper checks
|
||||||
|
def _check_kokoro_voices(self) -> Tuple[bool, List[str]]:
|
||||||
|
"""Return (ok, missing_list) for Kokoro voices check."""
|
||||||
|
missing = []
|
||||||
|
try:
|
||||||
|
from huggingface_hub import try_to_load_from_cache
|
||||||
|
|
||||||
|
for voice in VOICES_INTERNAL:
|
||||||
|
if not try_to_load_from_cache(
|
||||||
|
repo_id="hexgrad/Kokoro-82M", filename=f"voices/{voice}.pt"
|
||||||
|
):
|
||||||
|
missing.append(voice)
|
||||||
|
except Exception:
|
||||||
|
# If HF missing, report all as missing
|
||||||
|
return False, list(VOICES_INTERNAL)
|
||||||
|
return (len(missing) == 0), missing
|
||||||
|
|
||||||
|
def _check_kokoro_model(self) -> bool:
|
||||||
|
try:
|
||||||
|
from huggingface_hub import try_to_load_from_cache
|
||||||
|
|
||||||
|
return (
|
||||||
|
try_to_load_from_cache(
|
||||||
|
repo_id="hexgrad/Kokoro-82M", filename="kokoro-v1_0.pth"
|
||||||
|
)
|
||||||
|
is not None
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def _check_kokoro_config(self) -> bool:
|
||||||
|
try:
|
||||||
|
from huggingface_hub import try_to_load_from_cache
|
||||||
|
|
||||||
|
return (
|
||||||
|
try_to_load_from_cache(
|
||||||
|
repo_id="hexgrad/Kokoro-82M", filename="config.json"
|
||||||
|
)
|
||||||
|
is not None
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def _check_spacy_models(self) -> bool:
|
||||||
|
unique = _unique_sorted_models()
|
||||||
|
missing = [m for m in unique if not _is_package_installed(m)]
|
||||||
|
self._spacy_models_missing = missing
|
||||||
|
return len(missing) == 0
|
||||||
|
|
||||||
|
# Download control
|
||||||
|
def _start_download(self) -> None:
|
||||||
|
self.download_btn.setEnabled(False)
|
||||||
|
self.download_btn.setText("Downloading...")
|
||||||
|
# mark the start of downloads; this triggers the labels
|
||||||
|
self._on_progress("system", "starting", "Processing, please wait...")
|
||||||
|
self.worker = PreDownloadWorker(self)
|
||||||
|
self.worker.progress.connect(self._on_progress)
|
||||||
|
self.worker.category_done.connect(self._on_category_done)
|
||||||
|
self.worker.finished.connect(self._on_download_finished)
|
||||||
|
self.worker.error.connect(self._on_download_error)
|
||||||
|
self.worker.start()
|
||||||
|
|
||||||
|
def _on_progress(self, category: str, status: str, message: str) -> None:
|
||||||
|
"""Map worker (category, status, message) to UI label updates.
|
||||||
|
|
||||||
|
Status is one of: 'downloading', 'installed', 'downloaded', 'warning', 'starting'.
|
||||||
|
Category is one of: 'voice', 'model', 'spacy', 'config', or 'system'.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# If the category targets a specific label, update directly
|
||||||
|
if category in self.status_map:
|
||||||
|
lbl, prefix = self.status_map[category]
|
||||||
|
if not lbl:
|
||||||
|
return
|
||||||
|
# Compose message and set color based on status token
|
||||||
|
full_text = prefix + message
|
||||||
|
if len(full_text) > 60:
|
||||||
|
display_text = full_text[:57] + "..."
|
||||||
|
lbl.setText(display_text)
|
||||||
|
lbl.setToolTip(full_text)
|
||||||
|
else:
|
||||||
|
lbl.setText(full_text)
|
||||||
|
lbl.setToolTip("") # Clear tooltip if not needed
|
||||||
|
if status == "downloading":
|
||||||
|
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
|
||||||
|
elif status in ("installed", "downloaded"):
|
||||||
|
lbl.setStyleSheet(f"color: {COLORS['GREEN']};")
|
||||||
|
elif status == "warning":
|
||||||
|
lbl.setStyleSheet(f"color: {COLORS['RED']};")
|
||||||
|
elif status == "error":
|
||||||
|
lbl.setStyleSheet(f"color: {COLORS['RED']};")
|
||||||
|
return
|
||||||
|
|
||||||
|
# System-level messages
|
||||||
|
if category == "system":
|
||||||
|
if status == "starting":
|
||||||
|
for k in self.status_map:
|
||||||
|
lbl, prefix = self.status_map[k]
|
||||||
|
if lbl:
|
||||||
|
lbl.setText(prefix + "Processing, please wait...")
|
||||||
|
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
|
||||||
|
# other system statuses don't require action
|
||||||
|
return
|
||||||
|
except Exception:
|
||||||
|
# Do not let UI thread crash on unexpected worker message
|
||||||
|
pass
|
||||||
|
|
||||||
|
def _on_category_done(self, category: str) -> None:
|
||||||
|
for key in self.category_map.get(category, []):
|
||||||
|
self._set_status(key, "✓ Downloaded", COLORS["GREEN"])
|
||||||
|
|
||||||
|
def _on_download_finished(self) -> None:
|
||||||
|
self.has_missing = False
|
||||||
|
self.download_btn.setText("Download all")
|
||||||
|
self.download_btn.setEnabled(False)
|
||||||
|
|
||||||
|
def _on_download_error(self, error_msg: str) -> None:
|
||||||
|
self.download_btn.setText("Download all")
|
||||||
|
self.download_btn.setEnabled(True)
|
||||||
|
for key in self.status_map:
|
||||||
|
self._set_status(key, f"✗ Error - {error_msg}", COLORS["RED"])
|
||||||
|
|
||||||
|
def _handle_close(self) -> None:
|
||||||
|
if self.worker and self.worker.isRunning():
|
||||||
|
self.worker.cancel()
|
||||||
|
self.worker.wait(2000)
|
||||||
|
self.accept()
|
||||||
|
|
||||||
|
def closeEvent(self, event) -> None:
|
||||||
|
if self.worker and self.worker.isRunning():
|
||||||
|
self.worker.cancel()
|
||||||
|
self.worker.wait(2000)
|
||||||
|
super().closeEvent(event)
|
||||||
@@ -0,0 +1,860 @@
|
|||||||
|
# a simple window with a list of items in the queue, no checkboxes
|
||||||
|
# button to remove an item from the queue
|
||||||
|
# button to clear the queue
|
||||||
|
|
||||||
|
from PyQt6.QtWidgets import (
|
||||||
|
QDialog,
|
||||||
|
QVBoxLayout,
|
||||||
|
QHBoxLayout,
|
||||||
|
QDialogButtonBox,
|
||||||
|
QPushButton,
|
||||||
|
QListWidget,
|
||||||
|
QListWidgetItem,
|
||||||
|
QFileIconProvider,
|
||||||
|
QLabel,
|
||||||
|
QWidget,
|
||||||
|
QSizePolicy,
|
||||||
|
QAbstractItemView,
|
||||||
|
QCheckBox,
|
||||||
|
)
|
||||||
|
from PyQt6.QtCore import QFileInfo, Qt
|
||||||
|
from abogen.constants import COLORS
|
||||||
|
from copy import deepcopy
|
||||||
|
from PyQt6.QtGui import QFontMetrics
|
||||||
|
from abogen.utils import load_config, save_config
|
||||||
|
|
||||||
|
# Define attributes that are safe to override with global settings
|
||||||
|
OVERRIDE_FIELDS = [
|
||||||
|
"lang_code",
|
||||||
|
"speed",
|
||||||
|
"voice",
|
||||||
|
"save_option",
|
||||||
|
"output_folder",
|
||||||
|
"subtitle_mode",
|
||||||
|
"output_format",
|
||||||
|
"replace_single_newlines",
|
||||||
|
"use_silent_gaps",
|
||||||
|
"subtitle_speed_method",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class ElidedLabel(QLabel):
|
||||||
|
def __init__(self, text):
|
||||||
|
super().__init__(text)
|
||||||
|
self._full_text = text
|
||||||
|
self.setSizePolicy(QSizePolicy.Policy.Expanding, QSizePolicy.Policy.Preferred)
|
||||||
|
self.setTextFormat(Qt.TextFormat.PlainText)
|
||||||
|
|
||||||
|
def setText(self, text):
|
||||||
|
self._full_text = text
|
||||||
|
super().setText(text)
|
||||||
|
self.update()
|
||||||
|
|
||||||
|
def resizeEvent(self, event):
|
||||||
|
metrics = QFontMetrics(self.font())
|
||||||
|
elided = metrics.elidedText(
|
||||||
|
self._full_text, Qt.TextElideMode.ElideRight, self.width()
|
||||||
|
)
|
||||||
|
super().setText(elided)
|
||||||
|
super().resizeEvent(event)
|
||||||
|
|
||||||
|
def fullText(self):
|
||||||
|
return self._full_text
|
||||||
|
|
||||||
|
|
||||||
|
class QueueListItemWidget(QWidget):
|
||||||
|
def __init__(self, file_name, char_count):
|
||||||
|
super().__init__()
|
||||||
|
layout = QHBoxLayout()
|
||||||
|
layout.setContentsMargins(12, 0, 6, 0)
|
||||||
|
layout.setSpacing(0)
|
||||||
|
import os
|
||||||
|
|
||||||
|
name_label = ElidedLabel(os.path.basename(file_name))
|
||||||
|
char_label = QLabel(f"Chars: {char_count}")
|
||||||
|
char_label.setStyleSheet(f"color: {COLORS['LIGHT_DISABLED']};")
|
||||||
|
char_label.setAlignment(
|
||||||
|
Qt.AlignmentFlag.AlignRight | Qt.AlignmentFlag.AlignVCenter
|
||||||
|
)
|
||||||
|
char_label.setSizePolicy(
|
||||||
|
QSizePolicy.Policy.Minimum, QSizePolicy.Policy.Preferred
|
||||||
|
)
|
||||||
|
layout.addWidget(name_label, 1)
|
||||||
|
layout.addWidget(char_label, 0)
|
||||||
|
self.setLayout(layout)
|
||||||
|
|
||||||
|
|
||||||
|
class DroppableQueueListWidget(QListWidget):
|
||||||
|
def __init__(self, parent_dialog):
|
||||||
|
super().__init__()
|
||||||
|
self.parent_dialog = parent_dialog
|
||||||
|
self.setAcceptDrops(True)
|
||||||
|
# Overlay for drag hover
|
||||||
|
self.drag_overlay = QLabel("", self)
|
||||||
|
self.drag_overlay.setAlignment(Qt.AlignmentFlag.AlignCenter)
|
||||||
|
self.drag_overlay.setStyleSheet(
|
||||||
|
f"border:2px dashed {COLORS['BLUE_BORDER_HOVER']}; border-radius:5px; padding:20px; background:{COLORS['BLUE_BG_HOVER']};"
|
||||||
|
)
|
||||||
|
self.drag_overlay.setVisible(False)
|
||||||
|
self.drag_overlay.setAttribute(
|
||||||
|
Qt.WidgetAttribute.WA_TransparentForMouseEvents, True
|
||||||
|
)
|
||||||
|
|
||||||
|
def dragEnterEvent(self, event):
|
||||||
|
if event.mimeData().hasUrls():
|
||||||
|
for url in event.mimeData().urls():
|
||||||
|
file_path = url.toLocalFile().lower()
|
||||||
|
if url.isLocalFile() and (
|
||||||
|
file_path.endswith(".txt")
|
||||||
|
or file_path.endswith((".srt", ".ass", ".vtt"))
|
||||||
|
):
|
||||||
|
self.drag_overlay.resize(self.size())
|
||||||
|
self.drag_overlay.setVisible(True)
|
||||||
|
event.acceptProposedAction()
|
||||||
|
return
|
||||||
|
self.drag_overlay.setVisible(False)
|
||||||
|
event.ignore()
|
||||||
|
|
||||||
|
def dragMoveEvent(self, event):
|
||||||
|
if event.mimeData().hasUrls():
|
||||||
|
for url in event.mimeData().urls():
|
||||||
|
file_path = url.toLocalFile().lower()
|
||||||
|
if url.isLocalFile() and (
|
||||||
|
file_path.endswith(".txt")
|
||||||
|
or file_path.endswith((".srt", ".ass", ".vtt"))
|
||||||
|
):
|
||||||
|
event.acceptProposedAction()
|
||||||
|
return
|
||||||
|
event.ignore()
|
||||||
|
|
||||||
|
def dragLeaveEvent(self, event):
|
||||||
|
self.drag_overlay.setVisible(False)
|
||||||
|
event.accept()
|
||||||
|
|
||||||
|
def dropEvent(self, event):
|
||||||
|
self.drag_overlay.setVisible(False)
|
||||||
|
if event.mimeData().hasUrls():
|
||||||
|
file_paths = [
|
||||||
|
url.toLocalFile()
|
||||||
|
for url in event.mimeData().urls()
|
||||||
|
if url.isLocalFile()
|
||||||
|
and (
|
||||||
|
url.toLocalFile().lower().endswith(".txt")
|
||||||
|
or url.toLocalFile().lower().endswith((".srt", ".ass", ".vtt"))
|
||||||
|
)
|
||||||
|
]
|
||||||
|
if file_paths:
|
||||||
|
self.parent_dialog.add_files_from_paths(file_paths)
|
||||||
|
event.acceptProposedAction()
|
||||||
|
else:
|
||||||
|
event.ignore()
|
||||||
|
else:
|
||||||
|
event.ignore()
|
||||||
|
|
||||||
|
def resizeEvent(self, event):
|
||||||
|
super().resizeEvent(event)
|
||||||
|
if hasattr(self, "drag_overlay"):
|
||||||
|
self.drag_overlay.resize(self.size())
|
||||||
|
|
||||||
|
|
||||||
|
class QueueManager(QDialog):
|
||||||
|
def __init__(self, parent, queue: list, title="Queue Manager", size=(600, 700)):
|
||||||
|
super().__init__()
|
||||||
|
self.queue = queue
|
||||||
|
self._original_queue = deepcopy(
|
||||||
|
queue
|
||||||
|
) # Store a deep copy of the original queue
|
||||||
|
self.parent = parent
|
||||||
|
self.config = load_config() # Load config for persistence
|
||||||
|
|
||||||
|
layout = QVBoxLayout()
|
||||||
|
layout.setContentsMargins(15, 15, 15, 15) # set main layout margins
|
||||||
|
layout.setSpacing(12) # set spacing between widgets in main layout
|
||||||
|
# list of queued items
|
||||||
|
self.listwidget = DroppableQueueListWidget(self)
|
||||||
|
self.listwidget.setSelectionMode(
|
||||||
|
QAbstractItemView.SelectionMode.ExtendedSelection
|
||||||
|
)
|
||||||
|
self.listwidget.setAlternatingRowColors(True)
|
||||||
|
self.listwidget.setContextMenuPolicy(Qt.ContextMenuPolicy.CustomContextMenu)
|
||||||
|
self.listwidget.customContextMenuRequested.connect(self.show_context_menu)
|
||||||
|
# Add informative instructions at the top
|
||||||
|
instructions = QLabel(
|
||||||
|
"<h2>How Queue Works?</h2>"
|
||||||
|
"You can add text and subtitle files (.txt, .srt, .ass, .vtt) directly using the '<b>Add files</b>' button below. "
|
||||||
|
"To add PDF, EPUB or markdown files, use the input box in the main window and click the <b>'Add to Queue'</b> button. "
|
||||||
|
"By default, each file in the queue keeps the configuration settings active when they were added. "
|
||||||
|
"Enabling the <b>'Override item settings with current selection'</b> option below will force all items to use the configuration currently selected in the main window. "
|
||||||
|
"You can view each file's configuration by hovering over them."
|
||||||
|
)
|
||||||
|
instructions.setAlignment(Qt.AlignmentFlag.AlignLeft)
|
||||||
|
instructions.setWordWrap(True)
|
||||||
|
layout.addWidget(instructions)
|
||||||
|
|
||||||
|
# Override Checkbox
|
||||||
|
self.override_chk = QCheckBox("Override item settings with current selection")
|
||||||
|
self.override_chk.setToolTip(
|
||||||
|
"If checked, all items in the queue will be processed using the \n"
|
||||||
|
"settings currently selected in the main window, ignoring their saved state."
|
||||||
|
)
|
||||||
|
# Load saved state (default to False)
|
||||||
|
self.override_chk.setChecked(self.config.get("queue_override_settings", False))
|
||||||
|
# Trigger process_queue to update tooltips immediately when toggled
|
||||||
|
self.override_chk.stateChanged.connect(self.process_queue)
|
||||||
|
self.override_chk.setStyleSheet("margin-bottom: 8px;")
|
||||||
|
layout.addWidget(self.override_chk)
|
||||||
|
|
||||||
|
# Overlay label for empty queue
|
||||||
|
self.empty_overlay = QLabel(
|
||||||
|
"Drag and drop your text or subtitle files here or use the 'Add files' button.",
|
||||||
|
self.listwidget,
|
||||||
|
)
|
||||||
|
self.empty_overlay.setAlignment(Qt.AlignmentFlag.AlignCenter)
|
||||||
|
self.empty_overlay.setStyleSheet(
|
||||||
|
f"color: {COLORS['LIGHT_DISABLED']}; background: transparent; padding: 20px;"
|
||||||
|
)
|
||||||
|
self.empty_overlay.setWordWrap(True)
|
||||||
|
self.empty_overlay.setAttribute(
|
||||||
|
Qt.WidgetAttribute.WA_TransparentForMouseEvents, True
|
||||||
|
)
|
||||||
|
self.empty_overlay.hide()
|
||||||
|
# add queue items to the list
|
||||||
|
self.process_queue()
|
||||||
|
|
||||||
|
button_row = QHBoxLayout()
|
||||||
|
button_row.setContentsMargins(0, 0, 0, 0) # optional: no margins for button row
|
||||||
|
button_row.setSpacing(7) # set spacing between buttons
|
||||||
|
# Add files button
|
||||||
|
add_files_button = QPushButton("Add files")
|
||||||
|
add_files_button.setFixedHeight(40)
|
||||||
|
add_files_button.clicked.connect(self.add_more_files)
|
||||||
|
button_row.addWidget(add_files_button)
|
||||||
|
|
||||||
|
# Remove button
|
||||||
|
self.remove_button = QPushButton("Remove selected")
|
||||||
|
self.remove_button.setFixedHeight(40)
|
||||||
|
self.remove_button.clicked.connect(self.remove_item)
|
||||||
|
button_row.addWidget(self.remove_button)
|
||||||
|
|
||||||
|
# Clear button
|
||||||
|
self.clear_button = QPushButton("Clear Queue")
|
||||||
|
self.clear_button.setFixedHeight(40)
|
||||||
|
self.clear_button.clicked.connect(self.clear_queue)
|
||||||
|
button_row.addWidget(self.clear_button)
|
||||||
|
|
||||||
|
layout.addLayout(button_row)
|
||||||
|
layout.addWidget(self.listwidget)
|
||||||
|
|
||||||
|
# Connect selection change to update button state
|
||||||
|
self.listwidget.currentItemChanged.connect(self.update_button_states)
|
||||||
|
self.listwidget.itemSelectionChanged.connect(self.update_button_states)
|
||||||
|
|
||||||
|
buttons = QDialogButtonBox(
|
||||||
|
QDialogButtonBox.StandardButton.Ok | QDialogButtonBox.StandardButton.Cancel,
|
||||||
|
self,
|
||||||
|
)
|
||||||
|
buttons.accepted.connect(self.accept)
|
||||||
|
buttons.rejected.connect(self.reject)
|
||||||
|
|
||||||
|
layout.addWidget(buttons)
|
||||||
|
|
||||||
|
self.setLayout(layout)
|
||||||
|
|
||||||
|
self.setWindowTitle(title)
|
||||||
|
self.resize(*size)
|
||||||
|
|
||||||
|
self.update_button_states()
|
||||||
|
|
||||||
|
def process_queue(self):
|
||||||
|
"""Process the queue items."""
|
||||||
|
import os
|
||||||
|
|
||||||
|
self.listwidget.clear()
|
||||||
|
if not self.queue:
|
||||||
|
self.empty_overlay.show()
|
||||||
|
self.update_button_states()
|
||||||
|
return
|
||||||
|
else:
|
||||||
|
self.empty_overlay.hide()
|
||||||
|
|
||||||
|
# Get current global settings and checkbox state for overrides
|
||||||
|
current_global_settings = self.get_current_attributes()
|
||||||
|
is_override_active = self.override_chk.isChecked()
|
||||||
|
|
||||||
|
icon_provider = QFileIconProvider()
|
||||||
|
for item in self.queue:
|
||||||
|
# Dynamic Attribute Retrieval Helper
|
||||||
|
def get_val(attr, default=""):
|
||||||
|
# If override is ON and attr is overrideable, use global setting
|
||||||
|
if is_override_active and attr in OVERRIDE_FIELDS:
|
||||||
|
return current_global_settings.get(attr, default)
|
||||||
|
# Otherwise return the item's saved attribute
|
||||||
|
return getattr(item, attr, default)
|
||||||
|
|
||||||
|
# Determine display file path (prefer save_base_path for original file)
|
||||||
|
display_file_path = getattr(item, "save_base_path", None) or item.file_name
|
||||||
|
processing_file_path = item.file_name
|
||||||
|
|
||||||
|
# Normalize paths for consistent display (fixes Windows path separator issues)
|
||||||
|
display_file_path = (
|
||||||
|
os.path.normpath(display_file_path)
|
||||||
|
if display_file_path
|
||||||
|
else display_file_path
|
||||||
|
)
|
||||||
|
processing_file_path = (
|
||||||
|
os.path.normpath(processing_file_path)
|
||||||
|
if processing_file_path
|
||||||
|
else processing_file_path
|
||||||
|
)
|
||||||
|
|
||||||
|
# Only show the file name, not the full path
|
||||||
|
display_name = display_file_path
|
||||||
|
|
||||||
|
if os.path.sep in display_file_path:
|
||||||
|
display_name = os.path.basename(display_file_path)
|
||||||
|
# Get icon for the display file
|
||||||
|
icon = icon_provider.icon(QFileInfo(display_file_path))
|
||||||
|
list_item = QListWidgetItem()
|
||||||
|
|
||||||
|
# Tooltip Generation
|
||||||
|
tooltip = ""
|
||||||
|
# If override is active, add the warning header on its own line
|
||||||
|
if is_override_active:
|
||||||
|
tooltip += "<b style='color: #ff9900;'>(Global Override Active)</b><br>"
|
||||||
|
|
||||||
|
output_folder = get_val("output_folder")
|
||||||
|
# For plain .txt inputs we don't need to show a separate processing file
|
||||||
|
show_processing = True
|
||||||
|
try:
|
||||||
|
if isinstance(
|
||||||
|
display_file_path, str
|
||||||
|
) and display_file_path.lower().endswith(".txt"):
|
||||||
|
show_processing = False
|
||||||
|
except Exception:
|
||||||
|
show_processing = True
|
||||||
|
|
||||||
|
tooltip += f"<b>Input File:</b> {display_file_path}<br>"
|
||||||
|
if (
|
||||||
|
show_processing
|
||||||
|
and processing_file_path
|
||||||
|
and processing_file_path != display_file_path
|
||||||
|
):
|
||||||
|
tooltip += f"<b>Processing File:</b> {processing_file_path}<br>"
|
||||||
|
|
||||||
|
tooltip += (
|
||||||
|
f"<b>Language:</b> {get_val('lang_code')}<br>"
|
||||||
|
f"<b>Speed:</b> {get_val('speed')}<br>"
|
||||||
|
f"<b>Voice:</b> {get_val('voice')}<br>"
|
||||||
|
f"<b>Save Option:</b> {get_val('save_option')}<br>"
|
||||||
|
)
|
||||||
|
if output_folder not in (None, "", "None"):
|
||||||
|
tooltip += f"<b>Output Folder:</b> {output_folder}<br>"
|
||||||
|
tooltip += (
|
||||||
|
f"<b>Subtitle Mode:</b> {get_val('subtitle_mode')}<br>"
|
||||||
|
f"<b>Output Format:</b> {get_val('output_format')}<br>"
|
||||||
|
f"<b>Characters:</b> {getattr(item, 'total_char_count', '')}<br>"
|
||||||
|
f"<b>Replace Single Newlines:</b> {get_val('replace_single_newlines', True)}<br>"
|
||||||
|
f"<b>Use Silent Gaps:</b> {get_val('use_silent_gaps', False)}<br>"
|
||||||
|
f"<b>Speed Method:</b> {get_val('subtitle_speed_method', 'tts')}"
|
||||||
|
)
|
||||||
|
# Add book handler options if present (Preserve logic: specific to file structure)
|
||||||
|
save_chapters_separately = getattr(item, "save_chapters_separately", None)
|
||||||
|
merge_chapters_at_end = getattr(item, "merge_chapters_at_end", None)
|
||||||
|
if save_chapters_separately is not None:
|
||||||
|
tooltip += f"<br><b>Save chapters separately:</b> {'Yes' if save_chapters_separately else 'No'}"
|
||||||
|
# Only show merge option if saving chapters separately
|
||||||
|
if save_chapters_separately and merge_chapters_at_end is not None:
|
||||||
|
tooltip += f"<br><b>Merge chapters at the end:</b> {'Yes' if merge_chapters_at_end else 'No'}"
|
||||||
|
list_item.setToolTip(tooltip)
|
||||||
|
list_item.setIcon(icon)
|
||||||
|
# Store both paths for context menu
|
||||||
|
list_item.setData(
|
||||||
|
Qt.ItemDataRole.UserRole,
|
||||||
|
{
|
||||||
|
"display_path": display_file_path,
|
||||||
|
"processing_path": processing_file_path,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
# Use custom widget for display
|
||||||
|
char_count = getattr(item, "total_char_count", 0)
|
||||||
|
widget = QueueListItemWidget(display_file_path, char_count)
|
||||||
|
self.listwidget.addItem(list_item)
|
||||||
|
self.listwidget.setItemWidget(list_item, widget)
|
||||||
|
self.update_button_states()
|
||||||
|
|
||||||
|
def remove_item(self):
|
||||||
|
items = self.listwidget.selectedItems()
|
||||||
|
if not items:
|
||||||
|
return
|
||||||
|
from PyQt6.QtWidgets import QMessageBox
|
||||||
|
|
||||||
|
# Remove by index to ensure correct mapping
|
||||||
|
rows = sorted([self.listwidget.row(item) for item in items], reverse=True)
|
||||||
|
# Warn user if removing multiple files
|
||||||
|
if len(rows) > 1:
|
||||||
|
reply = QMessageBox.question(
|
||||||
|
self,
|
||||||
|
"Confirm Remove",
|
||||||
|
f"Are you sure you want to remove {len(rows)} selected items from the queue?",
|
||||||
|
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
|
||||||
|
QMessageBox.StandardButton.No,
|
||||||
|
)
|
||||||
|
if reply != QMessageBox.StandardButton.Yes:
|
||||||
|
return
|
||||||
|
for row in rows:
|
||||||
|
if 0 <= row < len(self.queue):
|
||||||
|
del self.queue[row]
|
||||||
|
self.process_queue()
|
||||||
|
self.update_button_states()
|
||||||
|
|
||||||
|
def clear_queue(self):
|
||||||
|
from PyQt6.QtWidgets import QMessageBox
|
||||||
|
|
||||||
|
if len(self.queue) > 1:
|
||||||
|
reply = QMessageBox.question(
|
||||||
|
self,
|
||||||
|
"Confirm Clear Queue",
|
||||||
|
f"Are you sure you want to clear {len(self.queue)} items from the queue?",
|
||||||
|
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
|
||||||
|
QMessageBox.StandardButton.No,
|
||||||
|
)
|
||||||
|
if reply != QMessageBox.StandardButton.Yes:
|
||||||
|
return
|
||||||
|
self.queue.clear()
|
||||||
|
self.listwidget.clear()
|
||||||
|
self.empty_overlay.resize(
|
||||||
|
self.listwidget.size()
|
||||||
|
) # Ensure overlay is sized correctly
|
||||||
|
self.empty_overlay.show() # Show the overlay when queue is empty
|
||||||
|
self.update_button_states()
|
||||||
|
|
||||||
|
def get_queue(self):
|
||||||
|
return self.queue
|
||||||
|
|
||||||
|
def get_current_attributes(self):
|
||||||
|
# Fetch current attribute values from the parent abogen GUI
|
||||||
|
attrs = {}
|
||||||
|
parent = self.parent
|
||||||
|
if parent is not None:
|
||||||
|
# lang_code: use parent's get_voice_formula and get_selected_lang
|
||||||
|
if hasattr(parent, "get_voice_formula") and hasattr(
|
||||||
|
parent, "get_selected_lang"
|
||||||
|
):
|
||||||
|
voice_formula = parent.get_voice_formula()
|
||||||
|
attrs["lang_code"] = parent.get_selected_lang(voice_formula)
|
||||||
|
attrs["voice"] = voice_formula
|
||||||
|
else:
|
||||||
|
attrs["lang_code"] = getattr(parent, "selected_lang", "")
|
||||||
|
attrs["voice"] = getattr(parent, "selected_voice", "")
|
||||||
|
# speed
|
||||||
|
if hasattr(parent, "speed_slider"):
|
||||||
|
attrs["speed"] = parent.speed_slider.value() / 100.0
|
||||||
|
else:
|
||||||
|
attrs["speed"] = getattr(parent, "speed", 1.0)
|
||||||
|
# save_option
|
||||||
|
attrs["save_option"] = getattr(parent, "save_option", "")
|
||||||
|
# output_folder
|
||||||
|
attrs["output_folder"] = getattr(parent, "selected_output_folder", "")
|
||||||
|
# subtitle_mode
|
||||||
|
if hasattr(parent, "get_actual_subtitle_mode"):
|
||||||
|
attrs["subtitle_mode"] = parent.get_actual_subtitle_mode()
|
||||||
|
else:
|
||||||
|
attrs["subtitle_mode"] = getattr(parent, "subtitle_mode", "")
|
||||||
|
# output_format
|
||||||
|
attrs["output_format"] = getattr(parent, "selected_format", "")
|
||||||
|
# total_char_count
|
||||||
|
attrs["total_char_count"] = getattr(parent, "char_count", "")
|
||||||
|
# replace_single_newlines
|
||||||
|
attrs["replace_single_newlines"] = getattr(
|
||||||
|
parent, "replace_single_newlines", True
|
||||||
|
)
|
||||||
|
# use_silent_gaps
|
||||||
|
attrs["use_silent_gaps"] = getattr(parent, "use_silent_gaps", False)
|
||||||
|
# subtitle_speed_method
|
||||||
|
attrs["subtitle_speed_method"] = getattr(
|
||||||
|
parent, "subtitle_speed_method", "tts"
|
||||||
|
)
|
||||||
|
# book handler options
|
||||||
|
attrs["save_chapters_separately"] = getattr(
|
||||||
|
parent, "save_chapters_separately", None
|
||||||
|
)
|
||||||
|
attrs["merge_chapters_at_end"] = getattr(
|
||||||
|
parent, "merge_chapters_at_end", None
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# fallback: empty values
|
||||||
|
attrs = {
|
||||||
|
k: ""
|
||||||
|
for k in [
|
||||||
|
"lang_code",
|
||||||
|
"speed",
|
||||||
|
"voice",
|
||||||
|
"save_option",
|
||||||
|
"output_folder",
|
||||||
|
"subtitle_mode",
|
||||||
|
"output_format",
|
||||||
|
"total_char_count",
|
||||||
|
"replace_single_newlines",
|
||||||
|
]
|
||||||
|
}
|
||||||
|
attrs["save_chapters_separately"] = None
|
||||||
|
attrs["merge_chapters_at_end"] = None
|
||||||
|
return attrs
|
||||||
|
|
||||||
|
def add_files_from_paths(self, file_paths):
|
||||||
|
from abogen.subtitle_utils import calculate_text_length
|
||||||
|
from PyQt6.QtWidgets import QMessageBox
|
||||||
|
import os
|
||||||
|
|
||||||
|
current_attrs = self.get_current_attributes()
|
||||||
|
duplicates = []
|
||||||
|
for file_path in file_paths:
|
||||||
|
|
||||||
|
class QueueItem:
|
||||||
|
pass
|
||||||
|
|
||||||
|
item = QueueItem()
|
||||||
|
item.file_name = file_path
|
||||||
|
item.save_base_path = (
|
||||||
|
file_path # For .txt files, processing and save paths are the same
|
||||||
|
)
|
||||||
|
for attr, value in current_attrs.items():
|
||||||
|
setattr(item, attr, value)
|
||||||
|
# Override subtitle_mode to "Disabled" for subtitle files
|
||||||
|
if file_path.lower().endswith((".srt", ".ass", ".vtt")):
|
||||||
|
item.subtitle_mode = "Disabled"
|
||||||
|
# Read file content and calculate total_char_count using calculate_text_length
|
||||||
|
try:
|
||||||
|
with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
|
||||||
|
file_content = f.read()
|
||||||
|
item.total_char_count = calculate_text_length(file_content)
|
||||||
|
except Exception:
|
||||||
|
item.total_char_count = 0
|
||||||
|
# Prevent adding duplicate items to the queue (check all attributes)
|
||||||
|
is_duplicate = False
|
||||||
|
for queued_item in self.queue:
|
||||||
|
if (
|
||||||
|
getattr(queued_item, "file_name", None)
|
||||||
|
== getattr(item, "file_name", None)
|
||||||
|
and getattr(queued_item, "lang_code", None)
|
||||||
|
== getattr(item, "lang_code", None)
|
||||||
|
and getattr(queued_item, "speed", None)
|
||||||
|
== getattr(item, "speed", None)
|
||||||
|
and getattr(queued_item, "voice", None)
|
||||||
|
== getattr(item, "voice", None)
|
||||||
|
and getattr(queued_item, "save_option", None)
|
||||||
|
== getattr(item, "save_option", None)
|
||||||
|
and getattr(queued_item, "output_folder", None)
|
||||||
|
== getattr(item, "output_folder", None)
|
||||||
|
and getattr(queued_item, "subtitle_mode", None)
|
||||||
|
== getattr(item, "subtitle_mode", None)
|
||||||
|
and getattr(queued_item, "output_format", None)
|
||||||
|
== getattr(item, "output_format", None)
|
||||||
|
and getattr(queued_item, "total_char_count", None)
|
||||||
|
== getattr(item, "total_char_count", None)
|
||||||
|
and getattr(queued_item, "replace_single_newlines", True)
|
||||||
|
== getattr(item, "replace_single_newlines", True)
|
||||||
|
and getattr(queued_item, "use_silent_gaps", False)
|
||||||
|
== getattr(item, "use_silent_gaps", False)
|
||||||
|
and getattr(queued_item, "subtitle_speed_method", "tts")
|
||||||
|
== getattr(item, "subtitle_speed_method", "tts")
|
||||||
|
and getattr(queued_item, "save_base_path", None)
|
||||||
|
== getattr(item, "save_base_path", None)
|
||||||
|
and getattr(queued_item, "save_chapters_separately", None)
|
||||||
|
== getattr(item, "save_chapters_separately", None)
|
||||||
|
and getattr(queued_item, "merge_chapters_at_end", None)
|
||||||
|
== getattr(item, "merge_chapters_at_end", None)
|
||||||
|
):
|
||||||
|
is_duplicate = True
|
||||||
|
break
|
||||||
|
if is_duplicate:
|
||||||
|
duplicates.append(os.path.basename(file_path))
|
||||||
|
continue
|
||||||
|
self.queue.append(item)
|
||||||
|
if duplicates:
|
||||||
|
QMessageBox.warning(
|
||||||
|
self,
|
||||||
|
"Duplicate Item(s)",
|
||||||
|
f"Skipping {len(duplicates)} file(s) with the same attributes, already in the queue.",
|
||||||
|
)
|
||||||
|
self.process_queue()
|
||||||
|
self.update_button_states()
|
||||||
|
|
||||||
|
def add_more_files(self):
|
||||||
|
from PyQt6.QtWidgets import QFileDialog
|
||||||
|
|
||||||
|
# Allow .txt, .srt, .ass, and .vtt files
|
||||||
|
files, _ = QFileDialog.getOpenFileNames(
|
||||||
|
self,
|
||||||
|
"Select text or subtitle files",
|
||||||
|
"",
|
||||||
|
"Supported Files (*.txt *.srt *.ass *.vtt)",
|
||||||
|
)
|
||||||
|
if not files:
|
||||||
|
return
|
||||||
|
self.add_files_from_paths(files)
|
||||||
|
|
||||||
|
def resizeEvent(self, event):
|
||||||
|
super().resizeEvent(event)
|
||||||
|
if hasattr(self, "empty_overlay"):
|
||||||
|
self.empty_overlay.resize(self.listwidget.size())
|
||||||
|
|
||||||
|
def update_button_states(self):
|
||||||
|
# Enable Remove if at least one item is selected, else disable
|
||||||
|
if hasattr(self, "remove_button"):
|
||||||
|
selected_count = len(self.listwidget.selectedItems())
|
||||||
|
self.remove_button.setEnabled(selected_count > 0)
|
||||||
|
if selected_count > 1:
|
||||||
|
self.remove_button.setText(f"Remove selected ({selected_count})")
|
||||||
|
else:
|
||||||
|
self.remove_button.setText("Remove selected")
|
||||||
|
# Disable Clear if queue is empty
|
||||||
|
if hasattr(self, "clear_button"):
|
||||||
|
self.clear_button.setEnabled(bool(self.queue))
|
||||||
|
|
||||||
|
def show_context_menu(self, pos):
|
||||||
|
from PyQt6.QtWidgets import QMenu
|
||||||
|
from PyQt6.QtGui import QAction, QDesktopServices
|
||||||
|
from PyQt6.QtCore import QUrl
|
||||||
|
import os
|
||||||
|
|
||||||
|
global_pos = self.listwidget.viewport().mapToGlobal(pos)
|
||||||
|
selected_items = self.listwidget.selectedItems()
|
||||||
|
menu = QMenu(self)
|
||||||
|
if len(selected_items) == 1:
|
||||||
|
# Add Remove action
|
||||||
|
remove_action = QAction("Remove this item", self)
|
||||||
|
remove_action.triggered.connect(self.remove_item)
|
||||||
|
menu.addAction(remove_action)
|
||||||
|
|
||||||
|
# Get paths for determining if it's a document input
|
||||||
|
item = selected_items[0]
|
||||||
|
paths = item.data(Qt.ItemDataRole.UserRole)
|
||||||
|
if isinstance(paths, dict):
|
||||||
|
display_path = paths.get("display_path", "")
|
||||||
|
processing_path = paths.get("processing_path", "")
|
||||||
|
else:
|
||||||
|
display_path = paths
|
||||||
|
processing_path = paths
|
||||||
|
|
||||||
|
doc_exts = (".md", ".markdown", ".pdf", ".epub")
|
||||||
|
is_document_input = (
|
||||||
|
isinstance(display_path, str)
|
||||||
|
and display_path.lower().endswith(doc_exts)
|
||||||
|
) or (
|
||||||
|
isinstance(processing_path, str)
|
||||||
|
and processing_path.lower().endswith(doc_exts)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Add Open file action(s)
|
||||||
|
def open_file_by_path(path_label: str):
|
||||||
|
from PyQt6.QtWidgets import QMessageBox
|
||||||
|
|
||||||
|
p = display_path if path_label == "display" else processing_path
|
||||||
|
if not p:
|
||||||
|
QMessageBox.warning(
|
||||||
|
self, "File Not Found", "Path is not available."
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
# Find the queue item and resolve the target path
|
||||||
|
target_path = None
|
||||||
|
for q in self.queue:
|
||||||
|
if (
|
||||||
|
getattr(q, "save_base_path", None) == display_path
|
||||||
|
or q.file_name == display_path
|
||||||
|
):
|
||||||
|
if path_label == "display":
|
||||||
|
target_path = (
|
||||||
|
getattr(q, "save_base_path", None) or q.file_name
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
target_path = q.file_name
|
||||||
|
break
|
||||||
|
if (
|
||||||
|
getattr(q, "save_base_path", None) == processing_path
|
||||||
|
or q.file_name == processing_path
|
||||||
|
):
|
||||||
|
if path_label == "display":
|
||||||
|
target_path = (
|
||||||
|
getattr(q, "save_base_path", None) or q.file_name
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
target_path = q.file_name
|
||||||
|
break
|
||||||
|
|
||||||
|
# Fallback to the raw path if resolution failed
|
||||||
|
if not target_path:
|
||||||
|
target_path = p
|
||||||
|
|
||||||
|
if not os.path.exists(target_path):
|
||||||
|
QMessageBox.warning(
|
||||||
|
self, "File Not Found", f"The file does not exist."
|
||||||
|
)
|
||||||
|
return
|
||||||
|
QDesktopServices.openUrl(QUrl.fromLocalFile(target_path))
|
||||||
|
|
||||||
|
if is_document_input:
|
||||||
|
# For documents, show two open options
|
||||||
|
open_processed_action = QAction("Open processed file", self)
|
||||||
|
open_processed_action.triggered.connect(
|
||||||
|
lambda: open_file_by_path("processing")
|
||||||
|
)
|
||||||
|
menu.addAction(open_processed_action)
|
||||||
|
|
||||||
|
open_input_action = QAction("Open input file", self)
|
||||||
|
open_input_action.triggered.connect(
|
||||||
|
lambda: open_file_by_path("display")
|
||||||
|
)
|
||||||
|
menu.addAction(open_input_action)
|
||||||
|
else:
|
||||||
|
# For plain text files, show single open option
|
||||||
|
open_file_action = QAction("Open file", self)
|
||||||
|
open_file_action.triggered.connect(lambda: open_file_by_path("display"))
|
||||||
|
menu.addAction(open_file_action)
|
||||||
|
|
||||||
|
# Add Go to folder action
|
||||||
|
# If the queued item represents a converted document (markdown, pdf, epub)
|
||||||
|
# show two actions: Go to processed file (the cached .txt) and Go to input file (original source)
|
||||||
|
|
||||||
|
from PyQt6.QtWidgets import QMessageBox
|
||||||
|
|
||||||
|
def open_folder_for(path_label: str):
|
||||||
|
# path_label should be either 'display' or 'processing'
|
||||||
|
p = display_path if path_label == "display" else processing_path
|
||||||
|
if not p:
|
||||||
|
QMessageBox.warning(
|
||||||
|
self, "File Not Found", "Path is not available."
|
||||||
|
)
|
||||||
|
return
|
||||||
|
# If the stored path is the display path (original) but the actual file may be
|
||||||
|
# stored on the queue object differently, try to resolve via the queue entry.
|
||||||
|
target_path = None
|
||||||
|
for q in self.queue:
|
||||||
|
if (
|
||||||
|
getattr(q, "save_base_path", None) == display_path
|
||||||
|
or q.file_name == display_path
|
||||||
|
):
|
||||||
|
if path_label == "display":
|
||||||
|
target_path = (
|
||||||
|
getattr(q, "save_base_path", None) or q.file_name
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
target_path = q.file_name
|
||||||
|
break
|
||||||
|
if (
|
||||||
|
getattr(q, "save_base_path", None) == processing_path
|
||||||
|
or q.file_name == processing_path
|
||||||
|
):
|
||||||
|
if path_label == "display":
|
||||||
|
target_path = (
|
||||||
|
getattr(q, "save_base_path", None) or q.file_name
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
target_path = q.file_name
|
||||||
|
break
|
||||||
|
# Fallback to the raw path if resolution failed
|
||||||
|
if not target_path:
|
||||||
|
target_path = p
|
||||||
|
|
||||||
|
if not os.path.exists(target_path):
|
||||||
|
QMessageBox.warning(
|
||||||
|
self,
|
||||||
|
"File Not Found",
|
||||||
|
f"The file does not exist: {target_path}",
|
||||||
|
)
|
||||||
|
return
|
||||||
|
folder = os.path.dirname(target_path)
|
||||||
|
if os.path.exists(folder):
|
||||||
|
QDesktopServices.openUrl(QUrl.fromLocalFile(folder))
|
||||||
|
|
||||||
|
if is_document_input:
|
||||||
|
processed_action = QAction("Go to processed file", self)
|
||||||
|
processed_action.triggered.connect(
|
||||||
|
lambda: open_folder_for("processing")
|
||||||
|
)
|
||||||
|
menu.addAction(processed_action)
|
||||||
|
|
||||||
|
input_action = QAction("Go to input file", self)
|
||||||
|
input_action.triggered.connect(lambda: open_folder_for("display"))
|
||||||
|
menu.addAction(input_action)
|
||||||
|
else:
|
||||||
|
# Default behavior for non-document inputs: single "Go to folder" action
|
||||||
|
go_to_folder_action = QAction("Go to folder", self)
|
||||||
|
|
||||||
|
def go_to_folder():
|
||||||
|
item = selected_items[0]
|
||||||
|
paths = item.data(Qt.ItemDataRole.UserRole)
|
||||||
|
if isinstance(paths, dict):
|
||||||
|
file_path = paths.get(
|
||||||
|
"display_path", paths.get("processing_path", "")
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
file_path = paths # Fallback for old format
|
||||||
|
# Find the queue item
|
||||||
|
for q in self.queue:
|
||||||
|
if (
|
||||||
|
getattr(q, "save_base_path", None) == file_path
|
||||||
|
or q.file_name == file_path
|
||||||
|
):
|
||||||
|
target_path = (
|
||||||
|
getattr(q, "save_base_path", None) or q.file_name
|
||||||
|
)
|
||||||
|
if not os.path.exists(target_path):
|
||||||
|
QMessageBox.warning(
|
||||||
|
self, "File Not Found", f"The file does not exist."
|
||||||
|
)
|
||||||
|
return
|
||||||
|
folder = os.path.dirname(target_path)
|
||||||
|
if os.path.exists(folder):
|
||||||
|
QDesktopServices.openUrl(QUrl.fromLocalFile(folder))
|
||||||
|
break
|
||||||
|
|
||||||
|
go_to_folder_action.triggered.connect(go_to_folder)
|
||||||
|
menu.addAction(go_to_folder_action)
|
||||||
|
|
||||||
|
elif len(selected_items) > 1:
|
||||||
|
remove_action = QAction(f"Remove selected ({len(selected_items)})", self)
|
||||||
|
remove_action.triggered.connect(self.remove_item)
|
||||||
|
menu.addAction(remove_action)
|
||||||
|
# Always add Clear Queue
|
||||||
|
clear_action = QAction("Clear Queue", self)
|
||||||
|
clear_action.triggered.connect(self.clear_queue)
|
||||||
|
menu.addAction(clear_action)
|
||||||
|
menu.exec(global_pos)
|
||||||
|
|
||||||
|
def accept(self):
|
||||||
|
# Save the override state to config so it persists globally
|
||||||
|
self.config["queue_override_settings"] = self.override_chk.isChecked()
|
||||||
|
save_config(self.config)
|
||||||
|
|
||||||
|
super().accept()
|
||||||
|
|
||||||
|
def reject(self):
|
||||||
|
# Cancel: restore original queue
|
||||||
|
from PyQt6.QtWidgets import QMessageBox
|
||||||
|
|
||||||
|
# Warn if user changed a lot (e.g., more than 1 items difference)
|
||||||
|
original_count = len(self._original_queue)
|
||||||
|
current_count = len(self.queue)
|
||||||
|
if abs(original_count - current_count) > 1:
|
||||||
|
reply = QMessageBox.question(
|
||||||
|
self,
|
||||||
|
"Confirm Cancel",
|
||||||
|
f"Are you sure you want to cancel and discard all changes?",
|
||||||
|
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
|
||||||
|
QMessageBox.StandardButton.No,
|
||||||
|
)
|
||||||
|
if reply != QMessageBox.StandardButton.Yes:
|
||||||
|
return
|
||||||
|
self.queue.clear()
|
||||||
|
self.queue.extend(deepcopy(self._original_queue))
|
||||||
|
super().reject()
|
||||||
|
|
||||||
|
def keyPressEvent(self, event):
|
||||||
|
from PyQt6.QtCore import Qt
|
||||||
|
|
||||||
|
if event.key() == Qt.Key.Key_Delete:
|
||||||
|
self.remove_item()
|
||||||
|
else:
|
||||||
|
super().keyPressEvent(event)
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# represents a queued item - book, chapters, voice, etc.
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class QueuedItem:
|
||||||
|
file_name: str
|
||||||
|
lang_code: str
|
||||||
|
speed: float
|
||||||
|
voice: str
|
||||||
|
save_option: str
|
||||||
|
output_folder: str
|
||||||
|
subtitle_mode: str
|
||||||
|
output_format: str
|
||||||
|
total_char_count: int
|
||||||
|
replace_single_newlines: bool = True
|
||||||
|
use_silent_gaps: bool = False
|
||||||
|
subtitle_speed_method: str = "tts"
|
||||||
|
save_base_path: str = None
|
||||||
|
save_chapters_separately: bool = None
|
||||||
|
merge_chapters_at_end: bool = None
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,9 +1,11 @@
|
|||||||
"""Legacy PyQt queue manager GUI removed."""
|
"""Backwards-compatible re-export of the PyQt queue manager.
|
||||||
|
|
||||||
|
The actual implementation lives in abogen.pyqt.queue_manager_gui.
|
||||||
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from abogen.pyqt.queue_manager_gui import * # noqa: F401, F403
|
||||||
|
from abogen.pyqt.queue_manager_gui import QueueManager
|
||||||
|
|
||||||
def __getattr__(name: str): # pragma: no cover - compatibility shim
|
__all__ = ["QueueManager"]
|
||||||
raise AttributeError(
|
|
||||||
"The PyQt queue manager GUI has been removed. Use the web dashboard instead."
|
|
||||||
)
|
|
||||||
|
|||||||
@@ -0,0 +1,161 @@
|
|||||||
|
"""
|
||||||
|
Lazy-loaded spaCy utilities for sentence segmentation.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Cached spaCy module and models (lazy loaded)
|
||||||
|
_spacy = None
|
||||||
|
_nlp_cache = {}
|
||||||
|
|
||||||
|
# Language code to spaCy model mapping
|
||||||
|
SPACY_MODELS = {
|
||||||
|
"a": "en_core_web_sm", # American English
|
||||||
|
"b": "en_core_web_sm", # British English
|
||||||
|
"e": "es_core_news_sm", # Spanish
|
||||||
|
"f": "fr_core_news_sm", # French
|
||||||
|
"i": "it_core_news_sm", # Italian
|
||||||
|
"p": "pt_core_news_sm", # Brazilian Portuguese
|
||||||
|
"z": "zh_core_web_sm", # Mandarin Chinese
|
||||||
|
"j": "ja_core_news_sm", # Japanese
|
||||||
|
"h": "xx_sent_ud_sm", # Hindi (multi-language model)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _load_spacy():
|
||||||
|
"""Lazy load spaCy module."""
|
||||||
|
global _spacy
|
||||||
|
if _spacy is None:
|
||||||
|
try:
|
||||||
|
import spacy
|
||||||
|
|
||||||
|
_spacy = spacy
|
||||||
|
except ImportError:
|
||||||
|
return None
|
||||||
|
return _spacy
|
||||||
|
|
||||||
|
|
||||||
|
def get_spacy_model(lang_code, log_callback=None):
|
||||||
|
"""
|
||||||
|
Get or load a spaCy model for the given language code.
|
||||||
|
Downloads the model automatically if not available.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
lang_code: Language code (a, b, e, f, etc.)
|
||||||
|
log_callback: Optional function to log messages
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Loaded spaCy model or None if unavailable
|
||||||
|
"""
|
||||||
|
|
||||||
|
def log(msg, is_error=False):
|
||||||
|
# Prefer GUI log callback when provided to avoid spamming stdout.
|
||||||
|
if log_callback:
|
||||||
|
color = "red" if is_error else "grey"
|
||||||
|
try:
|
||||||
|
log_callback((msg, color))
|
||||||
|
except Exception:
|
||||||
|
# Fallback to printing if callback misbehaves
|
||||||
|
print(msg)
|
||||||
|
else:
|
||||||
|
print(msg)
|
||||||
|
|
||||||
|
# Check if model is cached
|
||||||
|
if lang_code in _nlp_cache:
|
||||||
|
return _nlp_cache[lang_code]
|
||||||
|
|
||||||
|
# Check if language is supported
|
||||||
|
model_name = SPACY_MODELS.get(lang_code)
|
||||||
|
if not model_name:
|
||||||
|
log(f"\nspaCy: No model mapping for language '{lang_code}'...")
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Lazy load spaCy
|
||||||
|
spacy = _load_spacy()
|
||||||
|
if spacy is None:
|
||||||
|
log("\nspaCy: Module not installed, falling back to default segmentation...")
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Try to load the model
|
||||||
|
try:
|
||||||
|
log(f"\nLoading spaCy model '{model_name}'...")
|
||||||
|
# sentence segmentation involving parentheses, quotes, and complex structure.
|
||||||
|
# We only disable heavier components we don't need like NER.
|
||||||
|
nlp = spacy.load(
|
||||||
|
model_name,
|
||||||
|
disable=["ner", "tagger", "lemmatizer", "attribute_ruler"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# Ensure a sentence segmentation strategy is in place
|
||||||
|
# The parser provides sents, but if it's missing (unlikely for core models), fallback to sentencizer
|
||||||
|
if "parser" not in nlp.pipe_names and "sentencizer" not in nlp.pipe_names:
|
||||||
|
nlp.add_pipe("sentencizer")
|
||||||
|
|
||||||
|
_nlp_cache[lang_code] = nlp
|
||||||
|
return nlp
|
||||||
|
except OSError:
|
||||||
|
# Model not found, attempt download
|
||||||
|
log(f"\nspaCy: Downloading model '{model_name}'...")
|
||||||
|
try:
|
||||||
|
from spacy.cli import download
|
||||||
|
|
||||||
|
download(model_name)
|
||||||
|
# Retry loading with the same fix
|
||||||
|
nlp = spacy.load(
|
||||||
|
model_name,
|
||||||
|
disable=["ner", "tagger", "lemmatizer", "attribute_ruler"],
|
||||||
|
)
|
||||||
|
if "parser" not in nlp.pipe_names and "sentencizer" not in nlp.pipe_names:
|
||||||
|
nlp.add_pipe("sentencizer")
|
||||||
|
|
||||||
|
_nlp_cache[lang_code] = nlp
|
||||||
|
log(f"spaCy model '{model_name}' downloaded and loaded")
|
||||||
|
return nlp
|
||||||
|
except Exception as e:
|
||||||
|
log(
|
||||||
|
f"\nspaCy: Failed to download model '{model_name}': {e}...",
|
||||||
|
is_error=True,
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
except Exception as e:
|
||||||
|
log(f"\nspaCy: Error loading model '{model_name}': {e}...", is_error=True)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def segment_sentences(text, lang_code, log_callback=None):
|
||||||
|
"""
|
||||||
|
Segment text into sentences using spaCy.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Text to segment
|
||||||
|
lang_code: Language code
|
||||||
|
log_callback: Optional function to log messages
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of sentence strings, or None if spaCy unavailable
|
||||||
|
"""
|
||||||
|
nlp = get_spacy_model(lang_code, log_callback)
|
||||||
|
if nlp is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Ensure spaCy can handle large texts by adjusting max_length if necessary
|
||||||
|
try:
|
||||||
|
text_len = len(text or "")
|
||||||
|
if text_len and hasattr(nlp, "max_length") and text_len > nlp.max_length:
|
||||||
|
# increase a bit beyond the text length to be safe
|
||||||
|
nlp.max_length = text_len + 1000
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Process text and extract sentences
|
||||||
|
doc = nlp(text)
|
||||||
|
return [sent.text.strip() for sent in doc.sents if sent.text.strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def is_spacy_available():
|
||||||
|
"""Check if spaCy can be imported."""
|
||||||
|
return _load_spacy() is not None
|
||||||
|
|
||||||
|
|
||||||
|
def clear_cache():
|
||||||
|
"""Clear the model cache to free memory."""
|
||||||
|
global _nlp_cache
|
||||||
|
_nlp_cache.clear()
|
||||||
@@ -0,0 +1,459 @@
|
|||||||
|
import re
|
||||||
|
import platform
|
||||||
|
from abogen.utils import detect_encoding, load_config
|
||||||
|
from abogen.constants import SAMPLE_VOICE_TEXTS
|
||||||
|
|
||||||
|
# Pre-compile frequently used regex patterns for better performance
|
||||||
|
_METADATA_TAG_PATTERN = re.compile(r"<<METADATA_[^:]+:[^>]*>>")
|
||||||
|
_WHITESPACE_PATTERN = re.compile(r"[^\S\n]+")
|
||||||
|
_MULTIPLE_NEWLINES_PATTERN = re.compile(r"\n{3,}")
|
||||||
|
_SINGLE_NEWLINE_PATTERN = re.compile(r"(?<!\n)\n(?!\n)")
|
||||||
|
_CHAPTER_MARKER_PATTERN = re.compile(r"<<CHAPTER_MARKER:[^>]*>>")
|
||||||
|
_HTML_TAG_PATTERN = re.compile(r"<[^>]+>")
|
||||||
|
_VOICE_TAG_PATTERN = re.compile(r"{[^}]+}")
|
||||||
|
_ASS_STYLING_PATTERN = re.compile(r"\{[^}]+\}")
|
||||||
|
_ASS_NEWLINE_N_PATTERN = re.compile(r"\\N")
|
||||||
|
_ASS_NEWLINE_LOWER_N_PATTERN = re.compile(r"\\n")
|
||||||
|
_CHAPTER_MARKER_SEARCH_PATTERN = re.compile(r"<<CHAPTER_MARKER:(.*?)>>")
|
||||||
|
_WEBVTT_HEADER_PATTERN = re.compile(r"^WEBVTT.*?\n", re.MULTILINE)
|
||||||
|
_VTT_STYLE_PATTERN = re.compile(r"STYLE\s*\n.*?(?=\n\n|$)", re.DOTALL)
|
||||||
|
_VTT_NOTE_PATTERN = re.compile(r"NOTE\s*\n.*?(?=\n\n|$)", re.DOTALL)
|
||||||
|
_DOUBLE_NEWLINE_SPLIT_PATTERN = re.compile(r"\n\s*\n")
|
||||||
|
_VTT_TIMESTAMP_PATTERN = re.compile(r"([\d:.]+)\s*-->\s*([\d:.]+)")
|
||||||
|
_TIMESTAMP_ONLY_PATTERN = re.compile(r"^(\d{1,2}:\d{2}:\d{2}(?:[.,]\d{1,3})?)$")
|
||||||
|
_WINDOWS_ILLEGAL_CHARS_PATTERN = re.compile(r'[<>:"/\\|?*]')
|
||||||
|
_CONTROL_CHARS_PATTERN = re.compile(r"[\x00-\x1f]")
|
||||||
|
_LINUX_CONTROL_CHARS_PATTERN = re.compile(
|
||||||
|
r"[\x01-\x1f]"
|
||||||
|
) # Linux: exclude \x00 for separate handling
|
||||||
|
_MACOS_ILLEGAL_CHARS_PATTERN = re.compile(r"[:]")
|
||||||
|
_LINUX_ILLEGAL_CHARS_PATTERN = re.compile(r"[/\x00]")
|
||||||
|
|
||||||
|
|
||||||
|
def clean_subtitle_text(text):
|
||||||
|
"""Remove chapter markers and metadata tags from subtitle text."""
|
||||||
|
# Use pre-compiled patterns for better performance
|
||||||
|
text = _METADATA_TAG_PATTERN.sub("", text)
|
||||||
|
text = _CHAPTER_MARKER_PATTERN.sub("", text)
|
||||||
|
return text.strip()
|
||||||
|
|
||||||
|
def calculate_text_length(text):
|
||||||
|
# Use pre-compiled patterns for better performance
|
||||||
|
# Ignore chapter markers and metadata patterns in a single pass
|
||||||
|
text = _CHAPTER_MARKER_PATTERN.sub("", text)
|
||||||
|
text = _METADATA_TAG_PATTERN.sub("", text)
|
||||||
|
# Ignore newlines and leading/trailing spaces
|
||||||
|
text = text.replace("\n", "").strip()
|
||||||
|
# Calculate character count
|
||||||
|
char_count = len(text)
|
||||||
|
return char_count
|
||||||
|
|
||||||
|
def clean_text(text, *args, **kwargs):
|
||||||
|
# Remove metadata tags first
|
||||||
|
text = _METADATA_TAG_PATTERN.sub("", text)
|
||||||
|
# Load replace_single_newlines from config
|
||||||
|
cfg = load_config()
|
||||||
|
replace_single_newlines = cfg.get("replace_single_newlines", True)
|
||||||
|
# Collapse all whitespace (excluding newlines) into single spaces per line and trim edges
|
||||||
|
# Use pre-compiled pattern for better performance
|
||||||
|
lines = [_WHITESPACE_PATTERN.sub(" ", line).strip() for line in text.splitlines()]
|
||||||
|
text = "\n".join(lines)
|
||||||
|
# Standardize paragraph breaks (multiple newlines become exactly two) and trim overall whitespace
|
||||||
|
# Use pre-compiled pattern for better performance
|
||||||
|
text = _MULTIPLE_NEWLINES_PATTERN.sub("\n\n", text).strip()
|
||||||
|
# Optionally replace single newlines with spaces, but preserve double newlines
|
||||||
|
if replace_single_newlines:
|
||||||
|
# Use pre-compiled pattern for better performance
|
||||||
|
text = _SINGLE_NEWLINE_PATTERN.sub(" ", text)
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def parse_srt_file(file_path):
|
||||||
|
"""
|
||||||
|
Parse an SRT subtitle file and return a list of subtitle entries.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
file_path: Path to the SRT file
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of tuples: [(start_time_seconds, end_time_seconds, text), ...]
|
||||||
|
"""
|
||||||
|
encoding = detect_encoding(file_path)
|
||||||
|
with open(file_path, "r", encoding=encoding, errors="replace") as f:
|
||||||
|
content = f.read()
|
||||||
|
|
||||||
|
# Split by double newlines to get individual subtitle blocks
|
||||||
|
blocks = re.split(r"\n\s*\n", content.strip())
|
||||||
|
|
||||||
|
subtitles = []
|
||||||
|
for block in blocks:
|
||||||
|
if not block.strip():
|
||||||
|
continue
|
||||||
|
|
||||||
|
lines = block.strip().split("\n")
|
||||||
|
if len(lines) < 3:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# First line is index, second line is timestamp, rest is text
|
||||||
|
try:
|
||||||
|
timestamp_line = lines[1]
|
||||||
|
match = re.match(
|
||||||
|
r"(\d{2}:\d{2}:\d{2},\d{3})\s*-->\s*(\d{2}:\d{2}:\d{2},\d{3})",
|
||||||
|
timestamp_line,
|
||||||
|
)
|
||||||
|
if not match:
|
||||||
|
continue
|
||||||
|
|
||||||
|
start_str = match.group(1)
|
||||||
|
end_str = match.group(2)
|
||||||
|
text = "\n".join(lines[2:])
|
||||||
|
|
||||||
|
# Convert timestamp to seconds
|
||||||
|
def time_to_seconds(t):
|
||||||
|
h, m, s_ms = t.split(":")
|
||||||
|
s, ms = s_ms.split(",")
|
||||||
|
return int(h) * 3600 + int(m) * 60 + int(s) + int(ms) / 1000.0
|
||||||
|
|
||||||
|
start_sec = time_to_seconds(start_str)
|
||||||
|
end_sec = time_to_seconds(end_str)
|
||||||
|
|
||||||
|
# Clean text of any styling tags using pre-compiled pattern
|
||||||
|
text = _HTML_TAG_PATTERN.sub("", text)
|
||||||
|
# Remove chapter markers and metadata tags
|
||||||
|
text = clean_subtitle_text(text)
|
||||||
|
|
||||||
|
if text: # Only add non-empty subtitles
|
||||||
|
subtitles.append((start_sec, end_sec, text))
|
||||||
|
except (ValueError, IndexError):
|
||||||
|
continue
|
||||||
|
|
||||||
|
return subtitles
|
||||||
|
|
||||||
|
|
||||||
|
def parse_vtt_file(file_path):
|
||||||
|
"""
|
||||||
|
Parse a VTT (WebVTT) subtitle file and return a list of subtitle entries.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
file_path: Path to the VTT file
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of tuples: [(start_time_seconds, end_time_seconds, text), ...]
|
||||||
|
"""
|
||||||
|
encoding = detect_encoding(file_path)
|
||||||
|
with open(file_path, "r", encoding=encoding, errors="replace") as f:
|
||||||
|
content = f.read()
|
||||||
|
|
||||||
|
# Remove WEBVTT header and any style/note blocks using pre-compiled patterns
|
||||||
|
content = _WEBVTT_HEADER_PATTERN.sub("", content)
|
||||||
|
content = _VTT_STYLE_PATTERN.sub("", content)
|
||||||
|
content = _VTT_NOTE_PATTERN.sub("", content)
|
||||||
|
|
||||||
|
# Split by double newlines to get individual subtitle blocks using pre-compiled pattern
|
||||||
|
blocks = _DOUBLE_NEWLINE_SPLIT_PATTERN.split(content.strip())
|
||||||
|
|
||||||
|
subtitles = []
|
||||||
|
for block in blocks:
|
||||||
|
if not block.strip():
|
||||||
|
continue
|
||||||
|
|
||||||
|
lines = block.strip().split("\n")
|
||||||
|
if len(lines) < 2:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# VTT can have optional identifier on first line, timestamp on second or first
|
||||||
|
timestamp_line = None
|
||||||
|
text_start_idx = 0
|
||||||
|
|
||||||
|
# Check if first line is timestamp
|
||||||
|
if "-->" in lines[0]:
|
||||||
|
timestamp_line = lines[0]
|
||||||
|
text_start_idx = 1
|
||||||
|
elif len(lines) > 1 and "-->" in lines[1]:
|
||||||
|
timestamp_line = lines[1]
|
||||||
|
text_start_idx = 2
|
||||||
|
else:
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
# VTT format: 00:00:00.000 --> 00:00:05.000 or 00:00.000 --> 00:05.000
|
||||||
|
# Use pre-compiled pattern
|
||||||
|
match = _VTT_TIMESTAMP_PATTERN.match(timestamp_line)
|
||||||
|
if not match:
|
||||||
|
continue
|
||||||
|
|
||||||
|
start_str = match.group(1)
|
||||||
|
end_str = match.group(2)
|
||||||
|
text = "\n".join(lines[text_start_idx:])
|
||||||
|
|
||||||
|
# Convert timestamp to seconds
|
||||||
|
def time_to_seconds(t):
|
||||||
|
parts = t.split(":")
|
||||||
|
if len(parts) == 3: # HH:MM:SS.mmm
|
||||||
|
h, m, s = parts
|
||||||
|
s, ms = s.split(".")
|
||||||
|
return int(h) * 3600 + int(m) * 60 + int(s) + int(ms) / 1000.0
|
||||||
|
elif len(parts) == 2: # MM:SS.mmm
|
||||||
|
m, s = parts
|
||||||
|
s, ms = s.split(".")
|
||||||
|
return int(m) * 60 + int(s) + int(ms) / 1000.0
|
||||||
|
return 0
|
||||||
|
|
||||||
|
start_sec = time_to_seconds(start_str)
|
||||||
|
end_sec = time_to_seconds(end_str)
|
||||||
|
|
||||||
|
# Clean text of any styling tags and cue settings using pre-compiled patterns
|
||||||
|
text = _HTML_TAG_PATTERN.sub("", text)
|
||||||
|
text = _VOICE_TAG_PATTERN.sub("", text) # Remove voice tags
|
||||||
|
# Remove chapter markers and metadata tags
|
||||||
|
text = clean_subtitle_text(text)
|
||||||
|
|
||||||
|
if text: # Only add non-empty subtitles
|
||||||
|
subtitles.append((start_sec, end_sec, text))
|
||||||
|
except (ValueError, IndexError, AttributeError):
|
||||||
|
continue
|
||||||
|
|
||||||
|
return subtitles
|
||||||
|
|
||||||
|
|
||||||
|
def detect_timestamps_in_text(file_path):
|
||||||
|
"""Detect if text file contains timestamp markers (HH:MM:SS or HH:MM:SS,ms format) on separate lines."""
|
||||||
|
try:
|
||||||
|
encoding = detect_encoding(file_path)
|
||||||
|
with open(file_path, "r", encoding=encoding, errors="replace") as f:
|
||||||
|
lines = [
|
||||||
|
line.strip() for line in f.readlines()[:50] if line.strip()
|
||||||
|
] # Check first 50 non-empty lines
|
||||||
|
|
||||||
|
# Count lines that are ONLY timestamps (no other text)
|
||||||
|
# Supports HH:MM:SS or HH:MM:SS,ms format
|
||||||
|
# Use pre-compiled pattern for better performance
|
||||||
|
timestamp_lines = sum(
|
||||||
|
1 for line in lines if _TIMESTAMP_ONLY_PATTERN.match(line)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Must have at least 2 timestamp-only lines and they should be >5% of total lines
|
||||||
|
return timestamp_lines >= 2 and (timestamp_lines / max(len(lines), 1)) > 0.05
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def parse_timestamp_text_file(file_path):
|
||||||
|
"""Parse text file with timestamps. Returns list of (start_time, end_time, text) tuples.
|
||||||
|
Supports HH:MM:SS or HH:MM:SS,ms format. Returns time in seconds as float."""
|
||||||
|
encoding = detect_encoding(file_path)
|
||||||
|
with open(file_path, "r", encoding=encoding, errors="replace") as f:
|
||||||
|
content = f.read()
|
||||||
|
|
||||||
|
# Split by timestamp pattern (supports HH:MM:SS or HH:MM:SS,ms)
|
||||||
|
pattern = r"^(\d{1,2}:\d{2}:\d{2}(?:[.,]\d{1,3})?)$"
|
||||||
|
lines = content.split("\n")
|
||||||
|
|
||||||
|
def parse_time(time_str):
|
||||||
|
"""Convert HH:MM:SS or HH:MM:SS,ms to seconds as float."""
|
||||||
|
time_str = time_str.replace(",", ".")
|
||||||
|
parts = time_str.split(":")
|
||||||
|
return float(int(parts[0]) * 3600 + int(parts[1]) * 60 + float(parts[2]))
|
||||||
|
|
||||||
|
entries = []
|
||||||
|
current_time = None
|
||||||
|
current_text = []
|
||||||
|
pre_timestamp_text = [] # Text before first timestamp
|
||||||
|
|
||||||
|
for line in lines:
|
||||||
|
match = re.match(pattern, line.strip())
|
||||||
|
if match:
|
||||||
|
# Save previous entry
|
||||||
|
if current_time is not None and current_text:
|
||||||
|
text = "\n".join(current_text).strip()
|
||||||
|
if text:
|
||||||
|
entries.append((current_time, text))
|
||||||
|
elif current_time is None and pre_timestamp_text:
|
||||||
|
# First timestamp found, save pre-timestamp text with time 0
|
||||||
|
text = "\n".join(pre_timestamp_text).strip()
|
||||||
|
if text:
|
||||||
|
entries.append((0.0, text))
|
||||||
|
pre_timestamp_text = []
|
||||||
|
|
||||||
|
# Start new entry
|
||||||
|
time_str = match.group(1)
|
||||||
|
current_time = parse_time(time_str)
|
||||||
|
current_text = []
|
||||||
|
elif current_time is not None:
|
||||||
|
current_text.append(line)
|
||||||
|
else:
|
||||||
|
# Text before first timestamp
|
||||||
|
pre_timestamp_text.append(line)
|
||||||
|
|
||||||
|
# Save last entry
|
||||||
|
if current_time is not None and current_text:
|
||||||
|
text = "\n".join(current_text).strip()
|
||||||
|
if text:
|
||||||
|
entries.append((current_time, text))
|
||||||
|
elif not entries and pre_timestamp_text:
|
||||||
|
# No timestamps found at all, treat entire file as starting at 0
|
||||||
|
text = "\n".join(pre_timestamp_text).strip()
|
||||||
|
if text:
|
||||||
|
entries.append((0.0, text))
|
||||||
|
|
||||||
|
# Convert to subtitle format with end times
|
||||||
|
subtitles = []
|
||||||
|
for i, (start_time, text) in enumerate(entries):
|
||||||
|
end_time = entries[i + 1][0] if i + 1 < len(entries) else None
|
||||||
|
# Remove chapter markers and metadata tags
|
||||||
|
text = clean_subtitle_text(text)
|
||||||
|
if text: # Only add non-empty entries
|
||||||
|
subtitles.append((start_time, end_time, text))
|
||||||
|
|
||||||
|
return subtitles
|
||||||
|
|
||||||
|
|
||||||
|
def parse_ass_file(file_path):
|
||||||
|
"""
|
||||||
|
Parse an ASS/SSA subtitle file and return a list of subtitle entries.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
file_path: Path to the ASS/SSA file
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of tuples: [(start_time_seconds, end_time_seconds, text), ...]
|
||||||
|
"""
|
||||||
|
encoding = detect_encoding(file_path)
|
||||||
|
with open(file_path, "r", encoding=encoding, errors="replace") as f:
|
||||||
|
lines = f.readlines()
|
||||||
|
|
||||||
|
subtitles = []
|
||||||
|
in_events = False
|
||||||
|
format_indices = {}
|
||||||
|
|
||||||
|
for line in lines:
|
||||||
|
line = line.strip()
|
||||||
|
|
||||||
|
if line.startswith("[Events]"):
|
||||||
|
in_events = True
|
||||||
|
continue
|
||||||
|
|
||||||
|
if line.startswith("[") and in_events:
|
||||||
|
# New section, stop processing
|
||||||
|
break
|
||||||
|
|
||||||
|
if in_events and line.startswith("Format:"):
|
||||||
|
# Parse format line to know column positions
|
||||||
|
parts = line.split(":", 1)[1].strip().split(",")
|
||||||
|
for i, part in enumerate(parts):
|
||||||
|
format_indices[part.strip().lower()] = i
|
||||||
|
continue
|
||||||
|
|
||||||
|
if in_events and (line.startswith("Dialogue:") or line.startswith("Comment:")):
|
||||||
|
if line.startswith("Comment:"):
|
||||||
|
continue # Skip comments
|
||||||
|
|
||||||
|
parts = line.split(":", 1)[1].strip().split(",", len(format_indices) - 1)
|
||||||
|
|
||||||
|
if (
|
||||||
|
"start" in format_indices
|
||||||
|
and "end" in format_indices
|
||||||
|
and "text" in format_indices
|
||||||
|
):
|
||||||
|
start_str = parts[format_indices["start"]].strip()
|
||||||
|
end_str = parts[format_indices["end"]].strip()
|
||||||
|
text = parts[format_indices["text"]].strip()
|
||||||
|
|
||||||
|
# Convert timestamp to seconds (ASS format: H:MM:SS.CS where CS is centiseconds)
|
||||||
|
def ass_time_to_seconds(t):
|
||||||
|
parts = t.split(":")
|
||||||
|
if len(parts) == 3:
|
||||||
|
h, m, s = parts
|
||||||
|
s_parts = s.split(".")
|
||||||
|
seconds = float(s_parts[0])
|
||||||
|
centiseconds = float(s_parts[1]) if len(s_parts) > 1 else 0
|
||||||
|
return (
|
||||||
|
int(h) * 3600 + int(m) * 60 + seconds + centiseconds / 100.0
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
start_sec = ass_time_to_seconds(start_str)
|
||||||
|
end_sec = ass_time_to_seconds(end_str)
|
||||||
|
|
||||||
|
# Clean text of ASS styling tags using pre-compiled patterns
|
||||||
|
text = _ASS_STYLING_PATTERN.sub("", text) # Remove {tags}
|
||||||
|
text = _ASS_NEWLINE_N_PATTERN.sub("\n", text) # Convert \N to newline
|
||||||
|
text = _ASS_NEWLINE_LOWER_N_PATTERN.sub(
|
||||||
|
"\n", text
|
||||||
|
) # Convert \n to newline
|
||||||
|
# Remove chapter markers and metadata tags
|
||||||
|
text = clean_subtitle_text(text)
|
||||||
|
|
||||||
|
if text: # Only add non-empty subtitles
|
||||||
|
subtitles.append((start_sec, end_sec, text))
|
||||||
|
|
||||||
|
return subtitles
|
||||||
|
|
||||||
|
|
||||||
|
def get_sample_voice_text(lang_code):
|
||||||
|
return SAMPLE_VOICE_TEXTS.get(lang_code, SAMPLE_VOICE_TEXTS["a"])
|
||||||
|
|
||||||
|
|
||||||
|
def sanitize_name_for_os(name, is_folder=True):
|
||||||
|
"""
|
||||||
|
Sanitize a filename or folder name based on the operating system.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
name: The name to sanitize
|
||||||
|
is_folder: Whether this is a folder name (default: True)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Sanitized name safe for the current OS
|
||||||
|
"""
|
||||||
|
if not name:
|
||||||
|
return "audiobook"
|
||||||
|
|
||||||
|
system = platform.system()
|
||||||
|
|
||||||
|
if system == "Windows":
|
||||||
|
# Windows illegal characters: < > : " / \ | ? *
|
||||||
|
# Also can't end with space or dot
|
||||||
|
# Use pre-compiled pattern for better performance
|
||||||
|
sanitized = _WINDOWS_ILLEGAL_CHARS_PATTERN.sub("_", name)
|
||||||
|
# Remove control characters (0-31)
|
||||||
|
sanitized = _CONTROL_CHARS_PATTERN.sub("_", sanitized)
|
||||||
|
# Remove trailing spaces and dots
|
||||||
|
sanitized = sanitized.rstrip(". ")
|
||||||
|
# Windows reserved names (CON, PRN, AUX, NUL, COM1-9, LPT1-9)
|
||||||
|
reserved = (
|
||||||
|
["CON", "PRN", "AUX", "NUL"]
|
||||||
|
+ [f"COM{i}" for i in range(1, 10)]
|
||||||
|
+ [f"LPT{i}" for i in range(1, 10)]
|
||||||
|
)
|
||||||
|
if sanitized.upper() in reserved or sanitized.upper().split(".")[0] in reserved:
|
||||||
|
sanitized = f"_{sanitized}"
|
||||||
|
elif system == "Darwin": # macOS
|
||||||
|
# macOS illegal characters: : (colon is converted to / by the system)
|
||||||
|
# Also can't start with dot (hidden file) for folders typically
|
||||||
|
# Use pre-compiled pattern for better performance
|
||||||
|
sanitized = _MACOS_ILLEGAL_CHARS_PATTERN.sub("_", name)
|
||||||
|
# Remove control characters
|
||||||
|
sanitized = _CONTROL_CHARS_PATTERN.sub("_", sanitized)
|
||||||
|
# Avoid leading dot for folders (creates hidden folders)
|
||||||
|
if is_folder and sanitized.startswith("."):
|
||||||
|
sanitized = "_" + sanitized[1:]
|
||||||
|
else: # Linux and others
|
||||||
|
# Linux illegal characters: / and null character
|
||||||
|
# Though / is illegal, most other chars are technically allowed
|
||||||
|
# Use pre-compiled pattern for better performance
|
||||||
|
sanitized = _LINUX_ILLEGAL_CHARS_PATTERN.sub("_", name)
|
||||||
|
# Remove other control characters for safety (excluding \x00 which is already handled)
|
||||||
|
sanitized = _LINUX_CONTROL_CHARS_PATTERN.sub("_", sanitized)
|
||||||
|
# Avoid leading dot for folders (creates hidden folders)
|
||||||
|
if is_folder and sanitized.startswith("."):
|
||||||
|
sanitized = "_" + sanitized[1:]
|
||||||
|
|
||||||
|
# Ensure the name is not empty after sanitization
|
||||||
|
if not sanitized or sanitized.strip() == "":
|
||||||
|
sanitized = "audiobook"
|
||||||
|
|
||||||
|
# Limit length to 255 characters (common limit across filesystems)
|
||||||
|
if len(sanitized) > 255:
|
||||||
|
sanitized = sanitized[:255].rstrip(". ")
|
||||||
|
|
||||||
|
return sanitized
|
||||||
@@ -1,15 +1,11 @@
|
|||||||
"""Legacy PyQt voice formula dialog removed."""
|
"""Backwards-compatible re-export of the PyQt voice formula dialog.
|
||||||
|
|
||||||
|
The actual implementation lives in abogen.pyqt.voice_formula_gui.
|
||||||
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from abogen.pyqt.voice_formula_gui import * # noqa: F401, F403
|
||||||
class VoiceFormulaDialog: # pragma: no cover - legacy entry point
|
from abogen.pyqt.voice_formula_gui import VoiceFormulaDialog
|
||||||
"""Placeholder for removed PyQt dialog."""
|
|
||||||
|
|
||||||
def __init__(self, *_args, **_kwargs):
|
|
||||||
raise RuntimeError(
|
|
||||||
"The PyQt-based voice formula editor has been removed. Use the web tools instead."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["VoiceFormulaDialog"]
|
__all__ = ["VoiceFormulaDialog"]
|
||||||
|
|||||||
@@ -0,0 +1,90 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Build PyPI package (wheel and sdist) to `dist` folder for abogen."""
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import tempfile
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
script_dir = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
output_dir = os.path.join(script_dir, "dist")
|
||||||
|
|
||||||
|
print("🔧 abogen PyPI Package Builder")
|
||||||
|
print("=" * 40)
|
||||||
|
print(f"📁 Script directory: {script_dir}")
|
||||||
|
print(f"📦 Output directory: {output_dir}")
|
||||||
|
|
||||||
|
# Try to print package version if present
|
||||||
|
version = None
|
||||||
|
version_file = os.path.join(script_dir, "abogen", "VERSION")
|
||||||
|
if os.path.isfile(version_file):
|
||||||
|
try:
|
||||||
|
with open(version_file, "r", encoding="utf-8") as vf:
|
||||||
|
version = vf.read().strip()
|
||||||
|
except Exception:
|
||||||
|
version = None
|
||||||
|
if version:
|
||||||
|
print(f"🔖 Package version: {version}")
|
||||||
|
|
||||||
|
# Check if build module is installed, install if not
|
||||||
|
# Temporarily remove script_dir from sys.path to avoid importing local build.py
|
||||||
|
import sys
|
||||||
|
original_path = sys.path[:]
|
||||||
|
try:
|
||||||
|
sys.path = [p for p in sys.path if os.path.abspath(p) != script_dir]
|
||||||
|
import build
|
||||||
|
except ImportError:
|
||||||
|
print("📦 Installing build module...")
|
||||||
|
subprocess.run([sys.executable, "-m", "pip", "install", "build"], check=True)
|
||||||
|
finally:
|
||||||
|
sys.path = original_path
|
||||||
|
|
||||||
|
# Create output directory
|
||||||
|
print(f"📂 Preparing output directory: {output_dir}")
|
||||||
|
if os.path.exists(output_dir):
|
||||||
|
shutil.rmtree(output_dir)
|
||||||
|
os.makedirs(output_dir, exist_ok=True)
|
||||||
|
|
||||||
|
print("🏗️ Building PyPI package...")
|
||||||
|
print(" Using temporary directory to avoid module conflicts...")
|
||||||
|
|
||||||
|
# Run from temp directory to avoid local build.py shadowing the build module
|
||||||
|
with tempfile.TemporaryDirectory() as tmpdir:
|
||||||
|
print(f" Temp directory: {tmpdir}")
|
||||||
|
print(" Running: python -m build -o <output_dir> <source_dir>")
|
||||||
|
|
||||||
|
result = subprocess.run(
|
||||||
|
[sys.executable, "-m", "build", "-o", output_dir, script_dir],
|
||||||
|
check=False,
|
||||||
|
cwd=tmpdir,
|
||||||
|
)
|
||||||
|
|
||||||
|
print("\n" + "=" * 40)
|
||||||
|
if result.returncode == 0:
|
||||||
|
print("✅ Build successful!")
|
||||||
|
print(f"📦 Files created in {output_dir}:")
|
||||||
|
|
||||||
|
files = os.listdir(output_dir)
|
||||||
|
if files:
|
||||||
|
for f in files:
|
||||||
|
file_path = os.path.join(output_dir, f)
|
||||||
|
size = os.path.getsize(file_path)
|
||||||
|
print(f" 📄 {f} ({size:,} bytes)")
|
||||||
|
else:
|
||||||
|
print(" (No files found)")
|
||||||
|
|
||||||
|
print("\n🚀 Ready for upload with:\n")
|
||||||
|
print(" - To test on Test PyPI:")
|
||||||
|
print(f" python -m twine upload --repository testpypi {output_dir}/*")
|
||||||
|
print("\n - To upload to PyPI (when ready):")
|
||||||
|
print(f" python -m twine upload {output_dir}/*")
|
||||||
|
else:
|
||||||
|
print("❌ Build failed!")
|
||||||
|
print(f" Exit code: {result.returncode}")
|
||||||
|
sys.exit(result.returncode)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
+4
-2
@@ -34,7 +34,8 @@ dependencies = [
|
|||||||
"numpy>=1.24.0",
|
"numpy>=1.24.0",
|
||||||
"gpustat>=1.1.1",
|
"gpustat>=1.1.1",
|
||||||
"num2words>=0.5.13",
|
"num2words>=0.5.13",
|
||||||
"httpx>=0.27.0"
|
"httpx>=0.27.0",
|
||||||
|
"PyQt6>=6.5.0"
|
||||||
]
|
]
|
||||||
|
|
||||||
classifiers = [
|
classifiers = [
|
||||||
@@ -62,11 +63,12 @@ allow-direct-references = true
|
|||||||
|
|
||||||
|
|
||||||
[project.gui-scripts]
|
[project.gui-scripts]
|
||||||
abogen = "abogen.webui.app:main"
|
abogen = "abogen.pyqt.main:main"
|
||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
abogen-cli = "abogen.webui.app:main"
|
abogen-cli = "abogen.webui.app:main"
|
||||||
abogen-web = "abogen.webui.app:main"
|
abogen-web = "abogen.webui.app:main"
|
||||||
|
abogen-pyqt = "abogen.pyqt.main:main"
|
||||||
|
|
||||||
[tool.hatch.build.targets.sdist]
|
[tool.hatch.build.targets.sdist]
|
||||||
exclude = [
|
exclude = [
|
||||||
|
|||||||
@@ -0,0 +1,84 @@
|
|||||||
|
import unittest
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import shutil
|
||||||
|
import time
|
||||||
|
from PyQt6.QtWidgets import QApplication
|
||||||
|
|
||||||
|
# Ensure we can import the module
|
||||||
|
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
|
||||||
|
|
||||||
|
from abogen.book_handler import HandlerDialog
|
||||||
|
from ebooklib import epub
|
||||||
|
|
||||||
|
# We need a QApplication instance for QWriter/QDialog
|
||||||
|
app = QApplication(sys.argv)
|
||||||
|
|
||||||
|
class TestBookHandlerRegression(unittest.TestCase):
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.test_dir = "tests/test_data_handler"
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
os.makedirs(self.test_dir)
|
||||||
|
self.sample_epub_path = os.path.join(self.test_dir, "test_book.epub")
|
||||||
|
self._create_sample_epub()
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
HandlerDialog.clear_content_cache()
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
|
||||||
|
def _create_sample_epub(self):
|
||||||
|
book = epub.EpubBook()
|
||||||
|
book.set_identifier("id123456")
|
||||||
|
book.set_title("Sample Book")
|
||||||
|
book.set_language("en")
|
||||||
|
|
||||||
|
c1 = epub.EpubHtml(title="Intro", file_name="intro.xhtml", lang="en")
|
||||||
|
c1.content = "<h1>Introduction</h1><p>Welcome to the book.</p>"
|
||||||
|
book.add_item(c1)
|
||||||
|
book.spine = ["nav", c1]
|
||||||
|
book.add_item(epub.EpubNcx())
|
||||||
|
book.add_item(epub.EpubNav())
|
||||||
|
epub.write_epub(self.sample_epub_path, book)
|
||||||
|
|
||||||
|
def test_handler_initialization(self):
|
||||||
|
"""Test that HandlerDialog processes the book correctly."""
|
||||||
|
# HandlerDialog starts processing in a background thread in __init__
|
||||||
|
# We assume headless environment, so we won't show it.
|
||||||
|
# But we need to wait for the thread to finish.
|
||||||
|
|
||||||
|
dialog = HandlerDialog(self.sample_epub_path)
|
||||||
|
|
||||||
|
# Wait for thread to finish
|
||||||
|
# The dialog emits no signal publicly, but we can check internal state or thread
|
||||||
|
|
||||||
|
start_time = time.time()
|
||||||
|
while time.time() - start_time < 5:
|
||||||
|
# HandlerDialog logic:
|
||||||
|
# _loader_thread.finished connect to _on_load_finished
|
||||||
|
# _on_load_finished populates content_texts and content_lengths
|
||||||
|
|
||||||
|
# We can check if content_texts is populated
|
||||||
|
if dialog.content_texts:
|
||||||
|
break
|
||||||
|
app.processEvents() # Process Qt events to let thread signals propagate
|
||||||
|
time.sleep(0.1)
|
||||||
|
|
||||||
|
self.assertTrue(len(dialog.content_texts) > 0, "HandlerDialog failed to process content in time")
|
||||||
|
|
||||||
|
# Validate content similar to what we expect
|
||||||
|
# intro.xhtml should be there
|
||||||
|
found_intro = False
|
||||||
|
for key, text in dialog.content_texts.items():
|
||||||
|
if "Welcome to the book" in text:
|
||||||
|
found_intro = True
|
||||||
|
break
|
||||||
|
self.assertTrue(found_intro)
|
||||||
|
|
||||||
|
# Cleanup
|
||||||
|
dialog.close()
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,210 @@
|
|||||||
|
import unittest
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import shutil
|
||||||
|
import fitz # PyMuPDF
|
||||||
|
from ebooklib import epub
|
||||||
|
|
||||||
|
# Ensure we can import the module
|
||||||
|
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
|
||||||
|
|
||||||
|
from abogen.book_parser import get_book_parser, PdfParser, EpubParser, MarkdownParser
|
||||||
|
|
||||||
|
class TestBookParser(unittest.TestCase):
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.test_dir = "tests/test_data"
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
os.makedirs(self.test_dir)
|
||||||
|
|
||||||
|
self.sample_pdf_path = os.path.join(self.test_dir, "test_book.pdf")
|
||||||
|
self.sample_epub_path = os.path.join(self.test_dir, "test_book.epub")
|
||||||
|
self.sample_md_path = os.path.join(self.test_dir, "test_book.md")
|
||||||
|
|
||||||
|
self._create_sample_pdf()
|
||||||
|
self._create_sample_epub()
|
||||||
|
self._create_sample_md()
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
|
||||||
|
def _create_sample_pdf(self):
|
||||||
|
doc = fitz.open()
|
||||||
|
|
||||||
|
# Page 1
|
||||||
|
page1 = doc.new_page()
|
||||||
|
page1.insert_text((50, 50), "Page 1 content")
|
||||||
|
# Add pattern to be cleaned
|
||||||
|
page1.insert_text((50, 100), "[12]")
|
||||||
|
page1.insert_text((50, 200), "1") # Page number at bottom
|
||||||
|
|
||||||
|
# Page 2
|
||||||
|
page2 = doc.new_page()
|
||||||
|
page2.insert_text((50, 50), "Page 2 content")
|
||||||
|
|
||||||
|
doc.save(self.sample_pdf_path)
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
def _create_sample_epub(self):
|
||||||
|
book = epub.EpubBook()
|
||||||
|
book.set_identifier("id123456")
|
||||||
|
book.set_title("Sample Book")
|
||||||
|
book.set_language("en")
|
||||||
|
book.add_author("Test Author")
|
||||||
|
|
||||||
|
c1 = epub.EpubHtml(title="Intro", file_name="intro.xhtml", lang="en")
|
||||||
|
c1.content = "<h1>Introduction</h1><p>Welcome to the book.</p>"
|
||||||
|
|
||||||
|
c2 = epub.EpubHtml(title="Chapter 1", file_name="chap1.xhtml", lang="en")
|
||||||
|
c2.content = "<h1>Chapter 1</h1><ol><li>Item One</li><li>Item Two</li></ol>"
|
||||||
|
|
||||||
|
book.add_item(c1)
|
||||||
|
book.add_item(c2)
|
||||||
|
|
||||||
|
# Basic spine and nav
|
||||||
|
book.spine = ["nav", c1, c2]
|
||||||
|
|
||||||
|
# Add NCX and NAV for compatibility
|
||||||
|
book.add_item(epub.EpubNcx())
|
||||||
|
book.add_item(epub.EpubNav())
|
||||||
|
|
||||||
|
epub.write_epub(self.sample_epub_path, book)
|
||||||
|
|
||||||
|
def _create_sample_md(self):
|
||||||
|
content = "# Chapter 1\nSome text.\n# Chapter 2\nMore text."
|
||||||
|
with open(self.sample_md_path, "w") as f:
|
||||||
|
f.write(content)
|
||||||
|
|
||||||
|
def test_factory_returns_correct_class(self):
|
||||||
|
"""Test that get_book_parser returns the correct subclass based on extension."""
|
||||||
|
parser_pdf = get_book_parser(self.sample_pdf_path)
|
||||||
|
self.assertIsInstance(parser_pdf, PdfParser)
|
||||||
|
|
||||||
|
parser_md = get_book_parser(self.sample_md_path)
|
||||||
|
self.assertIsInstance(parser_md, MarkdownParser)
|
||||||
|
|
||||||
|
parser_epub = get_book_parser(self.sample_epub_path)
|
||||||
|
self.assertIsInstance(parser_epub, EpubParser)
|
||||||
|
|
||||||
|
def test_factory_explicit_type(self):
|
||||||
|
"""Test that explicit file type argument overrides extension."""
|
||||||
|
# 1. Copy sample epub to something.pdf
|
||||||
|
wrong_ext_path = os.path.join(self.test_dir, "actually_epub.pdf")
|
||||||
|
shutil.copy(self.sample_epub_path, wrong_ext_path)
|
||||||
|
|
||||||
|
# 2. Open it telling parser it IS epub
|
||||||
|
parser = get_book_parser(wrong_ext_path, file_type="epub")
|
||||||
|
self.assertIsInstance(parser, EpubParser)
|
||||||
|
|
||||||
|
# Should load successfully
|
||||||
|
parser.load()
|
||||||
|
self.assertTrue(parser.book is not None)
|
||||||
|
|
||||||
|
def test_pdf_parser_content(self):
|
||||||
|
"""Test PdfParser content extraction."""
|
||||||
|
parser = get_book_parser(self.sample_pdf_path)
|
||||||
|
parser.process_content()
|
||||||
|
|
||||||
|
self.assertIn("page_1", parser.content_texts)
|
||||||
|
self.assertIn("page_2", parser.content_texts)
|
||||||
|
|
||||||
|
text1 = parser.content_texts["page_1"]
|
||||||
|
self.assertIn("Page 1 content", text1)
|
||||||
|
self.assertNotIn("[12]", text1)
|
||||||
|
|
||||||
|
def test_markdown_parser_content(self):
|
||||||
|
"""Test MarkdownParser splitting logic."""
|
||||||
|
parser = get_book_parser(self.sample_md_path)
|
||||||
|
parser.process_content()
|
||||||
|
|
||||||
|
# Should have Chapter 1 and Chapter 2 keys (actual keys depend on ID generation)
|
||||||
|
# Markdown extensions might slugify IDs: "chapter-1"
|
||||||
|
self.assertIn("chapter-1", parser.content_texts)
|
||||||
|
self.assertIn("chapter-2", parser.content_texts)
|
||||||
|
|
||||||
|
self.assertIn("Some text", parser.content_texts["chapter-1"])
|
||||||
|
|
||||||
|
def test_epub_parser_content(self):
|
||||||
|
"""Test EpubParser processing."""
|
||||||
|
parser = get_book_parser(self.sample_epub_path)
|
||||||
|
parser.process_content()
|
||||||
|
|
||||||
|
self.assertIn("intro.xhtml", parser.content_texts)
|
||||||
|
self.assertIn("chap1.xhtml", parser.content_texts)
|
||||||
|
self.assertIn("Welcome to the book", parser.content_texts["intro.xhtml"])
|
||||||
|
|
||||||
|
def test_epub_metadata_extraction(self):
|
||||||
|
"""Test metadata extraction in EpubParser."""
|
||||||
|
parser = get_book_parser(self.sample_epub_path)
|
||||||
|
# Processing content triggers metadata extraction in current implementation
|
||||||
|
parser.process_content()
|
||||||
|
|
||||||
|
metadata = parser.get_metadata()
|
||||||
|
self.assertEqual(metadata.get("title"), "Sample Book")
|
||||||
|
self.assertEqual(metadata.get("author"), "Test Author")
|
||||||
|
|
||||||
|
def test_ordered_list_handling(self):
|
||||||
|
"""Test <ol> handling in EpubParser."""
|
||||||
|
parser = get_book_parser(self.sample_epub_path)
|
||||||
|
parser.process_content()
|
||||||
|
|
||||||
|
text = parser.content_texts.get("chap1.xhtml", "")
|
||||||
|
self.assertIn("1) Item One", text)
|
||||||
|
self.assertIn("2) Item Two", text)
|
||||||
|
|
||||||
|
def test_find_position_robust_logic(self):
|
||||||
|
"""Unit test for _find_position_robust on EpubParser."""
|
||||||
|
parser = EpubParser(self.sample_epub_path) # Instantiate directly
|
||||||
|
|
||||||
|
html = '<html><body><p>Start</p><h1 id="target">Heading</h1><p>End</p></body></html>'
|
||||||
|
parser.doc_content["dummy.html"] = html
|
||||||
|
|
||||||
|
# Test finding ID
|
||||||
|
pos = parser._find_position_robust("dummy.html", "target")
|
||||||
|
self.assertGreater(pos, 0)
|
||||||
|
self.assertTrue(html[pos:].startswith('<h1 id="target"'))
|
||||||
|
|
||||||
|
# Test missing ID
|
||||||
|
pos_missing = parser._find_position_robust("dummy.html", "missing")
|
||||||
|
self.assertEqual(pos_missing, 0)
|
||||||
|
|
||||||
|
def test_get_chapters(self):
|
||||||
|
"""Test get_chapters returns correct list for different parsers."""
|
||||||
|
# PDF
|
||||||
|
parser_pdf = get_book_parser(self.sample_pdf_path)
|
||||||
|
chapters = parser_pdf.get_chapters()
|
||||||
|
self.assertEqual(len(chapters), 2)
|
||||||
|
self.assertEqual(chapters[0], ("page_1", "Page 1"))
|
||||||
|
|
||||||
|
# MD
|
||||||
|
parser_md = get_book_parser(self.sample_md_path)
|
||||||
|
parser_md.process_content() # Must process to get structure
|
||||||
|
chapters_md = parser_md.get_chapters()
|
||||||
|
# Expecting chapter-1, chapter-2
|
||||||
|
ids = [c[0] for c in chapters_md]
|
||||||
|
self.assertIn("chapter-1", ids)
|
||||||
|
|
||||||
|
def test_get_formatted_text(self):
|
||||||
|
"""Test formatting of full text via BaseBookParser method."""
|
||||||
|
parser = get_book_parser(self.sample_md_path)
|
||||||
|
parser.process_content()
|
||||||
|
text = parser.get_formatted_text()
|
||||||
|
|
||||||
|
self.assertIn("<<CHAPTER_MARKER:Chapter 1>>", text)
|
||||||
|
self.assertIn("Some text", text)
|
||||||
|
|
||||||
|
def test_file_type_property(self):
|
||||||
|
"""Test that file_type property returns correct string for each parser."""
|
||||||
|
pdf_parser = PdfParser(self.sample_pdf_path)
|
||||||
|
self.assertEqual(pdf_parser.file_type, "pdf")
|
||||||
|
|
||||||
|
epub_parser = EpubParser(self.sample_epub_path)
|
||||||
|
self.assertEqual(epub_parser.file_type, "epub")
|
||||||
|
|
||||||
|
md_parser = MarkdownParser(self.sample_md_path)
|
||||||
|
self.assertEqual(md_parser.file_type, "markdown")
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,199 @@
|
|||||||
|
import unittest
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
from ebooklib import epub
|
||||||
|
|
||||||
|
# Ensure import path
|
||||||
|
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
|
||||||
|
|
||||||
|
from abogen.book_parser import get_book_parser
|
||||||
|
|
||||||
|
class TestEpubContentSlicing(unittest.TestCase):
|
||||||
|
"""
|
||||||
|
Tests for the complex content slicing logic in _execute_nav_parsing_logic.
|
||||||
|
This covers scenarios where multiple chapters/sections are contained within
|
||||||
|
a single physical HTML file, separated by anchors (fragments).
|
||||||
|
"""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.test_dir = "tests/test_data_slicing"
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
os.makedirs(self.test_dir)
|
||||||
|
self.epub_path = os.path.join(self.test_dir, "slicing_test.epub")
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
|
||||||
|
def test_single_file_multiple_chapters(self):
|
||||||
|
"""
|
||||||
|
Test splitting one XHTML file into two chapters using an anchor.
|
||||||
|
"""
|
||||||
|
book = epub.EpubBook()
|
||||||
|
book.set_identifier("slice123")
|
||||||
|
book.set_title("Slicing Test Book")
|
||||||
|
|
||||||
|
# Create a single content file with two sections
|
||||||
|
content_html = """
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<h1 id="chap1">Chapter 1</h1>
|
||||||
|
<p>Text for chapter 1.</p>
|
||||||
|
<hr/>
|
||||||
|
<h1 id="chap2">Chapter 2</h1>
|
||||||
|
<p>Text for chapter 2.</p>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
"""
|
||||||
|
c1 = epub.EpubHtml(title="Full Content", file_name="content.xhtml", lang="en")
|
||||||
|
c1.content = content_html
|
||||||
|
book.add_item(c1)
|
||||||
|
|
||||||
|
# Create Nav that points to anchors in the SAME file
|
||||||
|
# We use EpubHtml for Nav to control content exactly without ebooklib interference
|
||||||
|
nav_html = """
|
||||||
|
<nav epub:type="toc" id="toc">
|
||||||
|
<ol>
|
||||||
|
<li><a href="content.xhtml#chap1">Chapter 1</a></li>
|
||||||
|
<li><a href="content.xhtml#chap2">Chapter 2</a></li>
|
||||||
|
</ol>
|
||||||
|
</nav>
|
||||||
|
"""
|
||||||
|
nav = epub.EpubHtml(title="Nav", file_name="nav.xhtml")
|
||||||
|
nav.content = nav_html
|
||||||
|
book.add_item(nav)
|
||||||
|
|
||||||
|
book.spine = [nav, c1]
|
||||||
|
|
||||||
|
epub.write_epub(self.epub_path, book)
|
||||||
|
|
||||||
|
# OPF Patching to valid crash
|
||||||
|
import zipfile
|
||||||
|
patched = False
|
||||||
|
with zipfile.ZipFile(self.epub_path, 'r') as zin:
|
||||||
|
opf_content = zin.read('EPUB/content.opf').decode('utf-8')
|
||||||
|
if 'toc="ncx"' in opf_content:
|
||||||
|
opf_content = opf_content.replace('toc="ncx"', '')
|
||||||
|
patched = True
|
||||||
|
|
||||||
|
if patched:
|
||||||
|
TEMP_EPUB = self.epub_path + ".temp"
|
||||||
|
with zipfile.ZipFile(TEMP_EPUB, 'w') as zout:
|
||||||
|
for item in zin.infolist():
|
||||||
|
if item.filename == 'EPUB/content.opf':
|
||||||
|
zout.writestr(item, opf_content)
|
||||||
|
else:
|
||||||
|
zout.writestr(item, zin.read(item.filename))
|
||||||
|
|
||||||
|
if patched:
|
||||||
|
shutil.move(TEMP_EPUB, self.epub_path)
|
||||||
|
|
||||||
|
# Parse
|
||||||
|
parser = get_book_parser(self.epub_path)
|
||||||
|
parser.process_content()
|
||||||
|
chapters = parser.get_chapters()
|
||||||
|
|
||||||
|
# Filter Nav/Intro
|
||||||
|
chapters = [c for c in chapters if "Chapter" in c[1]]
|
||||||
|
|
||||||
|
self.assertEqual(len(chapters), 2)
|
||||||
|
self.assertEqual(chapters[0][1], "Chapter 1")
|
||||||
|
self.assertEqual(chapters[1][1], "Chapter 2")
|
||||||
|
|
||||||
|
# Check content of Chapter 1
|
||||||
|
# It should contain "Text for chapter 1" but NOT "Text for chapter 2"
|
||||||
|
# The parser logic slices from start_pos to next_pos
|
||||||
|
text1 = parser.content_texts[chapters[0][0]]
|
||||||
|
self.assertIn("Text for chapter 1", text1)
|
||||||
|
self.assertNotIn("Text for chapter 2", text1)
|
||||||
|
|
||||||
|
# Check content of Chapter 2
|
||||||
|
text2 = parser.content_texts[chapters[1][0]]
|
||||||
|
self.assertIn("Text for chapter 2", text2)
|
||||||
|
|
||||||
|
def test_list_renumbering(self):
|
||||||
|
"""
|
||||||
|
Test that ordered lists are re-numbered when slicing.
|
||||||
|
The parser has logic to reset <ol start="..."> or insert numbers.
|
||||||
|
"""
|
||||||
|
book = epub.EpubBook()
|
||||||
|
book.set_identifier("list123")
|
||||||
|
book.set_title("List Test Book")
|
||||||
|
|
||||||
|
content_html = """
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<h1 id="part1">Part 1</h1>
|
||||||
|
<ol>
|
||||||
|
<li>Item A</li>
|
||||||
|
<li>Item B</li>
|
||||||
|
</ol>
|
||||||
|
<h1 id="part2">Part 2</h1>
|
||||||
|
<ol start="3">
|
||||||
|
<li>Item C</li>
|
||||||
|
<li>Item D</li>
|
||||||
|
</ol>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
"""
|
||||||
|
c1 = epub.EpubHtml(title="Content", file_name="content.xhtml", lang="en")
|
||||||
|
c1.content = content_html
|
||||||
|
book.add_item(c1)
|
||||||
|
|
||||||
|
nav_html = """
|
||||||
|
<nav epub:type="toc">
|
||||||
|
<ol>
|
||||||
|
<li><a href="content.xhtml#part1">Part 1</a></li>
|
||||||
|
<li><a href="content.xhtml#part2">Part 2</a></li>
|
||||||
|
</ol>
|
||||||
|
</nav>
|
||||||
|
"""
|
||||||
|
nav = epub.EpubHtml(title="Nav", file_name="nav.xhtml")
|
||||||
|
nav.content = nav_html
|
||||||
|
book.add_item(nav)
|
||||||
|
book.spine = [nav, c1]
|
||||||
|
|
||||||
|
epub.write_epub(self.epub_path, book)
|
||||||
|
|
||||||
|
# Patch
|
||||||
|
import zipfile
|
||||||
|
patched = False
|
||||||
|
with zipfile.ZipFile(self.epub_path, 'r') as zin:
|
||||||
|
opf_content = zin.read('EPUB/content.opf').decode('utf-8')
|
||||||
|
if 'toc="ncx"' in opf_content:
|
||||||
|
opf_content = opf_content.replace('toc="ncx"', '')
|
||||||
|
patched = True
|
||||||
|
if patched:
|
||||||
|
TEMP_EPUB = self.epub_path + ".temp"
|
||||||
|
with zipfile.ZipFile(TEMP_EPUB, 'w') as zout:
|
||||||
|
for item in zin.infolist():
|
||||||
|
if item.filename == 'EPUB/content.opf':
|
||||||
|
zout.writestr(item, opf_content)
|
||||||
|
else:
|
||||||
|
zout.writestr(item, zin.read(item.filename))
|
||||||
|
if patched:
|
||||||
|
shutil.move(TEMP_EPUB, self.epub_path)
|
||||||
|
|
||||||
|
parser = get_book_parser(self.epub_path)
|
||||||
|
parser.process_content()
|
||||||
|
chapters = parser.get_chapters()
|
||||||
|
chapters = [c for c in chapters if "Part" in c[1]]
|
||||||
|
|
||||||
|
self.assertEqual(len(chapters), 2)
|
||||||
|
|
||||||
|
# Check Part 1 text
|
||||||
|
text1 = parser.content_texts[chapters[0][0]]
|
||||||
|
# The parser explicitly replaces li with "1) Item A" style text
|
||||||
|
self.assertIn("1) Item A", text1)
|
||||||
|
self.assertIn("2) Item B", text1)
|
||||||
|
|
||||||
|
# Check Part 2 text
|
||||||
|
text2 = parser.content_texts[chapters[1][0]]
|
||||||
|
# Should convert start="3" to "3) Item C"
|
||||||
|
self.assertIn("3) Item C", text2)
|
||||||
|
self.assertIn("4) Item D", text2)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
import unittest
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
from ebooklib import epub
|
||||||
|
|
||||||
|
# Ensure import path
|
||||||
|
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
|
||||||
|
|
||||||
|
from abogen.book_parser import get_book_parser
|
||||||
|
|
||||||
|
class TestEpubHeuristicNav(unittest.TestCase):
|
||||||
|
"""
|
||||||
|
Tests for the heuristic fallback in _identify_nav_item (Step 4),
|
||||||
|
where the parser scans ITEM_DOCUMENTs for <nav epub:type="toc">
|
||||||
|
when no explicit ITEM_NAVIGATION is found.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.test_dir = "tests/test_data_heuristic"
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
os.makedirs(self.test_dir)
|
||||||
|
self.epub_path = os.path.join(self.test_dir, "heuristic_test.epub")
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
|
||||||
|
def test_heuristic_nav_discovery(self):
|
||||||
|
book = epub.EpubBook()
|
||||||
|
book.set_identifier("heuristic123")
|
||||||
|
book.set_title("Heuristic Test Book")
|
||||||
|
|
||||||
|
# 1. Add Content
|
||||||
|
c1 = epub.EpubHtml(title="Chapter 1", file_name="chap1.xhtml", lang="en")
|
||||||
|
c1.content = "<h1>Chapter 1</h1><p>Text</p>"
|
||||||
|
book.add_item(c1)
|
||||||
|
|
||||||
|
# 2. Add a Nav file BUT as a regular EpubHtml (ITEM_DOCUMENT)
|
||||||
|
# We do NOT use EpubNav. We do NOT look like a standard nav file if possible,
|
||||||
|
# but content must contain the magical signature.
|
||||||
|
nav_content = """
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops">
|
||||||
|
<body>
|
||||||
|
<nav epub:type="toc" id="toc">
|
||||||
|
<h1>Hidden TOC</h1>
|
||||||
|
<ol>
|
||||||
|
<li><a href="chap1.xhtml">Chapter 1</a></li>
|
||||||
|
</ol>
|
||||||
|
</nav>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
"""
|
||||||
|
# Filename intentionally generic/obscure to avoid filename-based heuristics
|
||||||
|
# (though current code checks content, not just filename)
|
||||||
|
nav_file = epub.EpubHtml(title="Hidden Nav", file_name="content_toc.xhtml")
|
||||||
|
nav_file.content = nav_content
|
||||||
|
book.add_item(nav_file)
|
||||||
|
|
||||||
|
# 3. Setup Spine
|
||||||
|
book.spine = [nav_file, c1]
|
||||||
|
|
||||||
|
# 4. Write EPUB
|
||||||
|
epub.write_epub(self.epub_path, book)
|
||||||
|
|
||||||
|
# 5. Patch OPF to ensure ebooklib didn't sneakily add ITEM_NAVIGATION or toc="ncx"
|
||||||
|
import zipfile
|
||||||
|
patched = False
|
||||||
|
with zipfile.ZipFile(self.epub_path, 'r') as zin:
|
||||||
|
opf_content = zin.read('EPUB/content.opf').decode('utf-8')
|
||||||
|
|
||||||
|
# Remove toc="ncx" attribute if present (causes crash if no NCX)
|
||||||
|
if 'toc="ncx"' in opf_content:
|
||||||
|
opf_content = opf_content.replace('toc="ncx"', '')
|
||||||
|
patched = True
|
||||||
|
|
||||||
|
# Ideally we'd verify properties="nav" isn't there, but EpubHtml shouldn't add it.
|
||||||
|
# If ebooklib added it, we might need to strip it to force heuristic.
|
||||||
|
if 'properties="nav"' in opf_content:
|
||||||
|
opf_content = opf_content.replace('properties="nav"', '')
|
||||||
|
patched = True
|
||||||
|
|
||||||
|
if patched:
|
||||||
|
TEMP_EPUB = self.epub_path + ".temp"
|
||||||
|
with zipfile.ZipFile(TEMP_EPUB, 'w') as zout:
|
||||||
|
for item in zin.infolist():
|
||||||
|
if item.filename == 'EPUB/content.opf':
|
||||||
|
zout.writestr(item, opf_content)
|
||||||
|
else:
|
||||||
|
zout.writestr(item, zin.read(item.filename))
|
||||||
|
|
||||||
|
if patched:
|
||||||
|
shutil.move(TEMP_EPUB, self.epub_path)
|
||||||
|
|
||||||
|
# 6. Verify our setup: Ensure NO ITEM_NAVIGATION exists
|
||||||
|
# We can inspect using ebooklib again
|
||||||
|
import ebooklib
|
||||||
|
check_book = epub.read_epub(self.epub_path)
|
||||||
|
nav_items = list(check_book.get_items_of_type(ebooklib.ITEM_NAVIGATION))
|
||||||
|
self.assertEqual(len(nav_items), 0, "Setup failed: explicit navigation item found!")
|
||||||
|
|
||||||
|
# 7. Run Parser
|
||||||
|
parser = get_book_parser(self.epub_path)
|
||||||
|
parser.process_content()
|
||||||
|
chapters = parser.get_chapters()
|
||||||
|
|
||||||
|
# 8. Assertions
|
||||||
|
# Should have found the nav via content scanning
|
||||||
|
chapter_titles = [c[1] for c in chapters]
|
||||||
|
self.assertIn("Chapter 1", chapter_titles)
|
||||||
|
|
||||||
|
# Also verify we hit the "html" type in identification
|
||||||
|
# We can't easily check private variables, but success implies it worked.
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,184 @@
|
|||||||
|
import unittest
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
from ebooklib import epub
|
||||||
|
|
||||||
|
# Ensure import path
|
||||||
|
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
|
||||||
|
|
||||||
|
from abogen.book_parser import get_book_parser
|
||||||
|
|
||||||
|
class TestEpubHtmlNavParsing(unittest.TestCase):
|
||||||
|
"""
|
||||||
|
Tests for EPUB 3 HTML5 Navigation Document parsing logic (_parse_html_nav_li).
|
||||||
|
"""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.test_dir = "tests/test_data_nav"
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
os.makedirs(self.test_dir)
|
||||||
|
self.epub_path = os.path.join(self.test_dir, "nav_test.epub")
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
|
||||||
|
def _create_epub_with_custom_nav(self, nav_html_content):
|
||||||
|
"""
|
||||||
|
Creates an EPUB with a manually injected HTML Navigation Document.
|
||||||
|
"""
|
||||||
|
book = epub.EpubBook()
|
||||||
|
book.set_identifier("navtest123")
|
||||||
|
book.set_title("Nav Test Book")
|
||||||
|
|
||||||
|
# Add some content files
|
||||||
|
c1 = epub.EpubHtml(title="Chapter 1", file_name="chap1.xhtml", lang="en")
|
||||||
|
c1.content = "<h1>Chapter 1</h1><p>Text 1</p>"
|
||||||
|
book.add_item(c1)
|
||||||
|
|
||||||
|
c2 = epub.EpubHtml(title="Chapter 2", file_name="chap2.xhtml", lang="en")
|
||||||
|
c2.content = "<h1>Chapter 2</h1><p>Text 2</p>"
|
||||||
|
book.add_item(c2)
|
||||||
|
|
||||||
|
# Create the Nav item manually to control the HTML structure exactly
|
||||||
|
# Use EpubHtml + OPF patching because EpubNav forces auto-generation
|
||||||
|
nav = epub.EpubHtml(title="Nav", file_name="nav.xhtml")
|
||||||
|
nav.content = nav_html_content
|
||||||
|
book.add_item(nav)
|
||||||
|
|
||||||
|
# We must set spine manually
|
||||||
|
book.spine = [nav, c1, c2]
|
||||||
|
|
||||||
|
epub.write_epub(self.epub_path, book)
|
||||||
|
|
||||||
|
# Patch the OPF to remove toc="ncx" default which causes crash
|
||||||
|
# because we intentionally excluded the legacy NCX file.
|
||||||
|
import zipfile
|
||||||
|
|
||||||
|
with zipfile.ZipFile(self.epub_path, 'r') as zin:
|
||||||
|
opf_content = zin.read('EPUB/content.opf').decode('utf-8')
|
||||||
|
opf_content = opf_content.replace('toc="ncx"', '')
|
||||||
|
|
||||||
|
# Repack
|
||||||
|
TEMP_EPUB = self.epub_path + ".temp"
|
||||||
|
with zipfile.ZipFile(TEMP_EPUB, 'w') as zout:
|
||||||
|
for item in zin.infolist():
|
||||||
|
if item.filename == 'EPUB/content.opf':
|
||||||
|
zout.writestr(item, opf_content)
|
||||||
|
else:
|
||||||
|
zout.writestr(item, zin.read(item.filename))
|
||||||
|
|
||||||
|
shutil.move(TEMP_EPUB, self.epub_path)
|
||||||
|
|
||||||
|
def test_basic_html_nav_parsing(self):
|
||||||
|
"""
|
||||||
|
Test parsing of a standard flat list of links.
|
||||||
|
"""
|
||||||
|
nav_html = """
|
||||||
|
<nav epub:type="toc" id="toc">
|
||||||
|
<h1>Table of Contents</h1>
|
||||||
|
<ol>
|
||||||
|
<li><a href="chap1.xhtml">Chapter 1</a></li>
|
||||||
|
<li><a href="chap2.xhtml">Chapter 2</a></li>
|
||||||
|
</ol>
|
||||||
|
</nav>
|
||||||
|
"""
|
||||||
|
self._create_epub_with_custom_nav(nav_html)
|
||||||
|
|
||||||
|
parser = get_book_parser(self.epub_path)
|
||||||
|
parser.process_content()
|
||||||
|
chapters = parser.get_chapters()
|
||||||
|
|
||||||
|
# Filter out "Nav" or "Introduction" prefix content found from the Nav file itself
|
||||||
|
chapters = [c for c in chapters if "Chapter" in c[1] or "Section" in c[1]]
|
||||||
|
|
||||||
|
self.assertEqual(len(chapters), 2)
|
||||||
|
self.assertEqual(chapters[0][1], "Chapter 1")
|
||||||
|
self.assertEqual(chapters[1][1], "Chapter 2")
|
||||||
|
|
||||||
|
def test_nested_html_nav_parsing(self):
|
||||||
|
"""
|
||||||
|
Test parsing of nested lists (Sub-chapters).
|
||||||
|
"""
|
||||||
|
nav_html = """
|
||||||
|
<nav epub:type="toc">
|
||||||
|
<ol>
|
||||||
|
<li>
|
||||||
|
<a href="chap1.xhtml">Chapter 1</a>
|
||||||
|
<ol>
|
||||||
|
<li><a href="chap2.xhtml">Section 1.1</a></li>
|
||||||
|
</ol>
|
||||||
|
</li>
|
||||||
|
</ol>
|
||||||
|
</nav>
|
||||||
|
"""
|
||||||
|
# Note: In this test setup, chap2 is serving as "Section 1.1" effectively
|
||||||
|
self._create_epub_with_custom_nav(nav_html)
|
||||||
|
|
||||||
|
parser = get_book_parser(self.epub_path)
|
||||||
|
parser.process_content()
|
||||||
|
chapters = parser.get_chapters()
|
||||||
|
|
||||||
|
ids = [c[1] for c in chapters]
|
||||||
|
self.assertIn("Chapter 1", ids)
|
||||||
|
self.assertIn("Section 1.1", ids)
|
||||||
|
|
||||||
|
def test_span_header_parsing(self):
|
||||||
|
"""
|
||||||
|
Test parsing of <li><span>Header</span><ol>...</ol></li> pattern.
|
||||||
|
This represents a grouping header that isn't a link itself.
|
||||||
|
"""
|
||||||
|
nav_html = """
|
||||||
|
<nav epub:type="toc">
|
||||||
|
<ol>
|
||||||
|
<li>
|
||||||
|
<span>Part I</span>
|
||||||
|
<ol>
|
||||||
|
<li><a href="chap1.xhtml">Chapter 1</a></li>
|
||||||
|
</ol>
|
||||||
|
</li>
|
||||||
|
</ol>
|
||||||
|
</nav>
|
||||||
|
"""
|
||||||
|
self._create_epub_with_custom_nav(nav_html)
|
||||||
|
|
||||||
|
parser = get_book_parser(self.epub_path)
|
||||||
|
parser.process_content()
|
||||||
|
|
||||||
|
chapters = parser.get_chapters()
|
||||||
|
chapter_titles = [c[1] for c in chapters]
|
||||||
|
|
||||||
|
self.assertIn("Chapter 1", chapter_titles)
|
||||||
|
self.assertNotIn("Part I", chapter_titles)
|
||||||
|
|
||||||
|
# Check internal structure
|
||||||
|
# Find the node named "Part I" in the processed structure
|
||||||
|
root_node = next(node for node in parser.processed_nav_structure if node['title'] == "Part I")
|
||||||
|
|
||||||
|
self.assertEqual(root_node['title'], "Part I")
|
||||||
|
self.assertFalse(root_node['has_content'])
|
||||||
|
self.assertEqual(len(root_node['children']), 1)
|
||||||
|
self.assertEqual(root_node['children'][0]['title'], "Chapter 1")
|
||||||
|
|
||||||
|
|
||||||
|
def test_identify_nav_item(self):
|
||||||
|
"""Test the _identify_nav_item method specifically."""
|
||||||
|
nav_html = """
|
||||||
|
<nav epub:type="toc" id="toc"><h1>TOC</h1><ol><li><a href="c1.html">C1</a></li></ol></nav>
|
||||||
|
"""
|
||||||
|
self._create_epub_with_custom_nav(nav_html)
|
||||||
|
parser = get_book_parser(self.epub_path)
|
||||||
|
# Note: _identify_nav_item relies on self.book being loaded
|
||||||
|
# The parser constructor or process_content handles load()
|
||||||
|
# But here we can call load directly if needed, or rely on normal flow up until navigation
|
||||||
|
parser.load()
|
||||||
|
nav_item, nav_type = parser._identify_nav_item()
|
||||||
|
|
||||||
|
self.assertEqual(nav_type, "html")
|
||||||
|
self.assertIsNotNone(nav_item)
|
||||||
|
self.assertTrue("nav.xhtml" in nav_item.get_name())
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
import unittest
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import zipfile
|
||||||
|
import sys
|
||||||
|
import logging
|
||||||
|
from ebooklib import epub
|
||||||
|
|
||||||
|
# Ensure import path
|
||||||
|
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
|
||||||
|
|
||||||
|
from abogen.book_parser import get_book_parser
|
||||||
|
|
||||||
|
class TestEpubMissingFileErrorHandling(unittest.TestCase):
|
||||||
|
"""
|
||||||
|
Tests for robust error handling and recovery in the book parser.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.test_dir = "tests/test_data_errors"
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
os.makedirs(self.test_dir)
|
||||||
|
self.broken_epub_path = os.path.join(self.test_dir, "missing_file.epub")
|
||||||
|
|
||||||
|
# Suppress logging during tests to keep output clean,
|
||||||
|
# or capture it if we want to assert on warnings.
|
||||||
|
# For now, we just let it be or set level to ERROR.
|
||||||
|
logging.getLogger().setLevel(logging.ERROR)
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
|
||||||
|
def _create_broken_epub(self):
|
||||||
|
"""
|
||||||
|
Creates an EPUB where a file listed in the manifest is missing from the ZIP archive.
|
||||||
|
"""
|
||||||
|
book = epub.EpubBook()
|
||||||
|
book.set_identifier("broken123")
|
||||||
|
book.set_title("Broken Book")
|
||||||
|
|
||||||
|
# 1. Add a valid chapter
|
||||||
|
c1 = epub.EpubHtml(title="Chapter 1", file_name="chap1.xhtml", lang="en")
|
||||||
|
c1.content = "<h1>Chapter 1</h1><p>Survivable content.</p>"
|
||||||
|
book.add_item(c1)
|
||||||
|
|
||||||
|
# 2. Add a 'ghost' chapter that we will delete later
|
||||||
|
c2 = epub.EpubHtml(title="Ghost Chapter", file_name="ghost.xhtml", lang="en")
|
||||||
|
c2.content = "<h1>Ghost</h1><p>I will disappear.</p>"
|
||||||
|
book.add_item(c2)
|
||||||
|
|
||||||
|
book.spine = ["nav", c1, c2]
|
||||||
|
book.add_item(epub.EpubNcx())
|
||||||
|
book.add_item(epub.EpubNav())
|
||||||
|
|
||||||
|
temp_path = os.path.join(self.test_dir, "temp.epub")
|
||||||
|
epub.write_epub(temp_path, book)
|
||||||
|
|
||||||
|
# 3. Physically remove 'ghost.xhtml' from the ZIP
|
||||||
|
with zipfile.ZipFile(temp_path, 'r') as zin:
|
||||||
|
with zipfile.ZipFile(self.broken_epub_path, 'w') as zout:
|
||||||
|
for item in zin.infolist():
|
||||||
|
# Copy everything EXCEPT the ghost file
|
||||||
|
# Note: ebooklib might put files in OEPS/ or EPUB/ folders depending on version,
|
||||||
|
# so checking "ghost.xhtml" presence in filename is safer.
|
||||||
|
if "ghost.xhtml" not in item.filename:
|
||||||
|
zout.writestr(item, zin.read(item.filename))
|
||||||
|
|
||||||
|
def test_missing_file_recovery(self):
|
||||||
|
"""
|
||||||
|
Verify that the parser recovers gracefully when a referenced file is missing.
|
||||||
|
Should log a warning instead of raising KeyError.
|
||||||
|
"""
|
||||||
|
self._create_broken_epub()
|
||||||
|
|
||||||
|
try:
|
||||||
|
parser = get_book_parser(self.broken_epub_path)
|
||||||
|
parser.process_content()
|
||||||
|
|
||||||
|
# 1. Ensure process didn't crash
|
||||||
|
self.assertTrue(True, "Parser should not crash on missing file")
|
||||||
|
|
||||||
|
# 2. Ensure valid content was extracted
|
||||||
|
# Identify the ID for chap1.xhtml (usually file path based)
|
||||||
|
# Since IDs can vary, we check if ANY content contains our known string
|
||||||
|
chap1_found = False
|
||||||
|
for text in parser.content_texts.values():
|
||||||
|
if "Survivable content" in text:
|
||||||
|
chap1_found = True
|
||||||
|
break
|
||||||
|
self.assertTrue(chap1_found, "The valid chapter should still be processed")
|
||||||
|
|
||||||
|
except KeyError:
|
||||||
|
self.fail("Parser raised KeyError instead of handling the missing file!")
|
||||||
|
except Exception as e:
|
||||||
|
self.fail(f"Parser raised unexpected exception: {e}")
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,140 @@
|
|||||||
|
import unittest
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
from ebooklib import epub
|
||||||
|
|
||||||
|
# Ensure we can import the module
|
||||||
|
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
|
||||||
|
|
||||||
|
from abogen.book_parser import get_book_parser, EpubParser
|
||||||
|
|
||||||
|
class TestEpubNcxParsing(unittest.TestCase):
|
||||||
|
"""
|
||||||
|
Focused tests for NCX navigation scenarios, ensuring legacy/compatibility
|
||||||
|
modes work when HTML5 Navigation is missing.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.test_dir = "tests/test_data_ncx"
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
os.makedirs(self.test_dir)
|
||||||
|
self.ncx_only_epub_path = os.path.join(self.test_dir, "ncx_only.epub")
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
|
||||||
|
def _create_ncx_only_epub(self, chapters):
|
||||||
|
"""
|
||||||
|
Helper to create an EPUB with ONLY NCX table of contents (no HTML nav).
|
||||||
|
"""
|
||||||
|
book = epub.EpubBook()
|
||||||
|
book.set_identifier("ncx_test_123")
|
||||||
|
book.set_title("NCX Only Book")
|
||||||
|
book.set_language("en")
|
||||||
|
|
||||||
|
epub_chapters = []
|
||||||
|
for i, (title, content) in enumerate(chapters):
|
||||||
|
filename = f"chap{i+1}.xhtml"
|
||||||
|
c = epub.EpubHtml(title=title, file_name=filename, lang="en")
|
||||||
|
# Ensure content is substantial enough to not be skipped
|
||||||
|
c.content = f"<h1>{title}</h1><p>{content}</p>"
|
||||||
|
book.add_item(c)
|
||||||
|
epub_chapters.append(c)
|
||||||
|
|
||||||
|
# Define Table of Contents
|
||||||
|
book.toc = tuple(epub_chapters)
|
||||||
|
|
||||||
|
# Add default NCX and generic spine
|
||||||
|
book.add_item(epub.EpubNcx())
|
||||||
|
# IMPORTANT: Do NOT add EpubNav() here, that's what we are testing!
|
||||||
|
|
||||||
|
book.spine = ["nav"] + epub_chapters
|
||||||
|
|
||||||
|
epub.write_epub(self.ncx_only_epub_path, book)
|
||||||
|
|
||||||
|
def test_ncx_only_parsing(self):
|
||||||
|
"""
|
||||||
|
Verify that an EPUB with only an NCX file (no HTML nav) is parsed correctly.
|
||||||
|
Logic tested: _process_epub_content_nav (NCX branch), _parse_ncx_navpoint
|
||||||
|
"""
|
||||||
|
# 1. Setup Data
|
||||||
|
chapters_data = [
|
||||||
|
("Chapter 1", "This is the first chapter."),
|
||||||
|
("Chapter 2", "This is the second chapter.")
|
||||||
|
]
|
||||||
|
self._create_ncx_only_epub(chapters_data)
|
||||||
|
|
||||||
|
# 2. Run Parser
|
||||||
|
parser = get_book_parser(self.ncx_only_epub_path)
|
||||||
|
parser.process_content()
|
||||||
|
|
||||||
|
# 3. Verify Breakdown
|
||||||
|
# We expect detailed breakdown based on NCX
|
||||||
|
chapters = parser.get_chapters()
|
||||||
|
|
||||||
|
# Should find exactly 2 chapters based on the Toc
|
||||||
|
self.assertEqual(len(chapters), 2, "Should have 2 chapters extracted from NCX")
|
||||||
|
|
||||||
|
# Check Titles and Sequence
|
||||||
|
self.assertEqual(chapters[0][1], "Chapter 1")
|
||||||
|
self.assertEqual(chapters[1][1], "Chapter 2")
|
||||||
|
|
||||||
|
# Verify content was extracted
|
||||||
|
# Note: 'src' in chapters usually points to file_name if no fragments
|
||||||
|
id_1 = chapters[0][0]
|
||||||
|
self.assertIn("This is the first chapter", parser.content_texts[id_1])
|
||||||
|
|
||||||
|
def test_nested_ncx_parsing(self):
|
||||||
|
"""
|
||||||
|
Verify parsing of nested NCX structures (Chapters with Subchapters).
|
||||||
|
"""
|
||||||
|
book = epub.EpubBook()
|
||||||
|
book.set_identifier("nested_ncx")
|
||||||
|
book.set_title("Nested NCX")
|
||||||
|
|
||||||
|
# Create one big file with sections
|
||||||
|
c1 = epub.EpubHtml(title="Main Chapter", file_name="main.xhtml", lang="en")
|
||||||
|
c1.content = """
|
||||||
|
<h1 id="intro">Introduction</h1>
|
||||||
|
<p>Intro text.</p>
|
||||||
|
<h2 id="sect1">Section 1</h2>
|
||||||
|
<p>Section 1 text.</p>
|
||||||
|
"""
|
||||||
|
book.add_item(c1)
|
||||||
|
|
||||||
|
# Manually construct nested TOC because ebooklib's default helpers are simple
|
||||||
|
# EbookLib automatically builds NCX from book.toc
|
||||||
|
# Nested tuple structure: (Section, (Subsection, Sub-subsection))
|
||||||
|
|
||||||
|
# We need to link to Fragments for this to really test nested NCX pointing to same file
|
||||||
|
# EbookLib Link object: epub.Link(href, title, uid)
|
||||||
|
|
||||||
|
link_root = epub.Link("main.xhtml#intro", "Introduction", "intro")
|
||||||
|
link_sect = epub.Link("main.xhtml#sect1", "Section 1", "sect1")
|
||||||
|
|
||||||
|
# Structure: Intro -> Section 1 (as child)
|
||||||
|
book.toc = (
|
||||||
|
(link_root, (link_sect, )),
|
||||||
|
)
|
||||||
|
|
||||||
|
book.add_item(epub.EpubNcx())
|
||||||
|
book.spine = ["nav", c1]
|
||||||
|
|
||||||
|
epub.write_epub(self.ncx_only_epub_path, book)
|
||||||
|
|
||||||
|
# Parse
|
||||||
|
parser = get_book_parser(self.ncx_only_epub_path)
|
||||||
|
parser.process_content()
|
||||||
|
|
||||||
|
chapters = parser.get_chapters()
|
||||||
|
|
||||||
|
# Depending on how the parser flattens, we should see both entries
|
||||||
|
titles = [node[1] for node in chapters]
|
||||||
|
self.assertIn("Introduction", titles)
|
||||||
|
self.assertIn("Section 1", titles)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,130 @@
|
|||||||
|
import unittest
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
from ebooklib import epub
|
||||||
|
import ebooklib
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
# Ensure import path
|
||||||
|
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
|
||||||
|
|
||||||
|
from abogen.book_parser import get_book_parser
|
||||||
|
|
||||||
|
class TestEpubStandardNav(unittest.TestCase):
|
||||||
|
"""
|
||||||
|
Tests for the standard ITEM_NAVIGATION discovery in _identify_nav_item.
|
||||||
|
Refactored to explicitly test different discovery paths defined in the parser.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.test_dir = "tests/test_data_standard_nav"
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
os.makedirs(self.test_dir)
|
||||||
|
self.epub_path = os.path.join(self.test_dir, "standard_nav_test.epub")
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
if os.path.exists(self.test_dir):
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
|
||||||
|
def _create_and_load_epub(self):
|
||||||
|
"""Helper to create a basic EPUB and return a loaded parser."""
|
||||||
|
book = epub.EpubBook()
|
||||||
|
book.set_identifier("stdnav123")
|
||||||
|
book.set_title("Standard Nav Test")
|
||||||
|
|
||||||
|
c1 = epub.EpubHtml(title="Chapter 1", file_name="chap1.xhtml", lang="en")
|
||||||
|
c1.content = "<h1>Chapter 1</h1><p>Text 1</p>"
|
||||||
|
book.add_item(c1)
|
||||||
|
|
||||||
|
# Use Standard EpubNav
|
||||||
|
nav = epub.EpubNav()
|
||||||
|
book.add_item(nav)
|
||||||
|
book.spine = [nav, c1]
|
||||||
|
|
||||||
|
epub.write_epub(self.epub_path, book)
|
||||||
|
|
||||||
|
# "Zip Surgery" Patch:
|
||||||
|
# ebooklib unconditionally adds `toc="ncx"` to the spine, even for EPUB 3 files that purely use HTML Nav.
|
||||||
|
# This creates a dangling reference to a non-existent "ncx" item, causing ebooklib to crash on read.
|
||||||
|
# We manually remove this attribute to ensure the test EPUB is valid and readable.
|
||||||
|
# TODO - find real world examples of EPUB 3 files that use HTML Nav
|
||||||
|
import zipfile
|
||||||
|
patched = False
|
||||||
|
with zipfile.ZipFile(self.epub_path, 'r') as zin:
|
||||||
|
opf_content = zin.read('EPUB/content.opf').decode('utf-8')
|
||||||
|
if 'toc="ncx"' in opf_content:
|
||||||
|
opf_content = opf_content.replace('toc="ncx"', '')
|
||||||
|
patched = True
|
||||||
|
TEMP_EPUB = self.epub_path + ".temp"
|
||||||
|
with zipfile.ZipFile(TEMP_EPUB, 'w') as zout:
|
||||||
|
for item in zin.infolist():
|
||||||
|
if item.filename == 'EPUB/content.opf':
|
||||||
|
zout.writestr(item, opf_content)
|
||||||
|
else:
|
||||||
|
zout.writestr(item, zin.read(item.filename))
|
||||||
|
|
||||||
|
if patched:
|
||||||
|
shutil.move(TEMP_EPUB, self.epub_path)
|
||||||
|
|
||||||
|
parser = get_book_parser(self.epub_path)
|
||||||
|
parser.load()
|
||||||
|
return parser
|
||||||
|
|
||||||
|
def test_discovery_by_item_navigation_type(self):
|
||||||
|
"""
|
||||||
|
Scenario 1: The item is explicitly identified as ITEM_NAVIGATION (4).
|
||||||
|
This exercises the first branch of _identify_nav_item.
|
||||||
|
"""
|
||||||
|
parser = self._create_and_load_epub()
|
||||||
|
|
||||||
|
# Inject an item that mocks the ITEM_NAVIGATION type behavior
|
||||||
|
# (This simulates a library/parser that correctly types the item as 4)
|
||||||
|
mock_nav = MagicMock()
|
||||||
|
mock_nav.get_name.return_value = "nav.xhtml"
|
||||||
|
mock_nav.get_type.return_value = ebooklib.ITEM_NAVIGATION
|
||||||
|
|
||||||
|
# We append this mock to the book items to ensure get_items_of_type(ITEM_NAVIGATION) finds it
|
||||||
|
parser.book.items.append(mock_nav)
|
||||||
|
|
||||||
|
nav_item, nav_type = parser._identify_nav_item()
|
||||||
|
|
||||||
|
self.assertEqual(nav_type, "html")
|
||||||
|
self.assertEqual(nav_item.get_name(), "nav.xhtml")
|
||||||
|
# Verify we are getting the object we expect (implied by success)
|
||||||
|
|
||||||
|
def test_discovery_by_nav_property(self):
|
||||||
|
"""
|
||||||
|
Scenario 2: The item is ITEM_DOCUMENT (9) but has properties=['nav'].
|
||||||
|
This is the standard EPUB 3 behavior and exercises the fallback branch.
|
||||||
|
"""
|
||||||
|
parser = self._create_and_load_epub()
|
||||||
|
|
||||||
|
# Locate the generic 'nav' item loaded by ebooklib
|
||||||
|
original_nav = parser.book.get_item_with_id("nav")
|
||||||
|
self.assertIsNotNone(original_nav)
|
||||||
|
|
||||||
|
# "Fix" the object to match what we expect from a correct EPUB 3 read:
|
||||||
|
# It should have properties=['nav'].
|
||||||
|
# We use a real EpubNav object to ensure structural correctness.
|
||||||
|
proper_nav = epub.EpubNav(uid=original_nav.id, file_name=original_nav.file_name)
|
||||||
|
proper_nav.content = original_nav.content
|
||||||
|
proper_nav.properties = ['nav']
|
||||||
|
|
||||||
|
# Swap it into the book items list
|
||||||
|
try:
|
||||||
|
idx = parser.book.items.index(original_nav)
|
||||||
|
parser.book.items[idx] = proper_nav
|
||||||
|
except ValueError:
|
||||||
|
self.fail("Could not find original nav item to swap")
|
||||||
|
|
||||||
|
nav_item, nav_type = parser._identify_nav_item()
|
||||||
|
|
||||||
|
self.assertEqual(nav_type, "html")
|
||||||
|
self.assertEqual(nav_item.get_name(), "nav.xhtml")
|
||||||
|
# Check that we actually found the one with properties
|
||||||
|
self.assertEqual(getattr(nav_item, 'properties', []), ['nav'])
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
Reference in New Issue
Block a user