feat: Additive merge of webui branch with PyQt GUI support

- Add abogen/pyqt/ package with full PyQt6 desktop GUI
- Restore PyQt GUI files (gui.py, book_handler.py, queue_manager_gui.py, voice_formula_gui.py, conversion.py)
- Add new shared modules from webui: book_parser.py, subtitle_utils.py, spacy_utils.py
- Add Linux libraries (libxcb-cursor) for Qt platform plugin support
- Add new epub parsing tests from webui branch
- Update pyproject.toml with dual entry points:
  - abogen: PyQt6 desktop GUI
  - abogen-web: Flask Web UI
- Add PyQt6 to dependencies
- Re-export PyQt classes from root modules for backwards compatibility
- Merge CHANGELOG.md entries (1.2.0-1.2.5 from webui)
- Update README.md with dual interface documentation

Implements #26 - shared core with separate UI folders
This commit is contained in:
JB
2025-12-22 05:51:21 -08:00
parent ede5343e0c
commit e2b2f610a6
32 changed files with 14863 additions and 38 deletions
+57
View File
@@ -1,6 +1,63 @@
# Unreleased
- Added an EPUB 3 packaging pipeline that builds media-overlay EPUBs from generated audio and chunk metadata.
- Persisted chunk timing metadata in job artifacts and exercised the exporter with automated tests.
- Added Flask-based Web UI (`abogen-web`) for Docker and headless server deployments.
- Reorganized codebase to support both PyQt6 desktop GUI and Web UI from a shared core.
- Added Supertonic TTS engine support with GPU acceleration.
- Added entity analysis and pronunciation override system for proper nouns.
- Added speaker/role assignment for multi-voice "theatrical" audiobooks.
- Added Calibre OPDS and Audiobookshelf integration.
# 1.2.5
- Added new option: `Override item settings with current selection` in the queue manager. When enabled, all items in the queue will be processed using the current global settings selected in the main GUI, overriding their individual settings. When disabled, each item will retain its own specific settings.
- Fixed `Error "Could not load the Qt platform plugin "xcb"` error that occurred in some Linux distributions due to missing `libxcb-cursor0` library by conditionally loading the bundled library when the system version is unavailable, issue mentioned by @bmcgonag in #101.
- Fixed the `No module named pip` error that occurred for users who installed Abogen via the [**uv**](https://github.com/astral-sh/uv) installer.
- Fixed defaults for `replace_single_newlines` not being applied correctly in some cases.
- Fixed `Save chapters separately for queued epubs is ignored`, issue mentioned by @dymas-cz in #109.
- Fixed incorrect sentence segmentation when using spaCy, where text would erroneously split after opening parentheses.
- Improvements in code and documentation.
# 1.2.4
- **Subtitle generation is now available for all languages!** Abogen now supports subtitle generation for non-English languages using audio duration-based timing. Available modes include `Line`, `Sentence`, and `Sentence + Comma`. (Note: Word-level subtitle modes remain English-only due to Kokoro's timestamp token limitations.)
- New option: **"Use spaCy for sentence segmentation"** You can now use [spaCy](https://spacy.io/) to automatically detect sentence boundaries and produce cleaner, more readable subtitles.
- New option: **Pre-download models and voices for offline use** You can now pre-download all required Kokoro models, voices, and spaCy language models.
- Added support for `.` separator in timestamps (e.g. `HH:MM:SS.ms`) for timestamp-based text files.
- Optimized regex compilation and eliminated busy-wait loops.
- Possibly fixed `Silent truncation of long paragraphs` issue mentioned in #91 by @xklzlxr.
- Fixed unused regex patterns and variable naming conventions.
- Improvements in code and documentation.
# 1.2.3
- Same as 1.2.2, re-released to fix an issue with subtitle timing when using timestamp-based text files.
# 1.2.2
- **You can now voice your subtitle files!** Simply add `.srt`, `.ass` or `.vtt` files to generate timed audio.
- New option: **"Use silent gaps between subtitles"**: Prevents unnecessary audio speed-up by letting speech continue into the silent gaps between subtitles.
- New option: **"Subtitle speed adjustment method"**: Choose how to speed up audio when needed (TTS Regeneration or FFmpeg Time-stretch).
- Added support for embedding cover images in M4B files.
- Fixed `[WinError 1114] A dynamic link library (DLL) initialization routine failed` error on Windows.
- Potential fix for `CUDA GPU is not available` issue.
- Improvements in code and documentation.
# 1.2.1
- Upgraded Abogen's interface from PyQt5 to PyQt6 for better compatibility and long-term support.
- Added tooltip indicators in queue manager to display book handler options.
- Added `Open processed file` and `Open input file` options for items in the queue manager.
- Added loading gif animation to book handler window.
- Fixed light theme slider colors in voice mixer for better visibility (for non-Windows users).
- Fixed subtitle word-count splitting logic for more accurate segmentation.
- Improvements in code and documentation.
# 1.2.0
- Added `Line` option to subtitle generation modes, allowing subtitles to be generated based on line breaks in the text, by @mleg in #94.
- Added a loading indicator to the book handler window for better user experience during book preprocessing.
- Fixed `cannot access local variable 'is_narrow'` error when subtitle format `SRT` was selected, mentioned by @Kinasa0096 in #88.
- Fixed folder and filename sanitization to properly handle OS-specific illegal characters.
- Fixed `/` and `\` path display by normalizing paths.
- Fixed book reprocessing issue where books were being processed every time the chapters window was opened.
- Fixed taskbar icon not appearing correctly in Windows.
- Fixed "Go to folder" button not opening the chapter output directory when only separate chapters were generated.
- Improvements in code and documentation.
# 1.1.9
- Fixed the issue where spaces were deleted before punctuation marks while generating subtitles.
+17 -1
View File
@@ -9,6 +9,17 @@ Abogen is a web-first text-to-speech workstation. Drop in an EPUB, PDF, Markdown
- LLM-assisted text normalization with live previews and configurable prompts
- Runs well in Docker, ships a REST-style JSON API, and works across macOS, Linux, and Windows
## Interfaces
Abogen offers **two interfaces** to suit different workflows:
| Command | Interface | Best for |
|---------|-----------|----------|
| `abogen` | PyQt6 Desktop GUI | Local desktop use on Windows/macOS/Linux |
| `abogen-web` | Flask Web UI | Docker, headless servers, remote access |
Both interfaces share the same core processing engine and produce identical output.
## Quick start
Abogen supports Python 3.103.12.
@@ -19,11 +30,16 @@ source .venv/bin/activate # On Windows use: .venv\Scripts\activate
pip install abogen
```
### Launch the web app
### Launch the desktop app (PyQt6)
```bash
abogen
```
### Launch the web app (Flask)
```bash
abogen-web
```
Then open http://localhost:8808 and drag in your documents. Jobs run in the background worker and the browser updates automatically.
> **Tip:** Keep the terminal open while the server is running. Use `Ctrl+C` to stop it.
+7 -5
View File
@@ -1,9 +1,11 @@
"""Legacy PyQt-based chapter selection dialog has been removed."""
"""Backwards-compatible re-export of the PyQt book handler.
The actual implementation lives in abogen.pyqt.book_handler.
"""
from __future__ import annotations
from abogen.pyqt.book_handler import * # noqa: F401, F403
from abogen.pyqt.book_handler import HandlerDialog
def __getattr__(name: str):
raise AttributeError(
"The PyQt chapter selection dialog was removed. Use the web interface instead."
)
__all__ = ["HandlerDialog"]
+917
View File
@@ -0,0 +1,917 @@
import os
import re
import logging
import textwrap
import urllib.parse
from abc import ABC, abstractmethod
import ebooklib
from ebooklib import epub
from bs4 import BeautifulSoup, NavigableString
import fitz # PyMuPDF
import markdown
from abogen.utils import detect_encoding
from abogen.subtitle_utils import clean_text, calculate_text_length
# Pre-compile frequently used regex patterns
_BRACKETED_NUMBERS_PATTERN = re.compile(r"\[\s*\d+\s*\]")
_STANDALONE_PAGE_NUMBERS_PATTERN = re.compile(r"^\s*\d+\s*$", re.MULTILINE)
_PAGE_NUMBERS_AT_END_PATTERN = re.compile(r"\s+\d+\s*$", re.MULTILINE)
_PAGE_NUMBERS_WITH_DASH_PATTERN = re.compile(r"\s+[-–—]\s*\d+\s*[-–—]?\s*$", re.MULTILINE)
class BaseBookParser(ABC):
"""
Abstract base class for parsing different book formats.
"""
def __init__(self, book_path):
self.book_path = os.path.normpath(os.path.abspath(book_path))
self.content_texts = {}
self.content_lengths = {}
self.book_metadata = {}
# Unified structure for navigation: list of dicts
# { 'title': str, 'src': str, 'children': [], 'has_content': bool }
self.processed_nav_structure = []
self.load()
@abstractmethod
def load(self):
"""Load the book file."""
pass
@abstractmethod
def process_content(self, replace_single_newlines=True):
"""Process the book content to extract text and structure."""
pass
@property
@abstractmethod
def file_type(self):
"""Return the type of the file (pdf, epub, markdown)."""
pass
def get_chapters(self):
"""Return a list of chapter IDs and Names."""
chapters = []
if self.processed_nav_structure:
def flatten_nav(nodes):
for node in nodes:
if node.get("has_content"):
chapters.append((node["src"], node["title"]))
if node.get("children"):
flatten_nav(node["children"])
flatten_nav(self.processed_nav_structure)
else:
# Fallback for simple content without nav structure
for ch_id, content in self.content_texts.items():
# This could be improved, but serves as a generic fallback
chapters.append((ch_id, ch_id))
return chapters
def get_formatted_text(self):
"""
Returns the full text of the book formatted with chapter markers.
"""
chapters = self.get_chapters()
full_text = []
for chapter_id, chapter_name in chapters:
text = self.content_texts.get(chapter_id, "")
if text:
full_text.append(f"\n<<CHAPTER_MARKER:{chapter_name}>>\n")
full_text.append(text)
return "\n".join(full_text)
def get_metadata(self):
"""Return extracted metadata."""
return self.book_metadata
class PdfParser(BaseBookParser):
def __init__(self, book_path):
self.pdf_doc = None
super().__init__(book_path)
@property
def file_type(self):
return "pdf"
def load(self):
try:
self.pdf_doc = fitz.open(self.book_path)
except Exception as e:
logging.error(f"Error loading PDF {self.book_path}: {e}")
raise
def process_content(self, replace_single_newlines=True):
if not self.pdf_doc:
self.load()
for page_num in range(len(self.pdf_doc)):
text = clean_text(self.pdf_doc[page_num].get_text())
# Clean up common PDF artifacts:
# - Remove bracketed numbers often used for citations [1]
text = _BRACKETED_NUMBERS_PATTERN.sub("", text)
# - Remove standalone page numbers often found in headers/footers
text = _STANDALONE_PAGE_NUMBERS_PATTERN.sub("", text)
# - Remove page numbers at end of lines
text = _PAGE_NUMBERS_AT_END_PATTERN.sub("", text)
# - Remove page numbers with dashes - 4 -
text = _PAGE_NUMBERS_WITH_DASH_PATTERN.sub("", text)
page_id = f"page_{page_num + 1}"
self.content_texts[page_id] = text
self.content_lengths[page_id] = calculate_text_length(text)
return self.content_texts, self.content_lengths
def _extract_book_metadata(self):
# PDF metadata extraction can be added here if needed
# For now, base class metadata is empty dict
pass
def get_chapters(self):
# PDF specific implementation because it doesn't use nav structure
chapters = []
if self.pdf_doc:
for i in range(len(self.pdf_doc)):
chapters.append((f"page_{i+1}", f"Page {i+1}"))
return chapters
class MarkdownParser(BaseBookParser):
def __init__(self, book_path):
self.markdown_text = None
super().__init__(book_path)
@property
def file_type(self):
return "markdown"
def load(self):
try:
encoding = detect_encoding(self.book_path)
with open(self.book_path, "r", encoding=encoding, errors="replace") as f:
self.markdown_text = f.read()
except Exception as e:
logging.error(f"Error reading markdown file: {e}")
self.markdown_text = ""
def process_content(self, replace_single_newlines=True):
if self.markdown_text is None:
self.load()
self._process_markdown_content()
return self.content_texts, self.content_lengths
def _convert_markdown_toc_to_nav(self, toc_tokens):
nav_nodes = []
for token in toc_tokens:
node = {
"title": token["name"],
"src": token["id"],
"children": self._convert_markdown_toc_to_nav(
token.get("children", [])
),
"has_content": True,
}
nav_nodes.append(node)
return nav_nodes
def _process_markdown_content(self):
if not self.markdown_text:
return
original_text = textwrap.dedent(self.markdown_text)
md = markdown.Markdown(extensions=["toc", "fenced_code"])
html = md.convert(original_text)
markdown_toc = md.toc_tokens
# Convert markdown TOC tokens to our unified navigation structure
self.processed_nav_structure = self._convert_markdown_toc_to_nav(markdown_toc)
cleaned_full_text = clean_text(original_text)
# If no TOC found, treat as single chapter
if not self.processed_nav_structure:
chapter_id = "markdown_content"
self.content_texts[chapter_id] = cleaned_full_text
self.content_lengths[chapter_id] = calculate_text_length(cleaned_full_text)
return
soup = BeautifulSoup(html, "html.parser")
all_headers = []
def flatten_nav_internal(nodes):
for node in nodes:
all_headers.append(node)
if node.get("children"):
flatten_nav_internal(node["children"])
flatten_nav_internal(self.processed_nav_structure)
header_positions = []
for node in all_headers:
header_id = node["src"]
id_pattern = f'id="{header_id}"'
pos = html.find(id_pattern)
if pos != -1:
tag_start = html.rfind("<", 0, pos)
header_positions.append(
{"id": header_id, "start": tag_start, "name": node["title"]}
)
header_positions.sort(key=lambda x: x["start"])
for i, header_pos in enumerate(header_positions):
header_id = header_pos["id"]
header_name = header_pos["name"]
content_start = header_pos["start"]
content_end = (
header_positions[i + 1]["start"]
if i + 1 < len(header_positions)
else len(html)
)
section_html = html[content_start:content_end]
section_soup = BeautifulSoup(section_html, "html.parser")
header_tag = section_soup.find(attrs={"id": header_id})
if header_tag:
header_tag.decompose()
section_text = clean_text(section_soup.get_text()).strip()
chapter_id = header_id
if section_text:
full_content = f"{header_name}\n\n{section_text}"
self.content_texts[chapter_id] = full_content
self.content_lengths[chapter_id] = calculate_text_length(full_content)
else:
self.content_texts[chapter_id] = header_name
self.content_lengths[chapter_id] = calculate_text_length(header_name)
def get_chapters(self):
chapters = super().get_chapters()
if not chapters and "markdown_content" in self.content_texts:
chapters.append(("markdown_content", "Content"))
return chapters
class EpubParser(BaseBookParser):
def __init__(self, book_path):
self.book = None
self.doc_content = {}
super().__init__(book_path)
@property
def file_type(self):
return "epub"
def load(self):
try:
self.book = epub.read_epub(self.book_path)
except KeyError as e:
# TODO: should we just patch the ebooklib pre-emptively to avoid the need to catch this exception?
logging.warning(f"EPUB missing referenced file: {e}. Attempting to patch.")
# Patch ebooklib to skip missing files
import types
from ebooklib import epub as _epub_module
reader_class = _epub_module.EpubReader
orig_read_file = reader_class.read_file
def safe_read_file(self, name):
try:
return orig_read_file(self, name)
except KeyError:
logging.warning(f"Missing file in EPUB: {name}. Returning empty bytes.")
return b""
reader_class.read_file = safe_read_file
try:
self.book = epub.read_epub(self.book_path)
finally:
reader_class.read_file = orig_read_file
def process_content(self, replace_single_newlines=True):
if not self.book:
self.load()
self.book_metadata = self._extract_book_metadata()
try:
nav_item, nav_type = self._identify_nav_item()
self._execute_nav_parsing_logic(nav_item, nav_type)
except Exception as e:
logging.warning(f"EPUB nav processing failed: {e}. Falling back to spine.")
self._process_epub_content_spine_fallback()
return self.content_texts, self.content_lengths
def _extract_book_metadata(self):
metadata = {}
if not self.book:
return metadata
try:
metadata["title"] = self.book.get_metadata("DC", "title")[0][0]
except Exception:
metadata["title"] = os.path.splitext(os.path.basename(self.book_path))[0]
try:
metadata["author"] = self.book.get_metadata("DC", "creator")[0][0]
except Exception:
metadata["author"] = "Unknown Author"
try:
metadata["language"] = self.book.get_metadata("DC", "language")[0][0]
except Exception:
metadata["language"] = "en"
return metadata
def _find_doc_key(self, base_href, doc_order, doc_order_decoded):
candidates = [
base_href,
urllib.parse.unquote(base_href),
]
base_name = os.path.basename(base_href).lower()
for k in list(doc_order.keys()) + list(doc_order_decoded.keys()):
if os.path.basename(k).lower() == base_name:
candidates.append(k)
for candidate in candidates:
if candidate in doc_order:
return candidate, doc_order[candidate]
elif candidate in doc_order_decoded:
return candidate, doc_order_decoded[candidate]
return None, None
def _find_position_robust(self, doc_href, fragment_id):
if doc_href not in self.doc_content:
logging.warning(f"Document '{doc_href}' not found in cached content.")
return 0
html_content = self.doc_content[doc_href]
if not fragment_id:
return 0
try:
temp_soup = BeautifulSoup(f"<div>{html_content}</div>", "html.parser")
target_element = temp_soup.find(id=fragment_id)
if target_element:
tag_str = str(target_element)
pos = html_content.find(tag_str[: min(len(tag_str), 200)])
if pos != -1:
return pos
except Exception as e:
logging.warning(f"BeautifulSoup failed to find id='{fragment_id}': {e}")
safe_fragment_id = re.escape(fragment_id)
id_name_pattern = re.compile(
f"<[^>]+(?:id|name)\\s*=\\s*[\"']{safe_fragment_id}[\"']", re.IGNORECASE
)
match = id_name_pattern.search(html_content)
if match:
return match.start()
id_match_str = f'id="{fragment_id}"'
name_match_str = f'name="{fragment_id}"'
id_pos = html_content.find(id_match_str)
name_pos = html_content.find(name_match_str)
pos = -1
if id_pos != -1 and name_pos != -1:
pos = min(id_pos, name_pos)
elif id_pos != -1:
pos = id_pos
elif name_pos != -1:
pos = name_pos
if pos != -1:
tag_start_pos = html_content.rfind("<", 0, pos)
final_pos = tag_start_pos if tag_start_pos != -1 else 0
return final_pos
logging.warning(
f"Anchor '{fragment_id}' not found in {doc_href}. Defaulting to position 0."
)
return 0
def _parse_ncx_navpoint(
self,
nav_point,
ordered_entries,
doc_order,
doc_order_decoded,
tree_structure_list,
find_position_func,
):
"""
Recursive parsing of NCX navigation nodes.
Logic tested by: tests/test_epub_ncx_parsing.py
"""
nav_label = nav_point.find("navLabel")
content = nav_point.find("content")
title = (
nav_label.find("text").get_text(strip=True)
if nav_label and nav_label.find("text")
else "Untitled Section"
)
src = content["src"] if content and "src" in content.attrs else None
current_entry_node = {"title": title, "src": src, "children": []}
if src:
base_href, fragment = src.split("#", 1) if "#" in src else (src, None)
doc_key, doc_idx = self._find_doc_key(base_href, doc_order, doc_order_decoded)
if not doc_key:
current_entry_node["has_content"] = False
else:
position = find_position_func(doc_key, fragment)
entry_data = {
"src": src,
"title": title,
"doc_href": doc_key,
"position": position,
"doc_order": doc_idx,
}
ordered_entries.append(entry_data)
current_entry_node["has_content"] = True
else:
current_entry_node["has_content"] = False
child_navpoints = nav_point.find_all("navPoint", recursive=False)
if child_navpoints:
for child_np in child_navpoints:
self._parse_ncx_navpoint(
child_np,
ordered_entries,
doc_order,
doc_order_decoded,
current_entry_node["children"],
find_position_func,
)
if title and (
current_entry_node.get("has_content", False) or current_entry_node["children"]
):
tree_structure_list.append(current_entry_node)
def _extract_nav_li_title(self, li_element, link_element=None, span_element=None):
"""Helper to extract title from a nav <li> element, handling various structures."""
title = "Untitled Section"
if link_element:
title = link_element.get_text(strip=True) or title
elif span_element:
title = span_element.get_text(strip=True) or title
# Fallback to direct text if title is empty or default
# If we used link/span but got empty string, we try fallback.
# If we didn't use link/span, we try fallback.
if not title.strip() or title == "Untitled Section":
li_text = "".join(
t for t in li_element.contents if isinstance(t, NavigableString)
).strip()
if li_text:
title = li_text
# Second fallback: if we have a span but title is still empty, try span text again
# (covered by logic above mostly, but mirroring original logic's intense fallback)
if (not title.strip() or title == "Untitled Section") and span_text:
title = span_text.get_text(strip=True) or title
return title
def _parse_html_nav_li(
self,
li_element,
ordered_entries,
doc_order,
doc_order_decoded,
tree_structure_list,
find_position_func,
):
"""
Recursive parsing of HTML5 Navigation (li) nodes.
Logic tested by: tests/test_epub_html_nav_parsing.py
"""
link = li_element.find("a", recursive=False)
span_text = li_element.find("span", recursive=False)
src = None
current_entry_node = {"children": []}
if link and "href" in link.attrs:
src = link["href"]
title = self._extract_nav_li_title(li_element, link, span_text)
current_entry_node["title"] = title
current_entry_node["src"] = src
doc_key = None
doc_idx = None
position = 0
fragment = None
if src:
base_href, fragment = src.split("#", 1) if "#" in src else (src, None)
doc_key, doc_idx = self._find_doc_key(base_href, doc_order, doc_order_decoded)
if doc_key is not None:
position = find_position_func(doc_key, fragment)
entry_data = {
"src": src,
"title": title,
"doc_href": doc_key,
"position": position,
"doc_order": doc_idx,
}
ordered_entries.append(entry_data)
current_entry_node["has_content"] = True
else:
current_entry_node["has_content"] = False
else:
current_entry_node["has_content"] = False
for child_ol in li_element.find_all("ol", recursive=False):
for child_li in child_ol.find_all("li", recursive=False):
self._parse_html_nav_li(
child_li,
ordered_entries,
doc_order,
doc_order_decoded,
current_entry_node["children"],
find_position_func,
)
tree_structure_list.append(current_entry_node)
def _identify_nav_item(self):
"""Identify the navigation item (HTML Nav or NCX) and its type."""
nav_item = None
nav_type = None
# 1. Check ITEM_NAVIGATION
nav_items = list(self.book.get_items_of_type(ebooklib.ITEM_NAVIGATION))
# 1.1 Support for EPUB 3 EpubNav which might be ITEM_DOCUMENT (9) but with properties=['nav']
if not nav_items:
# Look in ITEM_DOCUMENT for items with 'nav' property
for item in self.book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
if hasattr(item, 'get_type') and item.get_type() == ebooklib.ITEM_DOCUMENT:
# Check properties - ebooklib stores opf properties in list
# Some versions use item.properties, some need checking
props = getattr(item, 'properties', [])
if 'nav' in props:
nav_items.append(item)
if nav_items:
nav_item = next(
(
item
for item in nav_items
if "nav" in item.get_name().lower()
and item.get_name().lower().endswith((".xhtml", ".html"))
),
None,
) or next(
(
item
for item in nav_items
if item.get_name().lower().endswith((".xhtml", ".html"))
),
None,
)
if nav_item:
nav_type = "html"
# 2. NCX in NAV
if not nav_item and nav_items:
ncx_in_nav = next(
(item for item in nav_items if item.get_name().lower().endswith(".ncx")),
None,
)
if ncx_in_nav:
nav_item = ncx_in_nav
nav_type = "ncx"
# 3. ITEM_NCX or Fallback
# If no explicit navigation item found, try to find a standard NCX file
if not nav_item:
ncx_constant = getattr(epub, "ITEM_NCX", None)
if ncx_constant is not None:
ncx_items = list(self.book.get_items_of_type(ncx_constant))
if ncx_items:
nav_item = ncx_items[0]
nav_type = "ncx"
# 4. Heuristic Search
# Scan documents for something that looks like a TOC if standard methods fail
if not nav_item:
for item in self.book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
try:
html_content = item.get_content().decode("utf-8", errors="ignore")
if "<nav" in html_content and 'epub:type="toc"' in html_content:
nav_item = item
nav_type = "html"
break
except Exception:
continue
if not nav_item or not nav_type:
raise ValueError("No navigation document found")
return nav_item, nav_type
def _execute_nav_parsing_logic(self, nav_item, nav_type):
"""Parse the identified navigation item and slice content accordingly."""
parser_type = "html.parser" if nav_type == "html" else "xml"
try:
nav_content = nav_item.get_content().decode("utf-8", errors="ignore")
nav_soup = BeautifulSoup(nav_content, parser_type)
except Exception as e:
raise ValueError(f"Failed to parse navigation content: {e}")
self.doc_content = {}
spine_docs = []
for spine_item_tuple in self.book.spine:
item_id = spine_item_tuple[0]
item = self.book.get_item_with_id(item_id)
if item:
spine_docs.append(item.get_name())
doc_order = {href: i for i, href in enumerate(spine_docs)}
doc_order_decoded = {
urllib.parse.unquote(href): i for href, i in doc_order.items()
}
self.content_texts = {}
self.content_lengths = {}
for item in self.book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
href = item.get_name()
if href in doc_order or any(
href in nav_point.get("src", "")
for nav_point in nav_soup.find_all(["content", "a"])
):
try:
self.doc_content[href] = item.get_content().decode(
"utf-8", errors="ignore"
)
except Exception:
self.doc_content[href] = ""
ordered_nav_entries = []
parse_successful = False
if nav_type == "ncx":
nav_map = nav_soup.find("navMap")
if nav_map:
for nav_point in nav_map.find_all("navPoint", recursive=False):
self._parse_ncx_navpoint(
nav_point,
ordered_nav_entries,
doc_order,
doc_order_decoded,
self.processed_nav_structure,
self._find_position_robust,
)
parse_successful = bool(ordered_nav_entries)
elif nav_type == "html":
toc_nav = nav_soup.find("nav", attrs={"epub:type": "toc"})
if not toc_nav:
for nav in nav_soup.find_all("nav"):
if nav.find("ol"):
toc_nav = nav
break
if toc_nav:
top_ol = toc_nav.find("ol", recursive=False)
if top_ol:
for li in top_ol.find_all("li", recursive=False):
self._parse_html_nav_li(
li,
ordered_nav_entries,
doc_order,
doc_order_decoded,
self.processed_nav_structure,
self._find_position_robust,
)
parse_successful = bool(ordered_nav_entries)
if not parse_successful:
raise ValueError("No valid navigation entries found after parsing")
ordered_nav_entries.sort(key=lambda x: (x["doc_order"], x["position"]))
num_entries = len(ordered_nav_entries)
for i in range(num_entries):
current_entry = ordered_nav_entries[i]
current_src = current_entry["src"]
current_doc = current_entry["doc_href"]
current_pos = current_entry["position"]
current_doc_html = self.doc_content.get(current_doc, "")
start_slice_pos = current_pos
slice_html = ""
next_entry = ordered_nav_entries[i + 1] if (i + 1) < num_entries else None
if next_entry:
next_doc = next_entry["doc_href"]
next_pos = next_entry["position"]
if current_doc == next_doc:
slice_html = current_doc_html[start_slice_pos:next_pos]
else:
slice_html = current_doc_html[start_slice_pos:]
docs_between = []
try:
idx_current = spine_docs.index(current_doc)
idx_next = spine_docs.index(next_doc)
if idx_current < idx_next:
docs_between = [
spine_docs[k] for k in range(idx_current + 1, idx_next)
]
elif idx_current > idx_next:
docs_between = [
spine_docs[k]
for k in range(idx_current + 1, len(spine_docs))
]
docs_between.extend(
[spine_docs[k] for k in range(0, idx_next)]
)
except ValueError:
pass
for doc_href in docs_between:
slice_html += self.doc_content.get(doc_href, "")
next_doc_html = self.doc_content.get(next_doc, "")
slice_html += next_doc_html[:next_pos]
else:
slice_html = current_doc_html[start_slice_pos:]
try:
idx_current = spine_docs.index(current_doc)
for doc_idx in range(idx_current + 1, len(spine_docs)):
slice_html += self.doc_content.get(spine_docs[doc_idx], "")
except ValueError:
pass
if not slice_html.strip() and current_doc_html:
slice_html = current_doc_html
if slice_html.strip():
slice_soup = BeautifulSoup(slice_html, "html.parser")
for tag in slice_soup.find_all(["p", "div"]):
tag.append("\n\n")
for ol in slice_soup.find_all("ol"):
start = int(ol.get("start", 1))
for idx, li in enumerate(ol.find_all("li", recursive=False)):
number_text = f"{start + idx}) "
if li.string:
li.string.replace_with(number_text + li.string)
else:
li.insert(0, NavigableString(number_text))
for tag in slice_soup.find_all(["sup", "sub"]):
tag.decompose()
text = clean_text(slice_soup.get_text()).strip()
if text:
self.content_texts[current_src] = text
self.content_lengths[current_src] = calculate_text_length(text)
else:
self.content_texts[current_src] = ""
self.content_lengths[current_src] = 0
else:
self.content_texts[current_src] = ""
self.content_lengths[current_src] = 0
if ordered_nav_entries:
first_entry = ordered_nav_entries[0]
first_doc_href = first_entry["doc_href"]
first_pos = first_entry["position"]
first_doc_order = first_entry["doc_order"]
prefix_html = ""
for doc_idx in range(first_doc_order):
if doc_idx < len(spine_docs):
intermediate_doc_href = spine_docs[doc_idx]
prefix_html += self.doc_content.get(intermediate_doc_href, "")
first_doc_html = self.doc_content.get(first_doc_href, "")
prefix_html += first_doc_html[:first_pos]
if prefix_html.strip():
prefix_soup = BeautifulSoup(prefix_html, "html.parser")
for tag in prefix_soup.find_all(["sup", "sub"]):
tag.decompose()
prefix_text = clean_text(prefix_soup.get_text()).strip()
if prefix_text:
prefix_chapter_src = "internal:prefix_content"
self.content_texts[prefix_chapter_src] = prefix_text
self.content_lengths[prefix_chapter_src] = len(prefix_text)
self.processed_nav_structure.insert(
0,
{
"src": prefix_chapter_src,
"title": "Introduction",
"children": [],
"has_content": True,
},
)
def _process_epub_content_spine_fallback(self):
"""
Process EPUB content using the spine (linear reading order)
when navigation processing fails.
"""
logging.info("Using spine fallback for EPUB processing.")
self.doc_content = {}
spine_docs = []
for spine_item_tuple in self.book.spine:
item_id = spine_item_tuple[0]
item = self.book.get_item_with_id(item_id)
if item:
spine_docs.append(item.get_name())
else:
logging.warning(f"Spine item with id '{item_id}' not found.")
for item in self.book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
href = item.get_name()
if href in spine_docs:
try:
html_content = item.get_content().decode("utf-8", errors="ignore")
self.doc_content[href] = html_content
except Exception:
self.doc_content[href] = ""
self.content_texts = {}
self.content_lengths = {}
for i, doc_href in enumerate(spine_docs):
html_content = self.doc_content.get(doc_href, "")
if html_content:
soup = BeautifulSoup(html_content, "html.parser")
# Handle ordered lists
for ol in soup.find_all("ol"):
start = int(ol.get("start", 1))
for idx, li in enumerate(ol.find_all("li", recursive=False)):
number_text = f"{start + idx}) "
if li.string:
li.string.replace_with(number_text + li.string)
else:
li.insert(0, NavigableString(number_text))
# Remove sup/sub
for tag in soup.find_all(["sup", "sub"]):
tag.decompose()
text = clean_text(soup.get_text()).strip()
if text:
self.content_texts[doc_href] = text
self.content_lengths[doc_href] = calculate_text_length(text)
def get_chapters(self):
chapters = super().get_chapters()
if not chapters:
# Use spine order fallback if no Nav structure
if self.book:
for spine_item_tuple in self.book.spine:
item_id = spine_item_tuple[0]
item = self.book.get_item_with_id(item_id)
if item:
href = item.get_name()
if href in self.content_texts:
chapters.append((href, href))
return chapters
def get_book_parser(book_path, file_type=None):
"""
Factory function to get the appropriate parser instance.
"""
book_path = os.path.normpath(os.path.abspath(book_path))
if not file_type:
if book_path.lower().endswith(".pdf"):
file_type = "pdf"
elif book_path.lower().endswith((".md", ".markdown")):
file_type = "markdown"
else:
file_type = "epub"
if file_type == "pdf":
return PdfParser(book_path)
elif file_type == "markdown":
return MarkdownParser(book_path)
elif file_type == "epub":
return EpubParser(book_path)
else:
raise ValueError(f"Unsupported file type: {file_type}")
+12 -5
View File
@@ -1,9 +1,16 @@
"""Legacy PyQt conversion helpers removed."""
"""Backwards-compatible re-export of conversion module.
The PyQt-based implementation lives in abogen.pyqt.conversion.
The web-based implementation is in abogen.webui.conversion_runner.
"""
from __future__ import annotations
def __getattr__(name: str): # pragma: no cover - compatibility shim
raise AttributeError(
"The PyQt-based conversion helpers were removed. Use the web service pipeline instead."
# Re-export PyQt conversion classes for backwards compatibility
from abogen.pyqt.conversion import ( # noqa: F401
ConversionThread,
VoicePreviewThread,
PlayAudioThread,
)
__all__ = ["ConversionThread", "VoicePreviewThread", "PlayAudioThread"]
+6 -10
View File
@@ -1,15 +1,11 @@
"""Legacy PyQt GUI module removed in favor of the web interface."""
"""Backwards-compatible re-export of the PyQt GUI.
The actual implementation lives in abogen.pyqt.gui.
"""
from __future__ import annotations
class abogen: # pragma: no cover - legacy entry point
"""Placeholder for the removed PyQt GUI class."""
def __init__(self, *_args, **_kwargs):
raise RuntimeError(
"The PyQt desktop interface has been removed. Please use the web UI instead."
)
from abogen.pyqt.gui import * # noqa: F401, F403
from abogen.pyqt.gui import abogen
__all__ = ["abogen"]
Binary file not shown.
Binary file not shown.
+590
View File
@@ -0,0 +1,590 @@
"""
Pre-download dialog and worker for Abogen
This module consolidates pre-download logic for Kokoro voices and model
and spaCy language models. The code favors clarity, avoids duplication,
and handles optional dependencies gracefully.
"""
from typing import List, Optional, Tuple
import importlib
import importlib.util
from PyQt6.QtWidgets import (
QDialog,
QVBoxLayout,
QHBoxLayout,
QLabel,
QPushButton,
QSpacerItem,
QSizePolicy,
)
from PyQt6.QtCore import QThread, pyqtSignal
from abogen.constants import COLORS, VOICES_INTERNAL
from abogen.spacy_utils import SPACY_MODELS
import abogen.hf_tracker
# Helpers
def _unique_sorted_models() -> List[str]:
"""Return a sorted list of unique spaCy model package names."""
return sorted(set(SPACY_MODELS.values()))
def _is_package_installed(pkg_name: str) -> bool:
"""Return True if a package with the given name can be imported (site-packages)."""
try:
return importlib.util.find_spec(pkg_name) is not None
except Exception:
return False
# NOTE: explicit HF cache helper removed; we use try_to_load_from_cache in-scope where needed
class PreDownloadWorker(QThread):
"""Worker thread to download required models/voices.
Emits human-readable messages via `progress`. Uses `category_done` to indicate
a category (voices/model/spacy) finished successfully. Emits `error` on exception
and `finished` after all work completes.
"""
# Emit (category, status, message)
progress = pyqtSignal(str, str, str)
category_done = pyqtSignal(str)
finished = pyqtSignal()
error = pyqtSignal(str)
def __init__(self, parent=None):
super().__init__(parent)
self._cancelled = False
# repo and filenames used for Kokoro model
self._repo_id = "hexgrad/Kokoro-82M"
self._model_files = ["kokoro-v1_0.pth", "config.json"]
# Track download success per category
self._voices_success = False
self._model_success = False
self._spacy_success = False
# Suppress HF tracker warnings during downloads
self._original_emitter = abogen.hf_tracker.show_warning_signal_emitter
def cancel(self) -> None:
self._cancelled = True
def run(self) -> None:
# Suppress HF tracker warnings during downloads
abogen.hf_tracker.show_warning_signal_emitter = None
try:
self._download_kokoro_voices()
if self._cancelled:
return
if self._voices_success:
self.category_done.emit("voices")
self._download_kokoro_model()
if self._cancelled:
return
if self._model_success:
self.category_done.emit("model")
self._download_spacy_models()
if self._cancelled:
return
if self._spacy_success:
self.category_done.emit("spacy")
self.finished.emit()
except Exception as exc: # pragma: no cover - best-effort reporting
self.error.emit(str(exc))
finally:
# Restore original emitter
abogen.hf_tracker.show_warning_signal_emitter = self._original_emitter
# Kokoro voices
def _download_kokoro_voices(self) -> None:
self._voices_success = True
try:
from huggingface_hub import hf_hub_download, try_to_load_from_cache
except Exception:
self.progress.emit(
"voice", "warning", "huggingface_hub not installed, skipping voices..."
)
self._voices_success = False
return
voice_list = VOICES_INTERNAL
for idx, voice in enumerate(voice_list, start=1):
if self._cancelled:
self._voices_success = False
return
filename = f"voices/{voice}.pt"
if try_to_load_from_cache(repo_id=self._repo_id, filename=filename):
self.progress.emit(
"voice",
"installed",
f"{idx}/{len(voice_list)}: {voice} already present",
)
continue
self.progress.emit(
"voice", "downloading", f"{idx}/{len(voice_list)}: {voice}..."
)
try:
hf_hub_download(repo_id=self._repo_id, filename=filename)
self.progress.emit("voice", "downloaded", f"{voice} downloaded")
except Exception as exc:
self.progress.emit(
"voice", "warning", f"could not download {voice}: {exc}"
)
self._voices_success = False
# Kokoro model
def _download_kokoro_model(self) -> None:
self._model_success = True
try:
from huggingface_hub import hf_hub_download, try_to_load_from_cache
except Exception:
self.progress.emit(
"model", "warning", "huggingface_hub not installed, skipping model..."
)
self._model_success = False
return
for fname in self._model_files:
if self._cancelled:
self._model_success = False
return
category = "config" if fname == "config.json" else "model"
if try_to_load_from_cache(repo_id=self._repo_id, filename=fname):
self.progress.emit(
category, "installed", f"file {fname} already present"
)
continue
self.progress.emit(category, "downloading", f"file {fname}...")
try:
hf_hub_download(repo_id=self._repo_id, filename=fname)
self.progress.emit(category, "downloaded", f"file {fname} downloaded")
except Exception as exc:
self.progress.emit(
category, "warning", f"could not download file {fname}: {exc}"
)
self._model_success = False
# spaCy models
def _download_spacy_models(self) -> None:
"""Download spaCy models. Prefer missing models provided by parent.
Parent dialog will populate _spacy_models_missing during checking.
"""
self._spacy_success = True
# Determine which models to process: prefer parent-provided missing list to avoid
# re-checking everything; otherwise use the full unique list.
parent = self.parent()
models_to_process: List[str] = _unique_sorted_models()
try:
if (
parent is not None
and hasattr(parent, "_spacy_models_missing")
and parent._spacy_models_missing
):
models_to_process = list(dict.fromkeys(parent._spacy_models_missing))
except Exception:
pass
# If spaCy is not available to run the CLI, skip gracefully
try:
import spacy.cli as _spacy_cli
except Exception:
self.progress.emit(
"spacy", "warning", "spaCy not available, skipping spaCy models..."
)
self._spacy_success = False
return
for idx, model_name in enumerate(models_to_process, start=1):
if self._cancelled:
self._spacy_success = False
return
if _is_package_installed(model_name):
self.progress.emit(
"spacy",
"installed",
f"{idx}/{len(models_to_process)}: {model_name} already installed",
)
continue
self.progress.emit(
"spacy",
"downloading",
f"{idx}/{len(models_to_process)}: {model_name}...",
)
try:
_spacy_cli.download(model_name)
self.progress.emit("spacy", "downloaded", f"{model_name} downloaded")
except Exception as exc:
self.progress.emit(
"spacy", "warning", f"could not download {model_name}: {exc}"
)
self._spacy_success = False
class PreDownloadDialog(QDialog):
"""Dialog to show and control pre-download process."""
VOICE_PREFIX = "Kokoro voices: "
MODEL_PREFIX = "Kokoro model: "
CONFIG_PREFIX = "Kokoro config: "
SPACY_PREFIX = "spaCy models: "
def __init__(self, parent=None):
super().__init__(parent)
self.setWindowTitle("Pre-download Models and Voices")
self.setMinimumWidth(500)
self.worker: Optional[PreDownloadWorker] = None
self.has_missing = False
self._spacy_models_checked: List[tuple] = []
self._spacy_models_missing: List[str] = []
self._status_worker = None
# Map keywords to (label, prefix) - labels filled after UI creation
self.status_map = {
"voice": (None, self.VOICE_PREFIX),
"spacy": (None, self.SPACY_PREFIX),
"model": (None, self.MODEL_PREFIX),
"config": (None, self.CONFIG_PREFIX),
}
self.category_map = {
"voices": ["voice"],
"model": ["model", "config"],
"spacy": ["spacy"],
}
self._setup_ui()
self._start_status_check()
def _setup_ui(self) -> None:
layout = QVBoxLayout(self)
layout.setSpacing(0)
layout.setContentsMargins(15, 0, 15, 15)
desc = QLabel(
"You can pre-download all required models and voices for offline use.\n"
"This includes Kokoro voices, Kokoro model (and config), and spaCy models."
)
desc.setWordWrap(True)
layout.addWidget(desc)
# Status rows
status_layout = QVBoxLayout()
status_title = QLabel("<b>Current Status:</b>")
status_layout.addWidget(status_title)
self.voices_status = QLabel(self.VOICE_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.voices_status)
row.addStretch()
status_layout.addLayout(row)
self.model_status = QLabel(self.MODEL_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.model_status)
row.addStretch()
status_layout.addLayout(row)
self.config_status = QLabel(self.CONFIG_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.config_status)
row.addStretch()
status_layout.addLayout(row)
self.spacy_status = QLabel(self.SPACY_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.spacy_status)
row.addStretch()
status_layout.addLayout(row)
# register labels
self.status_map["voice"] = (self.voices_status, self.VOICE_PREFIX)
self.status_map["model"] = (self.model_status, self.MODEL_PREFIX)
self.status_map["config"] = (self.config_status, self.CONFIG_PREFIX)
self.status_map["spacy"] = (self.spacy_status, self.SPACY_PREFIX)
layout.addLayout(status_layout)
layout.addItem(
QSpacerItem(0, 20, QSizePolicy.Policy.Minimum, QSizePolicy.Policy.Fixed)
)
# Buttons
button_row = QHBoxLayout()
button_row.setSpacing(10)
self.download_btn = QPushButton("Download all")
self.download_btn.setMinimumWidth(100)
self.download_btn.setMinimumHeight(35)
self.download_btn.setEnabled(False)
self.download_btn.clicked.connect(self._start_download)
button_row.addWidget(self.download_btn)
self.close_btn = QPushButton("Close")
self.close_btn.setMinimumWidth(100)
self.close_btn.setMinimumHeight(35)
self.close_btn.clicked.connect(self._handle_close)
button_row.addWidget(self.close_btn)
layout.addLayout(button_row)
self.adjustSize()
# Status checking worker
class StatusCheckWorker(QThread):
voices_checked = pyqtSignal(bool, list)
model_checked = pyqtSignal(bool)
config_checked = pyqtSignal(bool)
spacy_model_checking = pyqtSignal(str)
spacy_model_result = pyqtSignal(str, bool)
spacy_checked = pyqtSignal(bool, list)
def run(self):
parent = self.parent()
if parent is None:
return
voices_ok, missing_voices = parent._check_kokoro_voices()
self.voices_checked.emit(voices_ok, missing_voices)
model_ok = parent._check_kokoro_model()
self.model_checked.emit(model_ok)
config_ok = parent._check_kokoro_config()
self.config_checked.emit(config_ok)
# Check spaCy models by package name to detect site-package installs
unique = _unique_sorted_models()
missing: List[str] = []
for name in unique:
self.spacy_model_checking.emit(name)
ok = _is_package_installed(name)
self.spacy_model_result.emit(name, ok)
if not ok:
missing.append(name)
parent._spacy_models_missing = missing
self.spacy_checked.emit(len(missing) == 0, missing)
def _start_status_check(self) -> None:
self._status_worker = self.StatusCheckWorker(self)
self._status_worker.voices_checked.connect(self._update_voices_status)
self._status_worker.model_checked.connect(self._update_model_status)
self._status_worker.config_checked.connect(self._update_config_status)
self._status_worker.spacy_model_checking.connect(self._spacy_model_checking)
self._status_worker.spacy_model_result.connect(self._spacy_model_result)
self._status_worker.spacy_checked.connect(self._update_spacy_status)
# These are initialized in __init__ to keep consistent object state
# Set checking visual state
for lbl in (
self.voices_status,
self.model_status,
self.config_status,
self.spacy_status,
):
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
self.spacy_status.setText(self.SPACY_PREFIX + "⏳ Checking...")
self._status_worker.start()
# UI update callbacks
def _spacy_model_checking(self, name: str) -> None:
self.spacy_status.setText(f"{self.SPACY_PREFIX}Checking {name}...")
def _spacy_model_result(self, name: str, ok: bool) -> None:
self._spacy_models_checked.append((name, ok))
if not ok and name not in self._spacy_models_missing:
self._spacy_models_missing.append(name)
checked = len(self._spacy_models_checked)
missing_count = len(self._spacy_models_missing)
if missing_count:
self.spacy_status.setText(
f"{self.SPACY_PREFIX}{checked} checked, {missing_count} missing..."
)
else:
self.spacy_status.setText(f"{self.SPACY_PREFIX}{checked} checked...")
def _update_voices_status(self, ok: bool, missing: List[str]) -> None:
if ok:
self._set_status("voice", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
if missing:
self._set_status(
"voice", f"✗ Missing {len(missing)} voices", COLORS["RED"]
)
else:
self._set_status("voice", "✗ Not downloaded", COLORS["RED"])
def _update_model_status(self, ok: bool) -> None:
if ok:
self._set_status("model", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
self._set_status("model", "✗ Not downloaded", COLORS["RED"])
def _update_config_status(self, ok: bool) -> None:
if ok:
self._set_status("config", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
self._set_status("config", "✗ Not downloaded", COLORS["RED"])
def _update_spacy_status(self, ok: bool, missing: List[str]) -> None:
if ok:
self._set_status("spacy", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
if missing:
self._set_status(
"spacy", f"✗ Missing {len(missing)} model(s)", COLORS["RED"]
)
else:
self._set_status("spacy", "✗ Not downloaded", COLORS["RED"])
self.download_btn.setEnabled(self.has_missing)
def _set_status(self, key: str, text: str, color: str) -> None:
lbl, prefix = self.status_map.get(key, (None, ""))
if not lbl:
return
lbl.setText(prefix + text)
lbl.setStyleSheet(f"color: {color};")
# Helper checks
def _check_kokoro_voices(self) -> Tuple[bool, List[str]]:
"""Return (ok, missing_list) for Kokoro voices check."""
missing = []
try:
from huggingface_hub import try_to_load_from_cache
for voice in VOICES_INTERNAL:
if not try_to_load_from_cache(
repo_id="hexgrad/Kokoro-82M", filename=f"voices/{voice}.pt"
):
missing.append(voice)
except Exception:
# If HF missing, report all as missing
return False, list(VOICES_INTERNAL)
return (len(missing) == 0), missing
def _check_kokoro_model(self) -> bool:
try:
from huggingface_hub import try_to_load_from_cache
return (
try_to_load_from_cache(
repo_id="hexgrad/Kokoro-82M", filename="kokoro-v1_0.pth"
)
is not None
)
except Exception:
return False
def _check_kokoro_config(self) -> bool:
try:
from huggingface_hub import try_to_load_from_cache
return (
try_to_load_from_cache(
repo_id="hexgrad/Kokoro-82M", filename="config.json"
)
is not None
)
except Exception:
return False
def _check_spacy_models(self) -> bool:
unique = _unique_sorted_models()
missing = [m for m in unique if not _is_package_installed(m)]
self._spacy_models_missing = missing
return len(missing) == 0
# Download control
def _start_download(self) -> None:
self.download_btn.setEnabled(False)
self.download_btn.setText("Downloading...")
# mark the start of downloads; this triggers the labels
self._on_progress("system", "starting", "Processing, please wait...")
self.worker = PreDownloadWorker(self)
self.worker.progress.connect(self._on_progress)
self.worker.category_done.connect(self._on_category_done)
self.worker.finished.connect(self._on_download_finished)
self.worker.error.connect(self._on_download_error)
self.worker.start()
def _on_progress(self, category: str, status: str, message: str) -> None:
"""Map worker (category, status, message) to UI label updates.
Status is one of: 'downloading', 'installed', 'downloaded', 'warning', 'starting'.
Category is one of: 'voice', 'model', 'spacy', 'config', or 'system'.
"""
try:
# If the category targets a specific label, update directly
if category in self.status_map:
lbl, prefix = self.status_map[category]
if not lbl:
return
# Compose message and set color based on status token
full_text = prefix + message
if len(full_text) > 60:
display_text = full_text[:57] + "..."
lbl.setText(display_text)
lbl.setToolTip(full_text)
else:
lbl.setText(full_text)
lbl.setToolTip("") # Clear tooltip if not needed
if status == "downloading":
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
elif status in ("installed", "downloaded"):
lbl.setStyleSheet(f"color: {COLORS['GREEN']};")
elif status == "warning":
lbl.setStyleSheet(f"color: {COLORS['RED']};")
elif status == "error":
lbl.setStyleSheet(f"color: {COLORS['RED']};")
return
# System-level messages
if category == "system":
if status == "starting":
for k in self.status_map:
lbl, prefix = self.status_map[k]
if lbl:
lbl.setText(prefix + "Processing, please wait...")
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
# other system statuses don't require action
return
except Exception:
# Do not let UI thread crash on unexpected worker message
pass
def _on_category_done(self, category: str) -> None:
for key in self.category_map.get(category, []):
self._set_status(key, "✓ Downloaded", COLORS["GREEN"])
def _on_download_finished(self) -> None:
self.has_missing = False
self.download_btn.setText("Download all")
self.download_btn.setEnabled(False)
def _on_download_error(self, error_msg: str) -> None:
self.download_btn.setText("Download all")
self.download_btn.setEnabled(True)
for key in self.status_map:
self._set_status(key, f"✗ Error - {error_msg}", COLORS["RED"])
def _handle_close(self) -> None:
if self.worker and self.worker.isRunning():
self.worker.cancel()
self.worker.wait(2000)
self.accept()
def closeEvent(self, event) -> None:
if self.worker and self.worker.isRunning():
self.worker.cancel()
self.worker.wait(2000)
super().closeEvent(event)
+7
View File
@@ -0,0 +1,7 @@
"""PyQt6 Desktop GUI for abogen.
This package contains the traditional PyQt6-based desktop interface.
For the web-based interface, see abogen.webui.
"""
from __future__ import annotations
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+4049
View File
File diff suppressed because it is too large Load Diff
+187
View File
@@ -0,0 +1,187 @@
import os
import sys
import platform
import atexit
import signal
from abogen.utils import get_resource_path, load_config, prevent_sleep_end
# Fix PyTorch DLL loading issue ([WinError 1114]) on Windows before importing PyQt6
if platform.system() == "Windows":
import ctypes
from importlib.util import find_spec
try:
if (
(spec := find_spec("torch"))
and spec.origin
and os.path.exists(
dll_path := os.path.join(os.path.dirname(spec.origin), "lib", "c10.dll")
)
):
ctypes.CDLL(os.path.normpath(dll_path))
except Exception:
pass
# Qt platform plugin detection (fixes #59)
try:
from PyQt6.QtCore import QLibraryInfo
# Get the path to the plugins directory
plugins = QLibraryInfo.path(QLibraryInfo.LibraryPath.PluginsPath)
# Normalize path to use the OS-native separators and absolute path
platform_dir = os.path.normpath(os.path.join(plugins, "platforms"))
# Ensure we work with an absolute path for clarity
platform_dir = os.path.abspath(platform_dir)
if os.path.isdir(platform_dir):
os.environ["QT_QPA_PLATFORM_PLUGIN_PATH"] = platform_dir
print("QT_QPA_PLATFORM_PLUGIN_PATH set to:", platform_dir)
else:
print("PyQt6 platform plugins not found at", platform_dir)
except ImportError:
print("PyQt6 not installed.")
# Pre-load "libxcb-cursor" on Linux (fixes #101)
if platform.system() == "Linux":
arch = platform.machine().lower()
lib_filename = {"x86_64": "libxcb-cursor-amd64.so.0", "amd64": "libxcb-cursor-amd64.so.0", "aarch64": "libxcb-cursor-arm64.so.0", "arm64": "libxcb-cursor-arm64.so.0"}.get(arch)
if lib_filename:
import ctypes
try:
# Try to load the system libxcb-cursor.so.0 first
ctypes.CDLL('libxcb-cursor.so.0', mode=ctypes.RTLD_GLOBAL)
except OSError:
# System lib not available, load the bundled version
lib_path = get_resource_path('abogen.libs', lib_filename)
if lib_path:
try:
ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
except OSError:
# If it fails (e.g. wrong glibc version on very old systems),
# we simply ignore it and hope the system has the library.
pass
# Set application ID for Windows taskbar icon
if platform.system() == "Windows":
try:
from abogen.constants import PROGRAM_NAME, VERSION
import ctypes
app_id = f"{PROGRAM_NAME}.{VERSION}"
ctypes.windll.shell32.SetCurrentProcessExplicitAppUserModelID(app_id)
except Exception as e:
print("Warning: failed to set AppUserModelID:", e)
from PyQt6.QtWidgets import QApplication
from PyQt6.QtGui import QIcon
from PyQt6.QtCore import (
QLibraryInfo,
qInstallMessageHandler,
QtMsgType,
)
# Add the directory to Python path
sys.path.insert(0, os.path.join(os.path.dirname(__file__)))
# Set Hugging Face Hub environment variables
os.environ["HF_HUB_DISABLE_TELEMETRY"] = "1" # Disable Hugging Face telemetry
os.environ["HF_HUB_ETAG_TIMEOUT"] = "10" # Metadata request timeout (seconds)
os.environ["HF_HUB_DOWNLOAD_TIMEOUT"] = "10" # File download timeout (seconds)
os.environ["HF_HUB_DISABLE_SYMLINKS_WARNING"] = "1" # Disable symlinks warning
if load_config().get("disable_kokoro_internet", False):
print("INFO: Kokoro's internet access is disabled.")
os.environ["HF_HUB_OFFLINE"] = "1" # Disable Hugging Face Hub internet access
from abogen.pyqt.gui import abogen
from abogen.constants import PROGRAM_NAME, VERSION
# Set environment variables for AMD ROCm
os.environ["MIOPEN_FIND_MODE"] = "FAST"
os.environ["MIOPEN_CONV_PRECISE_ROCM_TUNING"] = "0"
# Reset sleep states
atexit.register(prevent_sleep_end)
# Also handle signals (Ctrl+C, kill, etc.)
def _cleanup_sleep(signum, frame):
prevent_sleep_end()
sys.exit(0)
signal.signal(signal.SIGINT, _cleanup_sleep)
signal.signal(signal.SIGTERM, _cleanup_sleep)
# Ensure sys.stdout and sys.stderr are valid in GUI mode
if sys.stdout is None:
sys.stdout = open(os.devnull, "w")
if sys.stderr is None:
sys.stderr = open(os.devnull, "w")
# Enable MPS GPU acceleration on Mac Apple Silicon
if platform.system() == "Darwin" and platform.processor() == "arm":
os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"
# Custom message handler to filter out specific Qt warnings
def qt_message_handler(mode, context, message):
# In PyQt6, the mode is an enum, so we compare with the enum members
if "Wayland does not support QWindow::requestActivate()" in message:
return # Suppress this specific message
if "setGrabPopup called with a parent, QtWaylandClient" in message:
return
if mode == QtMsgType.QtWarningMsg:
print(f"Qt Warning: {message}")
elif mode == QtMsgType.QtCriticalMsg:
print(f"Qt Critical: {message}")
elif mode == QtMsgType.QtFatalMsg:
print(f"Qt Fatal: {message}")
elif mode == QtMsgType.QtInfoMsg:
print(f"Qt Info: {message}")
# Install the custom message handler
qInstallMessageHandler(qt_message_handler)
# Handle Wayland on Linux GNOME
if platform.system() == "Linux":
xdg_session = os.environ.get("XDG_SESSION_TYPE", "").lower()
desktop = os.environ.get("XDG_CURRENT_DESKTOP", "").lower()
if (
"gnome" in desktop
and xdg_session == "wayland"
and "QT_QPA_PLATFORM" not in os.environ
):
os.environ["QT_QPA_PLATFORM"] = "wayland"
def main():
"""Main entry point for console usage."""
app = QApplication(sys.argv)
# Set application icon using get_resource_path from utils
icon_path = get_resource_path("abogen.assets", "icon.ico")
if icon_path:
app.setWindowIcon(QIcon(icon_path))
# Set the .desktop name on Linux
if platform.system() == "Linux":
try:
app.setDesktopFileName("abogen")
except AttributeError:
pass
ex = abogen()
ex.show()
sys.exit(app.exec())
if __name__ == "__main__":
main()
+590
View File
@@ -0,0 +1,590 @@
"""
Pre-download dialog and worker for Abogen
This module consolidates pre-download logic for Kokoro voices and model
and spaCy language models. The code favors clarity, avoids duplication,
and handles optional dependencies gracefully.
"""
from typing import List, Optional, Tuple
import importlib
import importlib.util
from PyQt6.QtWidgets import (
QDialog,
QVBoxLayout,
QHBoxLayout,
QLabel,
QPushButton,
QSpacerItem,
QSizePolicy,
)
from PyQt6.QtCore import QThread, pyqtSignal
from abogen.constants import COLORS, VOICES_INTERNAL
from abogen.spacy_utils import SPACY_MODELS
import abogen.hf_tracker
# Helpers
def _unique_sorted_models() -> List[str]:
"""Return a sorted list of unique spaCy model package names."""
return sorted(set(SPACY_MODELS.values()))
def _is_package_installed(pkg_name: str) -> bool:
"""Return True if a package with the given name can be imported (site-packages)."""
try:
return importlib.util.find_spec(pkg_name) is not None
except Exception:
return False
# NOTE: explicit HF cache helper removed; we use try_to_load_from_cache in-scope where needed
class PreDownloadWorker(QThread):
"""Worker thread to download required models/voices.
Emits human-readable messages via `progress`. Uses `category_done` to indicate
a category (voices/model/spacy) finished successfully. Emits `error` on exception
and `finished` after all work completes.
"""
# Emit (category, status, message)
progress = pyqtSignal(str, str, str)
category_done = pyqtSignal(str)
finished = pyqtSignal()
error = pyqtSignal(str)
def __init__(self, parent=None):
super().__init__(parent)
self._cancelled = False
# repo and filenames used for Kokoro model
self._repo_id = "hexgrad/Kokoro-82M"
self._model_files = ["kokoro-v1_0.pth", "config.json"]
# Track download success per category
self._voices_success = False
self._model_success = False
self._spacy_success = False
# Suppress HF tracker warnings during downloads
self._original_emitter = abogen.hf_tracker.show_warning_signal_emitter
def cancel(self) -> None:
self._cancelled = True
def run(self) -> None:
# Suppress HF tracker warnings during downloads
abogen.hf_tracker.show_warning_signal_emitter = None
try:
self._download_kokoro_voices()
if self._cancelled:
return
if self._voices_success:
self.category_done.emit("voices")
self._download_kokoro_model()
if self._cancelled:
return
if self._model_success:
self.category_done.emit("model")
self._download_spacy_models()
if self._cancelled:
return
if self._spacy_success:
self.category_done.emit("spacy")
self.finished.emit()
except Exception as exc: # pragma: no cover - best-effort reporting
self.error.emit(str(exc))
finally:
# Restore original emitter
abogen.hf_tracker.show_warning_signal_emitter = self._original_emitter
# Kokoro voices
def _download_kokoro_voices(self) -> None:
self._voices_success = True
try:
from huggingface_hub import hf_hub_download, try_to_load_from_cache
except Exception:
self.progress.emit(
"voice", "warning", "huggingface_hub not installed, skipping voices..."
)
self._voices_success = False
return
voice_list = VOICES_INTERNAL
for idx, voice in enumerate(voice_list, start=1):
if self._cancelled:
self._voices_success = False
return
filename = f"voices/{voice}.pt"
if try_to_load_from_cache(repo_id=self._repo_id, filename=filename):
self.progress.emit(
"voice",
"installed",
f"{idx}/{len(voice_list)}: {voice} already present",
)
continue
self.progress.emit(
"voice", "downloading", f"{idx}/{len(voice_list)}: {voice}..."
)
try:
hf_hub_download(repo_id=self._repo_id, filename=filename)
self.progress.emit("voice", "downloaded", f"{voice} downloaded")
except Exception as exc:
self.progress.emit(
"voice", "warning", f"could not download {voice}: {exc}"
)
self._voices_success = False
# Kokoro model
def _download_kokoro_model(self) -> None:
self._model_success = True
try:
from huggingface_hub import hf_hub_download, try_to_load_from_cache
except Exception:
self.progress.emit(
"model", "warning", "huggingface_hub not installed, skipping model..."
)
self._model_success = False
return
for fname in self._model_files:
if self._cancelled:
self._model_success = False
return
category = "config" if fname == "config.json" else "model"
if try_to_load_from_cache(repo_id=self._repo_id, filename=fname):
self.progress.emit(
category, "installed", f"file {fname} already present"
)
continue
self.progress.emit(category, "downloading", f"file {fname}...")
try:
hf_hub_download(repo_id=self._repo_id, filename=fname)
self.progress.emit(category, "downloaded", f"file {fname} downloaded")
except Exception as exc:
self.progress.emit(
category, "warning", f"could not download file {fname}: {exc}"
)
self._model_success = False
# spaCy models
def _download_spacy_models(self) -> None:
"""Download spaCy models. Prefer missing models provided by parent.
Parent dialog will populate _spacy_models_missing during checking.
"""
self._spacy_success = True
# Determine which models to process: prefer parent-provided missing list to avoid
# re-checking everything; otherwise use the full unique list.
parent = self.parent()
models_to_process: List[str] = _unique_sorted_models()
try:
if (
parent is not None
and hasattr(parent, "_spacy_models_missing")
and parent._spacy_models_missing
):
models_to_process = list(dict.fromkeys(parent._spacy_models_missing))
except Exception:
pass
# If spaCy is not available to run the CLI, skip gracefully
try:
import spacy.cli as _spacy_cli
except Exception:
self.progress.emit(
"spacy", "warning", "spaCy not available, skipping spaCy models..."
)
self._spacy_success = False
return
for idx, model_name in enumerate(models_to_process, start=1):
if self._cancelled:
self._spacy_success = False
return
if _is_package_installed(model_name):
self.progress.emit(
"spacy",
"installed",
f"{idx}/{len(models_to_process)}: {model_name} already installed",
)
continue
self.progress.emit(
"spacy",
"downloading",
f"{idx}/{len(models_to_process)}: {model_name}...",
)
try:
_spacy_cli.download(model_name)
self.progress.emit("spacy", "downloaded", f"{model_name} downloaded")
except Exception as exc:
self.progress.emit(
"spacy", "warning", f"could not download {model_name}: {exc}"
)
self._spacy_success = False
class PreDownloadDialog(QDialog):
"""Dialog to show and control pre-download process."""
VOICE_PREFIX = "Kokoro voices: "
MODEL_PREFIX = "Kokoro model: "
CONFIG_PREFIX = "Kokoro config: "
SPACY_PREFIX = "spaCy models: "
def __init__(self, parent=None):
super().__init__(parent)
self.setWindowTitle("Pre-download Models and Voices")
self.setMinimumWidth(500)
self.worker: Optional[PreDownloadWorker] = None
self.has_missing = False
self._spacy_models_checked: List[tuple] = []
self._spacy_models_missing: List[str] = []
self._status_worker = None
# Map keywords to (label, prefix) - labels filled after UI creation
self.status_map = {
"voice": (None, self.VOICE_PREFIX),
"spacy": (None, self.SPACY_PREFIX),
"model": (None, self.MODEL_PREFIX),
"config": (None, self.CONFIG_PREFIX),
}
self.category_map = {
"voices": ["voice"],
"model": ["model", "config"],
"spacy": ["spacy"],
}
self._setup_ui()
self._start_status_check()
def _setup_ui(self) -> None:
layout = QVBoxLayout(self)
layout.setSpacing(0)
layout.setContentsMargins(15, 0, 15, 15)
desc = QLabel(
"You can pre-download all required models and voices for offline use.\n"
"This includes Kokoro voices, Kokoro model (and config), and spaCy models."
)
desc.setWordWrap(True)
layout.addWidget(desc)
# Status rows
status_layout = QVBoxLayout()
status_title = QLabel("<b>Current Status:</b>")
status_layout.addWidget(status_title)
self.voices_status = QLabel(self.VOICE_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.voices_status)
row.addStretch()
status_layout.addLayout(row)
self.model_status = QLabel(self.MODEL_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.model_status)
row.addStretch()
status_layout.addLayout(row)
self.config_status = QLabel(self.CONFIG_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.config_status)
row.addStretch()
status_layout.addLayout(row)
self.spacy_status = QLabel(self.SPACY_PREFIX + "⏳ Checking...")
row = QHBoxLayout()
row.addWidget(self.spacy_status)
row.addStretch()
status_layout.addLayout(row)
# register labels
self.status_map["voice"] = (self.voices_status, self.VOICE_PREFIX)
self.status_map["model"] = (self.model_status, self.MODEL_PREFIX)
self.status_map["config"] = (self.config_status, self.CONFIG_PREFIX)
self.status_map["spacy"] = (self.spacy_status, self.SPACY_PREFIX)
layout.addLayout(status_layout)
layout.addItem(
QSpacerItem(0, 20, QSizePolicy.Policy.Minimum, QSizePolicy.Policy.Fixed)
)
# Buttons
button_row = QHBoxLayout()
button_row.setSpacing(10)
self.download_btn = QPushButton("Download all")
self.download_btn.setMinimumWidth(100)
self.download_btn.setMinimumHeight(35)
self.download_btn.setEnabled(False)
self.download_btn.clicked.connect(self._start_download)
button_row.addWidget(self.download_btn)
self.close_btn = QPushButton("Close")
self.close_btn.setMinimumWidth(100)
self.close_btn.setMinimumHeight(35)
self.close_btn.clicked.connect(self._handle_close)
button_row.addWidget(self.close_btn)
layout.addLayout(button_row)
self.adjustSize()
# Status checking worker
class StatusCheckWorker(QThread):
voices_checked = pyqtSignal(bool, list)
model_checked = pyqtSignal(bool)
config_checked = pyqtSignal(bool)
spacy_model_checking = pyqtSignal(str)
spacy_model_result = pyqtSignal(str, bool)
spacy_checked = pyqtSignal(bool, list)
def run(self):
parent = self.parent()
if parent is None:
return
voices_ok, missing_voices = parent._check_kokoro_voices()
self.voices_checked.emit(voices_ok, missing_voices)
model_ok = parent._check_kokoro_model()
self.model_checked.emit(model_ok)
config_ok = parent._check_kokoro_config()
self.config_checked.emit(config_ok)
# Check spaCy models by package name to detect site-package installs
unique = _unique_sorted_models()
missing: List[str] = []
for name in unique:
self.spacy_model_checking.emit(name)
ok = _is_package_installed(name)
self.spacy_model_result.emit(name, ok)
if not ok:
missing.append(name)
parent._spacy_models_missing = missing
self.spacy_checked.emit(len(missing) == 0, missing)
def _start_status_check(self) -> None:
self._status_worker = self.StatusCheckWorker(self)
self._status_worker.voices_checked.connect(self._update_voices_status)
self._status_worker.model_checked.connect(self._update_model_status)
self._status_worker.config_checked.connect(self._update_config_status)
self._status_worker.spacy_model_checking.connect(self._spacy_model_checking)
self._status_worker.spacy_model_result.connect(self._spacy_model_result)
self._status_worker.spacy_checked.connect(self._update_spacy_status)
# These are initialized in __init__ to keep consistent object state
# Set checking visual state
for lbl in (
self.voices_status,
self.model_status,
self.config_status,
self.spacy_status,
):
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
self.spacy_status.setText(self.SPACY_PREFIX + "⏳ Checking...")
self._status_worker.start()
# UI update callbacks
def _spacy_model_checking(self, name: str) -> None:
self.spacy_status.setText(f"{self.SPACY_PREFIX}Checking {name}...")
def _spacy_model_result(self, name: str, ok: bool) -> None:
self._spacy_models_checked.append((name, ok))
if not ok and name not in self._spacy_models_missing:
self._spacy_models_missing.append(name)
checked = len(self._spacy_models_checked)
missing_count = len(self._spacy_models_missing)
if missing_count:
self.spacy_status.setText(
f"{self.SPACY_PREFIX}{checked} checked, {missing_count} missing..."
)
else:
self.spacy_status.setText(f"{self.SPACY_PREFIX}{checked} checked...")
def _update_voices_status(self, ok: bool, missing: List[str]) -> None:
if ok:
self._set_status("voice", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
if missing:
self._set_status(
"voice", f"✗ Missing {len(missing)} voices", COLORS["RED"]
)
else:
self._set_status("voice", "✗ Not downloaded", COLORS["RED"])
def _update_model_status(self, ok: bool) -> None:
if ok:
self._set_status("model", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
self._set_status("model", "✗ Not downloaded", COLORS["RED"])
def _update_config_status(self, ok: bool) -> None:
if ok:
self._set_status("config", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
self._set_status("config", "✗ Not downloaded", COLORS["RED"])
def _update_spacy_status(self, ok: bool, missing: List[str]) -> None:
if ok:
self._set_status("spacy", "✓ Downloaded", COLORS["GREEN"])
else:
self.has_missing = True
if missing:
self._set_status(
"spacy", f"✗ Missing {len(missing)} model(s)", COLORS["RED"]
)
else:
self._set_status("spacy", "✗ Not downloaded", COLORS["RED"])
self.download_btn.setEnabled(self.has_missing)
def _set_status(self, key: str, text: str, color: str) -> None:
lbl, prefix = self.status_map.get(key, (None, ""))
if not lbl:
return
lbl.setText(prefix + text)
lbl.setStyleSheet(f"color: {color};")
# Helper checks
def _check_kokoro_voices(self) -> Tuple[bool, List[str]]:
"""Return (ok, missing_list) for Kokoro voices check."""
missing = []
try:
from huggingface_hub import try_to_load_from_cache
for voice in VOICES_INTERNAL:
if not try_to_load_from_cache(
repo_id="hexgrad/Kokoro-82M", filename=f"voices/{voice}.pt"
):
missing.append(voice)
except Exception:
# If HF missing, report all as missing
return False, list(VOICES_INTERNAL)
return (len(missing) == 0), missing
def _check_kokoro_model(self) -> bool:
try:
from huggingface_hub import try_to_load_from_cache
return (
try_to_load_from_cache(
repo_id="hexgrad/Kokoro-82M", filename="kokoro-v1_0.pth"
)
is not None
)
except Exception:
return False
def _check_kokoro_config(self) -> bool:
try:
from huggingface_hub import try_to_load_from_cache
return (
try_to_load_from_cache(
repo_id="hexgrad/Kokoro-82M", filename="config.json"
)
is not None
)
except Exception:
return False
def _check_spacy_models(self) -> bool:
unique = _unique_sorted_models()
missing = [m for m in unique if not _is_package_installed(m)]
self._spacy_models_missing = missing
return len(missing) == 0
# Download control
def _start_download(self) -> None:
self.download_btn.setEnabled(False)
self.download_btn.setText("Downloading...")
# mark the start of downloads; this triggers the labels
self._on_progress("system", "starting", "Processing, please wait...")
self.worker = PreDownloadWorker(self)
self.worker.progress.connect(self._on_progress)
self.worker.category_done.connect(self._on_category_done)
self.worker.finished.connect(self._on_download_finished)
self.worker.error.connect(self._on_download_error)
self.worker.start()
def _on_progress(self, category: str, status: str, message: str) -> None:
"""Map worker (category, status, message) to UI label updates.
Status is one of: 'downloading', 'installed', 'downloaded', 'warning', 'starting'.
Category is one of: 'voice', 'model', 'spacy', 'config', or 'system'.
"""
try:
# If the category targets a specific label, update directly
if category in self.status_map:
lbl, prefix = self.status_map[category]
if not lbl:
return
# Compose message and set color based on status token
full_text = prefix + message
if len(full_text) > 60:
display_text = full_text[:57] + "..."
lbl.setText(display_text)
lbl.setToolTip(full_text)
else:
lbl.setText(full_text)
lbl.setToolTip("") # Clear tooltip if not needed
if status == "downloading":
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
elif status in ("installed", "downloaded"):
lbl.setStyleSheet(f"color: {COLORS['GREEN']};")
elif status == "warning":
lbl.setStyleSheet(f"color: {COLORS['RED']};")
elif status == "error":
lbl.setStyleSheet(f"color: {COLORS['RED']};")
return
# System-level messages
if category == "system":
if status == "starting":
for k in self.status_map:
lbl, prefix = self.status_map[k]
if lbl:
lbl.setText(prefix + "Processing, please wait...")
lbl.setStyleSheet(f"color: {COLORS['ORANGE']};")
# other system statuses don't require action
return
except Exception:
# Do not let UI thread crash on unexpected worker message
pass
def _on_category_done(self, category: str) -> None:
for key in self.category_map.get(category, []):
self._set_status(key, "✓ Downloaded", COLORS["GREEN"])
def _on_download_finished(self) -> None:
self.has_missing = False
self.download_btn.setText("Download all")
self.download_btn.setEnabled(False)
def _on_download_error(self, error_msg: str) -> None:
self.download_btn.setText("Download all")
self.download_btn.setEnabled(True)
for key in self.status_map:
self._set_status(key, f"✗ Error - {error_msg}", COLORS["RED"])
def _handle_close(self) -> None:
if self.worker and self.worker.isRunning():
self.worker.cancel()
self.worker.wait(2000)
self.accept()
def closeEvent(self, event) -> None:
if self.worker and self.worker.isRunning():
self.worker.cancel()
self.worker.wait(2000)
super().closeEvent(event)
+860
View File
@@ -0,0 +1,860 @@
# a simple window with a list of items in the queue, no checkboxes
# button to remove an item from the queue
# button to clear the queue
from PyQt6.QtWidgets import (
QDialog,
QVBoxLayout,
QHBoxLayout,
QDialogButtonBox,
QPushButton,
QListWidget,
QListWidgetItem,
QFileIconProvider,
QLabel,
QWidget,
QSizePolicy,
QAbstractItemView,
QCheckBox,
)
from PyQt6.QtCore import QFileInfo, Qt
from abogen.constants import COLORS
from copy import deepcopy
from PyQt6.QtGui import QFontMetrics
from abogen.utils import load_config, save_config
# Define attributes that are safe to override with global settings
OVERRIDE_FIELDS = [
"lang_code",
"speed",
"voice",
"save_option",
"output_folder",
"subtitle_mode",
"output_format",
"replace_single_newlines",
"use_silent_gaps",
"subtitle_speed_method",
]
class ElidedLabel(QLabel):
def __init__(self, text):
super().__init__(text)
self._full_text = text
self.setSizePolicy(QSizePolicy.Policy.Expanding, QSizePolicy.Policy.Preferred)
self.setTextFormat(Qt.TextFormat.PlainText)
def setText(self, text):
self._full_text = text
super().setText(text)
self.update()
def resizeEvent(self, event):
metrics = QFontMetrics(self.font())
elided = metrics.elidedText(
self._full_text, Qt.TextElideMode.ElideRight, self.width()
)
super().setText(elided)
super().resizeEvent(event)
def fullText(self):
return self._full_text
class QueueListItemWidget(QWidget):
def __init__(self, file_name, char_count):
super().__init__()
layout = QHBoxLayout()
layout.setContentsMargins(12, 0, 6, 0)
layout.setSpacing(0)
import os
name_label = ElidedLabel(os.path.basename(file_name))
char_label = QLabel(f"Chars: {char_count}")
char_label.setStyleSheet(f"color: {COLORS['LIGHT_DISABLED']};")
char_label.setAlignment(
Qt.AlignmentFlag.AlignRight | Qt.AlignmentFlag.AlignVCenter
)
char_label.setSizePolicy(
QSizePolicy.Policy.Minimum, QSizePolicy.Policy.Preferred
)
layout.addWidget(name_label, 1)
layout.addWidget(char_label, 0)
self.setLayout(layout)
class DroppableQueueListWidget(QListWidget):
def __init__(self, parent_dialog):
super().__init__()
self.parent_dialog = parent_dialog
self.setAcceptDrops(True)
# Overlay for drag hover
self.drag_overlay = QLabel("", self)
self.drag_overlay.setAlignment(Qt.AlignmentFlag.AlignCenter)
self.drag_overlay.setStyleSheet(
f"border:2px dashed {COLORS['BLUE_BORDER_HOVER']}; border-radius:5px; padding:20px; background:{COLORS['BLUE_BG_HOVER']};"
)
self.drag_overlay.setVisible(False)
self.drag_overlay.setAttribute(
Qt.WidgetAttribute.WA_TransparentForMouseEvents, True
)
def dragEnterEvent(self, event):
if event.mimeData().hasUrls():
for url in event.mimeData().urls():
file_path = url.toLocalFile().lower()
if url.isLocalFile() and (
file_path.endswith(".txt")
or file_path.endswith((".srt", ".ass", ".vtt"))
):
self.drag_overlay.resize(self.size())
self.drag_overlay.setVisible(True)
event.acceptProposedAction()
return
self.drag_overlay.setVisible(False)
event.ignore()
def dragMoveEvent(self, event):
if event.mimeData().hasUrls():
for url in event.mimeData().urls():
file_path = url.toLocalFile().lower()
if url.isLocalFile() and (
file_path.endswith(".txt")
or file_path.endswith((".srt", ".ass", ".vtt"))
):
event.acceptProposedAction()
return
event.ignore()
def dragLeaveEvent(self, event):
self.drag_overlay.setVisible(False)
event.accept()
def dropEvent(self, event):
self.drag_overlay.setVisible(False)
if event.mimeData().hasUrls():
file_paths = [
url.toLocalFile()
for url in event.mimeData().urls()
if url.isLocalFile()
and (
url.toLocalFile().lower().endswith(".txt")
or url.toLocalFile().lower().endswith((".srt", ".ass", ".vtt"))
)
]
if file_paths:
self.parent_dialog.add_files_from_paths(file_paths)
event.acceptProposedAction()
else:
event.ignore()
else:
event.ignore()
def resizeEvent(self, event):
super().resizeEvent(event)
if hasattr(self, "drag_overlay"):
self.drag_overlay.resize(self.size())
class QueueManager(QDialog):
def __init__(self, parent, queue: list, title="Queue Manager", size=(600, 700)):
super().__init__()
self.queue = queue
self._original_queue = deepcopy(
queue
) # Store a deep copy of the original queue
self.parent = parent
self.config = load_config() # Load config for persistence
layout = QVBoxLayout()
layout.setContentsMargins(15, 15, 15, 15) # set main layout margins
layout.setSpacing(12) # set spacing between widgets in main layout
# list of queued items
self.listwidget = DroppableQueueListWidget(self)
self.listwidget.setSelectionMode(
QAbstractItemView.SelectionMode.ExtendedSelection
)
self.listwidget.setAlternatingRowColors(True)
self.listwidget.setContextMenuPolicy(Qt.ContextMenuPolicy.CustomContextMenu)
self.listwidget.customContextMenuRequested.connect(self.show_context_menu)
# Add informative instructions at the top
instructions = QLabel(
"<h2>How Queue Works?</h2>"
"You can add text and subtitle files (.txt, .srt, .ass, .vtt) directly using the '<b>Add files</b>' button below. "
"To add PDF, EPUB or markdown files, use the input box in the main window and click the <b>'Add to Queue'</b> button. "
"By default, each file in the queue keeps the configuration settings active when they were added. "
"Enabling the <b>'Override item settings with current selection'</b> option below will force all items to use the configuration currently selected in the main window. "
"You can view each file's configuration by hovering over them."
)
instructions.setAlignment(Qt.AlignmentFlag.AlignLeft)
instructions.setWordWrap(True)
layout.addWidget(instructions)
# Override Checkbox
self.override_chk = QCheckBox("Override item settings with current selection")
self.override_chk.setToolTip(
"If checked, all items in the queue will be processed using the \n"
"settings currently selected in the main window, ignoring their saved state."
)
# Load saved state (default to False)
self.override_chk.setChecked(self.config.get("queue_override_settings", False))
# Trigger process_queue to update tooltips immediately when toggled
self.override_chk.stateChanged.connect(self.process_queue)
self.override_chk.setStyleSheet("margin-bottom: 8px;")
layout.addWidget(self.override_chk)
# Overlay label for empty queue
self.empty_overlay = QLabel(
"Drag and drop your text or subtitle files here or use the 'Add files' button.",
self.listwidget,
)
self.empty_overlay.setAlignment(Qt.AlignmentFlag.AlignCenter)
self.empty_overlay.setStyleSheet(
f"color: {COLORS['LIGHT_DISABLED']}; background: transparent; padding: 20px;"
)
self.empty_overlay.setWordWrap(True)
self.empty_overlay.setAttribute(
Qt.WidgetAttribute.WA_TransparentForMouseEvents, True
)
self.empty_overlay.hide()
# add queue items to the list
self.process_queue()
button_row = QHBoxLayout()
button_row.setContentsMargins(0, 0, 0, 0) # optional: no margins for button row
button_row.setSpacing(7) # set spacing between buttons
# Add files button
add_files_button = QPushButton("Add files")
add_files_button.setFixedHeight(40)
add_files_button.clicked.connect(self.add_more_files)
button_row.addWidget(add_files_button)
# Remove button
self.remove_button = QPushButton("Remove selected")
self.remove_button.setFixedHeight(40)
self.remove_button.clicked.connect(self.remove_item)
button_row.addWidget(self.remove_button)
# Clear button
self.clear_button = QPushButton("Clear Queue")
self.clear_button.setFixedHeight(40)
self.clear_button.clicked.connect(self.clear_queue)
button_row.addWidget(self.clear_button)
layout.addLayout(button_row)
layout.addWidget(self.listwidget)
# Connect selection change to update button state
self.listwidget.currentItemChanged.connect(self.update_button_states)
self.listwidget.itemSelectionChanged.connect(self.update_button_states)
buttons = QDialogButtonBox(
QDialogButtonBox.StandardButton.Ok | QDialogButtonBox.StandardButton.Cancel,
self,
)
buttons.accepted.connect(self.accept)
buttons.rejected.connect(self.reject)
layout.addWidget(buttons)
self.setLayout(layout)
self.setWindowTitle(title)
self.resize(*size)
self.update_button_states()
def process_queue(self):
"""Process the queue items."""
import os
self.listwidget.clear()
if not self.queue:
self.empty_overlay.show()
self.update_button_states()
return
else:
self.empty_overlay.hide()
# Get current global settings and checkbox state for overrides
current_global_settings = self.get_current_attributes()
is_override_active = self.override_chk.isChecked()
icon_provider = QFileIconProvider()
for item in self.queue:
# Dynamic Attribute Retrieval Helper
def get_val(attr, default=""):
# If override is ON and attr is overrideable, use global setting
if is_override_active and attr in OVERRIDE_FIELDS:
return current_global_settings.get(attr, default)
# Otherwise return the item's saved attribute
return getattr(item, attr, default)
# Determine display file path (prefer save_base_path for original file)
display_file_path = getattr(item, "save_base_path", None) or item.file_name
processing_file_path = item.file_name
# Normalize paths for consistent display (fixes Windows path separator issues)
display_file_path = (
os.path.normpath(display_file_path)
if display_file_path
else display_file_path
)
processing_file_path = (
os.path.normpath(processing_file_path)
if processing_file_path
else processing_file_path
)
# Only show the file name, not the full path
display_name = display_file_path
if os.path.sep in display_file_path:
display_name = os.path.basename(display_file_path)
# Get icon for the display file
icon = icon_provider.icon(QFileInfo(display_file_path))
list_item = QListWidgetItem()
# Tooltip Generation
tooltip = ""
# If override is active, add the warning header on its own line
if is_override_active:
tooltip += "<b style='color: #ff9900;'>(Global Override Active)</b><br>"
output_folder = get_val("output_folder")
# For plain .txt inputs we don't need to show a separate processing file
show_processing = True
try:
if isinstance(
display_file_path, str
) and display_file_path.lower().endswith(".txt"):
show_processing = False
except Exception:
show_processing = True
tooltip += f"<b>Input File:</b> {display_file_path}<br>"
if (
show_processing
and processing_file_path
and processing_file_path != display_file_path
):
tooltip += f"<b>Processing File:</b> {processing_file_path}<br>"
tooltip += (
f"<b>Language:</b> {get_val('lang_code')}<br>"
f"<b>Speed:</b> {get_val('speed')}<br>"
f"<b>Voice:</b> {get_val('voice')}<br>"
f"<b>Save Option:</b> {get_val('save_option')}<br>"
)
if output_folder not in (None, "", "None"):
tooltip += f"<b>Output Folder:</b> {output_folder}<br>"
tooltip += (
f"<b>Subtitle Mode:</b> {get_val('subtitle_mode')}<br>"
f"<b>Output Format:</b> {get_val('output_format')}<br>"
f"<b>Characters:</b> {getattr(item, 'total_char_count', '')}<br>"
f"<b>Replace Single Newlines:</b> {get_val('replace_single_newlines', True)}<br>"
f"<b>Use Silent Gaps:</b> {get_val('use_silent_gaps', False)}<br>"
f"<b>Speed Method:</b> {get_val('subtitle_speed_method', 'tts')}"
)
# Add book handler options if present (Preserve logic: specific to file structure)
save_chapters_separately = getattr(item, "save_chapters_separately", None)
merge_chapters_at_end = getattr(item, "merge_chapters_at_end", None)
if save_chapters_separately is not None:
tooltip += f"<br><b>Save chapters separately:</b> {'Yes' if save_chapters_separately else 'No'}"
# Only show merge option if saving chapters separately
if save_chapters_separately and merge_chapters_at_end is not None:
tooltip += f"<br><b>Merge chapters at the end:</b> {'Yes' if merge_chapters_at_end else 'No'}"
list_item.setToolTip(tooltip)
list_item.setIcon(icon)
# Store both paths for context menu
list_item.setData(
Qt.ItemDataRole.UserRole,
{
"display_path": display_file_path,
"processing_path": processing_file_path,
},
)
# Use custom widget for display
char_count = getattr(item, "total_char_count", 0)
widget = QueueListItemWidget(display_file_path, char_count)
self.listwidget.addItem(list_item)
self.listwidget.setItemWidget(list_item, widget)
self.update_button_states()
def remove_item(self):
items = self.listwidget.selectedItems()
if not items:
return
from PyQt6.QtWidgets import QMessageBox
# Remove by index to ensure correct mapping
rows = sorted([self.listwidget.row(item) for item in items], reverse=True)
# Warn user if removing multiple files
if len(rows) > 1:
reply = QMessageBox.question(
self,
"Confirm Remove",
f"Are you sure you want to remove {len(rows)} selected items from the queue?",
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
QMessageBox.StandardButton.No,
)
if reply != QMessageBox.StandardButton.Yes:
return
for row in rows:
if 0 <= row < len(self.queue):
del self.queue[row]
self.process_queue()
self.update_button_states()
def clear_queue(self):
from PyQt6.QtWidgets import QMessageBox
if len(self.queue) > 1:
reply = QMessageBox.question(
self,
"Confirm Clear Queue",
f"Are you sure you want to clear {len(self.queue)} items from the queue?",
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
QMessageBox.StandardButton.No,
)
if reply != QMessageBox.StandardButton.Yes:
return
self.queue.clear()
self.listwidget.clear()
self.empty_overlay.resize(
self.listwidget.size()
) # Ensure overlay is sized correctly
self.empty_overlay.show() # Show the overlay when queue is empty
self.update_button_states()
def get_queue(self):
return self.queue
def get_current_attributes(self):
# Fetch current attribute values from the parent abogen GUI
attrs = {}
parent = self.parent
if parent is not None:
# lang_code: use parent's get_voice_formula and get_selected_lang
if hasattr(parent, "get_voice_formula") and hasattr(
parent, "get_selected_lang"
):
voice_formula = parent.get_voice_formula()
attrs["lang_code"] = parent.get_selected_lang(voice_formula)
attrs["voice"] = voice_formula
else:
attrs["lang_code"] = getattr(parent, "selected_lang", "")
attrs["voice"] = getattr(parent, "selected_voice", "")
# speed
if hasattr(parent, "speed_slider"):
attrs["speed"] = parent.speed_slider.value() / 100.0
else:
attrs["speed"] = getattr(parent, "speed", 1.0)
# save_option
attrs["save_option"] = getattr(parent, "save_option", "")
# output_folder
attrs["output_folder"] = getattr(parent, "selected_output_folder", "")
# subtitle_mode
if hasattr(parent, "get_actual_subtitle_mode"):
attrs["subtitle_mode"] = parent.get_actual_subtitle_mode()
else:
attrs["subtitle_mode"] = getattr(parent, "subtitle_mode", "")
# output_format
attrs["output_format"] = getattr(parent, "selected_format", "")
# total_char_count
attrs["total_char_count"] = getattr(parent, "char_count", "")
# replace_single_newlines
attrs["replace_single_newlines"] = getattr(
parent, "replace_single_newlines", True
)
# use_silent_gaps
attrs["use_silent_gaps"] = getattr(parent, "use_silent_gaps", False)
# subtitle_speed_method
attrs["subtitle_speed_method"] = getattr(
parent, "subtitle_speed_method", "tts"
)
# book handler options
attrs["save_chapters_separately"] = getattr(
parent, "save_chapters_separately", None
)
attrs["merge_chapters_at_end"] = getattr(
parent, "merge_chapters_at_end", None
)
else:
# fallback: empty values
attrs = {
k: ""
for k in [
"lang_code",
"speed",
"voice",
"save_option",
"output_folder",
"subtitle_mode",
"output_format",
"total_char_count",
"replace_single_newlines",
]
}
attrs["save_chapters_separately"] = None
attrs["merge_chapters_at_end"] = None
return attrs
def add_files_from_paths(self, file_paths):
from abogen.subtitle_utils import calculate_text_length
from PyQt6.QtWidgets import QMessageBox
import os
current_attrs = self.get_current_attributes()
duplicates = []
for file_path in file_paths:
class QueueItem:
pass
item = QueueItem()
item.file_name = file_path
item.save_base_path = (
file_path # For .txt files, processing and save paths are the same
)
for attr, value in current_attrs.items():
setattr(item, attr, value)
# Override subtitle_mode to "Disabled" for subtitle files
if file_path.lower().endswith((".srt", ".ass", ".vtt")):
item.subtitle_mode = "Disabled"
# Read file content and calculate total_char_count using calculate_text_length
try:
with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
file_content = f.read()
item.total_char_count = calculate_text_length(file_content)
except Exception:
item.total_char_count = 0
# Prevent adding duplicate items to the queue (check all attributes)
is_duplicate = False
for queued_item in self.queue:
if (
getattr(queued_item, "file_name", None)
== getattr(item, "file_name", None)
and getattr(queued_item, "lang_code", None)
== getattr(item, "lang_code", None)
and getattr(queued_item, "speed", None)
== getattr(item, "speed", None)
and getattr(queued_item, "voice", None)
== getattr(item, "voice", None)
and getattr(queued_item, "save_option", None)
== getattr(item, "save_option", None)
and getattr(queued_item, "output_folder", None)
== getattr(item, "output_folder", None)
and getattr(queued_item, "subtitle_mode", None)
== getattr(item, "subtitle_mode", None)
and getattr(queued_item, "output_format", None)
== getattr(item, "output_format", None)
and getattr(queued_item, "total_char_count", None)
== getattr(item, "total_char_count", None)
and getattr(queued_item, "replace_single_newlines", True)
== getattr(item, "replace_single_newlines", True)
and getattr(queued_item, "use_silent_gaps", False)
== getattr(item, "use_silent_gaps", False)
and getattr(queued_item, "subtitle_speed_method", "tts")
== getattr(item, "subtitle_speed_method", "tts")
and getattr(queued_item, "save_base_path", None)
== getattr(item, "save_base_path", None)
and getattr(queued_item, "save_chapters_separately", None)
== getattr(item, "save_chapters_separately", None)
and getattr(queued_item, "merge_chapters_at_end", None)
== getattr(item, "merge_chapters_at_end", None)
):
is_duplicate = True
break
if is_duplicate:
duplicates.append(os.path.basename(file_path))
continue
self.queue.append(item)
if duplicates:
QMessageBox.warning(
self,
"Duplicate Item(s)",
f"Skipping {len(duplicates)} file(s) with the same attributes, already in the queue.",
)
self.process_queue()
self.update_button_states()
def add_more_files(self):
from PyQt6.QtWidgets import QFileDialog
# Allow .txt, .srt, .ass, and .vtt files
files, _ = QFileDialog.getOpenFileNames(
self,
"Select text or subtitle files",
"",
"Supported Files (*.txt *.srt *.ass *.vtt)",
)
if not files:
return
self.add_files_from_paths(files)
def resizeEvent(self, event):
super().resizeEvent(event)
if hasattr(self, "empty_overlay"):
self.empty_overlay.resize(self.listwidget.size())
def update_button_states(self):
# Enable Remove if at least one item is selected, else disable
if hasattr(self, "remove_button"):
selected_count = len(self.listwidget.selectedItems())
self.remove_button.setEnabled(selected_count > 0)
if selected_count > 1:
self.remove_button.setText(f"Remove selected ({selected_count})")
else:
self.remove_button.setText("Remove selected")
# Disable Clear if queue is empty
if hasattr(self, "clear_button"):
self.clear_button.setEnabled(bool(self.queue))
def show_context_menu(self, pos):
from PyQt6.QtWidgets import QMenu
from PyQt6.QtGui import QAction, QDesktopServices
from PyQt6.QtCore import QUrl
import os
global_pos = self.listwidget.viewport().mapToGlobal(pos)
selected_items = self.listwidget.selectedItems()
menu = QMenu(self)
if len(selected_items) == 1:
# Add Remove action
remove_action = QAction("Remove this item", self)
remove_action.triggered.connect(self.remove_item)
menu.addAction(remove_action)
# Get paths for determining if it's a document input
item = selected_items[0]
paths = item.data(Qt.ItemDataRole.UserRole)
if isinstance(paths, dict):
display_path = paths.get("display_path", "")
processing_path = paths.get("processing_path", "")
else:
display_path = paths
processing_path = paths
doc_exts = (".md", ".markdown", ".pdf", ".epub")
is_document_input = (
isinstance(display_path, str)
and display_path.lower().endswith(doc_exts)
) or (
isinstance(processing_path, str)
and processing_path.lower().endswith(doc_exts)
)
# Add Open file action(s)
def open_file_by_path(path_label: str):
from PyQt6.QtWidgets import QMessageBox
p = display_path if path_label == "display" else processing_path
if not p:
QMessageBox.warning(
self, "File Not Found", "Path is not available."
)
return
# Find the queue item and resolve the target path
target_path = None
for q in self.queue:
if (
getattr(q, "save_base_path", None) == display_path
or q.file_name == display_path
):
if path_label == "display":
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
else:
target_path = q.file_name
break
if (
getattr(q, "save_base_path", None) == processing_path
or q.file_name == processing_path
):
if path_label == "display":
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
else:
target_path = q.file_name
break
# Fallback to the raw path if resolution failed
if not target_path:
target_path = p
if not os.path.exists(target_path):
QMessageBox.warning(
self, "File Not Found", f"The file does not exist."
)
return
QDesktopServices.openUrl(QUrl.fromLocalFile(target_path))
if is_document_input:
# For documents, show two open options
open_processed_action = QAction("Open processed file", self)
open_processed_action.triggered.connect(
lambda: open_file_by_path("processing")
)
menu.addAction(open_processed_action)
open_input_action = QAction("Open input file", self)
open_input_action.triggered.connect(
lambda: open_file_by_path("display")
)
menu.addAction(open_input_action)
else:
# For plain text files, show single open option
open_file_action = QAction("Open file", self)
open_file_action.triggered.connect(lambda: open_file_by_path("display"))
menu.addAction(open_file_action)
# Add Go to folder action
# If the queued item represents a converted document (markdown, pdf, epub)
# show two actions: Go to processed file (the cached .txt) and Go to input file (original source)
from PyQt6.QtWidgets import QMessageBox
def open_folder_for(path_label: str):
# path_label should be either 'display' or 'processing'
p = display_path if path_label == "display" else processing_path
if not p:
QMessageBox.warning(
self, "File Not Found", "Path is not available."
)
return
# If the stored path is the display path (original) but the actual file may be
# stored on the queue object differently, try to resolve via the queue entry.
target_path = None
for q in self.queue:
if (
getattr(q, "save_base_path", None) == display_path
or q.file_name == display_path
):
if path_label == "display":
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
else:
target_path = q.file_name
break
if (
getattr(q, "save_base_path", None) == processing_path
or q.file_name == processing_path
):
if path_label == "display":
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
else:
target_path = q.file_name
break
# Fallback to the raw path if resolution failed
if not target_path:
target_path = p
if not os.path.exists(target_path):
QMessageBox.warning(
self,
"File Not Found",
f"The file does not exist: {target_path}",
)
return
folder = os.path.dirname(target_path)
if os.path.exists(folder):
QDesktopServices.openUrl(QUrl.fromLocalFile(folder))
if is_document_input:
processed_action = QAction("Go to processed file", self)
processed_action.triggered.connect(
lambda: open_folder_for("processing")
)
menu.addAction(processed_action)
input_action = QAction("Go to input file", self)
input_action.triggered.connect(lambda: open_folder_for("display"))
menu.addAction(input_action)
else:
# Default behavior for non-document inputs: single "Go to folder" action
go_to_folder_action = QAction("Go to folder", self)
def go_to_folder():
item = selected_items[0]
paths = item.data(Qt.ItemDataRole.UserRole)
if isinstance(paths, dict):
file_path = paths.get(
"display_path", paths.get("processing_path", "")
)
else:
file_path = paths # Fallback for old format
# Find the queue item
for q in self.queue:
if (
getattr(q, "save_base_path", None) == file_path
or q.file_name == file_path
):
target_path = (
getattr(q, "save_base_path", None) or q.file_name
)
if not os.path.exists(target_path):
QMessageBox.warning(
self, "File Not Found", f"The file does not exist."
)
return
folder = os.path.dirname(target_path)
if os.path.exists(folder):
QDesktopServices.openUrl(QUrl.fromLocalFile(folder))
break
go_to_folder_action.triggered.connect(go_to_folder)
menu.addAction(go_to_folder_action)
elif len(selected_items) > 1:
remove_action = QAction(f"Remove selected ({len(selected_items)})", self)
remove_action.triggered.connect(self.remove_item)
menu.addAction(remove_action)
# Always add Clear Queue
clear_action = QAction("Clear Queue", self)
clear_action.triggered.connect(self.clear_queue)
menu.addAction(clear_action)
menu.exec(global_pos)
def accept(self):
# Save the override state to config so it persists globally
self.config["queue_override_settings"] = self.override_chk.isChecked()
save_config(self.config)
super().accept()
def reject(self):
# Cancel: restore original queue
from PyQt6.QtWidgets import QMessageBox
# Warn if user changed a lot (e.g., more than 1 items difference)
original_count = len(self._original_queue)
current_count = len(self.queue)
if abs(original_count - current_count) > 1:
reply = QMessageBox.question(
self,
"Confirm Cancel",
f"Are you sure you want to cancel and discard all changes?",
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
QMessageBox.StandardButton.No,
)
if reply != QMessageBox.StandardButton.Yes:
return
self.queue.clear()
self.queue.extend(deepcopy(self._original_queue))
super().reject()
def keyPressEvent(self, event):
from PyQt6.QtCore import Qt
if event.key() == Qt.Key.Key_Delete:
self.remove_item()
else:
super().keyPressEvent(event)
+21
View File
@@ -0,0 +1,21 @@
# represents a queued item - book, chapters, voice, etc.
from dataclasses import dataclass
@dataclass
class QueuedItem:
file_name: str
lang_code: str
speed: float
voice: str
save_option: str
output_folder: str
subtitle_mode: str
output_format: str
total_char_count: int
replace_single_newlines: bool = True
use_silent_gaps: bool = False
subtitle_speed_method: str = "tts"
save_base_path: str = None
save_chapters_separately: bool = None
merge_chapters_at_end: bool = None
File diff suppressed because it is too large Load Diff
+7 -5
View File
@@ -1,9 +1,11 @@
"""Legacy PyQt queue manager GUI removed."""
"""Backwards-compatible re-export of the PyQt queue manager.
The actual implementation lives in abogen.pyqt.queue_manager_gui.
"""
from __future__ import annotations
from abogen.pyqt.queue_manager_gui import * # noqa: F401, F403
from abogen.pyqt.queue_manager_gui import QueueManager
def __getattr__(name: str): # pragma: no cover - compatibility shim
raise AttributeError(
"The PyQt queue manager GUI has been removed. Use the web dashboard instead."
)
__all__ = ["QueueManager"]
+161
View File
@@ -0,0 +1,161 @@
"""
Lazy-loaded spaCy utilities for sentence segmentation.
"""
# Cached spaCy module and models (lazy loaded)
_spacy = None
_nlp_cache = {}
# Language code to spaCy model mapping
SPACY_MODELS = {
"a": "en_core_web_sm", # American English
"b": "en_core_web_sm", # British English
"e": "es_core_news_sm", # Spanish
"f": "fr_core_news_sm", # French
"i": "it_core_news_sm", # Italian
"p": "pt_core_news_sm", # Brazilian Portuguese
"z": "zh_core_web_sm", # Mandarin Chinese
"j": "ja_core_news_sm", # Japanese
"h": "xx_sent_ud_sm", # Hindi (multi-language model)
}
def _load_spacy():
"""Lazy load spaCy module."""
global _spacy
if _spacy is None:
try:
import spacy
_spacy = spacy
except ImportError:
return None
return _spacy
def get_spacy_model(lang_code, log_callback=None):
"""
Get or load a spaCy model for the given language code.
Downloads the model automatically if not available.
Args:
lang_code: Language code (a, b, e, f, etc.)
log_callback: Optional function to log messages
Returns:
Loaded spaCy model or None if unavailable
"""
def log(msg, is_error=False):
# Prefer GUI log callback when provided to avoid spamming stdout.
if log_callback:
color = "red" if is_error else "grey"
try:
log_callback((msg, color))
except Exception:
# Fallback to printing if callback misbehaves
print(msg)
else:
print(msg)
# Check if model is cached
if lang_code in _nlp_cache:
return _nlp_cache[lang_code]
# Check if language is supported
model_name = SPACY_MODELS.get(lang_code)
if not model_name:
log(f"\nspaCy: No model mapping for language '{lang_code}'...")
return None
# Lazy load spaCy
spacy = _load_spacy()
if spacy is None:
log("\nspaCy: Module not installed, falling back to default segmentation...")
return None
# Try to load the model
try:
log(f"\nLoading spaCy model '{model_name}'...")
# sentence segmentation involving parentheses, quotes, and complex structure.
# We only disable heavier components we don't need like NER.
nlp = spacy.load(
model_name,
disable=["ner", "tagger", "lemmatizer", "attribute_ruler"],
)
# Ensure a sentence segmentation strategy is in place
# The parser provides sents, but if it's missing (unlikely for core models), fallback to sentencizer
if "parser" not in nlp.pipe_names and "sentencizer" not in nlp.pipe_names:
nlp.add_pipe("sentencizer")
_nlp_cache[lang_code] = nlp
return nlp
except OSError:
# Model not found, attempt download
log(f"\nspaCy: Downloading model '{model_name}'...")
try:
from spacy.cli import download
download(model_name)
# Retry loading with the same fix
nlp = spacy.load(
model_name,
disable=["ner", "tagger", "lemmatizer", "attribute_ruler"],
)
if "parser" not in nlp.pipe_names and "sentencizer" not in nlp.pipe_names:
nlp.add_pipe("sentencizer")
_nlp_cache[lang_code] = nlp
log(f"spaCy model '{model_name}' downloaded and loaded")
return nlp
except Exception as e:
log(
f"\nspaCy: Failed to download model '{model_name}': {e}...",
is_error=True,
)
return None
except Exception as e:
log(f"\nspaCy: Error loading model '{model_name}': {e}...", is_error=True)
return None
def segment_sentences(text, lang_code, log_callback=None):
"""
Segment text into sentences using spaCy.
Args:
text: Text to segment
lang_code: Language code
log_callback: Optional function to log messages
Returns:
List of sentence strings, or None if spaCy unavailable
"""
nlp = get_spacy_model(lang_code, log_callback)
if nlp is None:
return None
# Ensure spaCy can handle large texts by adjusting max_length if necessary
try:
text_len = len(text or "")
if text_len and hasattr(nlp, "max_length") and text_len > nlp.max_length:
# increase a bit beyond the text length to be safe
nlp.max_length = text_len + 1000
except Exception:
pass
# Process text and extract sentences
doc = nlp(text)
return [sent.text.strip() for sent in doc.sents if sent.text.strip()]
def is_spacy_available():
"""Check if spaCy can be imported."""
return _load_spacy() is not None
def clear_cache():
"""Clear the model cache to free memory."""
global _nlp_cache
_nlp_cache.clear()
+459
View File
@@ -0,0 +1,459 @@
import re
import platform
from abogen.utils import detect_encoding, load_config
from abogen.constants import SAMPLE_VOICE_TEXTS
# Pre-compile frequently used regex patterns for better performance
_METADATA_TAG_PATTERN = re.compile(r"<<METADATA_[^:]+:[^>]*>>")
_WHITESPACE_PATTERN = re.compile(r"[^\S\n]+")
_MULTIPLE_NEWLINES_PATTERN = re.compile(r"\n{3,}")
_SINGLE_NEWLINE_PATTERN = re.compile(r"(?<!\n)\n(?!\n)")
_CHAPTER_MARKER_PATTERN = re.compile(r"<<CHAPTER_MARKER:[^>]*>>")
_HTML_TAG_PATTERN = re.compile(r"<[^>]+>")
_VOICE_TAG_PATTERN = re.compile(r"{[^}]+}")
_ASS_STYLING_PATTERN = re.compile(r"\{[^}]+\}")
_ASS_NEWLINE_N_PATTERN = re.compile(r"\\N")
_ASS_NEWLINE_LOWER_N_PATTERN = re.compile(r"\\n")
_CHAPTER_MARKER_SEARCH_PATTERN = re.compile(r"<<CHAPTER_MARKER:(.*?)>>")
_WEBVTT_HEADER_PATTERN = re.compile(r"^WEBVTT.*?\n", re.MULTILINE)
_VTT_STYLE_PATTERN = re.compile(r"STYLE\s*\n.*?(?=\n\n|$)", re.DOTALL)
_VTT_NOTE_PATTERN = re.compile(r"NOTE\s*\n.*?(?=\n\n|$)", re.DOTALL)
_DOUBLE_NEWLINE_SPLIT_PATTERN = re.compile(r"\n\s*\n")
_VTT_TIMESTAMP_PATTERN = re.compile(r"([\d:.]+)\s*-->\s*([\d:.]+)")
_TIMESTAMP_ONLY_PATTERN = re.compile(r"^(\d{1,2}:\d{2}:\d{2}(?:[.,]\d{1,3})?)$")
_WINDOWS_ILLEGAL_CHARS_PATTERN = re.compile(r'[<>:"/\\|?*]')
_CONTROL_CHARS_PATTERN = re.compile(r"[\x00-\x1f]")
_LINUX_CONTROL_CHARS_PATTERN = re.compile(
r"[\x01-\x1f]"
) # Linux: exclude \x00 for separate handling
_MACOS_ILLEGAL_CHARS_PATTERN = re.compile(r"[:]")
_LINUX_ILLEGAL_CHARS_PATTERN = re.compile(r"[/\x00]")
def clean_subtitle_text(text):
"""Remove chapter markers and metadata tags from subtitle text."""
# Use pre-compiled patterns for better performance
text = _METADATA_TAG_PATTERN.sub("", text)
text = _CHAPTER_MARKER_PATTERN.sub("", text)
return text.strip()
def calculate_text_length(text):
# Use pre-compiled patterns for better performance
# Ignore chapter markers and metadata patterns in a single pass
text = _CHAPTER_MARKER_PATTERN.sub("", text)
text = _METADATA_TAG_PATTERN.sub("", text)
# Ignore newlines and leading/trailing spaces
text = text.replace("\n", "").strip()
# Calculate character count
char_count = len(text)
return char_count
def clean_text(text, *args, **kwargs):
# Remove metadata tags first
text = _METADATA_TAG_PATTERN.sub("", text)
# Load replace_single_newlines from config
cfg = load_config()
replace_single_newlines = cfg.get("replace_single_newlines", True)
# Collapse all whitespace (excluding newlines) into single spaces per line and trim edges
# Use pre-compiled pattern for better performance
lines = [_WHITESPACE_PATTERN.sub(" ", line).strip() for line in text.splitlines()]
text = "\n".join(lines)
# Standardize paragraph breaks (multiple newlines become exactly two) and trim overall whitespace
# Use pre-compiled pattern for better performance
text = _MULTIPLE_NEWLINES_PATTERN.sub("\n\n", text).strip()
# Optionally replace single newlines with spaces, but preserve double newlines
if replace_single_newlines:
# Use pre-compiled pattern for better performance
text = _SINGLE_NEWLINE_PATTERN.sub(" ", text)
return text
def parse_srt_file(file_path):
"""
Parse an SRT subtitle file and return a list of subtitle entries.
Args:
file_path: Path to the SRT file
Returns:
List of tuples: [(start_time_seconds, end_time_seconds, text), ...]
"""
encoding = detect_encoding(file_path)
with open(file_path, "r", encoding=encoding, errors="replace") as f:
content = f.read()
# Split by double newlines to get individual subtitle blocks
blocks = re.split(r"\n\s*\n", content.strip())
subtitles = []
for block in blocks:
if not block.strip():
continue
lines = block.strip().split("\n")
if len(lines) < 3:
continue
# First line is index, second line is timestamp, rest is text
try:
timestamp_line = lines[1]
match = re.match(
r"(\d{2}:\d{2}:\d{2},\d{3})\s*-->\s*(\d{2}:\d{2}:\d{2},\d{3})",
timestamp_line,
)
if not match:
continue
start_str = match.group(1)
end_str = match.group(2)
text = "\n".join(lines[2:])
# Convert timestamp to seconds
def time_to_seconds(t):
h, m, s_ms = t.split(":")
s, ms = s_ms.split(",")
return int(h) * 3600 + int(m) * 60 + int(s) + int(ms) / 1000.0
start_sec = time_to_seconds(start_str)
end_sec = time_to_seconds(end_str)
# Clean text of any styling tags using pre-compiled pattern
text = _HTML_TAG_PATTERN.sub("", text)
# Remove chapter markers and metadata tags
text = clean_subtitle_text(text)
if text: # Only add non-empty subtitles
subtitles.append((start_sec, end_sec, text))
except (ValueError, IndexError):
continue
return subtitles
def parse_vtt_file(file_path):
"""
Parse a VTT (WebVTT) subtitle file and return a list of subtitle entries.
Args:
file_path: Path to the VTT file
Returns:
List of tuples: [(start_time_seconds, end_time_seconds, text), ...]
"""
encoding = detect_encoding(file_path)
with open(file_path, "r", encoding=encoding, errors="replace") as f:
content = f.read()
# Remove WEBVTT header and any style/note blocks using pre-compiled patterns
content = _WEBVTT_HEADER_PATTERN.sub("", content)
content = _VTT_STYLE_PATTERN.sub("", content)
content = _VTT_NOTE_PATTERN.sub("", content)
# Split by double newlines to get individual subtitle blocks using pre-compiled pattern
blocks = _DOUBLE_NEWLINE_SPLIT_PATTERN.split(content.strip())
subtitles = []
for block in blocks:
if not block.strip():
continue
lines = block.strip().split("\n")
if len(lines) < 2:
continue
# VTT can have optional identifier on first line, timestamp on second or first
timestamp_line = None
text_start_idx = 0
# Check if first line is timestamp
if "-->" in lines[0]:
timestamp_line = lines[0]
text_start_idx = 1
elif len(lines) > 1 and "-->" in lines[1]:
timestamp_line = lines[1]
text_start_idx = 2
else:
continue
try:
# VTT format: 00:00:00.000 --> 00:00:05.000 or 00:00.000 --> 00:05.000
# Use pre-compiled pattern
match = _VTT_TIMESTAMP_PATTERN.match(timestamp_line)
if not match:
continue
start_str = match.group(1)
end_str = match.group(2)
text = "\n".join(lines[text_start_idx:])
# Convert timestamp to seconds
def time_to_seconds(t):
parts = t.split(":")
if len(parts) == 3: # HH:MM:SS.mmm
h, m, s = parts
s, ms = s.split(".")
return int(h) * 3600 + int(m) * 60 + int(s) + int(ms) / 1000.0
elif len(parts) == 2: # MM:SS.mmm
m, s = parts
s, ms = s.split(".")
return int(m) * 60 + int(s) + int(ms) / 1000.0
return 0
start_sec = time_to_seconds(start_str)
end_sec = time_to_seconds(end_str)
# Clean text of any styling tags and cue settings using pre-compiled patterns
text = _HTML_TAG_PATTERN.sub("", text)
text = _VOICE_TAG_PATTERN.sub("", text) # Remove voice tags
# Remove chapter markers and metadata tags
text = clean_subtitle_text(text)
if text: # Only add non-empty subtitles
subtitles.append((start_sec, end_sec, text))
except (ValueError, IndexError, AttributeError):
continue
return subtitles
def detect_timestamps_in_text(file_path):
"""Detect if text file contains timestamp markers (HH:MM:SS or HH:MM:SS,ms format) on separate lines."""
try:
encoding = detect_encoding(file_path)
with open(file_path, "r", encoding=encoding, errors="replace") as f:
lines = [
line.strip() for line in f.readlines()[:50] if line.strip()
] # Check first 50 non-empty lines
# Count lines that are ONLY timestamps (no other text)
# Supports HH:MM:SS or HH:MM:SS,ms format
# Use pre-compiled pattern for better performance
timestamp_lines = sum(
1 for line in lines if _TIMESTAMP_ONLY_PATTERN.match(line)
)
# Must have at least 2 timestamp-only lines and they should be >5% of total lines
return timestamp_lines >= 2 and (timestamp_lines / max(len(lines), 1)) > 0.05
except Exception:
return False
def parse_timestamp_text_file(file_path):
"""Parse text file with timestamps. Returns list of (start_time, end_time, text) tuples.
Supports HH:MM:SS or HH:MM:SS,ms format. Returns time in seconds as float."""
encoding = detect_encoding(file_path)
with open(file_path, "r", encoding=encoding, errors="replace") as f:
content = f.read()
# Split by timestamp pattern (supports HH:MM:SS or HH:MM:SS,ms)
pattern = r"^(\d{1,2}:\d{2}:\d{2}(?:[.,]\d{1,3})?)$"
lines = content.split("\n")
def parse_time(time_str):
"""Convert HH:MM:SS or HH:MM:SS,ms to seconds as float."""
time_str = time_str.replace(",", ".")
parts = time_str.split(":")
return float(int(parts[0]) * 3600 + int(parts[1]) * 60 + float(parts[2]))
entries = []
current_time = None
current_text = []
pre_timestamp_text = [] # Text before first timestamp
for line in lines:
match = re.match(pattern, line.strip())
if match:
# Save previous entry
if current_time is not None and current_text:
text = "\n".join(current_text).strip()
if text:
entries.append((current_time, text))
elif current_time is None and pre_timestamp_text:
# First timestamp found, save pre-timestamp text with time 0
text = "\n".join(pre_timestamp_text).strip()
if text:
entries.append((0.0, text))
pre_timestamp_text = []
# Start new entry
time_str = match.group(1)
current_time = parse_time(time_str)
current_text = []
elif current_time is not None:
current_text.append(line)
else:
# Text before first timestamp
pre_timestamp_text.append(line)
# Save last entry
if current_time is not None and current_text:
text = "\n".join(current_text).strip()
if text:
entries.append((current_time, text))
elif not entries and pre_timestamp_text:
# No timestamps found at all, treat entire file as starting at 0
text = "\n".join(pre_timestamp_text).strip()
if text:
entries.append((0.0, text))
# Convert to subtitle format with end times
subtitles = []
for i, (start_time, text) in enumerate(entries):
end_time = entries[i + 1][0] if i + 1 < len(entries) else None
# Remove chapter markers and metadata tags
text = clean_subtitle_text(text)
if text: # Only add non-empty entries
subtitles.append((start_time, end_time, text))
return subtitles
def parse_ass_file(file_path):
"""
Parse an ASS/SSA subtitle file and return a list of subtitle entries.
Args:
file_path: Path to the ASS/SSA file
Returns:
List of tuples: [(start_time_seconds, end_time_seconds, text), ...]
"""
encoding = detect_encoding(file_path)
with open(file_path, "r", encoding=encoding, errors="replace") as f:
lines = f.readlines()
subtitles = []
in_events = False
format_indices = {}
for line in lines:
line = line.strip()
if line.startswith("[Events]"):
in_events = True
continue
if line.startswith("[") and in_events:
# New section, stop processing
break
if in_events and line.startswith("Format:"):
# Parse format line to know column positions
parts = line.split(":", 1)[1].strip().split(",")
for i, part in enumerate(parts):
format_indices[part.strip().lower()] = i
continue
if in_events and (line.startswith("Dialogue:") or line.startswith("Comment:")):
if line.startswith("Comment:"):
continue # Skip comments
parts = line.split(":", 1)[1].strip().split(",", len(format_indices) - 1)
if (
"start" in format_indices
and "end" in format_indices
and "text" in format_indices
):
start_str = parts[format_indices["start"]].strip()
end_str = parts[format_indices["end"]].strip()
text = parts[format_indices["text"]].strip()
# Convert timestamp to seconds (ASS format: H:MM:SS.CS where CS is centiseconds)
def ass_time_to_seconds(t):
parts = t.split(":")
if len(parts) == 3:
h, m, s = parts
s_parts = s.split(".")
seconds = float(s_parts[0])
centiseconds = float(s_parts[1]) if len(s_parts) > 1 else 0
return (
int(h) * 3600 + int(m) * 60 + seconds + centiseconds / 100.0
)
return 0
start_sec = ass_time_to_seconds(start_str)
end_sec = ass_time_to_seconds(end_str)
# Clean text of ASS styling tags using pre-compiled patterns
text = _ASS_STYLING_PATTERN.sub("", text) # Remove {tags}
text = _ASS_NEWLINE_N_PATTERN.sub("\n", text) # Convert \N to newline
text = _ASS_NEWLINE_LOWER_N_PATTERN.sub(
"\n", text
) # Convert \n to newline
# Remove chapter markers and metadata tags
text = clean_subtitle_text(text)
if text: # Only add non-empty subtitles
subtitles.append((start_sec, end_sec, text))
return subtitles
def get_sample_voice_text(lang_code):
return SAMPLE_VOICE_TEXTS.get(lang_code, SAMPLE_VOICE_TEXTS["a"])
def sanitize_name_for_os(name, is_folder=True):
"""
Sanitize a filename or folder name based on the operating system.
Args:
name: The name to sanitize
is_folder: Whether this is a folder name (default: True)
Returns:
Sanitized name safe for the current OS
"""
if not name:
return "audiobook"
system = platform.system()
if system == "Windows":
# Windows illegal characters: < > : " / \ | ? *
# Also can't end with space or dot
# Use pre-compiled pattern for better performance
sanitized = _WINDOWS_ILLEGAL_CHARS_PATTERN.sub("_", name)
# Remove control characters (0-31)
sanitized = _CONTROL_CHARS_PATTERN.sub("_", sanitized)
# Remove trailing spaces and dots
sanitized = sanitized.rstrip(". ")
# Windows reserved names (CON, PRN, AUX, NUL, COM1-9, LPT1-9)
reserved = (
["CON", "PRN", "AUX", "NUL"]
+ [f"COM{i}" for i in range(1, 10)]
+ [f"LPT{i}" for i in range(1, 10)]
)
if sanitized.upper() in reserved or sanitized.upper().split(".")[0] in reserved:
sanitized = f"_{sanitized}"
elif system == "Darwin": # macOS
# macOS illegal characters: : (colon is converted to / by the system)
# Also can't start with dot (hidden file) for folders typically
# Use pre-compiled pattern for better performance
sanitized = _MACOS_ILLEGAL_CHARS_PATTERN.sub("_", name)
# Remove control characters
sanitized = _CONTROL_CHARS_PATTERN.sub("_", sanitized)
# Avoid leading dot for folders (creates hidden folders)
if is_folder and sanitized.startswith("."):
sanitized = "_" + sanitized[1:]
else: # Linux and others
# Linux illegal characters: / and null character
# Though / is illegal, most other chars are technically allowed
# Use pre-compiled pattern for better performance
sanitized = _LINUX_ILLEGAL_CHARS_PATTERN.sub("_", name)
# Remove other control characters for safety (excluding \x00 which is already handled)
sanitized = _LINUX_CONTROL_CHARS_PATTERN.sub("_", sanitized)
# Avoid leading dot for folders (creates hidden folders)
if is_folder and sanitized.startswith("."):
sanitized = "_" + sanitized[1:]
# Ensure the name is not empty after sanitization
if not sanitized or sanitized.strip() == "":
sanitized = "audiobook"
# Limit length to 255 characters (common limit across filesystems)
if len(sanitized) > 255:
sanitized = sanitized[:255].rstrip(". ")
return sanitized
+6 -10
View File
@@ -1,15 +1,11 @@
"""Legacy PyQt voice formula dialog removed."""
"""Backwards-compatible re-export of the PyQt voice formula dialog.
The actual implementation lives in abogen.pyqt.voice_formula_gui.
"""
from __future__ import annotations
class VoiceFormulaDialog: # pragma: no cover - legacy entry point
"""Placeholder for removed PyQt dialog."""
def __init__(self, *_args, **_kwargs):
raise RuntimeError(
"The PyQt-based voice formula editor has been removed. Use the web tools instead."
)
from abogen.pyqt.voice_formula_gui import * # noqa: F401, F403
from abogen.pyqt.voice_formula_gui import VoiceFormulaDialog
__all__ = ["VoiceFormulaDialog"]
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env python3
"""Build PyPI package (wheel and sdist) to `dist` folder for abogen."""
import subprocess
import os
import shutil
import tempfile
def main():
script_dir = os.path.dirname(os.path.abspath(__file__))
output_dir = os.path.join(script_dir, "dist")
print("🔧 abogen PyPI Package Builder")
print("=" * 40)
print(f"📁 Script directory: {script_dir}")
print(f"📦 Output directory: {output_dir}")
# Try to print package version if present
version = None
version_file = os.path.join(script_dir, "abogen", "VERSION")
if os.path.isfile(version_file):
try:
with open(version_file, "r", encoding="utf-8") as vf:
version = vf.read().strip()
except Exception:
version = None
if version:
print(f"🔖 Package version: {version}")
# Check if build module is installed, install if not
# Temporarily remove script_dir from sys.path to avoid importing local build.py
import sys
original_path = sys.path[:]
try:
sys.path = [p for p in sys.path if os.path.abspath(p) != script_dir]
import build
except ImportError:
print("📦 Installing build module...")
subprocess.run([sys.executable, "-m", "pip", "install", "build"], check=True)
finally:
sys.path = original_path
# Create output directory
print(f"📂 Preparing output directory: {output_dir}")
if os.path.exists(output_dir):
shutil.rmtree(output_dir)
os.makedirs(output_dir, exist_ok=True)
print("🏗️ Building PyPI package...")
print(" Using temporary directory to avoid module conflicts...")
# Run from temp directory to avoid local build.py shadowing the build module
with tempfile.TemporaryDirectory() as tmpdir:
print(f" Temp directory: {tmpdir}")
print(" Running: python -m build -o <output_dir> <source_dir>")
result = subprocess.run(
[sys.executable, "-m", "build", "-o", output_dir, script_dir],
check=False,
cwd=tmpdir,
)
print("\n" + "=" * 40)
if result.returncode == 0:
print("✅ Build successful!")
print(f"📦 Files created in {output_dir}:")
files = os.listdir(output_dir)
if files:
for f in files:
file_path = os.path.join(output_dir, f)
size = os.path.getsize(file_path)
print(f" 📄 {f} ({size:,} bytes)")
else:
print(" (No files found)")
print("\n🚀 Ready for upload with:\n")
print(" - To test on Test PyPI:")
print(f" python -m twine upload --repository testpypi {output_dir}/*")
print("\n - To upload to PyPI (when ready):")
print(f" python -m twine upload {output_dir}/*")
else:
print("❌ Build failed!")
print(f" Exit code: {result.returncode}")
sys.exit(result.returncode)
if __name__ == "__main__":
main()
+4 -2
View File
@@ -34,7 +34,8 @@ dependencies = [
"numpy>=1.24.0",
"gpustat>=1.1.1",
"num2words>=0.5.13",
"httpx>=0.27.0"
"httpx>=0.27.0",
"PyQt6>=6.5.0"
]
classifiers = [
@@ -62,11 +63,12 @@ allow-direct-references = true
[project.gui-scripts]
abogen = "abogen.webui.app:main"
abogen = "abogen.pyqt.main:main"
[project.scripts]
abogen-cli = "abogen.webui.app:main"
abogen-web = "abogen.webui.app:main"
abogen-pyqt = "abogen.pyqt.main:main"
[tool.hatch.build.targets.sdist]
exclude = [
+84
View File
@@ -0,0 +1,84 @@
import unittest
import os
import sys
import shutil
import time
from PyQt6.QtWidgets import QApplication
# Ensure we can import the module
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from abogen.book_handler import HandlerDialog
from ebooklib import epub
# We need a QApplication instance for QWriter/QDialog
app = QApplication(sys.argv)
class TestBookHandlerRegression(unittest.TestCase):
def setUp(self):
self.test_dir = "tests/test_data_handler"
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
os.makedirs(self.test_dir)
self.sample_epub_path = os.path.join(self.test_dir, "test_book.epub")
self._create_sample_epub()
def tearDown(self):
HandlerDialog.clear_content_cache()
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
def _create_sample_epub(self):
book = epub.EpubBook()
book.set_identifier("id123456")
book.set_title("Sample Book")
book.set_language("en")
c1 = epub.EpubHtml(title="Intro", file_name="intro.xhtml", lang="en")
c1.content = "<h1>Introduction</h1><p>Welcome to the book.</p>"
book.add_item(c1)
book.spine = ["nav", c1]
book.add_item(epub.EpubNcx())
book.add_item(epub.EpubNav())
epub.write_epub(self.sample_epub_path, book)
def test_handler_initialization(self):
"""Test that HandlerDialog processes the book correctly."""
# HandlerDialog starts processing in a background thread in __init__
# We assume headless environment, so we won't show it.
# But we need to wait for the thread to finish.
dialog = HandlerDialog(self.sample_epub_path)
# Wait for thread to finish
# The dialog emits no signal publicly, but we can check internal state or thread
start_time = time.time()
while time.time() - start_time < 5:
# HandlerDialog logic:
# _loader_thread.finished connect to _on_load_finished
# _on_load_finished populates content_texts and content_lengths
# We can check if content_texts is populated
if dialog.content_texts:
break
app.processEvents() # Process Qt events to let thread signals propagate
time.sleep(0.1)
self.assertTrue(len(dialog.content_texts) > 0, "HandlerDialog failed to process content in time")
# Validate content similar to what we expect
# intro.xhtml should be there
found_intro = False
for key, text in dialog.content_texts.items():
if "Welcome to the book" in text:
found_intro = True
break
self.assertTrue(found_intro)
# Cleanup
dialog.close()
if __name__ == "__main__":
unittest.main()
+210
View File
@@ -0,0 +1,210 @@
import unittest
import os
import sys
import shutil
import fitz # PyMuPDF
from ebooklib import epub
# Ensure we can import the module
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from abogen.book_parser import get_book_parser, PdfParser, EpubParser, MarkdownParser
class TestBookParser(unittest.TestCase):
def setUp(self):
self.test_dir = "tests/test_data"
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
os.makedirs(self.test_dir)
self.sample_pdf_path = os.path.join(self.test_dir, "test_book.pdf")
self.sample_epub_path = os.path.join(self.test_dir, "test_book.epub")
self.sample_md_path = os.path.join(self.test_dir, "test_book.md")
self._create_sample_pdf()
self._create_sample_epub()
self._create_sample_md()
def tearDown(self):
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
def _create_sample_pdf(self):
doc = fitz.open()
# Page 1
page1 = doc.new_page()
page1.insert_text((50, 50), "Page 1 content")
# Add pattern to be cleaned
page1.insert_text((50, 100), "[12]")
page1.insert_text((50, 200), "1") # Page number at bottom
# Page 2
page2 = doc.new_page()
page2.insert_text((50, 50), "Page 2 content")
doc.save(self.sample_pdf_path)
doc.close()
def _create_sample_epub(self):
book = epub.EpubBook()
book.set_identifier("id123456")
book.set_title("Sample Book")
book.set_language("en")
book.add_author("Test Author")
c1 = epub.EpubHtml(title="Intro", file_name="intro.xhtml", lang="en")
c1.content = "<h1>Introduction</h1><p>Welcome to the book.</p>"
c2 = epub.EpubHtml(title="Chapter 1", file_name="chap1.xhtml", lang="en")
c2.content = "<h1>Chapter 1</h1><ol><li>Item One</li><li>Item Two</li></ol>"
book.add_item(c1)
book.add_item(c2)
# Basic spine and nav
book.spine = ["nav", c1, c2]
# Add NCX and NAV for compatibility
book.add_item(epub.EpubNcx())
book.add_item(epub.EpubNav())
epub.write_epub(self.sample_epub_path, book)
def _create_sample_md(self):
content = "# Chapter 1\nSome text.\n# Chapter 2\nMore text."
with open(self.sample_md_path, "w") as f:
f.write(content)
def test_factory_returns_correct_class(self):
"""Test that get_book_parser returns the correct subclass based on extension."""
parser_pdf = get_book_parser(self.sample_pdf_path)
self.assertIsInstance(parser_pdf, PdfParser)
parser_md = get_book_parser(self.sample_md_path)
self.assertIsInstance(parser_md, MarkdownParser)
parser_epub = get_book_parser(self.sample_epub_path)
self.assertIsInstance(parser_epub, EpubParser)
def test_factory_explicit_type(self):
"""Test that explicit file type argument overrides extension."""
# 1. Copy sample epub to something.pdf
wrong_ext_path = os.path.join(self.test_dir, "actually_epub.pdf")
shutil.copy(self.sample_epub_path, wrong_ext_path)
# 2. Open it telling parser it IS epub
parser = get_book_parser(wrong_ext_path, file_type="epub")
self.assertIsInstance(parser, EpubParser)
# Should load successfully
parser.load()
self.assertTrue(parser.book is not None)
def test_pdf_parser_content(self):
"""Test PdfParser content extraction."""
parser = get_book_parser(self.sample_pdf_path)
parser.process_content()
self.assertIn("page_1", parser.content_texts)
self.assertIn("page_2", parser.content_texts)
text1 = parser.content_texts["page_1"]
self.assertIn("Page 1 content", text1)
self.assertNotIn("[12]", text1)
def test_markdown_parser_content(self):
"""Test MarkdownParser splitting logic."""
parser = get_book_parser(self.sample_md_path)
parser.process_content()
# Should have Chapter 1 and Chapter 2 keys (actual keys depend on ID generation)
# Markdown extensions might slugify IDs: "chapter-1"
self.assertIn("chapter-1", parser.content_texts)
self.assertIn("chapter-2", parser.content_texts)
self.assertIn("Some text", parser.content_texts["chapter-1"])
def test_epub_parser_content(self):
"""Test EpubParser processing."""
parser = get_book_parser(self.sample_epub_path)
parser.process_content()
self.assertIn("intro.xhtml", parser.content_texts)
self.assertIn("chap1.xhtml", parser.content_texts)
self.assertIn("Welcome to the book", parser.content_texts["intro.xhtml"])
def test_epub_metadata_extraction(self):
"""Test metadata extraction in EpubParser."""
parser = get_book_parser(self.sample_epub_path)
# Processing content triggers metadata extraction in current implementation
parser.process_content()
metadata = parser.get_metadata()
self.assertEqual(metadata.get("title"), "Sample Book")
self.assertEqual(metadata.get("author"), "Test Author")
def test_ordered_list_handling(self):
"""Test <ol> handling in EpubParser."""
parser = get_book_parser(self.sample_epub_path)
parser.process_content()
text = parser.content_texts.get("chap1.xhtml", "")
self.assertIn("1) Item One", text)
self.assertIn("2) Item Two", text)
def test_find_position_robust_logic(self):
"""Unit test for _find_position_robust on EpubParser."""
parser = EpubParser(self.sample_epub_path) # Instantiate directly
html = '<html><body><p>Start</p><h1 id="target">Heading</h1><p>End</p></body></html>'
parser.doc_content["dummy.html"] = html
# Test finding ID
pos = parser._find_position_robust("dummy.html", "target")
self.assertGreater(pos, 0)
self.assertTrue(html[pos:].startswith('<h1 id="target"'))
# Test missing ID
pos_missing = parser._find_position_robust("dummy.html", "missing")
self.assertEqual(pos_missing, 0)
def test_get_chapters(self):
"""Test get_chapters returns correct list for different parsers."""
# PDF
parser_pdf = get_book_parser(self.sample_pdf_path)
chapters = parser_pdf.get_chapters()
self.assertEqual(len(chapters), 2)
self.assertEqual(chapters[0], ("page_1", "Page 1"))
# MD
parser_md = get_book_parser(self.sample_md_path)
parser_md.process_content() # Must process to get structure
chapters_md = parser_md.get_chapters()
# Expecting chapter-1, chapter-2
ids = [c[0] for c in chapters_md]
self.assertIn("chapter-1", ids)
def test_get_formatted_text(self):
"""Test formatting of full text via BaseBookParser method."""
parser = get_book_parser(self.sample_md_path)
parser.process_content()
text = parser.get_formatted_text()
self.assertIn("<<CHAPTER_MARKER:Chapter 1>>", text)
self.assertIn("Some text", text)
def test_file_type_property(self):
"""Test that file_type property returns correct string for each parser."""
pdf_parser = PdfParser(self.sample_pdf_path)
self.assertEqual(pdf_parser.file_type, "pdf")
epub_parser = EpubParser(self.sample_epub_path)
self.assertEqual(epub_parser.file_type, "epub")
md_parser = MarkdownParser(self.sample_md_path)
self.assertEqual(md_parser.file_type, "markdown")
if __name__ == "__main__":
unittest.main()
+199
View File
@@ -0,0 +1,199 @@
import unittest
import os
import shutil
import sys
from ebooklib import epub
# Ensure import path
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from abogen.book_parser import get_book_parser
class TestEpubContentSlicing(unittest.TestCase):
"""
Tests for the complex content slicing logic in _execute_nav_parsing_logic.
This covers scenarios where multiple chapters/sections are contained within
a single physical HTML file, separated by anchors (fragments).
"""
def setUp(self):
self.test_dir = "tests/test_data_slicing"
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
os.makedirs(self.test_dir)
self.epub_path = os.path.join(self.test_dir, "slicing_test.epub")
def tearDown(self):
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
def test_single_file_multiple_chapters(self):
"""
Test splitting one XHTML file into two chapters using an anchor.
"""
book = epub.EpubBook()
book.set_identifier("slice123")
book.set_title("Slicing Test Book")
# Create a single content file with two sections
content_html = """
<html>
<body>
<h1 id="chap1">Chapter 1</h1>
<p>Text for chapter 1.</p>
<hr/>
<h1 id="chap2">Chapter 2</h1>
<p>Text for chapter 2.</p>
</body>
</html>
"""
c1 = epub.EpubHtml(title="Full Content", file_name="content.xhtml", lang="en")
c1.content = content_html
book.add_item(c1)
# Create Nav that points to anchors in the SAME file
# We use EpubHtml for Nav to control content exactly without ebooklib interference
nav_html = """
<nav epub:type="toc" id="toc">
<ol>
<li><a href="content.xhtml#chap1">Chapter 1</a></li>
<li><a href="content.xhtml#chap2">Chapter 2</a></li>
</ol>
</nav>
"""
nav = epub.EpubHtml(title="Nav", file_name="nav.xhtml")
nav.content = nav_html
book.add_item(nav)
book.spine = [nav, c1]
epub.write_epub(self.epub_path, book)
# OPF Patching to valid crash
import zipfile
patched = False
with zipfile.ZipFile(self.epub_path, 'r') as zin:
opf_content = zin.read('EPUB/content.opf').decode('utf-8')
if 'toc="ncx"' in opf_content:
opf_content = opf_content.replace('toc="ncx"', '')
patched = True
if patched:
TEMP_EPUB = self.epub_path + ".temp"
with zipfile.ZipFile(TEMP_EPUB, 'w') as zout:
for item in zin.infolist():
if item.filename == 'EPUB/content.opf':
zout.writestr(item, opf_content)
else:
zout.writestr(item, zin.read(item.filename))
if patched:
shutil.move(TEMP_EPUB, self.epub_path)
# Parse
parser = get_book_parser(self.epub_path)
parser.process_content()
chapters = parser.get_chapters()
# Filter Nav/Intro
chapters = [c for c in chapters if "Chapter" in c[1]]
self.assertEqual(len(chapters), 2)
self.assertEqual(chapters[0][1], "Chapter 1")
self.assertEqual(chapters[1][1], "Chapter 2")
# Check content of Chapter 1
# It should contain "Text for chapter 1" but NOT "Text for chapter 2"
# The parser logic slices from start_pos to next_pos
text1 = parser.content_texts[chapters[0][0]]
self.assertIn("Text for chapter 1", text1)
self.assertNotIn("Text for chapter 2", text1)
# Check content of Chapter 2
text2 = parser.content_texts[chapters[1][0]]
self.assertIn("Text for chapter 2", text2)
def test_list_renumbering(self):
"""
Test that ordered lists are re-numbered when slicing.
The parser has logic to reset <ol start="..."> or insert numbers.
"""
book = epub.EpubBook()
book.set_identifier("list123")
book.set_title("List Test Book")
content_html = """
<html>
<body>
<h1 id="part1">Part 1</h1>
<ol>
<li>Item A</li>
<li>Item B</li>
</ol>
<h1 id="part2">Part 2</h1>
<ol start="3">
<li>Item C</li>
<li>Item D</li>
</ol>
</body>
</html>
"""
c1 = epub.EpubHtml(title="Content", file_name="content.xhtml", lang="en")
c1.content = content_html
book.add_item(c1)
nav_html = """
<nav epub:type="toc">
<ol>
<li><a href="content.xhtml#part1">Part 1</a></li>
<li><a href="content.xhtml#part2">Part 2</a></li>
</ol>
</nav>
"""
nav = epub.EpubHtml(title="Nav", file_name="nav.xhtml")
nav.content = nav_html
book.add_item(nav)
book.spine = [nav, c1]
epub.write_epub(self.epub_path, book)
# Patch
import zipfile
patched = False
with zipfile.ZipFile(self.epub_path, 'r') as zin:
opf_content = zin.read('EPUB/content.opf').decode('utf-8')
if 'toc="ncx"' in opf_content:
opf_content = opf_content.replace('toc="ncx"', '')
patched = True
if patched:
TEMP_EPUB = self.epub_path + ".temp"
with zipfile.ZipFile(TEMP_EPUB, 'w') as zout:
for item in zin.infolist():
if item.filename == 'EPUB/content.opf':
zout.writestr(item, opf_content)
else:
zout.writestr(item, zin.read(item.filename))
if patched:
shutil.move(TEMP_EPUB, self.epub_path)
parser = get_book_parser(self.epub_path)
parser.process_content()
chapters = parser.get_chapters()
chapters = [c for c in chapters if "Part" in c[1]]
self.assertEqual(len(chapters), 2)
# Check Part 1 text
text1 = parser.content_texts[chapters[0][0]]
# The parser explicitly replaces li with "1) Item A" style text
self.assertIn("1) Item A", text1)
self.assertIn("2) Item B", text1)
# Check Part 2 text
text2 = parser.content_texts[chapters[1][0]]
# Should convert start="3" to "3) Item C"
self.assertIn("3) Item C", text2)
self.assertIn("4) Item D", text2)
if __name__ == "__main__":
unittest.main()
+117
View File
@@ -0,0 +1,117 @@
import unittest
import os
import shutil
import sys
from ebooklib import epub
# Ensure import path
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from abogen.book_parser import get_book_parser
class TestEpubHeuristicNav(unittest.TestCase):
"""
Tests for the heuristic fallback in _identify_nav_item (Step 4),
where the parser scans ITEM_DOCUMENTs for <nav epub:type="toc">
when no explicit ITEM_NAVIGATION is found.
"""
def setUp(self):
self.test_dir = "tests/test_data_heuristic"
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
os.makedirs(self.test_dir)
self.epub_path = os.path.join(self.test_dir, "heuristic_test.epub")
def tearDown(self):
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
def test_heuristic_nav_discovery(self):
book = epub.EpubBook()
book.set_identifier("heuristic123")
book.set_title("Heuristic Test Book")
# 1. Add Content
c1 = epub.EpubHtml(title="Chapter 1", file_name="chap1.xhtml", lang="en")
c1.content = "<h1>Chapter 1</h1><p>Text</p>"
book.add_item(c1)
# 2. Add a Nav file BUT as a regular EpubHtml (ITEM_DOCUMENT)
# We do NOT use EpubNav. We do NOT look like a standard nav file if possible,
# but content must contain the magical signature.
nav_content = """
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops">
<body>
<nav epub:type="toc" id="toc">
<h1>Hidden TOC</h1>
<ol>
<li><a href="chap1.xhtml">Chapter 1</a></li>
</ol>
</nav>
</body>
</html>
"""
# Filename intentionally generic/obscure to avoid filename-based heuristics
# (though current code checks content, not just filename)
nav_file = epub.EpubHtml(title="Hidden Nav", file_name="content_toc.xhtml")
nav_file.content = nav_content
book.add_item(nav_file)
# 3. Setup Spine
book.spine = [nav_file, c1]
# 4. Write EPUB
epub.write_epub(self.epub_path, book)
# 5. Patch OPF to ensure ebooklib didn't sneakily add ITEM_NAVIGATION or toc="ncx"
import zipfile
patched = False
with zipfile.ZipFile(self.epub_path, 'r') as zin:
opf_content = zin.read('EPUB/content.opf').decode('utf-8')
# Remove toc="ncx" attribute if present (causes crash if no NCX)
if 'toc="ncx"' in opf_content:
opf_content = opf_content.replace('toc="ncx"', '')
patched = True
# Ideally we'd verify properties="nav" isn't there, but EpubHtml shouldn't add it.
# If ebooklib added it, we might need to strip it to force heuristic.
if 'properties="nav"' in opf_content:
opf_content = opf_content.replace('properties="nav"', '')
patched = True
if patched:
TEMP_EPUB = self.epub_path + ".temp"
with zipfile.ZipFile(TEMP_EPUB, 'w') as zout:
for item in zin.infolist():
if item.filename == 'EPUB/content.opf':
zout.writestr(item, opf_content)
else:
zout.writestr(item, zin.read(item.filename))
if patched:
shutil.move(TEMP_EPUB, self.epub_path)
# 6. Verify our setup: Ensure NO ITEM_NAVIGATION exists
# We can inspect using ebooklib again
import ebooklib
check_book = epub.read_epub(self.epub_path)
nav_items = list(check_book.get_items_of_type(ebooklib.ITEM_NAVIGATION))
self.assertEqual(len(nav_items), 0, "Setup failed: explicit navigation item found!")
# 7. Run Parser
parser = get_book_parser(self.epub_path)
parser.process_content()
chapters = parser.get_chapters()
# 8. Assertions
# Should have found the nav via content scanning
chapter_titles = [c[1] for c in chapters]
self.assertIn("Chapter 1", chapter_titles)
# Also verify we hit the "html" type in identification
# We can't easily check private variables, but success implies it worked.
if __name__ == "__main__":
unittest.main()
+184
View File
@@ -0,0 +1,184 @@
import unittest
import os
import shutil
import sys
from ebooklib import epub
# Ensure import path
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from abogen.book_parser import get_book_parser
class TestEpubHtmlNavParsing(unittest.TestCase):
"""
Tests for EPUB 3 HTML5 Navigation Document parsing logic (_parse_html_nav_li).
"""
def setUp(self):
self.test_dir = "tests/test_data_nav"
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
os.makedirs(self.test_dir)
self.epub_path = os.path.join(self.test_dir, "nav_test.epub")
def tearDown(self):
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
def _create_epub_with_custom_nav(self, nav_html_content):
"""
Creates an EPUB with a manually injected HTML Navigation Document.
"""
book = epub.EpubBook()
book.set_identifier("navtest123")
book.set_title("Nav Test Book")
# Add some content files
c1 = epub.EpubHtml(title="Chapter 1", file_name="chap1.xhtml", lang="en")
c1.content = "<h1>Chapter 1</h1><p>Text 1</p>"
book.add_item(c1)
c2 = epub.EpubHtml(title="Chapter 2", file_name="chap2.xhtml", lang="en")
c2.content = "<h1>Chapter 2</h1><p>Text 2</p>"
book.add_item(c2)
# Create the Nav item manually to control the HTML structure exactly
# Use EpubHtml + OPF patching because EpubNav forces auto-generation
nav = epub.EpubHtml(title="Nav", file_name="nav.xhtml")
nav.content = nav_html_content
book.add_item(nav)
# We must set spine manually
book.spine = [nav, c1, c2]
epub.write_epub(self.epub_path, book)
# Patch the OPF to remove toc="ncx" default which causes crash
# because we intentionally excluded the legacy NCX file.
import zipfile
with zipfile.ZipFile(self.epub_path, 'r') as zin:
opf_content = zin.read('EPUB/content.opf').decode('utf-8')
opf_content = opf_content.replace('toc="ncx"', '')
# Repack
TEMP_EPUB = self.epub_path + ".temp"
with zipfile.ZipFile(TEMP_EPUB, 'w') as zout:
for item in zin.infolist():
if item.filename == 'EPUB/content.opf':
zout.writestr(item, opf_content)
else:
zout.writestr(item, zin.read(item.filename))
shutil.move(TEMP_EPUB, self.epub_path)
def test_basic_html_nav_parsing(self):
"""
Test parsing of a standard flat list of links.
"""
nav_html = """
<nav epub:type="toc" id="toc">
<h1>Table of Contents</h1>
<ol>
<li><a href="chap1.xhtml">Chapter 1</a></li>
<li><a href="chap2.xhtml">Chapter 2</a></li>
</ol>
</nav>
"""
self._create_epub_with_custom_nav(nav_html)
parser = get_book_parser(self.epub_path)
parser.process_content()
chapters = parser.get_chapters()
# Filter out "Nav" or "Introduction" prefix content found from the Nav file itself
chapters = [c for c in chapters if "Chapter" in c[1] or "Section" in c[1]]
self.assertEqual(len(chapters), 2)
self.assertEqual(chapters[0][1], "Chapter 1")
self.assertEqual(chapters[1][1], "Chapter 2")
def test_nested_html_nav_parsing(self):
"""
Test parsing of nested lists (Sub-chapters).
"""
nav_html = """
<nav epub:type="toc">
<ol>
<li>
<a href="chap1.xhtml">Chapter 1</a>
<ol>
<li><a href="chap2.xhtml">Section 1.1</a></li>
</ol>
</li>
</ol>
</nav>
"""
# Note: In this test setup, chap2 is serving as "Section 1.1" effectively
self._create_epub_with_custom_nav(nav_html)
parser = get_book_parser(self.epub_path)
parser.process_content()
chapters = parser.get_chapters()
ids = [c[1] for c in chapters]
self.assertIn("Chapter 1", ids)
self.assertIn("Section 1.1", ids)
def test_span_header_parsing(self):
"""
Test parsing of <li><span>Header</span><ol>...</ol></li> pattern.
This represents a grouping header that isn't a link itself.
"""
nav_html = """
<nav epub:type="toc">
<ol>
<li>
<span>Part I</span>
<ol>
<li><a href="chap1.xhtml">Chapter 1</a></li>
</ol>
</li>
</ol>
</nav>
"""
self._create_epub_with_custom_nav(nav_html)
parser = get_book_parser(self.epub_path)
parser.process_content()
chapters = parser.get_chapters()
chapter_titles = [c[1] for c in chapters]
self.assertIn("Chapter 1", chapter_titles)
self.assertNotIn("Part I", chapter_titles)
# Check internal structure
# Find the node named "Part I" in the processed structure
root_node = next(node for node in parser.processed_nav_structure if node['title'] == "Part I")
self.assertEqual(root_node['title'], "Part I")
self.assertFalse(root_node['has_content'])
self.assertEqual(len(root_node['children']), 1)
self.assertEqual(root_node['children'][0]['title'], "Chapter 1")
def test_identify_nav_item(self):
"""Test the _identify_nav_item method specifically."""
nav_html = """
<nav epub:type="toc" id="toc"><h1>TOC</h1><ol><li><a href="c1.html">C1</a></li></ol></nav>
"""
self._create_epub_with_custom_nav(nav_html)
parser = get_book_parser(self.epub_path)
# Note: _identify_nav_item relies on self.book being loaded
# The parser constructor or process_content handles load()
# But here we can call load directly if needed, or rely on normal flow up until navigation
parser.load()
nav_item, nav_type = parser._identify_nav_item()
self.assertEqual(nav_type, "html")
self.assertIsNotNone(nav_item)
self.assertTrue("nav.xhtml" in nav_item.get_name())
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,100 @@
import unittest
import os
import shutil
import zipfile
import sys
import logging
from ebooklib import epub
# Ensure import path
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from abogen.book_parser import get_book_parser
class TestEpubMissingFileErrorHandling(unittest.TestCase):
"""
Tests for robust error handling and recovery in the book parser.
"""
def setUp(self):
self.test_dir = "tests/test_data_errors"
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
os.makedirs(self.test_dir)
self.broken_epub_path = os.path.join(self.test_dir, "missing_file.epub")
# Suppress logging during tests to keep output clean,
# or capture it if we want to assert on warnings.
# For now, we just let it be or set level to ERROR.
logging.getLogger().setLevel(logging.ERROR)
def tearDown(self):
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
def _create_broken_epub(self):
"""
Creates an EPUB where a file listed in the manifest is missing from the ZIP archive.
"""
book = epub.EpubBook()
book.set_identifier("broken123")
book.set_title("Broken Book")
# 1. Add a valid chapter
c1 = epub.EpubHtml(title="Chapter 1", file_name="chap1.xhtml", lang="en")
c1.content = "<h1>Chapter 1</h1><p>Survivable content.</p>"
book.add_item(c1)
# 2. Add a 'ghost' chapter that we will delete later
c2 = epub.EpubHtml(title="Ghost Chapter", file_name="ghost.xhtml", lang="en")
c2.content = "<h1>Ghost</h1><p>I will disappear.</p>"
book.add_item(c2)
book.spine = ["nav", c1, c2]
book.add_item(epub.EpubNcx())
book.add_item(epub.EpubNav())
temp_path = os.path.join(self.test_dir, "temp.epub")
epub.write_epub(temp_path, book)
# 3. Physically remove 'ghost.xhtml' from the ZIP
with zipfile.ZipFile(temp_path, 'r') as zin:
with zipfile.ZipFile(self.broken_epub_path, 'w') as zout:
for item in zin.infolist():
# Copy everything EXCEPT the ghost file
# Note: ebooklib might put files in OEPS/ or EPUB/ folders depending on version,
# so checking "ghost.xhtml" presence in filename is safer.
if "ghost.xhtml" not in item.filename:
zout.writestr(item, zin.read(item.filename))
def test_missing_file_recovery(self):
"""
Verify that the parser recovers gracefully when a referenced file is missing.
Should log a warning instead of raising KeyError.
"""
self._create_broken_epub()
try:
parser = get_book_parser(self.broken_epub_path)
parser.process_content()
# 1. Ensure process didn't crash
self.assertTrue(True, "Parser should not crash on missing file")
# 2. Ensure valid content was extracted
# Identify the ID for chap1.xhtml (usually file path based)
# Since IDs can vary, we check if ANY content contains our known string
chap1_found = False
for text in parser.content_texts.values():
if "Survivable content" in text:
chap1_found = True
break
self.assertTrue(chap1_found, "The valid chapter should still be processed")
except KeyError:
self.fail("Parser raised KeyError instead of handling the missing file!")
except Exception as e:
self.fail(f"Parser raised unexpected exception: {e}")
if __name__ == "__main__":
unittest.main()
+140
View File
@@ -0,0 +1,140 @@
import unittest
import os
import shutil
import sys
from ebooklib import epub
# Ensure we can import the module
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from abogen.book_parser import get_book_parser, EpubParser
class TestEpubNcxParsing(unittest.TestCase):
"""
Focused tests for NCX navigation scenarios, ensuring legacy/compatibility
modes work when HTML5 Navigation is missing.
"""
def setUp(self):
self.test_dir = "tests/test_data_ncx"
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
os.makedirs(self.test_dir)
self.ncx_only_epub_path = os.path.join(self.test_dir, "ncx_only.epub")
def tearDown(self):
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
def _create_ncx_only_epub(self, chapters):
"""
Helper to create an EPUB with ONLY NCX table of contents (no HTML nav).
"""
book = epub.EpubBook()
book.set_identifier("ncx_test_123")
book.set_title("NCX Only Book")
book.set_language("en")
epub_chapters = []
for i, (title, content) in enumerate(chapters):
filename = f"chap{i+1}.xhtml"
c = epub.EpubHtml(title=title, file_name=filename, lang="en")
# Ensure content is substantial enough to not be skipped
c.content = f"<h1>{title}</h1><p>{content}</p>"
book.add_item(c)
epub_chapters.append(c)
# Define Table of Contents
book.toc = tuple(epub_chapters)
# Add default NCX and generic spine
book.add_item(epub.EpubNcx())
# IMPORTANT: Do NOT add EpubNav() here, that's what we are testing!
book.spine = ["nav"] + epub_chapters
epub.write_epub(self.ncx_only_epub_path, book)
def test_ncx_only_parsing(self):
"""
Verify that an EPUB with only an NCX file (no HTML nav) is parsed correctly.
Logic tested: _process_epub_content_nav (NCX branch), _parse_ncx_navpoint
"""
# 1. Setup Data
chapters_data = [
("Chapter 1", "This is the first chapter."),
("Chapter 2", "This is the second chapter.")
]
self._create_ncx_only_epub(chapters_data)
# 2. Run Parser
parser = get_book_parser(self.ncx_only_epub_path)
parser.process_content()
# 3. Verify Breakdown
# We expect detailed breakdown based on NCX
chapters = parser.get_chapters()
# Should find exactly 2 chapters based on the Toc
self.assertEqual(len(chapters), 2, "Should have 2 chapters extracted from NCX")
# Check Titles and Sequence
self.assertEqual(chapters[0][1], "Chapter 1")
self.assertEqual(chapters[1][1], "Chapter 2")
# Verify content was extracted
# Note: 'src' in chapters usually points to file_name if no fragments
id_1 = chapters[0][0]
self.assertIn("This is the first chapter", parser.content_texts[id_1])
def test_nested_ncx_parsing(self):
"""
Verify parsing of nested NCX structures (Chapters with Subchapters).
"""
book = epub.EpubBook()
book.set_identifier("nested_ncx")
book.set_title("Nested NCX")
# Create one big file with sections
c1 = epub.EpubHtml(title="Main Chapter", file_name="main.xhtml", lang="en")
c1.content = """
<h1 id="intro">Introduction</h1>
<p>Intro text.</p>
<h2 id="sect1">Section 1</h2>
<p>Section 1 text.</p>
"""
book.add_item(c1)
# Manually construct nested TOC because ebooklib's default helpers are simple
# EbookLib automatically builds NCX from book.toc
# Nested tuple structure: (Section, (Subsection, Sub-subsection))
# We need to link to Fragments for this to really test nested NCX pointing to same file
# EbookLib Link object: epub.Link(href, title, uid)
link_root = epub.Link("main.xhtml#intro", "Introduction", "intro")
link_sect = epub.Link("main.xhtml#sect1", "Section 1", "sect1")
# Structure: Intro -> Section 1 (as child)
book.toc = (
(link_root, (link_sect, )),
)
book.add_item(epub.EpubNcx())
book.spine = ["nav", c1]
epub.write_epub(self.ncx_only_epub_path, book)
# Parse
parser = get_book_parser(self.ncx_only_epub_path)
parser.process_content()
chapters = parser.get_chapters()
# Depending on how the parser flattens, we should see both entries
titles = [node[1] for node in chapters]
self.assertIn("Introduction", titles)
self.assertIn("Section 1", titles)
if __name__ == "__main__":
unittest.main()
+130
View File
@@ -0,0 +1,130 @@
import unittest
import os
import shutil
import sys
from ebooklib import epub
import ebooklib
from unittest.mock import MagicMock
# Ensure import path
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from abogen.book_parser import get_book_parser
class TestEpubStandardNav(unittest.TestCase):
"""
Tests for the standard ITEM_NAVIGATION discovery in _identify_nav_item.
Refactored to explicitly test different discovery paths defined in the parser.
"""
def setUp(self):
self.test_dir = "tests/test_data_standard_nav"
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
os.makedirs(self.test_dir)
self.epub_path = os.path.join(self.test_dir, "standard_nav_test.epub")
def tearDown(self):
if os.path.exists(self.test_dir):
shutil.rmtree(self.test_dir)
def _create_and_load_epub(self):
"""Helper to create a basic EPUB and return a loaded parser."""
book = epub.EpubBook()
book.set_identifier("stdnav123")
book.set_title("Standard Nav Test")
c1 = epub.EpubHtml(title="Chapter 1", file_name="chap1.xhtml", lang="en")
c1.content = "<h1>Chapter 1</h1><p>Text 1</p>"
book.add_item(c1)
# Use Standard EpubNav
nav = epub.EpubNav()
book.add_item(nav)
book.spine = [nav, c1]
epub.write_epub(self.epub_path, book)
# "Zip Surgery" Patch:
# ebooklib unconditionally adds `toc="ncx"` to the spine, even for EPUB 3 files that purely use HTML Nav.
# This creates a dangling reference to a non-existent "ncx" item, causing ebooklib to crash on read.
# We manually remove this attribute to ensure the test EPUB is valid and readable.
# TODO - find real world examples of EPUB 3 files that use HTML Nav
import zipfile
patched = False
with zipfile.ZipFile(self.epub_path, 'r') as zin:
opf_content = zin.read('EPUB/content.opf').decode('utf-8')
if 'toc="ncx"' in opf_content:
opf_content = opf_content.replace('toc="ncx"', '')
patched = True
TEMP_EPUB = self.epub_path + ".temp"
with zipfile.ZipFile(TEMP_EPUB, 'w') as zout:
for item in zin.infolist():
if item.filename == 'EPUB/content.opf':
zout.writestr(item, opf_content)
else:
zout.writestr(item, zin.read(item.filename))
if patched:
shutil.move(TEMP_EPUB, self.epub_path)
parser = get_book_parser(self.epub_path)
parser.load()
return parser
def test_discovery_by_item_navigation_type(self):
"""
Scenario 1: The item is explicitly identified as ITEM_NAVIGATION (4).
This exercises the first branch of _identify_nav_item.
"""
parser = self._create_and_load_epub()
# Inject an item that mocks the ITEM_NAVIGATION type behavior
# (This simulates a library/parser that correctly types the item as 4)
mock_nav = MagicMock()
mock_nav.get_name.return_value = "nav.xhtml"
mock_nav.get_type.return_value = ebooklib.ITEM_NAVIGATION
# We append this mock to the book items to ensure get_items_of_type(ITEM_NAVIGATION) finds it
parser.book.items.append(mock_nav)
nav_item, nav_type = parser._identify_nav_item()
self.assertEqual(nav_type, "html")
self.assertEqual(nav_item.get_name(), "nav.xhtml")
# Verify we are getting the object we expect (implied by success)
def test_discovery_by_nav_property(self):
"""
Scenario 2: The item is ITEM_DOCUMENT (9) but has properties=['nav'].
This is the standard EPUB 3 behavior and exercises the fallback branch.
"""
parser = self._create_and_load_epub()
# Locate the generic 'nav' item loaded by ebooklib
original_nav = parser.book.get_item_with_id("nav")
self.assertIsNotNone(original_nav)
# "Fix" the object to match what we expect from a correct EPUB 3 read:
# It should have properties=['nav'].
# We use a real EpubNav object to ensure structural correctness.
proper_nav = epub.EpubNav(uid=original_nav.id, file_name=original_nav.file_name)
proper_nav.content = original_nav.content
proper_nav.properties = ['nav']
# Swap it into the book items list
try:
idx = parser.book.items.index(original_nav)
parser.book.items[idx] = proper_nav
except ValueError:
self.fail("Could not find original nav item to swap")
nav_item, nav_type = parser._identify_nav_item()
self.assertEqual(nav_type, "html")
self.assertEqual(nav_item.get_name(), "nav.xhtml")
# Check that we actually found the one with properties
self.assertEqual(getattr(nav_item, 'properties', []), ['nav'])
if __name__ == "__main__":
unittest.main()