abogen/docs/testing.md

Testing Guide

This document describes the testing strategy for Abogen's Plugin Architecture.

Test Categories

1. Contract Tests (tests/contracts/)

Purpose: Verify that every plugin satisfies the architectural contract. These tests ensure the Plugin Architecture's invariants are maintained.

What They Guarantee:

• Every plugin exports PLUGIN_MANIFEST, MODEL_REQUIREMENTS, create_engine
• create_engine is atomic (succeeds fully or raises and cleans up)
• Engine.createSession() returns valid EngineSession, transfers ownership
• Engine.dispose() is idempotent, never raises
• After dispose(), all methods raise EngineError
• EngineSession.synthesize() returns SynthesizedAudio or raises EngineError (session remains usable)
• EngineSession.dispose() is idempotent, never raises
• Capability interfaces (VoiceLister, PreviewGenerator, etc.) are correctly implemented
• Plugin Loader discovers, validates, and loads plugins correctly
• Plugin Manager creates, caches, and disposes engines correctly
• Value objects are immutable and have correct equality semantics
• Error hierarchy is preserved (EngineError base with subtypes)

Why They Exist:

• Provide compile-time-like guarantees for a dynamic plugin system
• Enable safe plugin ecosystem — host can trust any loaded plugin
• Catch architectural violations early (missing dispose, wrong return types, etc.)
• Document the contract in executable form

What Every New Plugin Must Pass:

pytest tests/contracts/ -v
# All tests must pass

Running Contract Tests:

# All contract tests
pytest tests/contracts/

# Specific contract
pytest tests/contracts/test_engine_contract.py

# With coverage
pytest tests/contracts/ --cov=abogen.tts_plugin

---

2. Behavioral Tests (tests/test_behavioral_regression.py)

Purpose: Verify external user-facing behavior using only public API. These tests are not coupled to internal implementation.

What They Test:

• Synthesis with various inputs (short, long, empty, unicode, mixed scripts)
• Voice selection and listing
• Parameter handling (speed, etc.)
• Error scenarios (unknown plugin, disposal, etc.)
• Resource cleanup (dispose idempotency, no leaks)
• Pipeline utility (create_pipeline)

Why They Test Public Behavior Only:

• Refactoring safety: Internal changes don't break tests
• Real-world usage: Tests match how consumers actually use the API
• Plugin agnostic: Parametrized across Kokoro, SuperTonic, and mock plugins
• Regression detection: Catch behavioral regressions regardless of implementation

What They Don't Test:

• Internal class structure
• Private methods
• Implementation details (how audio is generated, model loading internals)

Running Behavioral Tests:

# All behavioral tests
pytest tests/test_behavioral_regression.py -v

# With specific plugin (if installed)
pytest tests/test_behavioral_regression.py -v -k "kokoro"

---

3. Unit Tests (tests/)

Purpose: Test individual modules in isolation.

Examples:

• test_book_parser.py — EPUB/PDF/text parsing
• test_text_normalization.py — Text preprocessing
• test_chunk_helpers.py — Text chunking logic
• test_voice_cache.py — Voice caching

---

4. Integration Tests

Purpose: Test cross-component interactions.

Examples:

• test_kokoro_plugin.py — Full Kokoro plugin integration
• test_supertonic_plugin.py — Full SuperTonic plugin integration
• test_conversion_series.py — End-to-end conversion pipeline

---

Test Architecture

tests/
├── contracts/           # Contract tests (architectural compliance)
│   ├── conftest.py      # Shared fixtures (FakeEngine, FakeSession)
│   ├── test_manifest_contract.py
│   ├── test_plugin_contract.py
│   ├── test_engine_contract.py
│   ├── test_session_contract.py
│   ├── test_capabilities_contract.py
│   ├── test_loader_contract.py
│   ├── test_plugin_manager_contract.py
│   ├── test_types_contract.py
│   ├── test_errors_contract.py
│   ├── test_host_context_contract.py
│   └── test_integration.py
├── test_behavioral_regression.py  # Behavioral tests (public API)
├── test_kokoro_plugin.py          # Kokoro integration
├── test_supertonic_plugin.py      # SuperTonic integration
└── ...                            # Other unit/integration tests

---

Adding Tests for a New Plugin

Contract Tests (Required)

Create tests/contracts/test_your_plugin.py:

"""Contract tests for YourPlugin — verifies architectural compliance."""

from pathlib import Path
from abogen.tts_plugin.loader import load_plugin_from_dir
from abogen.tts_plugin.manifest import PluginManifest
from abogen.tts_plugin.engine import Engine

def test_your_plugin_loads():
    result = load_plugin_from_dir(Path("plugins/your_tts"))
    assert result.success
    assert isinstance(result.manifest, PluginManifest)
    assert result.manifest.id == "your_tts"
    assert callable(result.create_engine)

def test_your_plugin_creates_engine():
    result = load_plugin_from_dir(Path("plugins/your_tts"))
    # ... create HostContext, EngineConfig
    engine = result.create_engine(ctx, None, EngineConfig(device="cpu"))
    assert isinstance(engine, Engine)
    engine.dispose()

def test_your_plugin_capabilities():
    """If plugin declares capabilities, verify they're implemented."""
    result = load_plugin_from_dir(Path("plugins/your_tts"))
    # Check VoiceLister, PreviewGenerator, etc.
    ...

Behavioral Tests (Recommended)

Add parametrized tests to tests/test_behavioral_regression.py:

# In _plugin_ids list, add your plugin
_plugin_ids = ["kokoro", "supertonic", "your_tts"]
_plugin_engines["your_tts"] = _YourMockEngine
_plugin_default_voices["your_tts"] = "voice1"
_plugin_all_voices["your_tts"] = ["voice1", "voice2"]

All existing behavioral tests will automatically run against your plugin.

---

Continuous Integration

# .github/workflows/test.yml
- name: Contract Tests
  run: pytest tests/contracts/ -v

- name: Behavioral Tests
  run: pytest tests/test_behavioral_regression.py -v

- name: Unit & Integration Tests
  run: pytest tests/ -v --ignore=tests/test_behavioral_regression.py

---

Test Design Principles

Contract Tests

• No mocks for the system under test (test real plugin loading)
• Strict assertions on types and behavior
• Document architecture in test names and docstrings
• Fail fast on architectural violations

Behavioral Tests

• Only public API (create_pipeline, Engine, EngineSession, PluginManager)
• Parametrized across plugins
• Realistic scenarios (long text, unicode, mixed scripts)
• No implementation coupling (test behavior, not internals)

General

• Fast: Unit tests < 1s, Contract tests < 5s, Behavioral < 30s
• Isolated: No shared state between tests
• Deterministic: Same input → same output
• Descriptive names: test_<component>_<scenario>_<expected>