abogen/docs/testing.md

Testing Guide

This document describes the testing strategy for Abogen's Plugin Architecture.

Test Categories

0. Auto-Discovery Plugin Tests (tests/plugins/)

Purpose: Automatically test every plugin in plugins/ directory without manual test creation. These tests use discovery to find all plugins and run generic tests against each one.

What They Test:

• Manifest structure: Required fields, API version format, voices field
• Engine lifecycle: create_engine, dispose idempotency, post-dispose behavior
• Capability implementation: Declared capabilities are implemented (e.g., voice_list → VoiceLister)

How Auto-Discovery Works:

# tests/plugins/conftest.py
@pytest.fixture(scope="module")
def plugin_ids(plugins_dir: Path) -> list[str]:
    """Discovers all plugin directories with __init__.py"""
    return [item.name for item in plugins_dir.iterdir() 
            if item.is_dir() and (item / "__init__.py").exists()]

Test Structure:

tests/plugins/
├── conftest.py              # Fixtures: plugin_ids, loaded_plugin, host_context
└── test_all_plugins.py      # Generic tests for every plugin
    ├── TestAllPluginsManifest
    ├── TestAllPluginsEngine
    └── TestAllPluginsCapabilities

Running Auto-Discovery Tests:

# Test all plugins automatically
pytest tests/plugins/ -v

# Test specific plugin
pytest tests/plugins/ -v -k "kokoro"

# See which plugins were discovered
pytest tests/plugins/ --collect-only

Adding a New Plugin:

1. Create plugin directory: plugins/my_plugin/
2. Add __init__.py with PLUGIN_MANIFEST, MODEL_REQUIREMENTS, create_engine
3. Run pytest tests/plugins/ — tests automatically discover and test your plugin!

When to Add Plugin-Specific Tests: Auto-discovery tests cover generic contract validation. Create plugin-specific tests in tests/test_<plugin>_plugin.py for:

• Integration with real dependencies (e.g., KPipeline for Kokoro)
• Specific voice IDs and behavior
• Plugin-specific parameters and features

---

1. Contract Tests (tests/contracts/)

Purpose: Verify that every plugin satisfies the architectural contract. These tests ensure the Plugin Architecture's invariants are maintained.

What They Guarantee:

• Every plugin exports PLUGIN_MANIFEST, MODEL_REQUIREMENTS, create_engine
• create_engine is atomic (succeeds fully or raises and cleans up)
• Engine.createSession() returns valid EngineSession, transfers ownership
• Engine.dispose() is idempotent, never raises
• After dispose(), all methods raise EngineError
• EngineSession.synthesize() returns SynthesizedAudio or raises EngineError (session remains usable)
• EngineSession.dispose() is idempotent, never raises
• Capability interfaces (VoiceLister, PreviewGenerator, etc.) are correctly implemented
• Plugin Loader discovers, validates, and loads plugins correctly
• Plugin Manager creates, caches, and disposes engines correctly
• Value objects are immutable and have correct equality semantics
• Error hierarchy is preserved (EngineError base with subtypes)

Why They Exist:

• Provide compile-time-like guarantees for a dynamic plugin system
• Enable safe plugin ecosystem — host can trust any loaded plugin
• Catch architectural violations early (missing dispose, wrong return types, etc.)
• Document the contract in executable form

What Every New Plugin Must Pass:

pytest tests/contracts/ -v
# All tests must pass

Running Contract Tests:

# All contract tests
pytest tests/contracts/

# Specific contract
pytest tests/contracts/test_engine_contract.py

# With coverage
pytest tests/contracts/ --cov=abogen.tts_plugin

---

2. Behavioral Tests (tests/test_behavioral_regression.py)

Purpose: Verify external user-facing behavior using only public API. These tests are not coupled to internal implementation.

What They Test:

• Synthesis with various inputs (short, long, empty, unicode, mixed scripts)
• Voice selection and listing
• Parameter handling (speed, etc.)
• Error scenarios (unknown plugin, disposal, etc.)
• Resource cleanup (dispose idempotency, no leaks)
• Pipeline utility (create_pipeline)

Why They Test Public Behavior Only:

• Refactoring safety: Internal changes don't break tests
• Real-world usage: Tests match how consumers actually use the API
• Plugin agnostic: Parametrized across Kokoro, SuperTonic, and mock plugins
• Regression detection: Catch behavioral regressions regardless of implementation

What They Don't Test:

• Internal class structure
• Private methods
• Implementation details (how audio is generated, model loading internals)

Running Behavioral Tests:

# All behavioral tests
pytest tests/test_behavioral_regression.py -v

# With specific plugin (if installed)
pytest tests/test_behavioral_regression.py -v -k "kokoro"

---

4. Unit Tests (tests/)

Purpose: Test individual modules in isolation.

Examples:

• test_book_parser.py — EPUB/PDF/text parsing
• test_text_normalization.py — Text preprocessing
• test_chunk_helpers.py — Text chunking logic
• test_voice_cache.py — Voice caching

---

5. Integration Tests

Purpose: Test cross-component interactions.

Examples:

• test_kokoro_plugin.py — Full Kokoro plugin integration
• test_supertonic_plugin.py — Full SuperTonic plugin integration
• test_conversion_series.py — End-to-end conversion pipeline

---

Test Architecture

tests/
├── contracts/           # Contract tests (architectural compliance)
│   ├── conftest.py      # Shared fixtures (FakeEngine, FakeSession)
│   ├── test_manifest_contract.py
│   ├── test_plugin_contract.py
│   ├── test_engine_contract.py
│   ├── test_session_contract.py
│   ├── test_capabilities_contract.py
│   ├── test_loader_contract.py
│   ├── test_plugin_manager_contract.py
│   ├── test_types_contract.py
│   ├── test_errors_contract.py
│   ├── test_host_context_contract.py
│   └── test_integration.py
├── test_behavioral_regression.py  # Behavioral tests (public API)
├── test_kokoro_plugin.py          # Kokoro integration
├── test_supertonic_plugin.py      # SuperTonic integration
└── ...                            # Other unit/integration tests

---

Adding Tests for a New Plugin

Auto-Discovery Tests (Automatic!)

No manual test creation required! When you add a new plugin to plugins/:

1. Create plugin directory: plugins/my_plugin/
2. Add __init__.py with required exports:

   PLUGIN_MANIFEST = PluginManifest(...)
   MODEL_REQUIREMENTS = [...]
   def create_engine(...): ...
   

3. Run pytest tests/plugins/ — auto-discovery tests automatically find and test your plugin!

What's Tested Automatically:

• Manifest structure and required fields
• API version compatibility
• Engine creation and dispose contract
• Capability implementation (if declared)

Plugin-Specific Tests (Optional)

Create tests/test_my_plugin_plugin.py for:

• Integration with real backend (e.g., KPipeline for Kokoro)
• Specific voice IDs and behavior
• Plugin-specific parameters and features

Contract Tests (Deprecated for New Plugins)

Note: Auto-discovery tests (tests/plugins/) now cover contract validation for all plugins. Manual contract tests in tests/contracts/ are only needed for testing internal architecture components.

Behavioral Tests (Recommended)

Add parametrized tests to tests/test_behavioral_regression.py:

# In _plugin_ids list, add your plugin
_plugin_ids = ["kokoro", "supertonic", "my_plugin"]
_plugin_engines["my_plugin"] = _YourMockEngine
_plugin_default_voices["my_plugin"] = "voice1"
_plugin_all_voices["my_plugin"] = ["voice1", "voice2"]

All existing behavioral tests will automatically run against your plugin.

---

Continuous Integration

# .github/workflows/test.yml
- name: Contract Tests
  run: pytest tests/contracts/ -v

- name: Behavioral Tests
  run: pytest tests/test_behavioral_regression.py -v

- name: Unit & Integration Tests
  run: pytest tests/ -v --ignore=tests/test_behavioral_regression.py

---

Test Design Principles

Contract Tests

• No mocks for the system under test (test real plugin loading)
• Strict assertions on types and behavior
• Document architecture in test names and docstrings
• Fail fast on architectural violations

Behavioral Tests

• Only public API (create_pipeline, Engine, EngineSession, PluginManager)
• Parametrized across plugins
• Realistic scenarios (long text, unicode, mixed scripts)
• No implementation coupling (test behavior, not internals)

General

• Fast: Unit tests < 1s, Contract tests < 5s, Behavioral < 30s
• Isolated: No shared state between tests
• Deterministic: Same input → same output
• Descriptive names: test_<component>_<scenario>_<expected>