Skip to content

Devlog: Phase 3 Unit 2 - Data Models & Template Schema

Date: 2025-11-07 Unit: 2 of 9 Status: Complete Duration: ~3 hours Related: Phase 3 Plan, Unit 1 Research

Overview

Unit 2 implemented the core data models for Phase 3's extraction system. We created Pydantic models for templates, extracted content, and output generation, along with comprehensive test coverage and documentation.

Goals

✅ Create extraction models (ExtractionTemplate, ExtractedContent, ExtractionResult) ✅ Create output models (EpisodeMetadata, OutputFile, EpisodeOutput) ✅ Create base extractor abstraction ✅ Write comprehensive unit tests ✅ Document template schema design

Implementation

1. Extraction Models (src/inkwell/extraction/models.py)

Classes Implemented:

TemplateVariable - Defines customizable variables for prompt templates - Validates variable names as Python identifiers - Supports required/optional with defaults

ExtractionTemplate - Complete template configuration model - Required fields: name, version, description, prompts, format - Optional fields: category, priority, model preference, parameters - Validation: filesystem-safe names, valid Jinja2, temperature 0-1 - Cache key generation: template_name:version

ExtractedContent - Represents extracted content from a template - Supports both dict and string content - Quality metrics: confidence score, validation warnings - Property: is_valid (no warnings, confidence >= 0.7)

ExtractionResult - Operation result envelope - Tracks success/failure, cost, duration, tokens - Cache metadata (from_cache, cache_key) - Helper methods: is_successful, get_summary()

Key Design Decisions:

  1. Pydantic for Validation: Catch errors early with type checking
  2. Flexible Content Types: Support both structured (dict) and narrative (string)
  3. Quality Tracking: Confidence scores and warnings for transparency
  4. Cost Awareness: Track costs per extraction for user visibility
  5. Cache-Friendly: Include version in cache keys for invalidation

2. Output Models (src/inkwell/output/models.py)

Classes Implemented:

EpisodeMetadata - Episode information (podcast name, title, URL, date, duration) - Processing metadata (transcription source, templates applied) - Cost tracking (transcription cost, extraction cost, total cost) - Helper methods: duration_formatted, date_slug, add_template(), add_cost()

OutputFile - Single markdown file representation - Filename, template name, content - Optional YAML frontmatter - Size calculation: update_size() - Full content generation: frontmatter + content

EpisodeOutput - Complete episode output aggregation - Metadata + list of output files - Statistics: total files, total size - Helper methods: get_file(), get_file_by_name(), directory_name - Formatting: filesystem-safe directory names, size formatting

Key Design Decisions:

  1. Separation of Concerns: Metadata separate from content
  2. Aggregation: EpisodeOutput as container for all files
  3. Slugification: Auto-generate filesystem-safe names
  4. Size Tracking: Monitor output size for user feedback
  5. Cost Rollup: Aggregate costs at episode level

3. Base Extractor (src/inkwell/extraction/extractors/base.py)

Abstract Base Class:

BaseExtractor - Abstract methods: - extract() - Extract content from transcript - estimate_cost() - Calculate cost before extraction - supports_structured_output() - Check for JSON mode - Concrete methods: - build_prompt() - Render Jinja2 template with context - _count_tokens() - Estimate token count

Key Features: - Few-shot example formatting - Jinja2 template rendering with transcript + metadata - Token counting (rough approximation: 4 chars/token)

4. Test Coverage

Extraction Models Tests (tests/unit/test_extraction_models.py) - 35 tests across 4 test classes - 100% coverage of model logic - Tests cover: - Model creation (minimal and full) - Field validation (names, templates, ranges) - Property methods - Edge cases (warnings, low confidence)

Output Models Tests (tests/unit/test_output_models.py) - 33 tests across 3 test classes - 100% coverage of model logic - Tests cover: - Metadata formatting (duration, date slug) - File management (add, retrieve) - Directory name generation (slugification, truncation) - Size formatting (bytes, KB, MB)

Total: 68 comprehensive unit tests

5. Module Organization

Created Structure: 

src/inkwell/
├── extraction/
│   ├── __init__.py           # Export public models
│   ├── models.py             # Template and extraction models
│   └── extractors/
│       ├── __init__.py       # Export BaseExtractor
│       └── base.py           # Abstract base class
└── output/
    ├── __init__.py           # Export public models
    └── models.py             # Output and metadata models

Design Principles: - Clear module boundaries - Explicit init exports - Type hints throughout - Comprehensive docstrings

Documentation Created

Template Schema Design

Document: docs/research/template-schema-design.md

Contents: - Complete field reference (required and optional) - Validation rules and examples - Best practices for template authoring - Complete template examples (summary, quotes, tools) - Common pitfalls and how to avoid them - Template development workflow

Highlights: - All fields documented with examples - Good vs bad examples for each validation rule - Three complete production-ready templates - Testing and validation guidance

Key Insights

1. Pydantic Validation is Powerful

Pydantic caught numerous design issues during development: - Invalid variable names - Malformed Jinja2 templates - Out-of-range temperatures - Negative values

Impact: Errors caught at template load time, not runtime

2. Cache Key Versioning Critical

Including template version in cache key prevents subtle bugs: 

cache_key = f"{episode_url}:{template_name}:{template_version}"

Without version: - ❌ Update template → still gets old cached extraction - ❌ User gets wrong output - ❌ Hard to debug

With version: - ✅ Update template → cache miss → fresh extraction - ✅ Automatic invalidation - ✅ Clear semantics

3. Flexible Content Types Needed

Originally considered dict-only for ExtractedContent: 

content: dict[str, Any]  # Too restrictive

But some templates produce markdown strings: 

content: str | dict[str, Any]  # Flexible

Impact: Supports both structured (JSON) and narrative (markdown) templates

4. Quality Metrics Build Trust

Added confidence scoring and warnings: 

@property
def is_valid(self) -> bool:
    return len(self.warnings) == 0 and (
        self.confidence is None or self.confidence >= 0.7
    )

Benefits: - Users see quality indicators - Can retry low-confidence extractions - Warnings provide actionable feedback

5. Cost Tracking from Day One

Built-in cost tracking at every level: - Per extraction: ExtractionResult.cost_usd - Per episode: EpisodeMetadata.total_cost_usd - Aggregated: transcription + extraction costs

Impact: Users always know what they're spending

Design Decisions Validated

From Unit 1 Research:

✅ YAML for templates - Pydantic validates YAML → models ✅ Few-shot prompting - Built into ExtractionTemplate.few_shot_examples ✅ Template versioning - Version field + cache key integration ✅ Multi-provider support - model_preference field ready ✅ Quality validation - Confidence scores + warnings

New Decisions:

✅ Flexible content types - str | dict for ExtractedContent ✅ Property methods - Computed properties (is_valid, directory_name) ✅ Helper methods - add_template(), add_cost(), get_summary() ✅ Size tracking - Automatic size calculation for output files

Challenges and Solutions

Challenge 1: Jinja2 Template Validation

Problem: How to validate Jinja2 templates without executing them?

Solution: Parse with Jinja2.Template() during model validation 

@field_validator("user_prompt_template")
@classmethod
def validate_jinja_template(cls, v: str) -> str:
    from jinja2 import Template
    Template(v)  # Raises exception if invalid
    return v

Impact: Catches syntax errors early

Challenge 2: Directory Name Generation

Problem: Need filesystem-safe, human-readable directory names

Solution: Slugification with special character removal 

def directory_name(self) -> str:
    # podcast-name-YYYY-MM-DD-episode-title
    podcast_slug = re.sub(r"[^\w\s-]", "", podcast_name.lower())
    title_slug = re.sub(r"[^\w\s-]", "", episode_title.lower())
    # Truncate title to 50 chars
    return f"{podcast_slug}-{date_slug}-{title_slug[:50]}"

Impact: Clean, readable, filesystem-safe names

Challenge 3: Frontmatter Integration

Problem: How to combine YAML frontmatter with markdown content?

Solution: Computed property that combines both 

@property
def full_content(self) -> str:
    if not self.has_frontmatter:
        return self.content
    frontmatter_yaml = yaml.dump(self.frontmatter)
    return f"---\n{frontmatter_yaml}---\n\n{self.content}"

Impact: Automatic frontmatter generation

Testing Philosophy

Test-Driven Benefits:

  1. Immediate Feedback: Tests caught validation issues instantly
  2. Documentation: Tests show how to use models
  3. Regression Prevention: Changes won't break existing behavior
  4. Confidence: 68 passing tests = high confidence in models

Testing Strategy:

  • Positive Tests: Valid inputs produce expected outputs
  • Negative Tests: Invalid inputs raise appropriate errors
  • Edge Cases: Empty values, extreme values, boundary conditions
  • Integration: Models work together (EpisodeOutput + OutputFile)

Examples

Creating a Template

template = ExtractionTemplate(
    name="summary",
    version="1.0",
    description="Generate episode summary",
    system_prompt="You are an expert podcast analyst.",
    user_prompt_template="Summarize: {{ transcript }}",
    expected_format="markdown",
    max_tokens=2000,
    temperature=0.3,
    few_shot_examples=[{
        "input": "Sample transcript...",
        "output": "Sample summary..."
    }]
)

Creating an Extraction Result

content = ExtractedContent(
    template_name="summary",
    content="# Episode Summary\n\n...",
    confidence=0.95
)

result = ExtractionResult(
    episode_url="https://example.com/ep123",
    template_name="summary",
    success=True,
    extracted_content=content,
    duration_seconds=4.2,
    cost_usd=0.23,
    provider="claude"
)

print(result.get_summary())
# ✓ summary: Success (4.2s, $0.230)

Creating Episode Output

metadata = EpisodeMetadata(
    podcast_name="The Changelog",
    episode_title="Building Better Software",
    episode_url="https://example.com/ep123",
    transcription_source="youtube"
)

output = EpisodeOutput(
    metadata=metadata,
    output_dir=Path("~/podcasts")
)

output.add_file(OutputFile(
    filename="summary.md",
    template_name="summary",
    content="# Summary\n\n..."
))

print(output.directory_name)
# the-changelog-2025-11-07-building-better-software

Metrics

Code Written: - extraction/models.py: 215 lines - output/models.py: 258 lines - extractors/base.py: 113 lines - Total production code: ~586 lines

Tests Written: - test_extraction_models.py: 388 lines (35 tests) - test_output_models.py: 503 lines (33 tests) - Total test code: ~891 lines

Documentation: - Template schema design: ~800 lines - Total docs: ~800 lines

Test Coverage: 100% for model logic (validation, properties, methods)

Next Steps

Immediate (Unit 3):

✅ Implement template loading from YAML files ✅ Implement template selector (category detection) ✅ Create default templates (summary, quotes, key-concepts) ✅ Implement template validation CLI

Near-term (Units 4-5):

  • Implement Claude and Gemini extractors
  • Implement extraction engine with caching
  • Response parsing and validation
  • Integration testing

Reflection

What Went Well:

✅ Models are clean: Clear responsibilities, well-documented ✅ Tests are comprehensive: 68 tests cover all edge cases ✅ Validation catches errors: Pydantic prevents invalid data ✅ Documentation is thorough: Template schema guide is complete ✅ Design is flexible: Supports future enhancements

What Could Be Better:

⚠️ Test environment setup: pytest configuration needs work ⚠️ Example templates: Should create actual YAML template files ⚠️ Type hints: Could use more specific types (NewType, Literal)

Lessons Learned:

  1. Pydantic + type hints = robust code: Catches errors before runtime
  2. Property methods simplify usage: is_valid better than manual checks
  3. Helper methods improve UX: add_template() better than direct list manipulation
  4. Good defaults matter: temperature=0.3, max_tokens=2000 work well
  5. Versioning prevents bugs: Cache invalidation with template versions

Created This Unit: - src/inkwell/extraction/models.py - Extraction models - src/inkwell/output/models.py - Output models - src/inkwell/extraction/extractors/base.py - Base extractor - tests/unit/test_extraction_models.py - Extraction model tests - tests/unit/test_output_models.py - Output model tests - docs/research/template-schema-design.md - Template schema reference

Related from Unit 1: - Unit 1 Research - LLM Extraction Comparison - Template Format Evaluation - ADR-014: Template Format

Planning: - Phase 3 Detailed Plan

Unit 2 Status: ✅ COMPLETE

Next Unit: Unit 3 - Template Loading, Validation, and Selection System