From 88e92f44722e929dceb01504eaacfd1a8895f657 Mon Sep 17 00:00:00 2001 From: Magnus Hedemark Date: Sat, 23 May 2026 18:08:48 -0400 Subject: [PATCH] =?UTF-8?q?feat:=20add=20epub=20skill=20v2=20=E2=80=94=201?= =?UTF-8?q?1=20scripts,=209=20references,=2046=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Complete EPUB creation, editing, validation, and knowledge extraction skill for the Agent Skills open format. Built from spec research, real EPUB testing on 2.1MB commercial Apress title, and Apple Books compatibility verification on macOS 26. Scripts (11): epub-scaffold — Create valid EPUB3 with cover XHTML, Apple Books CSS epub-edit — Surgical editing (8 subcommands, epublib) epub-info — Structure/metadata dump as JSON epub-text — Clean text extraction, per-chapter or single-file epub-extract-knowledge — Heuristic + LLM extraction (env var auto-detect) epub-validate — EPUBCheck or Python fallback validation epub-images — List/extract all images with cover detection epub-batch — Multi-file processing (extract-text, validate, metadata) epub-convert — EPUB2→EPUB3 conversion with validation epub-repair — Diagnose & auto-fix common structural issues epub-cover — Add cover XHTML wrapper for Apple Books compatibility References (9): epub-format-internals.md, python-libraries.md, spec-and-validation.md, tutorials-and-guides.md, agent-capability-discovery.md, fixed-layout-epub.md, accessibility.md, media-overlays.md, apple-books-compatibility.md (NEW — verified on macOS 26) Test: 46/46 passing (test_epub_skill.sh) --- AGENTS.md | 2 +- README.md | 7 +- epub/SKILL.md | 474 ++++++++++++++++++ epub/references/accessibility.md | 127 +++++ epub/references/agent-capability-discovery.md | 182 +++++++ epub/references/apple-books-compatibility.md | 143 ++++++ epub/references/epub-format-internals.md | 185 +++++++ epub/references/fixed-layout-epub.md | 80 +++ epub/references/media-overlays.md | 103 ++++ epub/references/python-libraries.md | 325 ++++++++++++ epub/references/spec-and-validation.md | 114 +++++ epub/references/tutorials-and-guides.md | 224 +++++++++ epub/scripts/epub-batch | 161 ++++++ epub/scripts/epub-convert | 178 +++++++ epub/scripts/epub-cover | 147 ++++++ epub/scripts/epub-edit | 450 +++++++++++++++++ epub/scripts/epub-extract-knowledge | 431 ++++++++++++++++ epub/scripts/epub-images | 133 +++++ epub/scripts/epub-info | 219 ++++++++ epub/scripts/epub-repair | 323 ++++++++++++ epub/scripts/epub-scaffold | 298 +++++++++++ epub/scripts/epub-text | 193 +++++++ epub/scripts/epub-validate | 282 +++++++++++ epub/scripts/test_epub_skill.sh | 460 +++++++++++++++++ 24 files changed, 5239 insertions(+), 2 deletions(-) create mode 100644 epub/SKILL.md create mode 100644 epub/references/accessibility.md create mode 100644 epub/references/agent-capability-discovery.md create mode 100644 epub/references/apple-books-compatibility.md create mode 100644 epub/references/epub-format-internals.md create mode 100644 epub/references/fixed-layout-epub.md create mode 100644 epub/references/media-overlays.md create mode 100644 epub/references/python-libraries.md create mode 100644 epub/references/spec-and-validation.md create mode 100644 epub/references/tutorials-and-guides.md create mode 100755 epub/scripts/epub-batch create mode 100755 epub/scripts/epub-convert create mode 100755 epub/scripts/epub-cover create mode 100755 epub/scripts/epub-edit create mode 100755 epub/scripts/epub-extract-knowledge create mode 100755 epub/scripts/epub-images create mode 100755 epub/scripts/epub-info create mode 100755 epub/scripts/epub-repair create mode 100755 epub/scripts/epub-scaffold create mode 100755 epub/scripts/epub-text create mode 100755 epub/scripts/epub-validate create mode 100755 epub/scripts/test_epub_skill.sh diff --git a/AGENTS.md b/AGENTS.md index f5d6f02..90e77d3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -55,7 +55,7 @@ When the user mentions these keywords, load the corresponding skill: |---|---| | "build a CLI", "make a CLI tool", "agent-friendly CLI", "add --json flag" | [cli-builder](cli-builder/SKILL.md) | | "debug this", "root cause", "why is this broken", "fix this bug" | [systematic-debugging](systematic-debugging/SKILL.md) | -| "epub", "ebook", "EPUB file", "read epub", "write epub", "create ebook", "extract from epub", "epub to text", "ebook format" | [epub](epub/SKILL.md) | +| "epub", "ebook", "EPUB file", "ebook format", "read epub", "write epub", "create ebook", "extract from epub", "epub to text", "edit epub", "repair epub", "convert epub2", "epub images", "batch epub", "ebook metadata" | [epub](epub/SKILL.md) | | "weather", "forecast", "temperature", "is it raining", "Tempest" | [tempest-cli](tempest-cli/SKILL.md) | | "reverse-engineer", "understand this codebase", "PRD from code", "architecture document" | [software-architecture-analysis](software-architecture-analysis/SKILL.md) | || "data architecture", "data platform", "data strategy", "data mesh", "governance" | [data-architect](data-architect/SKILL.md) | diff --git a/README.md b/README.md index 2fa6685..1c89a6b 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,12 @@ PhD-level expertise in data science, statistics, and machine learning. Rigorous ### [epub](epub/SKILL.md) -EPUB file format expert — read, write, and edit EPUB2/EPUB3 ebooks. Extract text, metadata, structure, and knowledge into formatted output (JSON, vault atoms, memory entries). Create valid EPUBs from scratch. Validate against the EPUB specification. Five CLI scripts with a capability-discovery pipeline for portable knowledge extraction across any agent harness. +EPUB file format expert — read, write, edit, convert, and repair EPUB2/EPUB3 ebooks. +Ten CLI scripts: structure inspection, text extraction, knowledge extraction (LLM mode +via env vars), scaffold creation, surgical editing, image extraction, batch processing, +EPUB2→3 conversion, repair, and validation. Eight reference files covering format +internals, Python libraries, spec/validation, tutorials, capability discovery, +fixed-layout, accessibility, and media overlays. Portable across any AgentSkills harness. ### [forgejo-cli](forgejo-cli/SKILL.md) diff --git a/epub/SKILL.md b/epub/SKILL.md new file mode 100644 index 0000000..512f8ac --- /dev/null +++ b/epub/SKILL.md @@ -0,0 +1,474 @@ +--- +name: epub +description: >- + EPUB file format expert — read, write, and edit EPUB2/EPUB3 ebooks. Extract text, + metadata, structure, and knowledge from EPUB files for enrichment or memory. Create + valid EPUBs from scratch. Validate against the EPUB specification. Use when the user + mentions epub, ebook, EPUB file, ebook format, read epub, write epub, create ebook, + extract from epub, epub to text, or ebook structure. +license: MIT +compatibility: >- + Python 3.8+ required. Core scripts use EbookLib (pip install EbookLib) for reading and + creating EPUBs. Optional epublib (pip install epublib) for non-intrusive editing. + beautifulsoup4 (pip install beautifulsoup4) for text extraction. Optional LLM mode via + EPUB_LLM_URL + EPUB_LLM_KEY env vars (any OpenAI-compatible provider). EPUBCheck + (Java, optional) for authoritative validation. Portable across all AgentSkills-compatible + harnesses — scripts are platform-agnostic. +metadata: + skills: [epub, ebook, publishing, knowledge-extraction, document-processing] + tags: [epub, ebook, epub3, epub2, publishing, document-extraction, knowledge] +--- + +# EPUB — Ebook Creation, Extraction & Enrichment + +Expert-level EPUB handling: read, write, edit, validate, and extract knowledge +from EPUB 2 and EPUB 3 files. Ships with five Python CLI scripts and five +detailed references covering the entire EPUB domain. + +## EPUB Format Essentials + +An EPUB file is a **ZIP archive** (Open Container Format, OCF) with a specific +internal layout. The W3C EPUB 3.3 standard uses a "three planes" model: + +| Plane | Contains | Key Rule | +|-------|----------|----------| +| **Manifest** | All resources (XHTML, images, CSS, fonts) | Every file must be listed in OPF `` | +| **Spine** | Linear reading order | Only XHTML/SVG by default; other types need fallbacks | +| **Content** | Resources embedded within documents | Core media types guaranteed; foreign types need fallbacks | + +The package document (`content.opf`) holds metadata (Dublin Core), manifest +(every resource), and spine (reading order). EPUB3 uses an XHTML `nav` document +for navigation; EPUB2 uses `.ncx` XML. Both can coexist for compatibility. + +**Critical rules:** `mimetype` must be the first ZIP entry, stored uncompressed. +All content documents must be well-formed XML (XHTML, not HTML5). The manifest +must list every file used in rendering. Read `references/epub-format-internals.md` +for the full structure reference. + +## Decision Table — Which Script to Use + +| Task | Script | Notes | +|------|--------|-------| +| See structure, metadata, manifest, spine, TOC | `epub-info` | JSON output, `--summary` for compact | +| Extract clean reading-order text | `epub-text` | Per-chapter or single file | +| Create minimal valid EPUB from scratch | `epub-scaffold` | No dependencies needed | +| Extract facts, quotes, definitions, arguments | `epub-extract-knowledge` | Heuristic or LLM mode (env var auto-detect) | +| Validate against EPUB spec | `epub-validate` | EPUBCheck or Python fallback | +| **Edit EPUB** (metadata, chapters, spine, CSS) | **`epub-edit`** | **v2 flagship — 8 subcommands, non-intrusive** | +| Extract images | `epub-images` | List or extract to directory | +| Batch process multiple EPUBs | `epub-batch` | Wrap existing scripts across globs | +| Convert EPUB2 → EPUB3 | `epub-convert` | Add NAV, update NS, keep NCX | +| Diagnose & repair structural issues | `epub-repair` | Auto-fix common validation failures | + +## Scripts + +All scripts live in `scripts/` relative to this skill's directory. Each follows +cli-builder conventions: `--json` for machine output, `--dry-run` to preview, +non-interactive, errors to stderr. Run with `--help` for full flag details. + +### epub-info — Structure & Metadata Dump + +```bash +scripts/epub-info book.epub --json +scripts/epub-info book.epub --summary # compact manifest +scripts/epub-info book.epub --dry-run # preview +``` + +Outputs: EPUB version, metadata (title, author, language, identifier), +manifest (all items with id/href/media-type), spine (reading order), +TOC (nested structure). `--summary` reduces manifest to id+href+media-type. + +### epub-text — Clean Text Extraction + +```bash +scripts/epub-text book.epub # plain text to stdout +scripts/epub-text book.epub --json # JSON with chapter array +scripts/epub-text book.epub --chapters # one .txt per chapter +scripts/epub-text book.epub --output book.txt # single file +scripts/epub-text book.epub --format markdown # markdown output +``` + +Extracts text from spine-ordered content documents. Strips HTML tags, preserves +paragraph structure. Requires beautifulsoup4. Respects spine linearity — only +processes documents in the reading order. + +### epub-scaffold — Create Valid EPUB from Scratch + +```bash +scripts/epub-scaffold --title "My Book" --author "Jane Doe" +scripts/epub-scaffold --title "Novel" --author "Me" --chapters 12 --output novel.epub +scripts/epub-scaffold --title "Guide" --author "Me" --cover cover.jpg +scripts/epub-scaffold --title "Guide" --author "Me" --cover cover.jpg --toc-hidden --dry-run +``` + +Creates a valid EPUB3 with all content inside `OEBPS/` — required for Apple +Books compatibility. No external dependencies — Python stdlib only. + +**Cover handling:** When `--cover` is provided, the scaffold automatically: +- Copies the image to `OEBPS/Images/cover.{ext}` +- Generates `OEBPS/Text/cover.xhtml` — an XHTML wrapper page with full-viewport CSS +- Adds the cover page to the spine as the first item +- Sets `properties="cover-image"` on the raw image for library thumbnails + +This follows the Apple Books requirement that covers must be XHTML pages in +the spine, not raw image references (raw images render as blank pages). + +**Nav visibility:** `--toc-hidden` sets `linear="no"` on the nav spine item, +hiding it from the reading flow (still accessible via the app's built-in TOC +browser). Default is `--toc-visible` (nav renders as a page). + +**CSS:** Ships `OEBPS/Styles/default.css` with Apple Books-compatible typography: +no deprecated `page-break-before`, `margin: 0` on body (padding for whitespace), +proper heading hierarchy, and responsive styling. + +**Cover art guidance:** The `--cover` flag accepts a pre-existing image file. If +the user does not have a cover image, offer to generate one using the agent's +image_gen capability. See `references/apple-books-compatibility.md` for the full +cover XHTML and CSS conventions. + +### epub-cover — Add Cover to Existing EPUB + +```bash +scripts/epub-cover wrap book.epub --image cover.png --output with-cover.epub +scripts/epub-cover wrap book.epub --image cover.png --in-place +``` + +Adds a cover XHTML wrapper page to an EPUB that already has a cover image in +its manifest. Use when the image exists but isn't rendering in Apple Books. +Requires epublib. + +### epub-extract-knowledge — Knowledge Extraction Pipeline + +```bash +# LLM mode — set env vars first (see below), then run without flags: +scripts/epub-extract-knowledge book.epub --format json +scripts/epub-extract-knowledge book.epub --format atoms +scripts/epub-extract-knowledge book.epub --format memory + +# Force heuristic mode (ignore env vars): +scripts/epub-extract-knowledge book.epub --no-llm --format json + +# Custom prompt override: +scripts/epub-extract-knowledge book.epub --prompt "Extract all definitions" --format json +``` + +Extracts knowledge from EPUB content: facts, definitions, key points, and +arguments. Two modes, auto-selected: + +- **LLM mode (auto-detected):** When `EPUB_LLM_URL` and `EPUB_LLM_KEY` env vars + are set, calls the configured LLM with the chapter text and extraction prompt. + Produces high-quality structured insights. Falls back to heuristic if the LLM + call fails. +- **Heuristic mode (fallback):** When env vars are NOT set, or `--no-llm` is + passed, uses pattern matching (headings, emphasis markers, definition language, + paragraph density) to identify knowledge-bearing passages. No LLM required. + +Output formats: +- `json` — raw structured JSON with types, content, and source chapters +- `atoms` — Obsidian vault atom templates (YAML frontmatter + body) +- `memory` — key-value memory entries suitable for agent persistence + +### LLM Configuration Convention + +Several scripts in this skill support optional LLM-powered features. Any script +that does auto-detects LLM availability via environment variables. Set them once +and all scripts inherit: + +```bash +# Required for LLM mode: +export EPUB_LLM_URL="https://your-provider.example.com/v1" # OpenAI-compatible endpoint +export EPUB_LLM_KEY="sk-..." # API key + +# Optional: +export EPUB_LLM_MODEL="model-name" # Defaults to provider default +``` + +**How it works:** +- If `EPUB_LLM_URL` and `EPUB_LLM_KEY` are both set → LLM mode enabled +- If either is missing → heuristic/deterministic mode (no LLM) +- `--no-llm` flag forces heuristic mode even when env vars are set +- The scripts make OpenAI-compatible `POST /chat/completions` calls — any + OpenAI-compatible provider works (OpenAI, OpenCode, Anthropic via proxy, + local llama.cpp, Ollama, vLLM, etc.) + +**Which scripts support this:** +| Script | LLM Feature | Fallback | +|--------|------------|----------| +| `epub-extract-knowledge` | Structured knowledge extraction | Heuristic pattern matching | +| `epub-validate` | LLM-generated repair suggestions for errors | Error codes only | +| *(more scripts can adopt this pattern as features are added)* | | | + +**Agent instructions:** Before running any extraction pipeline, set these +env vars in your environment. They are inherited by subprocesses, so every +script in the pipeline auto-detects the same LLM configuration. If your +harness provides an LLM natively (e.g., you *are* the LLM), you can skip +the env vars — the heuristic mode is designed for that case. But if you +have access to an external LLM API, wiring it through these env vars +unlocks dramatically better extraction quality without requiring the +agent to manually chunk, prompt, parse, and re-inject results. + +### epub-edit — Surgical EPUB Editing (v2) + +```bash +scripts/epub-edit info book.epub --json +scripts/epub-edit metadata book.epub --title "New Title" --output out.epub +scripts/epub-edit add-chapter book.epub --content new.xhtml --after chapter3 --output out.epub +scripts/epub-edit remove-chapter book.epub --id chapter5 --output out.epub +scripts/epub-edit reorder-spine book.epub --order chapter3,chapter1,chapter2 --dry-run +scripts/epub-edit rename-resource book.epub --from Images/old.jpg --to Images/new.jpg +scripts/epub-edit inject-css book.epub --css dark.css --output out.epub +scripts/epub-edit update-manifest book.epub --output out.epub +``` + +Non-intrusive EPUB editing via epublib. Eight subcommands covering the full +edit surface. Never overwrites original — defaults to `--output out.epub`; +use `--in-place` to commit. All subcommands support `--json`, `--dry-run`. + +### epub-images — Image Extraction + +```bash +scripts/epub-images book.epub --list --json # list all images +scripts/epub-images book.epub --extract images/ # extract all to directory +scripts/epub-images book.epub --type cover --extract . # cover image only +``` + +### epub-batch — Multi-File Processing + +```bash +scripts/epub-batch extract-text "books/*.epub" --output texts/ +scripts/epub-batch validate "books/*.epub" --json +scripts/epub-batch metadata "books/*.epub" --set-author "Author" --output-dir fixed/ +scripts/epub-batch info "books/*.epub" --json +``` + +### epub-convert — EPUB2 → EPUB3 + +```bash +scripts/epub-convert old.epub --output new-v3.epub +scripts/epub-convert old.epub --validate --json +``` + +### epub-repair — Diagnose & Fix + +```bash +scripts/epub-repair broken.epub --diagnose --json # list fixable issues +scripts/epub-repair broken.epub --output fixed.epub # auto-fix +``` + +### epub-validate — Structural Validation + +```bash +scripts/epub-validate book.epub --json +scripts/epub-validate book.epub --dry-run +``` + +Tries EPUBCheck (Java JAR) first for authoritative validation. Falls back to +pure-Python structural checks: mimetype position/compression/content, +container.xml parseability, OPF schema, manifest completeness, spine reference +integrity, required metadata, and NAV document presence. + +## Capability Discovery & Pipeline Construction + +Before executing a multi-step EPUB pipeline, **discover what tools are available** +on your agent platform. This skill is portable — the exact pipeline shape +depends on your harness's capabilities. + +### Discovery Protocol + +1. **Enumerate:** What tools do you have? File write? Web access? Subagents? + Cron? Persistent memory? Vector DB? Vault? LLM? +2. **Classify:** Map available tools to pipeline stages (Ingest → Parse → + Extract → Format → Sink) +3. **Construct:** Build a specific pipeline from available pieces +4. **Propose:** Present the plan to the user before executing + +### Pipeline Stages + +``` +EPUB file + ↓ INGEST — local path, URL download, or user provides file + ↓ PARSE — epub-text, epub-info, or stdlib zipfile+XML + ↓ EXTRACT — epub-extract-knowledge (heuristic or LLM) + ↓ FORMAT — vault atoms, memory entries, JSON, markdown + ↓ SINK — file write, memory tool, vector DB, vault, terminal +``` + +### Example Pipelines + +**Full (Hermes Agent):** +Ingest EPUB → epub-text (JSON) → epub-extract-knowledge → vault atoms → +wiki-link-verification → LightRAG ingest + +**Minimal (any harness with terminal + file write):** +Ingest EPUB → epub-text → epub-extract-knowledge --no-llm → write output.md + +**Batch (multi-file):** +Find *.epub → for each: epub-text --chapters → epub-extract-knowledge --no-llm → collect +results → summary report + +See `references/agent-capability-discovery.md` for the full protocol with +worked examples for different platforms. + +## Knowledge Extraction Deep Dive + +EPUB files are dense sources of structured knowledge. The extraction process +targets specific knowledge types: + +| Type | Detection | Example | +|------|-----------|---------| +| **Fact** | Headings, list items, named entities | "Python 3.13 added the `@override` decorator" | +| **Definition** | Paragraphs with definition markers | "A coroutine is defined as a function that can suspend execution" | +| **Key point** | Emphasized text (bold, italic) | Important conclusions, takeaways | +| **Argument** | Dense paragraphs (>200 chars) | Multi-sentence reasoning chains | + +### Prompt Design for LLM Mode + +When using LLM extraction, provide a focused prompt: + +``` +Extract from this chapter: +1. All technical definitions (term + definition) +2. Key facts (concise, standalone statements) +3. Notable quotes (exact wording) +4. Core arguments (the main thesis and supporting points) + +Format as JSON with fields: type, content, context +``` + +### Sink Options by Platform + +| Sink | Platform | Format to use | +|------|----------|---------------| +| Vault atoms | Obsidian | `--format atoms` | +| Agent memory | Most harnesses | `--format memory` | +| Vector DB | LightRAG, Chroma | `--format json` → insert | +| Plain files | Any | `--output DIR` | + +## Common Workflows + +### Create EPUB from Markdown Files + +```bash +# 1. Scaffold the EPUB +scripts/epub-scaffold --title "My Book" --author "Me" --chapters 3 --output book.epub + +# 2. Use epublib (Python) to inject real content into each chapter +# See references/tutorials-and-guides.md for the editing pattern +``` + +### Extract All Images from EPUB + +```bash +scripts/epub-images book.epub --extract images/ +``` + +### Edit an EPUB + +```bash +# Update metadata +scripts/epub-edit metadata book.epub --title "New Title" --output revised.epub + +# Add a chapter +scripts/epub-edit add-chapter book.epub --content new.xhtml --output expanded.epub + +# Inject dark theme +scripts/epub-edit inject-css book.epub --css dark.css --output dark.epub +``` + +### Fix a Broken EPUB + +```bash +scripts/epub-repair broken.epub --diagnose --json # see what's broken +scripts/epub-repair broken.epub --output fixed.epub # auto-fix +``` + +### Batch Extract Text from a Library + +```bash +scripts/epub-batch extract-text "books/*.epub" --output texts/ +``` + +### Convert EPUB2 to EPUB3 + +```bash +scripts/epub-convert old.epub --output old-v3.epub --validate +``` + +## Apple Books Compatibility + +Apple Books on macOS/iOS enforces requirements beyond the EPUB spec. These +rules were verified by building and testing on macOS 26. + +| Rule | Why | +|------|-----| +| All content inside `OEBPS/` directory | Files at ZIP root render as blank pages | +| Cover must be XHTML page in spine | Raw `` to `image/png` renders blank | +| `margin: 0` on body, use `padding` | Apple Books applies its own margins; they stack | +| No `page-break-before` (use `break-before: page` or omit) | Deprecated; Apple Books ignores it | +| `xmlns:epub` only on nav document | Unused namespace declarations can trigger parser failures | +| Cover image keeps `properties="cover-image"` | Used for library thumbnail on raw image, not on wrapper page | +| `linear="no"` hides page from reading flow | Nav still accessible via app's built-in TOC browser | + +**Spine ordering patterns:** +| Pattern | Order | Use case | +|---------|-------|----------| +| No cover | `nav → chapters` | Simplest | +| Cover only | `cover-page → nav(linear="no") → chapters` | Cover visible, ToC via app browser | +| Full | `cover-page → nav → chapters` | Most commercial ebooks (scaffold default) | + +See `references/apple-books-compatibility.md` for the full reference with +CSS examples, cover XHTML template, and validation quirks. + +## Gotchas + +- **mimetype compression:** Python's `zipfile` compresses by default. Always use + `ZIP_STORED` for the mimetype entry. A compressed mimetype silently breaks + reading systems. +- **XHTML ≠ HTML5:** Content documents must be well-formed XML. Self-closing + tags required (`
` not `
`). Use `xmlns:epub="http://www.idpf.org/2007/ops"`. +- **Manifest is exhaustive:** Every file in the EPUB must be listed. Missing + manifest entries cause validation failures. Images, CSS, fonts — no exceptions. +- **AGPL boundary:** EbookLib is AGPL; scripts call it at runtime but don't + bundle it. Users install it themselves via pip. This skill and its scripts + are MIT. +- **EPUBCheck needs Java:** The authoritative validator requires Java. The + fallback Python checks catch structural issues but not XHTML schema violations + or CSS validity. +- **Spine references manifest IDs:** An `idref` in the spine must match an `id` + in the manifest. Broken references cause the EPUB to fail validation. +- **Navigation document:** EPUB3 requires a NAV with `properties="nav"`. + Without it, reading systems may not show a table of contents. +- **Language is required:** Both `` and `xml:lang`/`lang` + attributes on content documents. Missing language = invalid EPUB. + +## Pitfalls + +- **Don't skip capability discovery.** Assuming a tool exists that doesn't + leads to broken pipelines. Always check before building. +- **Don't assume LLM availability.** Always offer `--no-llm` fallback for + extraction. Heuristic mode works surprisingly well for well-structured books. +- **Don't modify EPUBs in place without backup.** EPUB editing is surgery — + always keep the original. +- **Don't mix EbookLib and epublib on the same file in the same session.** + They have different memory models and may conflict. Pick one library per task. +- **Don't assume the OPF is at `OEBPS/content.opf`.** Always read + `container.xml` to find the actual path. The root directory varies. + +## References + +- `references/epub-format-internals.md` — Full structural reference (OCF, OPF, + XHTML, NCX/NAV, spine, three planes, core media types, EPUB2 vs EPUB3) +- `references/python-libraries.md` — EbookLib vs epublib comparison, code + examples, when to use which, license notes +- `references/spec-and-validation.md` — W3C EPUB 3.3 spec access points, + EPUBCheck usage, Ace accessibility validation, key constraints summary +- `references/tutorials-and-guides.md` — Beginner to advanced guides, common + workflows, manual OPF editing, batch operations, pitfalls +- `references/agent-capability-discovery.md` — Protocol for probing agent tools, + constructing extraction pipelines, worked examples across platforms +- `references/fixed-layout-epub.md` — Fixed-layout detection and properties + (rendition:layout, orientation, spread, viewport meta) +- `references/accessibility.md` — WCAG alignment, alt text, heading hierarchy, + ARIA roles, Ace integration, accessibility metadata +- `references/media-overlays.md` — SMIL synchronization, audio-text pairing, + skippability/escapability, detection from manifest diff --git a/epub/references/accessibility.md b/epub/references/accessibility.md new file mode 100644 index 0000000..7d8e1aa --- /dev/null +++ b/epub/references/accessibility.md @@ -0,0 +1,127 @@ +# EPUB Accessibility + +Accessible EPUBs can be read by people using screen readers, braille displays, +and other assistive technologies. The W3C EPUB Accessibility 1.1 specification +defines conformance requirements. + +## Key Requirements + +### 1. Alternative Text for Images + +Every `` element must have an `alt` attribute: + +```xhtml +Flowchart showing data pipeline stages +``` + +Decorative images should use `alt=""` so screen readers skip them. + +### 2. Heading Hierarchy + +Headings must follow a logical nesting order — no skipping levels: + +```xhtml +

Chapter Title

+

Section

+

Subsection

+

Another Section

+``` + +Do not use `

` without a preceding `

`. + +### 3. Language Tagging + +Every XHTML document must declare its language: + +```xhtml + +``` + +For language shifts within a document: + +```xhtml +

English text texte français more English.

+``` + +### 4. ARIA Roles + +ARIA landmarks help screen reader navigation: + +```xhtml + +
...
+ +``` + +Key roles: `doc-toc`, `doc-chapter`, `doc-cover`, `doc-footnote`, `doc-glossary`, +`doc-index`, `doc-bibliography`, `doc-acknowledgments`. + +### 5. Accessibility Metadata + +The OPF must declare accessibility conformance: + +```xml +textual +visual +alternativeText +longDescription +none + + This publication meets WCAG 2.0 Level AA. + +``` + +## Ace by DAISY + +[Ace](https://daisy.org/activities/software/ace/) is the official accessibility +validator for EPUB. It checks: + +- Image alt text presence and quality +- Heading hierarchy correctness +- Language declarations +- ARIA roles and landmarks +- Accessibility metadata completeness +- Color contrast (basic) + +```bash +# Install: npm install -g @daisy/ace +ace book.epub --outdir ace-report/ +``` + +The report is an HTML dashboard showing violations by type and severity. + +## WCAG Alignment + +EPUB Accessibility 1.1 maps to WCAG 2.x Level AA. The four principles: + +| Principle | EPUB Meaning | +|-----------|-------------| +| **Perceivable** | All content has text alternatives, captions, sufficient contrast | +| **Operable** | All navigation is keyboard-accessible, no seizure-inducing content | +| **Understandable** | Language is declared, reading order is logical, predictable | +| **Robust** | Content is well-formed XHTML, compatible with assistive technologies | + +## Common Failures + +| Failure | Detection | Fix | +|---------|----------|-----| +| Missing alt text | Ace / manual inspection | Add `alt` to every `` | +| Skipped heading levels | Ace | Re-level headings | +| Language not declared | `xml:lang` + `lang` missing on `` | Add both attributes | +| No accessibility metadata | OPF missing `schema:accessibilityFeature` | Add required meta tags | +| Color-only information | Content relies on color for meaning | Add text labels or patterns | + +## Script Support + +Our EPUB scripts do NOT perform accessibility validation. Use Ace for that. +However, `epub-validate` checks for structural issues (XHTML well-formedness, +metadata completeness) that are prerequisites for accessibility. + +`epub-edit` can help remediate some issues: adding metadata fields, injecting +CSS for contrast, or adding `alt` attributes via DOM manipulation. + +## References + +- W3C EPUB Accessibility 1.1: https://www.w3.org/TR/epub-a11y-11/ +- W3C EPUB Accessibility Techniques 1.1: https://www.w3.org/TR/epub-a11y-tech-11/ +- Ace by DAISY: https://daisy.org/activities/software/ace/ diff --git a/epub/references/agent-capability-discovery.md b/epub/references/agent-capability-discovery.md new file mode 100644 index 0000000..496c45b --- /dev/null +++ b/epub/references/agent-capability-discovery.md @@ -0,0 +1,182 @@ +# Agent Capability Discovery & Pipeline Construction + +This reference teaches the agent how to discover its own capabilities before +constructing an EPUB extraction or construction pipeline. The skill must be +portable across any agent harness — this discovery step is how it adapts. + +## Why Discovery Matters + +Different agent platforms have different capabilities: + +| Platform | File I/O | Subagents | Cron | Kanban | Vector DB | Vault | +|----------|----------|-----------|------|--------|-----------|-------| +| Hermes Agent | terminal/write_file | delegate_task | cronjob | kanban | LightRAG | Obsidian vault | +| Claude Code | Bash/Write | Task tool | Scheduled tasks | — | — | — | +| OpenCode | Terminal/File | Subagents | — | — | — | — | +| GitHub Copilot | Terminal/File | — | — | — | — | — | +| Cursor | Terminal/File | — | — | — | — | — | +| Basic harness | Terminal only | — | — | — | — | — | + +The extraction pipeline works on any of these — but the *shape* of the pipeline +changes based on what's available. + +## Step 1: Enumerate Available Capabilities + +Before building a pipeline, probe the agent's toolset. The exact method depends +on the platform, but the principle is the same: check what's available. + +Several scripts in this skill use an **LLM auto-detection convention:** if the +environment variables `EPUB_LLM_URL` and `EPUB_LLM_KEY` are set, the scripts +enable LLM-powered features automatically. If they're absent, the scripts fall +back to heuristic/deterministic mode. This means you don't need to pass +`--no-llm` flags or state — the scripts detect it. + +```bash +# Set once for all scripts: +export EPUB_LLM_URL="https://your-provider.example.com/v1" +export EPUB_LLM_KEY="sk-..." +export EPUB_LLM_MODEL="model-name" # optional +``` + +Any OpenAI-compatible provider works — OpenAI, Anthropic via proxy, local +llama.cpp, Ollama, vLLM, OpenCode, etc. + +### What to check + +| Capability | How to detect | +|-----------|---------------| +| **Terminal / shell access** | Can you run shell commands? | +| **File write** | Can you create files on disk? | +| **Web access** | Can you make HTTP requests, scrape URLs, search the web? | +| **Subagents / task delegation** | Can you spawn child agents for parallel work? | +| **Scheduling / cron** | Can you schedule jobs to run later? | +| **Persistent memory** | Can you save facts that survive this session? | +| **Vector / semantic search** | Can you search by meaning, not keywords? | +| **Kanban / workflow boards** | Can you create and track tasks on boards? | +| **Vault / knowledge base** | Is there a structured note system (Obsidian, etc.)? | +| **LLM access** | Can you call an LLM for text extraction/classification? | +| **Browser automation** | Can you interact with web pages dynamically? | + +### Decision: Do you have what this skill's scripts need? + +At minimum, you need: +- Terminal/shell access (to run Python scripts) +- File write (to save output) + +Everything else enriches the pipeline but isn't required. + +## Step 2: Classify Into Pipeline Stages + +An EPUB knowledge extraction pipeline has five stages: + +``` +EPUB file + ↓ +[INGEST] — get the EPUB (local file, URL download, etc.) + ↓ +[PARSE] — extract content (text, images, metadata, structure) + ↓ +[EXTRACT] — identify knowledge (key passages, facts, quotes, arguments) + ↓ +[FORMAT] — structure the output (atoms, memory entries, JSON, markdown) + ↓ +[SINK] — deliver to destination (vault, memory, vector DB, file, etc.) +``` + +### INGEST options + +| If you have... | Use... | +|---------------|--------| +| Local file access | Direct path to EPUB | +| Web access + download | `curl` or web tools to fetch EPUB | +| User provides path | Prompt user for file location | + +### PARSE options + +| If you have... | Use... | +|---------------|--------| +| Python + pip | `scripts/epub-text` or `scripts/epub-info` | +| Python only (no pip) | `zipfile` + `xml.etree.ElementTree` (stdlib) | +| Neither | Ask user to unzip and provide the OPF/XHTML files | + +### EXTRACT options + +| If you have... | Use... | +|---------------|--------| +| LLM access | `scripts/epub-extract-knowledge` with LLM mode | +| No LLM | `scripts/epub-extract-knowledge --no-llm` (heuristic mode) | +| | Manual keyword/phrase extraction | + +### FORMAT options + +| If you have... | Output format | +|---------------|---------------| +| Vault (Obsidian) | Atom/molecule templates | +| Persistent memory | Memory entry format | +| Vector DB / LightRAG | JSON with title + content + source | +| File only | Markdown files, one per chapter | + +### SINK options + +| If you have... | Destination | +|---------------|------------| +| Vault write | Obsidian vault atoms/molecules | +| Memory tool | Direct memory entries | +| Vector DB ingestion | LightRAG / equivalent | +| File write | `.md`, `.json`, `.txt` files | +| None | Print to terminal (ephemeral) | + +## Step 3: Construct and Propose a Pipeline Plan + +Based on what's available, build a specific plan. Present it to the user for +approval before executing. + +### Example: Full Hermes Agent Pipeline + +``` +INGEST: Read EPUB from local path +PARSE: epub-text --json → extract per-chapter text +EXTRACT: epub-extract-knowledge --format atoms → identify facts/quotes/arguments +FORMAT: Render as Obsidian vault atom templates +SINK: write_file → vault atoms in 1 - Atoms/ +VERIFY: obsidian-wiki-link-verification → backlink audit +ENRICH: LightRAG insert → semantic association +``` + +### Example: Minimal Pipeline (Terminal Only) + +``` +INGEST: User provides EPUB path +PARSE: epub-text → plain text file +EXTRACT: epub-extract-knowledge --no-llm → heuristic extraction to JSON +FORMAT: JSON → markdown summary +SINK: write_file → output.md +``` + +### Example: Batch Pipeline (with Delegation) + +``` +INGEST: Find all EPUBs in directory +FOR EACH EPUB: + PARSE: Delegate to subagent → epub-text + epub-info + EXTRACT: Delegate to subagent → epub-extract-knowledge --no-llm + SINK: Collect outputs, write summary +``` + +## Step 4: Execute with User Approval + +1. **Present the pipeline plan** — what stages, what tools, what outputs +2. **Ask for confirmation** — or proceed if the user has said to go ahead +3. **Execute each stage** — report progress at each boundary +4. **Handle failures** — if a tool is missing, fall back to the next option + +## Principles + +1. **Don't assume capabilities exist.** Check before building the pipeline. +2. **Default to the simplest pipeline that works.** Extra complexity without + extra value is noise. +3. **Fall back gracefully.** If LLM isn't available, use heuristics. If pip + isn't available, use stdlib. If nothing works, explain what's missing. +4. **Report at stage boundaries.** Don't make the user ask "what's happening?" +5. **This skill ships the scripts.** All parsing and extraction logic is in + `scripts/`. The agent uses them — it doesn't reimplement them. diff --git a/epub/references/apple-books-compatibility.md b/epub/references/apple-books-compatibility.md new file mode 100644 index 0000000..86a57ca --- /dev/null +++ b/epub/references/apple-books-compatibility.md @@ -0,0 +1,143 @@ +# Apple Books Compatibility + +Apple Books (macOS/iOS) is the most widely used EPUB reading system on Apple +platforms. It enforces several requirements beyond the EPUB 3.3 spec that +cause rendering failures if violated. All rules below were verified by +building, opening, and debugging EPUBs in Apple Books on macOS 26. + +## Required Directory Structure + +**All content must live inside the `OEBPS/` directory.** Apple Books ignores +files at the ZIP root (except `mimetype` and `META-INF/`). + +``` +book.epub +├── mimetype +├── META-INF/ +│ └── container.xml → points to OEBPS/content.opf +└── OEBPS/ + ├── content.opf ← all manifest hrefs are relative to here + ├── nav.xhtml + ├── Styles/default.css + ├── Images/cover.png + └── Text/ + ├── cover.xhtml ← XHTML wrapper for cover image + ├── chapter1.xhtml + └── chapter2.xhtml +``` + +**Wrong** (causes blank pages): +``` +├── Text/ ← ZIP root — Apple Books ignores these +│ └── chapter1.xhtml +``` + +## Cover Image Rules + +1. **The cover must be an XHTML page in the spine, not a raw image reference.** + A `` pointing to `image/png` or `image/jpeg` renders as a blank page. + +2. **Correct pattern:** + - Manifest: `` + - Manifest: `` + - Spine: `` (NOT `cover-img`) + +3. **Cover XHTML must use full-viewport CSS:** + ```css + body { margin: 0; padding: 0; text-align: center; + display: flex; align-items: center; justify-content: center; + min-height: 100vh; } + img { max-width: 100%; max-height: 100vh; width: auto; height: auto; } + ``` + +4. **The `properties="cover-image"` stays on the raw image** — Apple Books uses + it for the library thumbnail. The spine references the wrapper page. + +## CSS Compatibility + +### Deprecated properties (silently ignored by Apple Books) +| Deprecated | Replacement | +|-----------|-------------| +| `page-break-before: always` | `break-before: page` (or remove entirely) | +| `page-break-after: always` | `break-after: page` | +| `page-break-inside: avoid` | `break-inside: avoid` | + +### Margin vs Padding on body +Apple Books applies its own reading margins. Using `margin` on `` stacks +with these, creating excessive whitespace. Use `padding` instead: + +```css +body { + margin: 0; /* ✓ no margin stacking */ + padding: 0 0.5em; /* ✓ internal whitespace only */ +} +``` + +### Namespace selectors +CSS selectors with namespace prefixes (`nav[epub|type="toc"]`) require a +`@namespace` declaration. Some readers fail silently if it's missing. Prefer +generic selectors or keep `@namespace epub` at the top of every CSS file. + +### Avoid `text-indent: 0` on body paragraphs +Apple Books sometimes applies its own `text-indent`, and setting it to zero +can conflict. Omit it unless specifically needed. + +## Navigation (TOC) Visibility + +The `linear` attribute on spine `` controls whether a page appears in +the reading flow: + +| Setting | Behavior | +|---------|----------| +| `linear="yes"` (default) | Page visible in reading flow | +| `linear="no"` | Page hidden from reading flow, accessible via app TOC browser | + +Apple Books still shows hidden pages in its built-in table of contents browser. +`linear="no"` only removes them from the swipe/page-turn reading order. + +## XHTML Requirements + +1. **No `xmlns:epub` on content chapters** — only needed on the navigation + document. Unused namespace declarations can trigger strict XML validation + failures in Apple Books' parser. + +2. **All XHTML must be well-formed XML** — self-closing tags (`
`, ``), + properly escaped ampersands (`&`), no bare `<` in text content. + +3. **CSS `` in every ``** — unstyled chapters render with Apple + Books' defaults, which may differ dramatically from your intended appearance. + Always include the stylesheet link in chapter XHTML. + +## Spine Ordering Conventions + +### Pattern 1: No Cover (simplest) +``` +nav → chapters +``` +No cover in the book. ToC is the first thing the reader sees. + +### Pattern 2: Cover Only (ToC via app browser) +``` +cover-page → nav(linear="no") → chapters +``` +Cover is page 1. ToC accessible via app's built-in browser. + +### Pattern 3: Full (most commercial ebooks) +``` +cover-page → nav → chapters +``` +Cover is page 1, ToC is page 2, chapters follow. This is the `epub-scaffold` +default when `--cover` is provided. + +## Validation Quirks + +EPUBCheck validates against the EPUB spec, not Apple Books' additional +requirements. A valid EPUB can still render blank pages. Always test in +Apple Books before declaring a build complete. + +## References Tested Against + +These rules were verified on: +- macOS 26.5, Apple Books (native) +- `The Spider Blueprint` test EPUB (5 build iterations) +- `Agentic AI in Enterprise` commercial Apress EPUB (2.1MB, passed all checks) diff --git a/epub/references/epub-format-internals.md b/epub/references/epub-format-internals.md new file mode 100644 index 0000000..87b8948 --- /dev/null +++ b/epub/references/epub-format-internals.md @@ -0,0 +1,185 @@ +# EPUB Format Internals + +An EPUB file is a **ZIP archive** (Open Container Format, OCF) containing +structured web content packaged as a single distributable file. This reference +covers EPUB 3.3 (the current W3C standard) and EPUB 2.0 compatibility points. + +## Container Structure (OCF) + +``` +book.epub (ZIP archive) +├── mimetype ← "application/epub+zip" (MUST be first, uncompressed) +├── META-INF/ +│ └── container.xml ← Points to the OPF package document +└── OEBPS/ (or custom root) + ├── content.opf ← Package document (metadata, manifest, spine) + ├── nav.xhtml ← EPUB3 navigation document + ├── toc.ncx ← EPUB2 table of contents (legacy) + ├── Text/ + │ ├── chapter1.xhtml ← XHTML content documents + │ └── chapter2.xhtml + ├── Styles/ + │ └── style.css + └── Images/ + └── cover.jpg +``` + +### Critical ZIP Rules + +- **`mimetype` must be the first file**, stored **uncompressed** (STORE method). + This is a hard requirement — reading systems identify the format from this. +- All other files may be compressed (DEFLATE). +- The `mimetype` file contains exactly: `application/epub+zip` +- No extra bytes, no trailing newline. + +### container.xml + +```xml + + + + + + +``` + +The `full-path` attribute points to the OPF package document. This is the only +required file in `META-INF/`. + +## The Three Planes + +EPUB 3.3 uses a "three planes" model to classify resources: + +| Plane | What it contains | Key rules | +|-------|-----------------|-----------| +| **Manifest plane** | All resources that contribute to rendering | Listed in OPF ``. Exhaustive — every file in the EPUB that's used for rendering must appear here. | +| **Spine plane** | Resources in the default reading order | Defined by OPF ``. Only XHTML and SVG are allowed by default (EPUB content documents). Other formats require manifest fallbacks. | +| **Content plane** | Resources embedded within content documents (images, CSS, scripts, fonts, audio, video) | Core media types are guaranteed supported. Foreign resources require fallbacks. | + +A resource can appear on multiple planes. For example, an XHTML chapter is on +all three — it's in the manifest, in the spine, and can embed resources from +the content plane. + +## Package Document (OPF) + +The OPF file (typically `content.opf`) is the "table of contents for the +container." It tells reading systems what's in the EPUB and how to render it. + +```xml + + + + urn:uuid:123e4567-e89b-12d3-a456-426614174000 + Book Title + en + Author Name + 2026-01-01T00:00:00Z + + + + + + + + + + + + +``` + +### Metadata + +Dublin Core elements are used for standard metadata: +- `dc:title` — title (required) +- `dc:creator` — author (optional but expected) +- `dc:language` — language code (required) +- `dc:identifier` — unique identifier (required) +- `dc:date` — publication date +- `dc:publisher`, `dc:rights`, `dc:description`, `dc:subject` — optional + +Custom metadata uses `` elements with `property` attributes. + +### Manifest + +Every publication resource (file used in rendering) must have an `` in +the manifest. Each item requires: +- `id` — unique identifier within the OPF (used by spine and other references) +- `href` — relative path to the file within the ZIP +- `media-type` — MIME type of the resource + +Optional `properties` attribute: space-separated list. Key values: +- `nav` — this is the EPUB3 navigation document +- `cover-image` — this is the cover image +- `scripted` — contains JavaScript +- `mathml` — contains MathML +- `remote-resources` — references remote resources + +### Spine + +Defines the linear reading order. Each `` references a manifest item +by its `id`. Only EPUB content documents (XHTML, SVG) should appear here by +default. The `linear` attribute can be `"no"` for non-linear content (accessible +but not part of the default reading flow). + +## EPUB 2 vs EPUB 3 + +| Feature | EPUB 2 | EPUB 3 | +|---------|--------|--------| +| Navigation | NCX file (XML-based TOC) | NAV document (XHTML with `