diff --git a/AGENTS.md b/AGENTS.md index f5d6f02..90e77d3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -55,7 +55,7 @@ When the user mentions these keywords, load the corresponding skill: |---|---| | "build a CLI", "make a CLI tool", "agent-friendly CLI", "add --json flag" | [cli-builder](cli-builder/SKILL.md) | | "debug this", "root cause", "why is this broken", "fix this bug" | [systematic-debugging](systematic-debugging/SKILL.md) | -| "epub", "ebook", "EPUB file", "read epub", "write epub", "create ebook", "extract from epub", "epub to text", "ebook format" | [epub](epub/SKILL.md) | +| "epub", "ebook", "EPUB file", "ebook format", "read epub", "write epub", "create ebook", "extract from epub", "epub to text", "edit epub", "repair epub", "convert epub2", "epub images", "batch epub", "ebook metadata" | [epub](epub/SKILL.md) | | "weather", "forecast", "temperature", "is it raining", "Tempest" | [tempest-cli](tempest-cli/SKILL.md) | | "reverse-engineer", "understand this codebase", "PRD from code", "architecture document" | [software-architecture-analysis](software-architecture-analysis/SKILL.md) | || "data architecture", "data platform", "data strategy", "data mesh", "governance" | [data-architect](data-architect/SKILL.md) | diff --git a/README.md b/README.md index 2fa6685..1c89a6b 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,12 @@ PhD-level expertise in data science, statistics, and machine learning. Rigorous ### [epub](epub/SKILL.md) -EPUB file format expert — read, write, and edit EPUB2/EPUB3 ebooks. Extract text, metadata, structure, and knowledge into formatted output (JSON, vault atoms, memory entries). Create valid EPUBs from scratch. Validate against the EPUB specification. Five CLI scripts with a capability-discovery pipeline for portable knowledge extraction across any agent harness. +EPUB file format expert — read, write, edit, convert, and repair EPUB2/EPUB3 ebooks. +Ten CLI scripts: structure inspection, text extraction, knowledge extraction (LLM mode +via env vars), scaffold creation, surgical editing, image extraction, batch processing, +EPUB2→3 conversion, repair, and validation. Eight reference files covering format +internals, Python libraries, spec/validation, tutorials, capability discovery, +fixed-layout, accessibility, and media overlays. Portable across any AgentSkills harness. ### [forgejo-cli](forgejo-cli/SKILL.md) diff --git a/epub/SKILL.md b/epub/SKILL.md new file mode 100644 index 0000000..512f8ac --- /dev/null +++ b/epub/SKILL.md @@ -0,0 +1,474 @@ +--- +name: epub +description: >- + EPUB file format expert — read, write, and edit EPUB2/EPUB3 ebooks. Extract text, + metadata, structure, and knowledge from EPUB files for enrichment or memory. Create + valid EPUBs from scratch. Validate against the EPUB specification. Use when the user + mentions epub, ebook, EPUB file, ebook format, read epub, write epub, create ebook, + extract from epub, epub to text, or ebook structure. +license: MIT +compatibility: >- + Python 3.8+ required. Core scripts use EbookLib (pip install EbookLib) for reading and + creating EPUBs. Optional epublib (pip install epublib) for non-intrusive editing. + beautifulsoup4 (pip install beautifulsoup4) for text extraction. Optional LLM mode via + EPUB_LLM_URL + EPUB_LLM_KEY env vars (any OpenAI-compatible provider). EPUBCheck + (Java, optional) for authoritative validation. Portable across all AgentSkills-compatible + harnesses — scripts are platform-agnostic. +metadata: + skills: [epub, ebook, publishing, knowledge-extraction, document-processing] + tags: [epub, ebook, epub3, epub2, publishing, document-extraction, knowledge] +--- + +# EPUB — Ebook Creation, Extraction & Enrichment + +Expert-level EPUB handling: read, write, edit, validate, and extract knowledge +from EPUB 2 and EPUB 3 files. Ships with five Python CLI scripts and five +detailed references covering the entire EPUB domain. + +## EPUB Format Essentials + +An EPUB file is a **ZIP archive** (Open Container Format, OCF) with a specific +internal layout. The W3C EPUB 3.3 standard uses a "three planes" model: + +| Plane | Contains | Key Rule | +|-------|----------|----------| +| **Manifest** | All resources (XHTML, images, CSS, fonts) | Every file must be listed in OPF `` | +| **Spine** | Linear reading order | Only XHTML/SVG by default; other types need fallbacks | +| **Content** | Resources embedded within documents | Core media types guaranteed; foreign types need fallbacks | + +The package document (`content.opf`) holds metadata (Dublin Core), manifest +(every resource), and spine (reading order). EPUB3 uses an XHTML `nav` document +for navigation; EPUB2 uses `.ncx` XML. Both can coexist for compatibility. + +**Critical rules:** `mimetype` must be the first ZIP entry, stored uncompressed. +All content documents must be well-formed XML (XHTML, not HTML5). The manifest +must list every file used in rendering. Read `references/epub-format-internals.md` +for the full structure reference. + +## Decision Table — Which Script to Use + +| Task | Script | Notes | +|------|--------|-------| +| See structure, metadata, manifest, spine, TOC | `epub-info` | JSON output, `--summary` for compact | +| Extract clean reading-order text | `epub-text` | Per-chapter or single file | +| Create minimal valid EPUB from scratch | `epub-scaffold` | No dependencies needed | +| Extract facts, quotes, definitions, arguments | `epub-extract-knowledge` | Heuristic or LLM mode (env var auto-detect) | +| Validate against EPUB spec | `epub-validate` | EPUBCheck or Python fallback | +| **Edit EPUB** (metadata, chapters, spine, CSS) | **`epub-edit`** | **v2 flagship — 8 subcommands, non-intrusive** | +| Extract images | `epub-images` | List or extract to directory | +| Batch process multiple EPUBs | `epub-batch` | Wrap existing scripts across globs | +| Convert EPUB2 → EPUB3 | `epub-convert` | Add NAV, update NS, keep NCX | +| Diagnose & repair structural issues | `epub-repair` | Auto-fix common validation failures | + +## Scripts + +All scripts live in `scripts/` relative to this skill's directory. Each follows +cli-builder conventions: `--json` for machine output, `--dry-run` to preview, +non-interactive, errors to stderr. Run with `--help` for full flag details. + +### epub-info — Structure & Metadata Dump + +```bash +scripts/epub-info book.epub --json +scripts/epub-info book.epub --summary # compact manifest +scripts/epub-info book.epub --dry-run # preview +``` + +Outputs: EPUB version, metadata (title, author, language, identifier), +manifest (all items with id/href/media-type), spine (reading order), +TOC (nested structure). `--summary` reduces manifest to id+href+media-type. + +### epub-text — Clean Text Extraction + +```bash +scripts/epub-text book.epub # plain text to stdout +scripts/epub-text book.epub --json # JSON with chapter array +scripts/epub-text book.epub --chapters # one .txt per chapter +scripts/epub-text book.epub --output book.txt # single file +scripts/epub-text book.epub --format markdown # markdown output +``` + +Extracts text from spine-ordered content documents. Strips HTML tags, preserves +paragraph structure. Requires beautifulsoup4. Respects spine linearity — only +processes documents in the reading order. + +### epub-scaffold — Create Valid EPUB from Scratch + +```bash +scripts/epub-scaffold --title "My Book" --author "Jane Doe" +scripts/epub-scaffold --title "Novel" --author "Me" --chapters 12 --output novel.epub +scripts/epub-scaffold --title "Guide" --author "Me" --cover cover.jpg +scripts/epub-scaffold --title "Guide" --author "Me" --cover cover.jpg --toc-hidden --dry-run +``` + +Creates a valid EPUB3 with all content inside `OEBPS/` — required for Apple +Books compatibility. No external dependencies — Python stdlib only. + +**Cover handling:** When `--cover` is provided, the scaffold automatically: +- Copies the image to `OEBPS/Images/cover.{ext}` +- Generates `OEBPS/Text/cover.xhtml` — an XHTML wrapper page with full-viewport CSS +- Adds the cover page to the spine as the first item +- Sets `properties="cover-image"` on the raw image for library thumbnails + +This follows the Apple Books requirement that covers must be XHTML pages in +the spine, not raw image references (raw images render as blank pages). + +**Nav visibility:** `--toc-hidden` sets `linear="no"` on the nav spine item, +hiding it from the reading flow (still accessible via the app's built-in TOC +browser). Default is `--toc-visible` (nav renders as a page). + +**CSS:** Ships `OEBPS/Styles/default.css` with Apple Books-compatible typography: +no deprecated `page-break-before`, `margin: 0` on body (padding for whitespace), +proper heading hierarchy, and responsive styling. + +**Cover art guidance:** The `--cover` flag accepts a pre-existing image file. If +the user does not have a cover image, offer to generate one using the agent's +image_gen capability. See `references/apple-books-compatibility.md` for the full +cover XHTML and CSS conventions. + +### epub-cover — Add Cover to Existing EPUB + +```bash +scripts/epub-cover wrap book.epub --image cover.png --output with-cover.epub +scripts/epub-cover wrap book.epub --image cover.png --in-place +``` + +Adds a cover XHTML wrapper page to an EPUB that already has a cover image in +its manifest. Use when the image exists but isn't rendering in Apple Books. +Requires epublib. + +### epub-extract-knowledge — Knowledge Extraction Pipeline + +```bash +# LLM mode — set env vars first (see below), then run without flags: +scripts/epub-extract-knowledge book.epub --format json +scripts/epub-extract-knowledge book.epub --format atoms +scripts/epub-extract-knowledge book.epub --format memory + +# Force heuristic mode (ignore env vars): +scripts/epub-extract-knowledge book.epub --no-llm --format json + +# Custom prompt override: +scripts/epub-extract-knowledge book.epub --prompt "Extract all definitions" --format json +``` + +Extracts knowledge from EPUB content: facts, definitions, key points, and +arguments. Two modes, auto-selected: + +- **LLM mode (auto-detected):** When `EPUB_LLM_URL` and `EPUB_LLM_KEY` env vars + are set, calls the configured LLM with the chapter text and extraction prompt. + Produces high-quality structured insights. Falls back to heuristic if the LLM + call fails. +- **Heuristic mode (fallback):** When env vars are NOT set, or `--no-llm` is + passed, uses pattern matching (headings, emphasis markers, definition language, + paragraph density) to identify knowledge-bearing passages. No LLM required. + +Output formats: +- `json` — raw structured JSON with types, content, and source chapters +- `atoms` — Obsidian vault atom templates (YAML frontmatter + body) +- `memory` — key-value memory entries suitable for agent persistence + +### LLM Configuration Convention + +Several scripts in this skill support optional LLM-powered features. Any script +that does auto-detects LLM availability via environment variables. Set them once +and all scripts inherit: + +```bash +# Required for LLM mode: +export EPUB_LLM_URL="https://your-provider.example.com/v1" # OpenAI-compatible endpoint +export EPUB_LLM_KEY="sk-..." # API key + +# Optional: +export EPUB_LLM_MODEL="model-name" # Defaults to provider default +``` + +**How it works:** +- If `EPUB_LLM_URL` and `EPUB_LLM_KEY` are both set → LLM mode enabled +- If either is missing → heuristic/deterministic mode (no LLM) +- `--no-llm` flag forces heuristic mode even when env vars are set +- The scripts make OpenAI-compatible `POST /chat/completions` calls — any + OpenAI-compatible provider works (OpenAI, OpenCode, Anthropic via proxy, + local llama.cpp, Ollama, vLLM, etc.) + +**Which scripts support this:** +| Script | LLM Feature | Fallback | +|--------|------------|----------| +| `epub-extract-knowledge` | Structured knowledge extraction | Heuristic pattern matching | +| `epub-validate` | LLM-generated repair suggestions for errors | Error codes only | +| *(more scripts can adopt this pattern as features are added)* | | | + +**Agent instructions:** Before running any extraction pipeline, set these +env vars in your environment. They are inherited by subprocesses, so every +script in the pipeline auto-detects the same LLM configuration. If your +harness provides an LLM natively (e.g., you *are* the LLM), you can skip +the env vars — the heuristic mode is designed for that case. But if you +have access to an external LLM API, wiring it through these env vars +unlocks dramatically better extraction quality without requiring the +agent to manually chunk, prompt, parse, and re-inject results. + +### epub-edit — Surgical EPUB Editing (v2) + +```bash +scripts/epub-edit info book.epub --json +scripts/epub-edit metadata book.epub --title "New Title" --output out.epub +scripts/epub-edit add-chapter book.epub --content new.xhtml --after chapter3 --output out.epub +scripts/epub-edit remove-chapter book.epub --id chapter5 --output out.epub +scripts/epub-edit reorder-spine book.epub --order chapter3,chapter1,chapter2 --dry-run +scripts/epub-edit rename-resource book.epub --from Images/old.jpg --to Images/new.jpg +scripts/epub-edit inject-css book.epub --css dark.css --output out.epub +scripts/epub-edit update-manifest book.epub --output out.epub +``` + +Non-intrusive EPUB editing via epublib. Eight subcommands covering the full +edit surface. Never overwrites original — defaults to `--output out.epub`; +use `--in-place` to commit. All subcommands support `--json`, `--dry-run`. + +### epub-images — Image Extraction + +```bash +scripts/epub-images book.epub --list --json # list all images +scripts/epub-images book.epub --extract images/ # extract all to directory +scripts/epub-images book.epub --type cover --extract . # cover image only +``` + +### epub-batch — Multi-File Processing + +```bash +scripts/epub-batch extract-text "books/*.epub" --output texts/ +scripts/epub-batch validate "books/*.epub" --json +scripts/epub-batch metadata "books/*.epub" --set-author "Author" --output-dir fixed/ +scripts/epub-batch info "books/*.epub" --json +``` + +### epub-convert — EPUB2 → EPUB3 + +```bash +scripts/epub-convert old.epub --output new-v3.epub +scripts/epub-convert old.epub --validate --json +``` + +### epub-repair — Diagnose & Fix + +```bash +scripts/epub-repair broken.epub --diagnose --json # list fixable issues +scripts/epub-repair broken.epub --output fixed.epub # auto-fix +``` + +### epub-validate — Structural Validation + +```bash +scripts/epub-validate book.epub --json +scripts/epub-validate book.epub --dry-run +``` + +Tries EPUBCheck (Java JAR) first for authoritative validation. Falls back to +pure-Python structural checks: mimetype position/compression/content, +container.xml parseability, OPF schema, manifest completeness, spine reference +integrity, required metadata, and NAV document presence. + +## Capability Discovery & Pipeline Construction + +Before executing a multi-step EPUB pipeline, **discover what tools are available** +on your agent platform. This skill is portable — the exact pipeline shape +depends on your harness's capabilities. + +### Discovery Protocol + +1. **Enumerate:** What tools do you have? File write? Web access? Subagents? + Cron? Persistent memory? Vector DB? Vault? LLM? +2. **Classify:** Map available tools to pipeline stages (Ingest → Parse → + Extract → Format → Sink) +3. **Construct:** Build a specific pipeline from available pieces +4. **Propose:** Present the plan to the user before executing + +### Pipeline Stages + +``` +EPUB file + ↓ INGEST — local path, URL download, or user provides file + ↓ PARSE — epub-text, epub-info, or stdlib zipfile+XML + ↓ EXTRACT — epub-extract-knowledge (heuristic or LLM) + ↓ FORMAT — vault atoms, memory entries, JSON, markdown + ↓ SINK — file write, memory tool, vector DB, vault, terminal +``` + +### Example Pipelines + +**Full (Hermes Agent):** +Ingest EPUB → epub-text (JSON) → epub-extract-knowledge → vault atoms → +wiki-link-verification → LightRAG ingest + +**Minimal (any harness with terminal + file write):** +Ingest EPUB → epub-text → epub-extract-knowledge --no-llm → write output.md + +**Batch (multi-file):** +Find *.epub → for each: epub-text --chapters → epub-extract-knowledge --no-llm → collect +results → summary report + +See `references/agent-capability-discovery.md` for the full protocol with +worked examples for different platforms. + +## Knowledge Extraction Deep Dive + +EPUB files are dense sources of structured knowledge. The extraction process +targets specific knowledge types: + +| Type | Detection | Example | +|------|-----------|---------| +| **Fact** | Headings, list items, named entities | "Python 3.13 added the `@override` decorator" | +| **Definition** | Paragraphs with definition markers | "A coroutine is defined as a function that can suspend execution" | +| **Key point** | Emphasized text (bold, italic) | Important conclusions, takeaways | +| **Argument** | Dense paragraphs (>200 chars) | Multi-sentence reasoning chains | + +### Prompt Design for LLM Mode + +When using LLM extraction, provide a focused prompt: + +``` +Extract from this chapter: +1. All technical definitions (term + definition) +2. Key facts (concise, standalone statements) +3. Notable quotes (exact wording) +4. Core arguments (the main thesis and supporting points) + +Format as JSON with fields: type, content, context +``` + +### Sink Options by Platform + +| Sink | Platform | Format to use | +|------|----------|---------------| +| Vault atoms | Obsidian | `--format atoms` | +| Agent memory | Most harnesses | `--format memory` | +| Vector DB | LightRAG, Chroma | `--format json` → insert | +| Plain files | Any | `--output DIR` | + +## Common Workflows + +### Create EPUB from Markdown Files + +```bash +# 1. Scaffold the EPUB +scripts/epub-scaffold --title "My Book" --author "Me" --chapters 3 --output book.epub + +# 2. Use epublib (Python) to inject real content into each chapter +# See references/tutorials-and-guides.md for the editing pattern +``` + +### Extract All Images from EPUB + +```bash +scripts/epub-images book.epub --extract images/ +``` + +### Edit an EPUB + +```bash +# Update metadata +scripts/epub-edit metadata book.epub --title "New Title" --output revised.epub + +# Add a chapter +scripts/epub-edit add-chapter book.epub --content new.xhtml --output expanded.epub + +# Inject dark theme +scripts/epub-edit inject-css book.epub --css dark.css --output dark.epub +``` + +### Fix a Broken EPUB + +```bash +scripts/epub-repair broken.epub --diagnose --json # see what's broken +scripts/epub-repair broken.epub --output fixed.epub # auto-fix +``` + +### Batch Extract Text from a Library + +```bash +scripts/epub-batch extract-text "books/*.epub" --output texts/ +``` + +### Convert EPUB2 to EPUB3 + +```bash +scripts/epub-convert old.epub --output old-v3.epub --validate +``` + +## Apple Books Compatibility + +Apple Books on macOS/iOS enforces requirements beyond the EPUB spec. These +rules were verified by building and testing on macOS 26. + +| Rule | Why | +|------|-----| +| All content inside `OEBPS/` directory | Files at ZIP root render as blank pages | +| Cover must be XHTML page in spine | Raw `` to `image/png` renders blank | +| `margin: 0` on body, use `padding` | Apple Books applies its own margins; they stack | +| No `page-break-before` (use `break-before: page` or omit) | Deprecated; Apple Books ignores it | +| `xmlns:epub` only on nav document | Unused namespace declarations can trigger parser failures | +| Cover image keeps `properties="cover-image"` | Used for library thumbnail on raw image, not on wrapper page | +| `linear="no"` hides page from reading flow | Nav still accessible via app's built-in TOC browser | + +**Spine ordering patterns:** +| Pattern | Order | Use case | +|---------|-------|----------| +| No cover | `nav → chapters` | Simplest | +| Cover only | `cover-page → nav(linear="no") → chapters` | Cover visible, ToC via app browser | +| Full | `cover-page → nav → chapters` | Most commercial ebooks (scaffold default) | + +See `references/apple-books-compatibility.md` for the full reference with +CSS examples, cover XHTML template, and validation quirks. + +## Gotchas + +- **mimetype compression:** Python's `zipfile` compresses by default. Always use + `ZIP_STORED` for the mimetype entry. A compressed mimetype silently breaks + reading systems. +- **XHTML ≠ HTML5:** Content documents must be well-formed XML. Self-closing + tags required (`
` not `
`). Use `xmlns:epub="http://www.idpf.org/2007/ops"`. +- **Manifest is exhaustive:** Every file in the EPUB must be listed. Missing + manifest entries cause validation failures. Images, CSS, fonts — no exceptions. +- **AGPL boundary:** EbookLib is AGPL; scripts call it at runtime but don't + bundle it. Users install it themselves via pip. This skill and its scripts + are MIT. +- **EPUBCheck needs Java:** The authoritative validator requires Java. The + fallback Python checks catch structural issues but not XHTML schema violations + or CSS validity. +- **Spine references manifest IDs:** An `idref` in the spine must match an `id` + in the manifest. Broken references cause the EPUB to fail validation. +- **Navigation document:** EPUB3 requires a NAV with `properties="nav"`. + Without it, reading systems may not show a table of contents. +- **Language is required:** Both `` and `xml:lang`/`lang` + attributes on content documents. Missing language = invalid EPUB. + +## Pitfalls + +- **Don't skip capability discovery.** Assuming a tool exists that doesn't + leads to broken pipelines. Always check before building. +- **Don't assume LLM availability.** Always offer `--no-llm` fallback for + extraction. Heuristic mode works surprisingly well for well-structured books. +- **Don't modify EPUBs in place without backup.** EPUB editing is surgery — + always keep the original. +- **Don't mix EbookLib and epublib on the same file in the same session.** + They have different memory models and may conflict. Pick one library per task. +- **Don't assume the OPF is at `OEBPS/content.opf`.** Always read + `container.xml` to find the actual path. The root directory varies. + +## References + +- `references/epub-format-internals.md` — Full structural reference (OCF, OPF, + XHTML, NCX/NAV, spine, three planes, core media types, EPUB2 vs EPUB3) +- `references/python-libraries.md` — EbookLib vs epublib comparison, code + examples, when to use which, license notes +- `references/spec-and-validation.md` — W3C EPUB 3.3 spec access points, + EPUBCheck usage, Ace accessibility validation, key constraints summary +- `references/tutorials-and-guides.md` — Beginner to advanced guides, common + workflows, manual OPF editing, batch operations, pitfalls +- `references/agent-capability-discovery.md` — Protocol for probing agent tools, + constructing extraction pipelines, worked examples across platforms +- `references/fixed-layout-epub.md` — Fixed-layout detection and properties + (rendition:layout, orientation, spread, viewport meta) +- `references/accessibility.md` — WCAG alignment, alt text, heading hierarchy, + ARIA roles, Ace integration, accessibility metadata +- `references/media-overlays.md` — SMIL synchronization, audio-text pairing, + skippability/escapability, detection from manifest diff --git a/epub/references/accessibility.md b/epub/references/accessibility.md new file mode 100644 index 0000000..7d8e1aa --- /dev/null +++ b/epub/references/accessibility.md @@ -0,0 +1,127 @@ +# EPUB Accessibility + +Accessible EPUBs can be read by people using screen readers, braille displays, +and other assistive technologies. The W3C EPUB Accessibility 1.1 specification +defines conformance requirements. + +## Key Requirements + +### 1. Alternative Text for Images + +Every `` element must have an `alt` attribute: + +```xhtml +Flowchart showing data pipeline stages +``` + +Decorative images should use `alt=""` so screen readers skip them. + +### 2. Heading Hierarchy + +Headings must follow a logical nesting order — no skipping levels: + +```xhtml +

Chapter Title

+

Section

+

Subsection

+

Another Section

+``` + +Do not use `

` without a preceding `

`. + +### 3. Language Tagging + +Every XHTML document must declare its language: + +```xhtml + +``` + +For language shifts within a document: + +```xhtml +

English text texte français more English.

+``` + +### 4. ARIA Roles + +ARIA landmarks help screen reader navigation: + +```xhtml + +
...
+ +``` + +Key roles: `doc-toc`, `doc-chapter`, `doc-cover`, `doc-footnote`, `doc-glossary`, +`doc-index`, `doc-bibliography`, `doc-acknowledgments`. + +### 5. Accessibility Metadata + +The OPF must declare accessibility conformance: + +```xml +textual +visual +alternativeText +longDescription +none + + This publication meets WCAG 2.0 Level AA. + +``` + +## Ace by DAISY + +[Ace](https://daisy.org/activities/software/ace/) is the official accessibility +validator for EPUB. It checks: + +- Image alt text presence and quality +- Heading hierarchy correctness +- Language declarations +- ARIA roles and landmarks +- Accessibility metadata completeness +- Color contrast (basic) + +```bash +# Install: npm install -g @daisy/ace +ace book.epub --outdir ace-report/ +``` + +The report is an HTML dashboard showing violations by type and severity. + +## WCAG Alignment + +EPUB Accessibility 1.1 maps to WCAG 2.x Level AA. The four principles: + +| Principle | EPUB Meaning | +|-----------|-------------| +| **Perceivable** | All content has text alternatives, captions, sufficient contrast | +| **Operable** | All navigation is keyboard-accessible, no seizure-inducing content | +| **Understandable** | Language is declared, reading order is logical, predictable | +| **Robust** | Content is well-formed XHTML, compatible with assistive technologies | + +## Common Failures + +| Failure | Detection | Fix | +|---------|----------|-----| +| Missing alt text | Ace / manual inspection | Add `alt` to every `` | +| Skipped heading levels | Ace | Re-level headings | +| Language not declared | `xml:lang` + `lang` missing on `` | Add both attributes | +| No accessibility metadata | OPF missing `schema:accessibilityFeature` | Add required meta tags | +| Color-only information | Content relies on color for meaning | Add text labels or patterns | + +## Script Support + +Our EPUB scripts do NOT perform accessibility validation. Use Ace for that. +However, `epub-validate` checks for structural issues (XHTML well-formedness, +metadata completeness) that are prerequisites for accessibility. + +`epub-edit` can help remediate some issues: adding metadata fields, injecting +CSS for contrast, or adding `alt` attributes via DOM manipulation. + +## References + +- W3C EPUB Accessibility 1.1: https://www.w3.org/TR/epub-a11y-11/ +- W3C EPUB Accessibility Techniques 1.1: https://www.w3.org/TR/epub-a11y-tech-11/ +- Ace by DAISY: https://daisy.org/activities/software/ace/ diff --git a/epub/references/agent-capability-discovery.md b/epub/references/agent-capability-discovery.md new file mode 100644 index 0000000..496c45b --- /dev/null +++ b/epub/references/agent-capability-discovery.md @@ -0,0 +1,182 @@ +# Agent Capability Discovery & Pipeline Construction + +This reference teaches the agent how to discover its own capabilities before +constructing an EPUB extraction or construction pipeline. The skill must be +portable across any agent harness — this discovery step is how it adapts. + +## Why Discovery Matters + +Different agent platforms have different capabilities: + +| Platform | File I/O | Subagents | Cron | Kanban | Vector DB | Vault | +|----------|----------|-----------|------|--------|-----------|-------| +| Hermes Agent | terminal/write_file | delegate_task | cronjob | kanban | LightRAG | Obsidian vault | +| Claude Code | Bash/Write | Task tool | Scheduled tasks | — | — | — | +| OpenCode | Terminal/File | Subagents | — | — | — | — | +| GitHub Copilot | Terminal/File | — | — | — | — | — | +| Cursor | Terminal/File | — | — | — | — | — | +| Basic harness | Terminal only | — | — | — | — | — | + +The extraction pipeline works on any of these — but the *shape* of the pipeline +changes based on what's available. + +## Step 1: Enumerate Available Capabilities + +Before building a pipeline, probe the agent's toolset. The exact method depends +on the platform, but the principle is the same: check what's available. + +Several scripts in this skill use an **LLM auto-detection convention:** if the +environment variables `EPUB_LLM_URL` and `EPUB_LLM_KEY` are set, the scripts +enable LLM-powered features automatically. If they're absent, the scripts fall +back to heuristic/deterministic mode. This means you don't need to pass +`--no-llm` flags or state — the scripts detect it. + +```bash +# Set once for all scripts: +export EPUB_LLM_URL="https://your-provider.example.com/v1" +export EPUB_LLM_KEY="sk-..." +export EPUB_LLM_MODEL="model-name" # optional +``` + +Any OpenAI-compatible provider works — OpenAI, Anthropic via proxy, local +llama.cpp, Ollama, vLLM, OpenCode, etc. + +### What to check + +| Capability | How to detect | +|-----------|---------------| +| **Terminal / shell access** | Can you run shell commands? | +| **File write** | Can you create files on disk? | +| **Web access** | Can you make HTTP requests, scrape URLs, search the web? | +| **Subagents / task delegation** | Can you spawn child agents for parallel work? | +| **Scheduling / cron** | Can you schedule jobs to run later? | +| **Persistent memory** | Can you save facts that survive this session? | +| **Vector / semantic search** | Can you search by meaning, not keywords? | +| **Kanban / workflow boards** | Can you create and track tasks on boards? | +| **Vault / knowledge base** | Is there a structured note system (Obsidian, etc.)? | +| **LLM access** | Can you call an LLM for text extraction/classification? | +| **Browser automation** | Can you interact with web pages dynamically? | + +### Decision: Do you have what this skill's scripts need? + +At minimum, you need: +- Terminal/shell access (to run Python scripts) +- File write (to save output) + +Everything else enriches the pipeline but isn't required. + +## Step 2: Classify Into Pipeline Stages + +An EPUB knowledge extraction pipeline has five stages: + +``` +EPUB file + ↓ +[INGEST] — get the EPUB (local file, URL download, etc.) + ↓ +[PARSE] — extract content (text, images, metadata, structure) + ↓ +[EXTRACT] — identify knowledge (key passages, facts, quotes, arguments) + ↓ +[FORMAT] — structure the output (atoms, memory entries, JSON, markdown) + ↓ +[SINK] — deliver to destination (vault, memory, vector DB, file, etc.) +``` + +### INGEST options + +| If you have... | Use... | +|---------------|--------| +| Local file access | Direct path to EPUB | +| Web access + download | `curl` or web tools to fetch EPUB | +| User provides path | Prompt user for file location | + +### PARSE options + +| If you have... | Use... | +|---------------|--------| +| Python + pip | `scripts/epub-text` or `scripts/epub-info` | +| Python only (no pip) | `zipfile` + `xml.etree.ElementTree` (stdlib) | +| Neither | Ask user to unzip and provide the OPF/XHTML files | + +### EXTRACT options + +| If you have... | Use... | +|---------------|--------| +| LLM access | `scripts/epub-extract-knowledge` with LLM mode | +| No LLM | `scripts/epub-extract-knowledge --no-llm` (heuristic mode) | +| | Manual keyword/phrase extraction | + +### FORMAT options + +| If you have... | Output format | +|---------------|---------------| +| Vault (Obsidian) | Atom/molecule templates | +| Persistent memory | Memory entry format | +| Vector DB / LightRAG | JSON with title + content + source | +| File only | Markdown files, one per chapter | + +### SINK options + +| If you have... | Destination | +|---------------|------------| +| Vault write | Obsidian vault atoms/molecules | +| Memory tool | Direct memory entries | +| Vector DB ingestion | LightRAG / equivalent | +| File write | `.md`, `.json`, `.txt` files | +| None | Print to terminal (ephemeral) | + +## Step 3: Construct and Propose a Pipeline Plan + +Based on what's available, build a specific plan. Present it to the user for +approval before executing. + +### Example: Full Hermes Agent Pipeline + +``` +INGEST: Read EPUB from local path +PARSE: epub-text --json → extract per-chapter text +EXTRACT: epub-extract-knowledge --format atoms → identify facts/quotes/arguments +FORMAT: Render as Obsidian vault atom templates +SINK: write_file → vault atoms in 1 - Atoms/ +VERIFY: obsidian-wiki-link-verification → backlink audit +ENRICH: LightRAG insert → semantic association +``` + +### Example: Minimal Pipeline (Terminal Only) + +``` +INGEST: User provides EPUB path +PARSE: epub-text → plain text file +EXTRACT: epub-extract-knowledge --no-llm → heuristic extraction to JSON +FORMAT: JSON → markdown summary +SINK: write_file → output.md +``` + +### Example: Batch Pipeline (with Delegation) + +``` +INGEST: Find all EPUBs in directory +FOR EACH EPUB: + PARSE: Delegate to subagent → epub-text + epub-info + EXTRACT: Delegate to subagent → epub-extract-knowledge --no-llm + SINK: Collect outputs, write summary +``` + +## Step 4: Execute with User Approval + +1. **Present the pipeline plan** — what stages, what tools, what outputs +2. **Ask for confirmation** — or proceed if the user has said to go ahead +3. **Execute each stage** — report progress at each boundary +4. **Handle failures** — if a tool is missing, fall back to the next option + +## Principles + +1. **Don't assume capabilities exist.** Check before building the pipeline. +2. **Default to the simplest pipeline that works.** Extra complexity without + extra value is noise. +3. **Fall back gracefully.** If LLM isn't available, use heuristics. If pip + isn't available, use stdlib. If nothing works, explain what's missing. +4. **Report at stage boundaries.** Don't make the user ask "what's happening?" +5. **This skill ships the scripts.** All parsing and extraction logic is in + `scripts/`. The agent uses them — it doesn't reimplement them. diff --git a/epub/references/apple-books-compatibility.md b/epub/references/apple-books-compatibility.md new file mode 100644 index 0000000..86a57ca --- /dev/null +++ b/epub/references/apple-books-compatibility.md @@ -0,0 +1,143 @@ +# Apple Books Compatibility + +Apple Books (macOS/iOS) is the most widely used EPUB reading system on Apple +platforms. It enforces several requirements beyond the EPUB 3.3 spec that +cause rendering failures if violated. All rules below were verified by +building, opening, and debugging EPUBs in Apple Books on macOS 26. + +## Required Directory Structure + +**All content must live inside the `OEBPS/` directory.** Apple Books ignores +files at the ZIP root (except `mimetype` and `META-INF/`). + +``` +book.epub +├── mimetype +├── META-INF/ +│ └── container.xml → points to OEBPS/content.opf +└── OEBPS/ + ├── content.opf ← all manifest hrefs are relative to here + ├── nav.xhtml + ├── Styles/default.css + ├── Images/cover.png + └── Text/ + ├── cover.xhtml ← XHTML wrapper for cover image + ├── chapter1.xhtml + └── chapter2.xhtml +``` + +**Wrong** (causes blank pages): +``` +├── Text/ ← ZIP root — Apple Books ignores these +│ └── chapter1.xhtml +``` + +## Cover Image Rules + +1. **The cover must be an XHTML page in the spine, not a raw image reference.** + A `` pointing to `image/png` or `image/jpeg` renders as a blank page. + +2. **Correct pattern:** + - Manifest: `` + - Manifest: `` + - Spine: `` (NOT `cover-img`) + +3. **Cover XHTML must use full-viewport CSS:** + ```css + body { margin: 0; padding: 0; text-align: center; + display: flex; align-items: center; justify-content: center; + min-height: 100vh; } + img { max-width: 100%; max-height: 100vh; width: auto; height: auto; } + ``` + +4. **The `properties="cover-image"` stays on the raw image** — Apple Books uses + it for the library thumbnail. The spine references the wrapper page. + +## CSS Compatibility + +### Deprecated properties (silently ignored by Apple Books) +| Deprecated | Replacement | +|-----------|-------------| +| `page-break-before: always` | `break-before: page` (or remove entirely) | +| `page-break-after: always` | `break-after: page` | +| `page-break-inside: avoid` | `break-inside: avoid` | + +### Margin vs Padding on body +Apple Books applies its own reading margins. Using `margin` on `` stacks +with these, creating excessive whitespace. Use `padding` instead: + +```css +body { + margin: 0; /* ✓ no margin stacking */ + padding: 0 0.5em; /* ✓ internal whitespace only */ +} +``` + +### Namespace selectors +CSS selectors with namespace prefixes (`nav[epub|type="toc"]`) require a +`@namespace` declaration. Some readers fail silently if it's missing. Prefer +generic selectors or keep `@namespace epub` at the top of every CSS file. + +### Avoid `text-indent: 0` on body paragraphs +Apple Books sometimes applies its own `text-indent`, and setting it to zero +can conflict. Omit it unless specifically needed. + +## Navigation (TOC) Visibility + +The `linear` attribute on spine `` controls whether a page appears in +the reading flow: + +| Setting | Behavior | +|---------|----------| +| `linear="yes"` (default) | Page visible in reading flow | +| `linear="no"` | Page hidden from reading flow, accessible via app TOC browser | + +Apple Books still shows hidden pages in its built-in table of contents browser. +`linear="no"` only removes them from the swipe/page-turn reading order. + +## XHTML Requirements + +1. **No `xmlns:epub` on content chapters** — only needed on the navigation + document. Unused namespace declarations can trigger strict XML validation + failures in Apple Books' parser. + +2. **All XHTML must be well-formed XML** — self-closing tags (`
`, ``), + properly escaped ampersands (`&`), no bare `<` in text content. + +3. **CSS `` in every ``** — unstyled chapters render with Apple + Books' defaults, which may differ dramatically from your intended appearance. + Always include the stylesheet link in chapter XHTML. + +## Spine Ordering Conventions + +### Pattern 1: No Cover (simplest) +``` +nav → chapters +``` +No cover in the book. ToC is the first thing the reader sees. + +### Pattern 2: Cover Only (ToC via app browser) +``` +cover-page → nav(linear="no") → chapters +``` +Cover is page 1. ToC accessible via app's built-in browser. + +### Pattern 3: Full (most commercial ebooks) +``` +cover-page → nav → chapters +``` +Cover is page 1, ToC is page 2, chapters follow. This is the `epub-scaffold` +default when `--cover` is provided. + +## Validation Quirks + +EPUBCheck validates against the EPUB spec, not Apple Books' additional +requirements. A valid EPUB can still render blank pages. Always test in +Apple Books before declaring a build complete. + +## References Tested Against + +These rules were verified on: +- macOS 26.5, Apple Books (native) +- `The Spider Blueprint` test EPUB (5 build iterations) +- `Agentic AI in Enterprise` commercial Apress EPUB (2.1MB, passed all checks) diff --git a/epub/references/epub-format-internals.md b/epub/references/epub-format-internals.md new file mode 100644 index 0000000..87b8948 --- /dev/null +++ b/epub/references/epub-format-internals.md @@ -0,0 +1,185 @@ +# EPUB Format Internals + +An EPUB file is a **ZIP archive** (Open Container Format, OCF) containing +structured web content packaged as a single distributable file. This reference +covers EPUB 3.3 (the current W3C standard) and EPUB 2.0 compatibility points. + +## Container Structure (OCF) + +``` +book.epub (ZIP archive) +├── mimetype ← "application/epub+zip" (MUST be first, uncompressed) +├── META-INF/ +│ └── container.xml ← Points to the OPF package document +└── OEBPS/ (or custom root) + ├── content.opf ← Package document (metadata, manifest, spine) + ├── nav.xhtml ← EPUB3 navigation document + ├── toc.ncx ← EPUB2 table of contents (legacy) + ├── Text/ + │ ├── chapter1.xhtml ← XHTML content documents + │ └── chapter2.xhtml + ├── Styles/ + │ └── style.css + └── Images/ + └── cover.jpg +``` + +### Critical ZIP Rules + +- **`mimetype` must be the first file**, stored **uncompressed** (STORE method). + This is a hard requirement — reading systems identify the format from this. +- All other files may be compressed (DEFLATE). +- The `mimetype` file contains exactly: `application/epub+zip` +- No extra bytes, no trailing newline. + +### container.xml + +```xml + + + + + + +``` + +The `full-path` attribute points to the OPF package document. This is the only +required file in `META-INF/`. + +## The Three Planes + +EPUB 3.3 uses a "three planes" model to classify resources: + +| Plane | What it contains | Key rules | +|-------|-----------------|-----------| +| **Manifest plane** | All resources that contribute to rendering | Listed in OPF ``. Exhaustive — every file in the EPUB that's used for rendering must appear here. | +| **Spine plane** | Resources in the default reading order | Defined by OPF ``. Only XHTML and SVG are allowed by default (EPUB content documents). Other formats require manifest fallbacks. | +| **Content plane** | Resources embedded within content documents (images, CSS, scripts, fonts, audio, video) | Core media types are guaranteed supported. Foreign resources require fallbacks. | + +A resource can appear on multiple planes. For example, an XHTML chapter is on +all three — it's in the manifest, in the spine, and can embed resources from +the content plane. + +## Package Document (OPF) + +The OPF file (typically `content.opf`) is the "table of contents for the +container." It tells reading systems what's in the EPUB and how to render it. + +```xml + + + + urn:uuid:123e4567-e89b-12d3-a456-426614174000 + Book Title + en + Author Name + 2026-01-01T00:00:00Z + + + + + + + + + + + + +``` + +### Metadata + +Dublin Core elements are used for standard metadata: +- `dc:title` — title (required) +- `dc:creator` — author (optional but expected) +- `dc:language` — language code (required) +- `dc:identifier` — unique identifier (required) +- `dc:date` — publication date +- `dc:publisher`, `dc:rights`, `dc:description`, `dc:subject` — optional + +Custom metadata uses `` elements with `property` attributes. + +### Manifest + +Every publication resource (file used in rendering) must have an `` in +the manifest. Each item requires: +- `id` — unique identifier within the OPF (used by spine and other references) +- `href` — relative path to the file within the ZIP +- `media-type` — MIME type of the resource + +Optional `properties` attribute: space-separated list. Key values: +- `nav` — this is the EPUB3 navigation document +- `cover-image` — this is the cover image +- `scripted` — contains JavaScript +- `mathml` — contains MathML +- `remote-resources` — references remote resources + +### Spine + +Defines the linear reading order. Each `` references a manifest item +by its `id`. Only EPUB content documents (XHTML, SVG) should appear here by +default. The `linear` attribute can be `"no"` for non-linear content (accessible +but not part of the default reading flow). + +## EPUB 2 vs EPUB 3 + +| Feature | EPUB 2 | EPUB 3 | +|---------|--------|--------| +| Navigation | NCX file (XML-based TOC) | NAV document (XHTML with `