magnus919_agent-skills/gutenberg/references/epub-extraction.md

EPUB Text Extraction for Illustrated Project Gutenberg Books

When a Project Gutenberg plain text download is under 50KB for a known substantial book, the text has likely been silently truncated — the real content lives in the EPUB or HTML editions, which may be 1-2MB. This is common for illustrated, scientific, or mathematical works.

Detection

Signal	Action
Plain text > 50KB	Use plain text — it's fine
Plain text < 50KB for a substantial book	Switch to EPUB extraction
Book has diagrams, figures, equations	Prefer EPUB from the start

Mathematical/scientific works (Euclid's Elements, Newton's Principia, etc.) almost always need EPUB extraction because their figures are image-based and the plain text only captures captions.

Extraction Method

EPUB files are standard ZIP archives containing XHTML. The CLI script (scripts/gutenberg) handles this automatically for the pipeline and extract --format epub commands.

Manual Python Extraction

If you're working outside the CLI, this pattern extracts clean text from an EPUB:

import zipfile
from html.parser import HTMLParser

class TextExtractor(HTMLParser):
    def __init__(self):
        super().__init__()
        self.text = []
        self.skip = False
    def handle_starttag(self, tag, attrs):
        if tag in ('script', 'style', 'svg'):
            self.skip = True
    def handle_endtag(self, tag):
        if tag in ('script', 'style', 'svg'):
            self.skip = False
        if tag in ('p', 'h1', 'h2', 'h3', 'h4', 'div', 'br', 'li'):
            self.text.append('\n')
    def handle_data(self, data):
        if not self.skip:
            self.text.append(data)

all_text = []
with zipfile.ZipFile('/tmp/book.epub', 'r') as z:
    for name in sorted(z.namelist()):
        if not (name.endswith('.html') or name.endswith('.xhtml')):
            continue
        content = z.read(name).decode('utf-8', errors='replace')
        parser = TextExtractor()
        parser.feed(content)
        all_text.append(''.join(parser.text))

full_text = '\n'.join(all_text)

Expected Output

• 484K+ characters for a substantial multi-book work like Euclid's Elements
• Multiple HTML files (one per chapter/book/section) merged into a single text
• Some formatting artifacts from HTML stripping — clean with re.sub(r'\n{3,}', '\n\n', text) if needed

Format Choice Decision Tree

Is the book illustrated, mathematical, or scientific?
├── YES → Prefer EPUB. The plain text may omit diagrams, figures, equations.
│         Use: `gutenberg download <id> --format epub`
│              `gutenberg extract <id> --format epub`
└── NO  → Plain text is fine for novels, essays, poetry, drama.
          Use: `gutenberg download <id> --format txt`
               `gutenberg extract <id>`

The pipeline command handles this automatically: it downloads plain text first, checks size, and falls back to EPUB extraction when the text is suspiciously small.