mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-17 14:36:29 +03:00
Magnus HedemarkandGitHub
035e58d3e3
Squash-merge verified routing remediation at exact head 690f9c14b0. Required validate and paired evaluation checks passed; advisory droid review had no blocking findings.
183 lines
9.0 KiB
Markdown
183 lines
9.0 KiB
Markdown
---
|
|
name: gutenberg
|
|
description: >-
|
|
Search, download, and extract public-domain books from Project Gutenberg. Look up books
|
|
by ID or keyword via gutendex, download plain-text and EPUB editions, strip licensing
|
|
boilerplate, extract clean text from EPUB for illustrated works, and classify fiction vs
|
|
non-fiction. Ships a portable CLI script with zero external dependencies. Use when the
|
|
user says "gutenberg", "public domain", "download a book", "classic literature", "free
|
|
ebook", "gutenberg.org", or names any public-domain title or author. Do not use this
|
|
skill for unrelated requests; route to the nearest named specialist.
|
|
license: MIT
|
|
compatibility: Python 3.8+ with zero external dependencies. The CLI uses only the
|
|
Python standard library (urllib.request, json, html.parser, zipfile, re, sys). For
|
|
EPUB extraction, Python 3.8+ with only stdlib is required (zipfile + html.parser).
|
|
The gutendex API (https://gutendex.com) requires no API key or registration. No
|
|
env vars needed for basic operation.
|
|
metadata:
|
|
tags: gutenberg, project-gutenberg, books, public-domain, literature, classics,
|
|
ebooks, text-extraction, epub
|
|
sources: https://gutendex.com, https://www.gutenberg.org
|
|
skills: books, public-domain, literature, text-mining, ebooks
|
|
---
|
|
|
|
# Gutenberg — Public Domain Book Toolkit
|
|
|
|
Search, download, and extract clean text from [Project Gutenberg](https://www.gutenberg.org) — 70,000+ free public-domain ebooks. Ships a portable Python CLI with zero external dependencies.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Search for books
|
|
python3 scripts/gutenberg search "Moby Dick"
|
|
|
|
# Download by Gutenberg ID (plain text)
|
|
python3 scripts/gutenberg download 2701 --format txt
|
|
|
|
# Download EPUB (for illustrated books)
|
|
python3 scripts/gutenberg download 2701 --format epub
|
|
|
|
# Extract clean text (strips PG boilerplate)
|
|
python3 scripts/gutenberg extract 2701
|
|
|
|
# Classify fiction vs non-fiction
|
|
python3 scripts/gutenberg classify 2701
|
|
|
|
# Full pipeline: search → download → extract
|
|
python3 scripts/gutenberg pipeline "Alice's Adventures in Wonderland"
|
|
```
|
|
|
|
## How It Works
|
|
|
|
Project Gutenberg provides **70,000+ free public-domain ebooks** in multiple formats. The gutendex API (https://gutendex.com) offers a free, unauthenticated JSON catalog. No API key required — just curl or this CLI.
|
|
|
|
### Data Flow
|
|
|
|
```
|
|
User provides title/ID/author
|
|
↓
|
|
gutendex API search → pick book by ID
|
|
↓
|
|
Download plain text (preferred) or EPUB (fallback for illustrated books)
|
|
↓
|
|
Strip PG boilerplate → clean text
|
|
↓
|
|
Classify fiction/non-fiction → extract content
|
|
```
|
|
|
|
## CLI Reference
|
|
|
|
### `search` — Find books by keyword
|
|
|
|
```bash
|
|
python3 scripts/gutenberg search "Moby Dick"
|
|
python3 scripts/gutenberg search "Dracula" --limit 5
|
|
python3 scripts/gutenberg search "Sherlock Holmes" --json
|
|
python3 scripts/gutenberg search "Alice" --language en
|
|
```
|
|
|
|
Returns: ID, title, author (with life dates), language, subjects, download count. Results sorted by download count (most popular first).
|
|
|
|
### `metadata` — Get full metadata for a book by ID
|
|
|
|
```bash
|
|
python3 scripts/gutenberg metadata 2701 # Moby Dick
|
|
python3 scripts/gutenberg metadata 11 # Alice's Adventures
|
|
python3 scripts/gutenberg metadata 1342 # Pride and Prejudice
|
|
python3 scripts/gutenberg metadata 1342 --json # JSON-only output
|
|
```
|
|
|
|
Returns: title, author(s), language(s), subjects, bookshelves, summaries, copyright status, download count, and all available format URLs.
|
|
|
|
### `download` — Download a book by Gutenberg ID
|
|
|
|
```bash
|
|
# Plain text (UTF-8, preferred — works for most books)
|
|
python3 scripts/gutenberg download 2701 --format txt
|
|
|
|
# EPUB with images (for illustrated/scientific books)
|
|
python3 scripts/gutenberg download 2701 --format epub
|
|
|
|
# HTML (alternative fallback)
|
|
python3 scripts/gutenberg download 2701 --format html
|
|
|
|
# Specify output directory
|
|
python3 scripts/gutenberg download 2701 --format txt --output ./books/
|
|
```
|
|
|
|
The file is saved to `./gutenberg-<id>.<ext>` (or `--output` path). Large books may take a moment.
|
|
|
|
### `extract` — Strip PG boilerplate and produce clean text
|
|
|
|
```bash
|
|
python3 scripts/gutenberg extract 2701 # from downloaded txt
|
|
python3 scripts/gutenberg extract 2701 --input ./gutenberg-2701.txt
|
|
python3 scripts/gutenberg extract 2701 --format epub # extract from EPUB
|
|
```
|
|
|
|
Output: clean text without the Project Gutenberg license header/footer. For EPUB extraction (illustrated books), extracts text from all XHTML files and merges them into a single cleaned document.
|
|
|
|
Size detection: if a plain-text download is under 50KB for a known substantial book, warns that the text may be truncated and recommends EPUB mode.
|
|
|
|
### `classify` — Classify fiction vs non-fiction
|
|
|
|
```bash
|
|
python3 scripts/gutenberg classify 2701
|
|
python3 scripts/gutenberg classify 2701 --json
|
|
```
|
|
|
|
Uses the book's subjects and bookshelves to classify:
|
|
- **Fiction signals:** "Fiction", "novels", "short stories", "poetry", "drama", "fantasy", "horror"
|
|
- **Non-fiction signals:** "Essays", "History", "Philosophy", "Biography", "Science", "Religion"
|
|
|
|
Returns: `fiction`, `non-fiction`, or `ambiguous` (with explanation of why).
|
|
|
|
### `pipeline` — Full fetch pipeline
|
|
|
|
```bash
|
|
python3 scripts/gutenberg pipeline "Moby Dick" # search first
|
|
python3 scripts/gutenberg pipeline 2701 # by known ID
|
|
python3 scripts/gutenberg pipeline 2701 --clean /tmp/pipeline-output/ # save cleaned text
|
|
```
|
|
|
|
Runs: search (if title) → metadata → download (txt) → check size → extract (or EPUB fallback) → classify. Prints an executive summary at the end.
|
|
|
|
### Global Flags
|
|
|
|
| Flag | Effect |
|
|
|------|--------|
|
|
| `--json` | Output machine-readable JSON instead of human-readable text |
|
|
| `--quiet` | Suppress diagnostic output |
|
|
| `--dry-run` | Show what would be done without executing |
|
|
| `--output ./dir` | Save downloads to a specific directory |
|
|
| `--timeout 30` | Override API timeout (default 15s) |
|
|
|
|
## Fiction vs Non-Fiction Handling
|
|
|
|
When the classified result is **fiction**, the extracted text comes from an authored imagination. Consider splitting analysis into two tracks:
|
|
|
|
| Track | What it covers | Example claims |
|
|
|-------|----------------|----------------|
|
|
| **Canon** | Facts within the fictional world — named entities, quoted lines, story events, world rules | "In Stoker's text, Dracula can assume wolf, bat, and mist forms" |
|
|
| **Craft** | Real-world technique — how the author achieves effect, publication history, literary influence | "Stoker's epistolary form forces the reader to piece together the narrative like an investigator" |
|
|
| **Negative space** | Deliberate omissions — what the author notably leaves unspecified | "Dracula is never granted interior voice in the novel" |
|
|
|
|
When classified as **non-fiction**, claims can be treated as real-world factual assertions about the subject matter.
|
|
|
|
## Known Gotchas
|
|
|
|
- **Plain text truncation for illustrated books** — Books with diagrams, figures, or equations (geometry texts, scientific works, art books) may have plain-text downloads silently cut to 5-10KB (just the PG header). Always check file size. Under 50KB for a known substantial book → switch to EPUB extraction. The `pipeline` command does this check automatically.
|
|
- **Gutendex can be slow or timeout** — The API is a free service and can be slow for less popular books. The CLI uses a 15-second default timeout. Use `--timeout 30` for slow responses, or navigate directly to `https://www.gutenberg.org/ebooks/<id>` as a fallback.
|
|
- **HTML downloads include navigation markup** — HTML downloads contain site navigation and formatting. Prefer plain text or EPUB for clean text extraction.
|
|
- **Rare books may 404 on certain format URLs** — Not every book has every format. The CLI tries UTF-8 plain text first, falls back to US-ASCII, then to the `-0.txt` file path, then to EPUB, then to HTML. The `download` command reports which format was actually retrieved.
|
|
- **Rate limiting** — Gutendex is unauthenticated but rate-limited. Batch requests with `sleep 1` between calls for more than 10 rapid-fire requests.
|
|
- **utf-8 vs us-ascii** — Gutendex returns both a `text/plain; charset=utf-8` and a `text/plain; charset=us-ascii` URL. Prefer UTF-8; fall back to US-ASCII if the UTF-8 URL returns a 404.
|
|
- **Fiction classification ambiguity** — Books with both fiction and non-fiction subjects (e.g. "Historical Fiction" + "History") are marked `ambiguous`. Use `--json` to inspect the subject list and decide manually.
|
|
|
|
## References
|
|
|
|
- [scripts/gutenberg](scripts/gutenberg) — Portable Python CLI. Zero external dependencies (stdlib only). Covers all major Gutenberg workflows: search, download (txt/epub/html), boilerplate stripping, EPUB text extraction, fiction classification, and the full pipeline.
|
|
- [references/epub-extraction.md](references/epub-extraction.md) — EPUB text extraction details for illustrated books, with expanded Python walkthrough and format detection tips.
|
|
- [Project Gutenberg](https://www.gutenberg.org) — 70,000+ free ebooks.
|
|
- [Gutendex API](https://gutendex.com) — JSON web API for the Project Gutenberg catalog.
|