diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 9edce07..8be6de6 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -192,7 +192,7 @@ "./cli-builder" ], "strict": false, - "description": "Build or refactor CLI tools designed for AI agent consumption: non-interactive, flag-driven, idempotent, with --json output and --dry-run preview. Use when creating a new script the agent will call, adding agent-friendly flags to an existing tool, or debugging why an agent keeps failing to use your CLI." + "description": "Build or refactor agent-facing CLI tools with non-interactive commands, stable --help and --json contracts, idempotent operations, and --dry-run previews. Use for CLI design, agent-friction refactors, output/exit-code debugging, or automation safety. Do not use for GUI/TUI design, conversational tools, MCP server design, or general API architecture; use api-design-and-evolution for the latter." }, { "name": "cncf-landscape", @@ -255,7 +255,7 @@ "./crowdsec" ], "strict": false, - "description": "Deploy, configure, and manage CrowdSec — the open-source, collaborative IPS/IDPS/WAF. Covers Security Engine setup (Linux, Docker), cscli hub management, remediation components, AppSec WAF, profiles, notifications, blocklists, CTI, and metrics. Use when setting up or troubleshooting CrowdSec." + "description": "Deploy, configure, and operate CrowdSec Security Engine, cscli, remediation components, acquisition pipelines, and AppSec WAF. Use for Linux or Docker installation, detection-to-blocking design, incident review, and safe changes. Do not use for generic firewall, Kubernetes, or reverse-proxy design; route those to the named platform skill and use this skill for CrowdSec integration." }, { "name": "cyberpunk", diff --git a/cli-builder/README.md b/cli-builder/README.md index 85078a3..2585882 100644 --- a/cli-builder/README.md +++ b/cli-builder/README.md @@ -16,9 +16,9 @@ When your agent loads this skill, it can **design, build, and refactor CLI tools | Directory | Purpose | |-----------|---------| -| `SKILL.md` | 500+ line reference: 10 design patterns, 3-phase build workflow, agent-compatibility test suite | -| `templates/` | Python API client scaffold and bash CLI scaffold | -| `references/` | Agent-compatibility test suite, Python API client pattern | +| `SKILL.md` | Concise workflow for discovery, predictable contracts, verification, and maintenance | +| `templates/` | Bash CLI scaffold for local wrappers | +| `references/` | Python API clients, advanced patterns, readiness checklist, wrapper example, MCP decisions, and improvement cycle | ## Triggers diff --git a/cli-builder/SKILL.md b/cli-builder/SKILL.md index c112bab..6efcc15 100644 --- a/cli-builder/SKILL.md +++ b/cli-builder/SKILL.md @@ -1,9 +1,12 @@ --- name: cli-builder -description: 'Build or refactor CLI tools designed for AI agent consumption: non-interactive, - flag-driven, idempotent, with --json output and --dry-run preview. Use when creating - a new script the agent will call, adding agent-friendly flags to an existing tool, - or debugging why an agent keeps failing to use your CLI.' +description: >- + Build or refactor agent-facing CLI tools with non-interactive commands, stable + --help and --json contracts, idempotent operations, and --dry-run previews. + Use for CLI design, agent-friction refactors, output/exit-code debugging, or + automation safety. Do not use for GUI/TUI design, conversational tools, MCP + server design, or general API architecture; use api-design-and-evolution for + the latter. license: MIT compatibility: Requires bash, Python 3.8+, jq, and standard Unix CLI environment. metadata: @@ -12,488 +15,89 @@ metadata: https://github.com/ComposioHQ/awesome-agent-clis, https://ronnierocha.dev/blog/dont-build-mcps-build-cli-tools --- -# CLI Builder — Agent-Friendly Tool Design -## Overview -A CLI tool is a **contract** between your code and the agent that calls it. Every design decision is part of that contract: +# CLI Builder -| CLI Element | Contract Purpose | -|---|---| -| `--help` output | Schema — what the tool offers, what flags it accepts | -| Subcommand structure | API surface — the operations the agent can perform | -| `--json` output fields | Data contract — guaranteed keys and their types | -| Exit codes | Status signals — success, usage error, runtime failure | -| Stderr messages | Error contract — what went wrong and how to fix it | -| `--dry-run` output | Preview contract — what would happen | - -An agent discovers this contract by calling `--help`. The tool needs to be **predictable, structured, and complete** — no interactive surprises, no missing examples, no silent failures. +Treat a CLI as a contract between the tool and the agent. The contract includes +command names, `--help`, flags, output schemas, exit codes, stderr, and previews. +Keep this file as the workflow; load the linked references only when their +specialized guidance is needed. ## When to Use -- Building a new script the agent will call -- Refactoring an existing tool that causes agent friction (interactive prompts, unclear errors, non-idempotent operations) -- Adding `--json`, `--dry-run`, or `--yes` flags to an existing script -- Designing a CLI subcommand for an agent framework +- Build a new CLI that an agent will discover and call. +- Refactor prompts, ambiguous commands, parser-hostile output, or false success codes. +- Add or review `--json`, `--dry-run`, `--yes`, idempotency, or lazy authentication. -**Don't use for:** One-off terminal commands the human runs interactively. The principles here optimize for machine consumption, which can make human-facing CLIs feel overly verbose. +## One Workflow -## Build Workflow +### 1. Discover the real contract -A CLI tool is built in three phases: +Define one CLI per service (except services sharing vendor authentication). Inspect +real non-health read endpoints before coding, verify the actual authentication +header and response shape, and capture the command tree and failure cases. Do not +invent flags from API documentation alone. For HTTP clients, read +[the Python API client pattern](references/python-api-client.md). -``` -Phase 1: Discover → Phase 2: Build → Phase 3: Verify -curl the live server → Implement → Run test suite -Confirm auth → Wire client auth → Test against live -Inventory endpoints → Write help text → Verify dry-run paths -Capture data shapes → Run tests as-you-go → Fix failures -``` +Choose Bash for local wrappers and filesystem pipelines. Choose Python for HTTP, +JSON, authentication state, or three-level command trees. -## Phase 1: Plan — Before Writing Code +### 2. Build a predictable surface -### Architecture: One CLI Per Service +Use one consistent verb/resource convention. Every command should be non-interactive +and flag-driven, reject unknown flags, and provide useful subcommand help with +concrete examples. Make normal output human-readable, but support `--json` with a +stable curated schema, normalized types, deterministic ordering, and no other stdout. +Send diagnostics to stderr with truthful non-zero exit codes. -Each API or data source gets its own CLI. Do not combine disparate services into one tool. +For any state change, implement `--dry-run` before data-fetching or mutation, and +require an explicit `--yes`/`--force` gate for destructive work. Guard creation, +updates, and deletion so reruns are safe. Authentication must be lazy: `--help` and +safe dry-runs work without credentials. -**Correct:** `tmdb` (TMDb only), `ghost` (Ghost CMS only) -**Wrong:** `media-cli` (combines TMDb + Trakt + Radarr) +Use the [advanced patterns](references/advanced-patterns.md) for chained dry-runs, +third-party JSON, version-dependent behavior, and text matching. Use the +[Bash scaffold](templates/bash-cli-scaffold.sh) when a local shell wrapper is the +right fit. -Exception: services from the same vendor sharing auth (e.g. Radarr + Sonarr). +### 3. Verify the contract -### Live-Server Discovery +Run syntax checks, every command's `--help`, parse every `--json` response, check +missing arguments and unknown flags, confirm errors are on stderr, exercise dry-run +without credentials, and prove a second identical run is a no-op. Against a live +service, test a real authenticated read and dry-run each mutation. Use the +[agent-readiness checklist](references/agent-readiness-checklist.md) for the final +review. -Before writing any code, verify against the actual server: +### 4. Wrap and maintain -```bash -# 1. Test auth against a NON-WHITELISTED endpoint -curl -s -w "\nHTTP: %{http_code}" \ - -H "X-API-Key: $KEY" \ - https://server.example.com/api/items +Keep the entry-point skill concise and put conditional detail in references. A +wrapper should explain what the CLI is for, setup, common commands, output meaning, +and gotchas, not duplicate every flag. See the +[skill-wrapper example](references/skill-wrapper-example.md). After real usage, +record failures and prioritize fixes with the [improvement cycle](references/improvement-cycle.md). -# 2. Try alternate auth mechanisms if that 401s -curl -s -w "\nHTTP: %{http_code}" \ - -H "Authorization: Bearer $TOKEN" \ - https://server.example.com/api/items +## Core Contracts -# 3. Capture response shapes for a read endpoint -curl -s -H "X-API-Key: $KEY" \ - https://server.example.com/api/items?limit=1 | head -c 2000 -``` +- `--help` is the discoverable schema and includes examples. +- `--json` is parseable on stdout alone; errors have stable machine-readable codes. +- `--dry-run` describes exact intended changes and performs no writes or prerequisite lookups. +- Mutations are explicitly authorized, idempotent, and report meaningful status. +- Exit status distinguishes success, usage failure, and runtime failure. -**Why this matters:** The health endpoint is often whitelisted and won't catch a wrong auth header. Test against a real data endpoint. Field names in the live response are the only truth — docs are often for a different version. +## When Not to Use -### Bash vs Python Decision - -| Concern | Bash | Python | -|---------|------|--------| -| HTTP requests | Pipe to curl, parse with jq | `requests` library, proper error handling | -| JSON handling | jq, fragile escaping | Native `json` module | -| Auth tokens | Write/read files | Class with `_token` state | -| Multipart uploads | curl -F, painful | requests `files=` param | -| Subcommands | Case statements | argparse subparsers | -| Testing | bats, shunit2 | pytest | - -**Use Python** when: the tool sends HTTP requests, manages auth state, parses JSON responses, or has 3+ subcommand levels. - -**Use Bash** when: the tool wraps local binaries, does filesystem operations, or pipes commands together. Bash must have `set -euo pipefail` at the top. - -## Phase 2: Build — Design Patterns - -### Pattern 1: Non-Interactive by Default - -No prompts mid-execution. Everything passable as a flag or environment variable. - -```bash -# BAD — agent blocks forever -read -p "Are you sure? (y/n) " confirm - -# GOOD — flag-driven -FORCE=${FORCE:-false} -if [[ "$1" == "--force" || "$FORCE" == "true" ]]; then - : # proceed -fi -``` - -### Pattern 2: Progressive Help Discovery - -Every subcommand's `--help` includes concrete examples. Agents pattern-match off examples faster than prose. - -```bash -usage() { - case "${1:-}" in - create) - echo "Usage: $0 create --name [OPTIONS]" - echo "" - echo "Examples:" - echo " $0 create --name my-resource" - echo " $0 create --name my-resource --dry-run" - echo " $0 create --name my-resource --force" - ;; - *) - echo "Commands:" - echo " list List resources" - echo " create Create a resource" - echo "Run '$0 --help' for command-specific options." - ;; - esac -} -``` - -### Pattern 3: `--json` for Machine-Readable Output - -Support `--json` flag. Humans get text; agents get parseable data. - -```bash -if [[ "$JSON_OUTPUT" == "true" ]]; then - cat <&2; } -die() { echo "Error: $*" >&2; exit 1; } -info() { [[ "$VERBOSE" == "true" ]] && echo "[info] $*" >&2 || true; } -``` - -**Critical:** `log()` must be suppressed in `--json` mode. A stray "Processing..." line before the JSON payload breaks all consumers. `emit()` handles this correctly; the danger is auxiliary `log()`/`print()` calls that don't go through `emit()`. - -### Pattern 8: `--force` / `--yes` to Skip Confirmations - -Safe default, bypassable for automation. - -```bash -for arg in "$@"; do - case "$arg" in - --force|--yes|-y) FORCE=true ;; - --dry-run|-n) DRY_RUN=true ;; - --json) JSON_OUTPUT=true ;; - esac -done -``` - -### Pattern 9: Consistent Subcommand Structure - -Pick `resource verb` or `verb resource` and stick to it everywhere. - -``` -tool service list ✓ -tool service create ✓ -tool service delete ✓ -tool config list ✓ (agent can guess this pattern) - -tool list services ✗ (verb resource — inconsistent) -tool create-service ✗ (hyphenated verb-resource) -``` - -### Pattern 10: Lazy Auth — Help Works Without Credentials - -Credentials checked at request time, not client creation time. `--help` and `--dry-run` work without any key. - -```python -class MyClient: - def __init__(self, api_key=""): - self.api_key = api_key # Accept empty key — don't check yet - - def _request(self, method, path, ...): - if not self.api_key and not DRY_RUN: - die("API key not found. Set MYTOOL_API_KEY in your environment.") - if DRY_RUN: - return {"dry_run": True} # Safe empty response - # Real HTTP call follows -``` - -This way `tool cmd --help` and `tool cmd --dry-run` never need credentials. - -## Phase 3: QA — Agent Compatibility Testing - -### Essential Test Suite - -```bash -# 1. Syntax check -python3 -c "import py_compile; py_compile.compile('./tool.py', doraise=True)" - -# 2. --help on every subcommand has examples -./tool.sh create --help | grep -qi "example" && echo "PASS" - -# 3. --json output is valid parseable JSON -./tool.sh list --json 2>/dev/null | jq . >/dev/null && echo "PASS" - -# 4. Missing required args → immediate error with corrective usage -result=$(./tool.sh create 2>&1 || true) -echo "$result" | grep -qi "\-\-name" && echo "PASS" - -# 5. --dry-run returns meaningful preview -result=$(./tool.sh delete --name x --dry-run 2>&1 || true) -echo "$result" | grep -qi "dry-run\|would" && echo "PASS" - -# 6. Errors to stderr, data to stdout -result=$(./tool.sh create 2>&1 1>/dev/null || true) -echo "$result" | grep -qi "Error" && echo "PASS: errors to stderr" - -# 7. Idempotent — second call succeeds -result=$(./tool.sh create --name test 2>&1 || true) -result=$(./tool.sh create --name test 2>&1 || true) -echo "$result" | grep -qi "no-op\|already\|exists" && echo "PASS" - -# 8. --dry-run on chained commands doesn't crash -result=$(./tool.sh list --dry-run 2>&1 || true) -echo "$result" | grep -qi "dry-run\|would\|preview" && echo "PASS" -``` - -### Live-Server Verification - -The syntax tests above catch coding errors. They don't catch API mismatches. Run every read command against a real server: - -```bash -# Auth verification (non-whitelisted endpoint) -curl -s -H "X-API-Key: $KEY" https://api.example.com/items?limit=1 - -# Read command smoke test -./tool.sh list --json > /dev/null && echo "PASS" - -# Dry-run every mutating command to verify payload structure -./tool.sh create --name test --dry-run --json 2>/dev/null -``` - -Common bugs only found this way: -- **Wrong auth header** — health endpoint was whitelisted, you never tested a real endpoint -- **Wrong field names** — API returns `items[].id` but you wrote `entity_name` -- **Wrong content-type** — login needs form data, not JSON -- **Wrong nesting** — Swagger shows `daily` at top level, real API nests it under `forecast` - -## Gotchas — The Eight Most Common Agent-CLI Bugs - -These are the failures observed across every agent-built CLI: - -1. **Errors on stdout** — `echo "Error"` (no `>&2`) breaks pipeline consumers. Always use `die()` which writes to stderr. - -2. **No examples in `--help`** — Agents can't guess argument order from a field description. Every subcommand needs at least two concrete examples. - -3. **`--json` output has auxiliary text** — A "Processing..." line before the JSON payload makes `json.load()` fail. Gate ALL output through `emit()`. - -4. **No `--dry-run` for chained commands** — Handler fetches data first (e.g. station ID lookup), dry-run crashes before reaching the preview. Short-circuit BEFORE data-fetching logic. - -5. **Auth header format guessed wrong** — Some APIs use `X-API-Key`, others use `Authorization: Bearer`, some use both for different auth mechanisms. Always curl a non-whitelisted endpoint first. - -6. **Content-type mismatch on login** — Login/oauth endpoints usually use `application/x-www-form-urlencoded`, not JSON. Build `_form_post()` separately. - -7. **Idempotency not checked** — `create` called twice creates duplicate state. Always guard creation/deletion with an existence check. - -8. **Hyphenated positional arguments** — Python argparse converts `--flag-name` to `args.flag_name` for flags, but `parser.add_argument("resource-id")` stays as `getattr(args, "resource-id")`, not `args.resource_id`. - -## Phase 4: Skillify — Wrap Your CLI for Agent Discovery - -A CLI tool that an agent doesn't know exists is useless. The final phase creates a [compliant Agent Skill](https://agentskills.io) wrapper — a `SKILL.md` that acts as the trigger surface, letting the agent discover and reach for your CLI at the right moment. - -### The Two-Layer Architecture - -Your CLI lives in the skill's `scripts/` directory, alongside SKILL.md: - -``` -servicex-cli/ -├── scripts/ -│ └── servicex-cli # Your CLI binary (Phases 1-3) -├── SKILL.md # The skill wrapper (Phase 4) -└── references/ # Supporting documentation -``` - -The two layers serve distinct roles: - -| Layer | File | Purpose | -|-------|------|---------| -| **Trigger** | `SKILL.md` | Tells the agent when to use this tool, what data to pass, what the output means, known gotchas | -| **Execute** | `scripts/servicex-cli` | Provides `--help` as schema, `--json` as data contract, `--dry-run` as preview, `--force` as automation bypass | - -The skill **triggers** the tool. The tool **executes** the contract. Neither is complete without the other. - -### Frontmatter Conventions - -The `description` field is your skill's only trigger mechanism. Craft it to match the agent's vocabulary: - -```yaml ---- -name: tool-name # matches the CLI binary name -description: >- - Interact with ServiceX: search, create, and manage resources via - the ServiceX API. Use when the user mentions ServiceX, their service - status, or asks to look up records, create resources, or check - service health. -license: MIT -compatibility: Requires CLI on PATH, API key in - SERVICEX_API_KEY env var (or ~/.servicex.env) -metadata: - tags: [servicex, api-client, automation] ---- -``` - -**Rules:** -- `name` matches the CLI binary name — the agent may need to call it -- `description` lists concrete trigger keywords the user might say -- `compatibility` documents what the agent needs to have set up -- `metadata.tags` adds secondary retrieval surface - -### Body Structure - -The skill body does NOT duplicate the CLI's `--help`. Instead, it teaches the agent *what to use the tool FOR* and *how to interpret the results*: - -```markdown -# ToolName CLI - -## When to Use - -- User asks "what's the status of X" or "check on Y" -- User asks to create, update, or delete resources -- User asks about unusual behavior from the service - -## Setup - -Credentials are read from the `SERVICEX_API_KEY` env var or -`~/.servicex.env`. If the agent gets a 401, guide the user to -set up credentials before retrying. - -## Essential Commands - -### list — List resources - -```bash -tool-name list # human-readable table -tool-name list --json | jq '.[].id' # machine-readable -``` - -### create — Create a resource - -```bash -tool-name create --name "My Resource" --type standard -tool-name create --name "My Resource" --type standard --dry-run -``` - -### get — Get details by id - -```bash -tool-name get --id abc123 --json -``` - -## Known Gotchas - -- Rate limit: 100 req/min. On 429, back off and retry. -- The `status` field uses the API's raw labels (`provisioning`, `active`, `error`). -- Names are case-sensitive. `My Resource` ≠ `my resource`. -``` - -### What NOT to Put in the Skill Body - -| Don't | Why | -|-------|-----| -| Full flag reference | That's what `--help` is for. Reference it, don't duplicate it. | -| Installation instructions | For distribution via this repo, the CLI lives in `scripts/` within the skill directory — the skill documents invocation patterns, not setup. For global installs (PATH), deployment is separate. | -| API architecture details | The skill teaches *usage*, not *architecture*. Gotchas are the exception. | -| Every possible subcommand | Cover the 3-5 most common. Agents discover the rest via `--help`. | - -### The Completed Architecture - -The two-layer pattern follows the [Agent Skills open format](https://agentskills.io) directory structure: - -``` -servicex-cli/ -├── scripts/ -│ └── servicex-cli # The CLI binary (built with Phases 1-3) -├── SKILL.md # The skill wrapper (built in Phase 4) -└── references/ # Supporting documentation (optional) - -Agent opens session: - ├── Loads all SKILL.md descriptions at startup - ├── User says "check my servicex resources" - ├── skill-triggered: "servicex" in user message matches description - │ └── Agent loads skill body - │ ├── Reads "use scripts/servicex-cli list --json" - │ ├── Runs scripts/servicex-cli list --json - │ └── Reads output, tells user - │ - Deeper questions → agent reads CLI --help for specifics -``` - -#### When to use `scripts/` vs global PATH - -| Approach | Best for | Cmd invocation | -|----------|----------|----------------| -| **`scripts/` inside skill** | Distribution via this repo — self-contained, portable, format-compliant. The agent references the script by relative path from the skill root. | `scripts/servicex-cli list --json` | -| **Global PATH** | When the CLI is useful beyond this skill (other agents, human users, scripts). Install to `~/.hermes/scripts/` (Hermes) or a system PATH directory. | `servicex-cli list --json` | - -The default for this repo is **`scripts/` inside the skill** — it follows the Agent Skills specification for progressive disclosure and keeps the skill self-contained. Add a note in the skill body when the CLI is also available on global PATH for broader use. - -### Skill Wrapper Template - -## Agent-Readiness Checklist - -Use [the agent-readiness checklist](references/agent-readiness-checklist.md) before shipping a CLI. - -## When not to use - -Do not use this skill to design conversational agent tools or MCP servers — [references/mcp-vs-cli.md](references/mcp-vs-cli.md) carries that decision framework — and route general API design questions to [api-design-and-evolution](../api-design-and-evolution/SKILL.md). This skill also does not cover GUI, TUI, or web-app interface design. +Do not use this skill for one-off interactive human commands, GUI/TUI or web UI +design, conversational agent tools, or MCP servers. Read +[the MCP-vs-CLI decision guide](references/mcp-vs-cli.md) for tool-boundary choices, +and route general API contract design to +[api-design-and-evolution](../api-design-and-evolution/SKILL.md). ## References -- [templates/bash-cli-scaffold.sh](templates/bash-cli-scaffold.sh) — Full bash project template with pre-wired global flags, logging helpers, and subcommand dispatch. Use as a starting point for any bash CLI. -- [references/python-api-client.md](references/python-api-client.md) — Complete Python API client pattern with lazy auth, centralized error handling, form-login support, and argparse dispatch with pre-parsed global flags. Read when building a Python CLI that wraps an HTTP API. -- [references/advanced-patterns.md](references/advanced-patterns.md) — Edge case patterns: morphological text matching, version-dependent imports, robust JSON consumption from third-party tools, dry-run short-circuit for chained APIs. Read when a specific edge case from the gotchas section bites you. -- [references/skill-wrapper-example.md](references/skill-wrapper-example.md) — Complete worked example of a skill wrapper around a hypothetical `weather-cli`, including frontmatter, essential commands, gotchas, and auth wiring. Read in Phase 4 as a template for wrapping your own CLI. -- [references/mcp-vs-cli.md](references/mcp-vs-cli.md) — Summary of the MCP-vs-CLI discourse with a decision framework. Read when debating whether to build a CLI or an MCP server for a new integration. -- [references/improvement-cycle.md](references/improvement-cycle.md) — Structured feedback schema and HALO-style prioritization for improving CLIs over time. Read after shipping your first version and collecting usage traces. +- [Python API client](references/python-api-client.md) +- [Advanced patterns](references/advanced-patterns.md) +- [Agent-readiness checklist](references/agent-readiness-checklist.md) +- [Skill wrapper example](references/skill-wrapper-example.md) +- [MCP vs CLI](references/mcp-vs-cli.md) +- [Improvement cycle](references/improvement-cycle.md) +- [Bash scaffold](templates/bash-cli-scaffold.sh) diff --git a/crowdsec/README.md b/crowdsec/README.md index 3779ac0..a0694b2 100644 --- a/crowdsec/README.md +++ b/crowdsec/README.md @@ -16,8 +16,9 @@ When your agent loads this skill, it becomes a **CrowdSec security engineer** wh | Directory | Purpose | |-----------|---------| -| `SKILL.md` | Architecture overview, installation guide, quick reference | -| `references/` | 7 reference files: config deep dive, AppSec WAF, Docker deployment, Traefik integration, database backends, hub collections, troubleshooting | +| `SKILL.md` | Safety-gated workflow for deployment, detection-to-blocking, cscli operations, AppSec, and troubleshooting | +| `references/` | Focused guides for configuration, cscli, AppSec WAF, Docker, bouncers, operations, databases, hardening, collections, and troubleshooting | +| `evals/` | Five representative deployment, triage, WAF, mutation-safety, and no-data cases | ## Quick Start diff --git a/crowdsec/SKILL.md b/crowdsec/SKILL.md index 4df4d27..aa9fb30 100644 --- a/crowdsec/SKILL.md +++ b/crowdsec/SKILL.md @@ -1,498 +1,141 @@ --- name: crowdsec -description: Deploy, configure, and manage CrowdSec — the open-source, collaborative - IPS/IDPS/WAF. Covers Security Engine setup (Linux, Docker), cscli hub management, - remediation components, AppSec WAF, profiles, notifications, blocklists, CTI, and - metrics. Use when setting up or troubleshooting CrowdSec. +description: >- + Deploy, configure, and operate CrowdSec Security Engine, cscli, remediation + components, acquisition pipelines, and AppSec WAF. Use for Linux or Docker + installation, detection-to-blocking design, incident review, and safe changes. + Do not use for generic firewall, Kubernetes, or reverse-proxy design; route + those to the named platform skill and use this skill for CrowdSec integration. license: MIT -compatibility: Any agent supporting Agent Skills format — commands use standard shell - and CLI tools +compatibility: Requires CrowdSec/cscli for live operations; Docker is optional for container deployment. metadata: source: https://docs.crowdsec.net - version: 0.0.2 + version: 0.0.3 --- -# CrowdSec Skill -CrowdSec is an open-source, collaborative security engine that detects and blocks malicious actors. It analyzes logs and HTTP requests using behavior-based patterns (scenarios) and enforces blocks through remediation components (bouncers). -## Architecture Overview -CrowdSec has a modular, API-centric architecture. The main components: -| Component | Role | -|-----------|------| -| **Security Engine** (crowdsec) | Reads logs, parses them, evaluates scenarios, and produces alerts/decisions. Runs the Log Processor and Local API (LAPI). | -| **Local API (LAPI)** | HTTP API that stores decisions, serves remediation components, and communicates with the Central API. Runs inside the Security Engine. | -| **Central API (CAPI)** | CrowdSec's cloud service — receives signals from all instances and distributes community blocklists. | -| **Remediation Components** (formerly "bouncers") | Connect to LAPI to fetch decisions and enforce blocks at various levels (firewall, reverse proxy, web server). | -| **AppSec Component** | WAF subsystem that inspects HTTP requests in real-time. Lives in the Security Engine. | -| **cscli** | Command-line tool to manage the entire CrowdSec stack. | -**Data flow:** Logs → Parsers (s00-raw, s01-parse, s02-enrich) → Scenarios → Alerts → LAPI → Decisions → Remediation Components → Block -> **Important:** The Security Engine alone only *detects* — it does NOT block. You must add at least one remediation component to enforce decisions. +# CrowdSec -## Quick Reference +CrowdSec detects hostile behavior from logs and HTTP requests, then exposes +alerts and decisions through LAPI. The engine alone does not block traffic: +install and verify at least one remediation component (bouncer) before claiming +protection. -| Task | Command | -|------|---------| -| Install engine | `curl -s https://install.crowdsec.net \| sudo sh` then `sudo apt install crowdsec` | -| Install firewall bouncer | `sudo apt install crowdsec-firewall-bouncer-iptables` (or `-nftables`) | -| Add bouncer API key | `sudo cscli bouncers add ` | -| List bouncers | `sudo cscli bouncers list` | -| Install collection | `sudo cscli collections install crowdsecurity/nginx` | -| List collections | `sudo cscli collections list` | -| View metrics | `sudo cscli metrics` | -| List alerts | `sudo cscli alerts list` | -| List decisions | `sudo cscli decisions list` | -| Manually ban IP | `sudo cscli decisions add --ip ` | -| Manually unban IP | `sudo cscli decisions delete --ip ` (or `remove --ip`, which is an alias) | -| View status | `sudo systemctl status crowdsec` | -| Reload config | `sudo systemctl reload crowdsec` | +## Safety Gate -## Installation +Before any mutation, confirm the target host/container, scope, backup or rollback, +and maintenance window. Prefer read-only inspection and simulation first. Never +manually delete decisions, collections, or data without recording the reason and +an undo path. Save bouncer keys when created; they are shown once. Use +`simulation: true` while tuning scenarios so detections are observed without +enforcement, then verify allowlists before live blocking. -### Linux (Debian/Ubuntu) +## Choose a Deployment + +For Debian/Ubuntu, add the CrowdSec repository, install `crowdsec`, then install +a remediation package such as `crowdsec-firewall-bouncer-iptables` or +`-nftables`. For Docker Compose, expose LAPI (`127.0.0.1:8080`), metrics +(`127.0.0.1:6060`), and AppSec (`127.0.0.1:7422`) only to required networks, +mount `/etc/crowdsec`, `/var/lib/crowdsec/data`, and logs read-only, and pin a +reviewed image version. Persist the data directory, mandatory for v1.7.0+. +Load [the Docker deployment guide](references/docker-deployment.md) for a full +compose example and remote-agent caveats. + +After installation, verify `systemctl status crowdsec` (or container health), +then `cscli version`, `cscli collections list`, acquisition metrics, and +bouncer connectivity. Do not expose LAPI or AppSec publicly without an explicit +network and authentication design. + +## The Detection-to-Blocking Workflow + +1. Select collections for the actual log format, for example + `crowdsecurity/linux`, `sshd`, `nginx`, `traefik`, or `base-http-scenarios`. +2. Configure acquisition in `/etc/crowdsec/acquis.yaml` or `acquis.d/`; every + source needs `labels.type` so the correct parser runs. Use + `poll_without_inotify: true` for unreliable NFS/SMB or bind mounts and + `use_time_machine: true` for buffered logs. +3. Check parser/scenario hits and unparsed lines with `cscli metrics -o json`. +4. Use profiles to map alerts to decisions. Keep `profiles.yaml.local` and + remember YAML sequences replace rather than merge. +5. Add a bouncer with `cscli bouncers add NAME`, store its one-time key securely, + and verify `cscli bouncers list` plus a harmless test decision. +6. Confirm the reverse proxy/firewall is actually enforcing decisions; an alert + or LAPI decision alone is not proof of a blocked request. + +For complete configuration directives, database choices, and hardening, read +[config-reference](references/config-reference.md), +[database-config](references/database-config.md), and +[production-hardening](references/production-hardening.md). + +## cscli Essentials + +Use `cscli -o json` for automation and capture command output, version, host, +and time as evidence. Read-only triage commonly uses: ```bash -# Add repository -curl -s https://install.crowdsec.net | sudo sh -sudo apt update -sudo apt install crowdsec - -# Optionally install firewall bouncer -sudo apt install crowdsec-firewall-bouncer-iptables +cscli hub update +cscli collections list +cscli alerts list --contain "scenario:ssh-bf" +cscli decisions list -o json +cscli metrics -o json +cscli explain --file /path/to/sample.log ``` -During installation, CrowdSec auto-detects running services (SSH, nginx, etc.) and installs appropriate collections + acquisition config. +Manage hub items with `collections|parsers|scenarios install/list/upgrade/inspect`. +Manage alerts and decisions with `alerts list/inspect` and +`decisions add/list/delete`; mutation commands require the safety gate above. +Manage bouncers and machines with `bouncers add/list/delete` and +`machines add/list/delete`. Use `console status`, `console enroll`, and +`lapi register` only after confirming the destination and credentials. Load the +[full cscli reference](references/cscli-command-reference.md) for flags, +output modes, and less common commands. -### Docker / Docker Compose +## Acquisition and AppSec WAF + +A minimal file acquisition entry is: ```yaml -services: - crowdsec: - image: crowdsecurity/crowdsec:latest - restart: always - ports: - - 127.0.0.1:8080:8080 # LAPI - - 127.0.0.1:6060:6060 # Prometheus metrics - - 127.0.0.1:7422:7422 # AppSec WAF - environment: - COLLECTIONS: "crowdsecurity/linux crowdsecurity/nginx" - GID: "${GID-1000}" - TZ: "UTC" - volumes: - - ./crowdsec/config:/etc/crowdsec - - ./crowdsec/data:/var/lib/crowdsec/data - - /var/log:/var/log:ro +filenames: [/var/log/nginx/*.log] +labels: {type: nginx} ``` -> **Version note:** Persisting `/var/lib/crowdsec/data` is **mandatory since v1.7.0**. On older versions (v1.6.x and earlier), the container uses this directory for the SQLite database but does not require it. However, persisting it is always recommended to avoid data loss on container restart. Use a named volume or bind mount; tmpfs is only suitable for throwaway/non-production deployments. - -**Key environment variables:** - -| Variable | Default | Description | -|----------|---------|-------------| -| `COLLECTIONS` | *(none)* | Space-separated list of collections to install | -| `DISABLE_LOCAL_API` | `false` | Set `true` to run as log processor only | -| `DISABLE_AGENT` | `false` | Set `true` to run as LAPI only | -| `BOUNCER_KEY_` | *(none)* | Seed API key for a bouncer | -| `TZ` | `UTC` | Timezone | -| `CONFIG_FILE` | `/etc/crowdsec/config.yaml` | Path to main config | - -## Configuration - -### Main config file (`/etc/crowdsec/config.yaml`) - -Key sections: `common`, `config_paths`, `crowdsec_service`, `db_config`, `api`, `prometheus`. - -Use `config.yaml.local` for local overrides — values here take precedence over `config.yaml` and survive package upgrades. Supports environment variable substitution (`${VAR}`). - -### Acquisition (`/etc/crowdsec/acquis.yaml` or `/etc/crowdsec/acquis.d/*.yaml`) - -Tells CrowdSec which log files to read: - -```yaml -filenames: - - /var/log/nginx/*.log -labels: - type: nginx ---- -filenames: - - /var/log/auth.log - - /var/log/syslog -labels: - type: syslog ---- -source: docker -container_name_regexp: - - .*caddy* -labels: - type: caddy -``` - -The `labels.type` field is **mandatory** — it determines which parsers handle the logs. - -> **Note:** For log files on network shares (NFS, SMB) or Docker bind mounts where inotify doesn't work reliably, add `poll_without_inotify: true` to the acquisition entry. This polls the file at intervals instead of relying on filesystem events. - -### Profiles (`/etc/crowdsec/profiles.yaml`) - -Controls what remediation action is taken when a scenario triggers: - -```yaml -name: default_ip_remediation -filters: - - Alert.Remediation == true && Alert.GetScope() == "Ip" -decisions: - - type: ban - duration: 4h -on_success: break -``` - -Override values via `profiles.yaml.local`. Files are read sequentially (not merged). - -### Simulation mode (`/etc/crowdsec/simulation.yaml`) - -When enabled, CrowdSec still detects but does not enforce: - -```yaml -simulation: true -exclusions: - - crowdsecurity/ssh-bf -``` - -## cscli Command Reference - -`cscli` [global flags] `` [subcommand] [options] - -**Global flags:** `-c ` (config path), `-o json|human|raw` (output format), `--debug`, `--color` - -| Category | Key Commands | Load detail | -|----------|-------------|-------------| -| Hub Management | `cscli hub update`, `cscli collections install/list/upgrade/inspect`, `cscli parsers install/list/upgrade`, `cscli scenarios install/list/upgrade` | `references/cscli-command-reference.md` | -| Decisions & Alerts | `cscli decisions add/list/delete`, `cscli alerts list/inspect` | `references/cscli-command-reference.md` | -| Bouncers & Agents | `cscli bouncers add/list/delete`, `cscli machines add/list/delete` | `references/cscli-command-reference.md` | -| Metrics | `cscli metrics`, `cscli metrics show appsec\|bouncers` | `references/cscli-command-reference.md` | -| Console & LAPI | `cscli console status/enroll`, `cscli lapi register` | `references/cscli-command-reference.md` | -| Additional | `cscli version`, `cscli config`, `cscli explain`, `cscli simulation`, `cscli allowlists` | `references/cscli-command-reference.md` | - -See the full command reference at `references/cscli-command-reference.md`. - -## Hub Collections - -Collections bundle parsers + scenarios for a service. **This is the primary way to add protection:** - -| Collection | Protects | -|------------|----------| -| `crowdsecurity/linux` | Linux syslog, SSH, sudo | -| `crowdsecurity/sshd` | SSH brute force detection | -| `crowdsecurity/nginx` | Nginx web server | -| `crowdsecurity/traefik` | Traefik reverse proxy | -| `crowdsecurity/caddy` | Caddy web server | -| `crowdsecurity/apache2` | Apache httpd | -| `crowdsecurity/base-http-scenarios` | Generic HTTP attacks | -| `crowdsecurity/http-cve` | CVE-based HTTP attack detection | -| `crowdsecurity/whitelist-good-actors` | Whitelist known good actors (search engines, CDNs) | -| `crowdsecurity/appsec-virtual-patching` | AppSec virtual patching rules | -| `crowdsecurity/appsec-crs` | OWASP CRS rules for AppSec | -| `crowdsecurity/appsec-generic-rules` | Generic AppSec WAF rules | -| `crowdsecurity/mysql` | MySQL database | -| `crowdsecurity/postgres` | PostgreSQL | -| `crowdsecurity/cloudflare` | Cloudflare-protected sites | - -Browse all collections at: https://app.crowdsec.net/hub/collections - -## Remediation Components (Bouncers) - -After installing a bouncer, add it to LAPI: - -```bash -sudo cscli bouncers add my-bouncer-name -# Save the API key returned — it won't be shown again -``` - -### Firewall Bouncer (iptables/nftables) - -Blocks IPs at the network level. Best for SSH, databases, SMTP. - -```bash -sudo apt install crowdsec-firewall-bouncer-iptables -# or -sudo apt install crowdsec-firewall-bouncer-nftables -``` - -Config at: `/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml` - -### Traefik Bouncer (Plugin) - -Block at the reverse proxy level. Supports AppSec WAF forwarding. - -**Static config (`traefik.yaml`):** -```yaml -experimental: - plugins: - bouncer: - moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin - version: v1.6.0 -``` - -**Dynamic config (middleware):** -```yaml -middlewares: - crowdsec: - plugin: - bouncer: - enabled: true - crowdsecMode: live - crowdsecLapiScheme: http - crowdsecLapiHost: crowdsec:8080 - crowdsecLapiKey: "" - forwardedHeadersTrustedIPs: - - 10.0.0.0/8 - - 172.16.0.0/12 - - 192.168.0.0/16 -``` - -### Nginx Bouncer - -The Nginx bouncer uses Lua directives to check requests against CrowdSec decisions. - -```bash -sudo apt install crowdsec-firewall-bouncer-nginx -``` - -**Bouncer config** (`/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.yaml`): -```yaml -api_url: http://127.0.0.1:8080 -api_key: "" -mode: stream # stream (push) or live (pull on each request) -update_frequency: 10s # How often to refresh decisions in stream mode -``` - -**Nginx config** (add to `server {}` block or `nginx.conf`): -```nginx -lua_package_path "/usr/lib/crowdsec/lua/?.lua;;"; -lua_shared_dict crowdsec_cache 10m; - -init_by_lua_block { - local bouncer = require "crowdsec" - bouncer.init() -} - -access_by_lua_block { - local bouncer = require "crowdsec" - if bouncer.check() then - return ngx.exit(ngx.FORBIDDEN) - end -} -``` - -After setup: `sudo systemctl reload nginx && sudo systemctl restart crowdsec`. - -Config at `/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.yaml`. See the full configuration guide with `nginx.conf` directives and Cloudflare CDN support in `references/nginx-bouncer.md`. - -### Other Bouncers - -- **Caddy:** Uses the crowdsec module for Caddy -- **HAProxy:** SPOE-based integration -- **Blocklist Mirror:** Provides a downloadable blocklist for firewalls/routers -- **Custom Bouncer:** Build your own via the LAPI HTTP API - -Full list: https://hub.crowdsec.net/browse/#remediation-components - -## AppSec (WAF) - -The AppSec Component turns CrowdSec into a full WAF with virtual patching. - -### Enable AppSec - -In `acquis.yaml` (or `acquis.yaml.local`): -```yaml -source: appsec -listen_addr: 0.0.0.0:7422 -appsec_config: crowdsecurity/appsec-default -labels: - type: appsec -``` - -Install AppSec collections: -```bash -sudo cscli collections install crowdsecurity/appsec-virtual-patching -sudo cscli collections install crowdsecurity/appsec-crs -sudo cscli collections install crowdsecurity/appsec-generic-rules -``` - -### How AppSec Works - -1. Web server forwards HTTP request to the CrowdSec engine (port 7422) -2. In-band rules are evaluated first — if triggered, request is blocked (403) or captcha'd -3. Out-of-band rules evaluate asynchronously — non-blocking, used for behavioral detection -4. When rules trigger, events feed into scenarios for longer-term decisions - -### AppSec Rule Types - -- **In-band rules:** Blocking — return `ban` or `captcha` immediately. Used for SQLi, XSS, path traversal, CVE exploitation. -- **Out-of-band rules:** Non-blocking — emit events for scenario processing. Used for enumeration, scraping, spam. - -## Notifications - -### Configure Notifications - -1. **Enable in profiles:** Add `notification` to profile in `/etc/crowdsec/profiles.yaml` -2. **Create notification config:** File in `/etc/crowdsec/notifications/.yaml` - -Supported plugins: Slack, HTTP/Webhook, Email (SMTP), Splunk, Telegram, Sentry - -### Test Notifications - -```bash -sudo cscli notifications test -sudo cscli notifications list -``` - -### Example: HTTP Webhook - -```yaml -# /etc/crowdsec/notifications/http.yaml -type: http -name: http_default -log_level: info -format: json -url: https://hooks.example.com/crowdsec -method: POST -headers: - Content-Type: application/json -``` - -## Blocklists - -Blocklists are curated threat feeds you subscribe to via the CrowdSec Console. They augment community blocklists with third-party intelligence. - -Two tiers in `config.yaml` under `api.server.online_client.pull`: -- `community: true/false` — Pull from the CrowdSec community network -- `blocklists: true/false` — Pull from subscribed third-party blocklists - -## CTI (Cyber Threat Intelligence) - -CrowdSec provides an IP reputation API. Configure in `config.yaml`: -```yaml -api: - cti: - key: "" - cache_timeout: "60m" - cache_size: 50 - enabled: true -``` - -Use `cscli decisions list -o json` to see CTI-enriched output. - -## Data Sources / Acquisition - -CrowdSec supports many log sources: - -| Source | Config Type | Stream | One-shot | -|--------|-------------|--------|----------| -| File | `filenames:` | Yes | Yes | -| Docker | `source: docker` | Yes | Yes | -| Journald | `source: journald` | Yes | Yes | -| Syslog | `source: syslog` | Yes | No | -| HTTP | `source: http` | Yes | No | -| Kafka | `source: kafka` | Yes | No | -| AWS CloudWatch | `source: cloudwatch` | Yes | Yes | -| AWS S3 | `source: s3` | Yes | Yes | -| Loki | `source: loki` | Yes | Yes | -| Windows Event | `source: windows_evt_log` | Yes | Yes | - -Common acquisition parameters: -- `log_level`: Per-source log level -- `transform`: Expression to modify events pre-parsing -- `use_time_machine: true` — Use log timestamps instead of read time (important for buffered logs like IIS, S3) -- `labels.type`: **Required** — determines which parser handles the logs - -## Database Backends - -CrowdSec supports multiple database backends in `/etc/crowdsec/config.yaml`: - -```yaml -db_config: - type: sqlite # or mysql, postgresql, pgx - db_path: /var/lib/crowdsec/data/crowdsec.db - use_wal: true # SQLite WAL mode for better concurrency - max_open_conns: 100 - flush: - max_items: 50000 # Max alerts before purge; lower if disk-constrained - max_age: 7d # Alert retention — both max_items and max_age act independently - metrics_max_age: 90d -``` - -**Flush tuning guidance:** -- Both `max_items` and `max_age` act as independent thresholds — whichever triggers first causes a flush. Set both for belt-and-suspenders control. -- On low-power devices (Raspberry Pi, SD cards), set `max_items: 10000` and `decision_bulk_size: 2000` to reduce write frequency. -- For high-traffic deployments, increase `max_items` to 100000+ but monitor disk usage. -- SQLite flush does NOT reclaim disk space — run `VACUUM` periodically on the SQLite database file to shrink it after large flushes. - -## Metrics & Observability - -### Built-in metrics - -```bash -sudo cscli metrics # Full metrics dashboard -sudo cscli metrics -o json # JSON for programmatic use -``` - -Metrics include: Acquisition stats, parser hits/unparsed, scenario counts, alert counts, decisions (local vs CAPI), bouncer activity. - -### Prometheus - -Enable in `config.yaml`: -```yaml -prometheus: - enabled: true - level: full # or "aggregated" for low cardinality - listen_addr: 0.0.0.0 - listen_port: 6060 -``` - -CrowdSec provides Grafana dashboards: https://github.com/crowdsecurity/grafana-dashboards - -### CrowdSec Console (Web UI) - -Free web console at https://app.crowdsec.net — provides: -- Alert dashboard with IP reputation, MITRE ATT&CK TTPs -- Decision management -- Blocklist subscriptions -- Security Engine enrollment -- Stack health monitoring -- Remediation metrics - -Enroll: `sudo cscli console enroll ` - -## TLS / mTLS - -CrowdSec supports TLS for LAPI communication: - -```yaml -api: - server: - tls: - cert_file: "/path/to/cert.pem" - key_file: "/path/to/key.pem" - client_verification: "RequireAndVerifyClientCert" - ca_cert_path: "/path/to/ca.pem" - agents_allowed_ou: - - agents_ou - bouncers_allowed_ou: - - bouncers_ou -``` +For AppSec, install the relevant virtual-patching/CRS collections and add an +`appsec` acquisition source listening on `7422` with +`appsec_config: crowdsecurity/appsec-default` and `labels.type: appsec`. Route +requests from the proxy to AppSec and decide failure behavior deliberately: +fail-open preserves availability but can bypass protection; fail-closed protects +more strongly but can cause an outage. Test with benign fixtures and inspect +AppSec metrics before enabling blocking. In-band rules block or captcha the +current request; out-of-band rules emit events for later scenarios. Load +[the AppSec deep dive](references/appsec-deep-dive.md) and the relevant +[bouncer guide](references/traefik-bouncer.md) or +[nginx-bouncer](references/nginx-bouncer.md). + +## Operations and Troubleshooting + +Check service logs, `cscli metrics`, parser/unparsed counts, scenario hits, +active decisions, and bouncer last-pull time in that order. Distinguish “no +logs acquired”, “logs acquired but unparsed”, “parsed but no scenario hit”, +“decision exists but bouncer is stale”, and “bouncer enforced but proxy routing +is wrong”. Do not interpret an empty alert query as proof of safety. Use +[the troubleshooting guide](references/troubleshooting.md) and the +[operations checklist](references/operations-checklist.md) for a bounded +verification packet. + +Use profiles and notifications deliberately. Test notification plugins with +`cscli notifications test NAME`; never place webhook secrets or CTI keys in +examples. Enable TLS/mTLS for LAPI across trust boundaries and review +community/blocklist pulls before relying on them. ## References -Load the following reference files for deeper coverage of specific topics: -| Reference | Load when | File | -|-----------|-----------|------| -| Full config.yaml reference | You need every configuration directive explained | `references/config-reference.md` | -| cscli command reference | You need every cscli subcommand and flag | `references/cscli-command-reference.md` | -| AppSec WAF deep dive | Setting up or troubleshooting AppSec | `references/appsec-deep-dive.md` | -| Docker deployment guide | Running CrowdSec in Docker Compose | `references/docker-deployment.md` | -| Traefik bouncer setup | Integrating with Traefik reverse proxy | `references/traefik-bouncer.md` | -| Nginx bouncer setup | Configuring the Nginx bouncer with nginx.conf directives | `references/nginx-bouncer.md` | -| Database configuration | Choosing between SQLite, MySQL, PostgreSQL | `references/database-config.md` | -| Production hardening | Security, TLS, performance tuning | `references/production-hardening.md` | -| Hub collections list | You need to know which collection protects what | `references/hub-collections.md` | -| Troubleshooting guide | Something isn't working | `references/troubleshooting.md` | -| Production operations checklist | Verifying or operating a deployment | `references/operations-checklist.md` | +- [Docker deployment](references/docker-deployment.md) +- [Configuration](references/config-reference.md) +- [cscli commands](references/cscli-command-reference.md) +- [AppSec WAF](references/appsec-deep-dive.md) +- [Operations checklist](references/operations-checklist.md) +- [Troubleshooting](references/troubleshooting.md) +- [Production hardening](references/production-hardening.md) +- [Database configuration](references/database-config.md) +- [Hub collections](references/hub-collections.md) +- [Traefik bouncer](references/traefik-bouncer.md) +- [Nginx bouncer](references/nginx-bouncer.md) diff --git a/crowdsec/evals/evals.json b/crowdsec/evals/evals.json new file mode 100644 index 0000000..f06e1f4 --- /dev/null +++ b/crowdsec/evals/evals.json @@ -0,0 +1,66 @@ +{ + "schema_version": 1, + "skill_name": "crowdsec", + "evals": [ + { + "id": "docker-deployment-safe-networking", + "prompt": "Deploy CrowdSec with Docker Compose for an nginx reverse proxy. Include the volumes, networks, ports, collections, and post-deployment checks that avoid losing state or exposing the API.", + "expected_output": "A Compose-oriented deployment plan that persists /etc/crowdsec and /var/lib/crowdsec/data, mounts logs read-only, installs collections matching nginx and Linux logs, restricts LAPI/metrics/AppSec bindings and networks, pins or reviews the image, and verifies engine health, acquisition metrics, collections, and bouncer connectivity. It explains that the engine detects but does not block without a remediation component.", + "assertions": [ + "The deployment persists CrowdSec configuration and data volumes", + "Log mounts are read-only and collections match the source logs", + "The plan restricts API and metrics exposure and verifies network placement", + "The response states that a remediation component is required for blocking", + "Post-deployment checks include health, acquisition, collections, and bouncer status" + ] + }, + { + "id": "cscli-readonly-triage", + "prompt": "CrowdSec is installed but I am not seeing bans. Give me a read-only cscli triage sequence that distinguishes missing logs, parser failures, no scenario matches, stale decisions, and a broken bouncer.", + "expected_output": "A bounded read-only sequence using cscli version, collections, metrics -o json, alerts, decisions, and bouncers, with interpretation for each layer. It checks acquisition and unparsed counts before scenario hits, then confirms decisions and bouncer last-pull/enforcement, without deleting or adding state.", + "assertions": [ + "The sequence starts with version or service health and installed collections", + "Metrics distinguish missing acquisition from unparsed logs and parser hits", + "Alerts and decisions are checked separately from bouncer status", + "The response identifies stale bouncer or proxy enforcement as distinct failures", + "All commands in the triage path are read-only" + ] + }, + { + "id": "acquisition-and-appsec-waf", + "prompt": "Configure CrowdSec to read nginx logs and inspect HTTP requests with AppSec. Explain the acquisition labels, AppSec endpoint, collections, proxy failure mode, and how to verify safely before blocking.", + "expected_output": "A configuration plan with a typed nginx acquisition entry and an appsec acquisition on port 7422, relevant AppSec collections, proxy forwarding, an explicit fail-open versus fail-closed decision, and benign verification using metrics and test requests. It distinguishes in-band blocking from out-of-band event generation and warns that labels.type is required.", + "assertions": [ + "The nginx acquisition includes labels.type for parser selection", + "The AppSec source uses the AppSec endpoint and an AppSec configuration", + "Relevant AppSec collections and proxy forwarding are specified", + "Fail-open versus fail-closed behavior is an explicit operational decision", + "Verification uses benign traffic and distinguishes in-band from out-of-band rules" + ] + }, + { + "id": "mutation-safety-decision", + "prompt": "An operator wants to manually ban an IP and later remove it after testing. Give a safe CrowdSec procedure with confirmation, rollback, and verification, including what not to assume from an alert or decision.", + "expected_output": "A procedure that confirms host/container and scope, records the reason and rollback, uses simulation or a narrowly scoped short-duration decision where appropriate, requires explicit authorization before cscli decisions add/delete, verifies the decision and bouncer enforcement, and removes only the intended decision. It states that alerts or LAPI decisions alone do not prove traffic was blocked.", + "assertions": [ + "The procedure confirms target, scope, reason, and rollback before mutation", + "The manual decision is narrowly scoped and explicitly authorized", + "The response verifies both the LAPI decision and remediation enforcement", + "Removal targets only the intended decision and is treated as a mutation", + "The response does not equate an alert or decision with confirmed blocking" + ] + }, + { + "id": "crowdsec-no-data-diagnosis", + "prompt": "After changing an acquisition file, CrowdSec shows zero alerts. Diagnose the situation without concluding that the system is safe, and list the evidence to capture before changing profiles or collections.", + "expected_output": "A diagnosis that treats zero alerts as ambiguous and checks service logs, acquisition paths and permissions, labels.type, parser/unparsed metrics, scenario hits, time windows and buffered-source settings, active decisions, and bouncer pulls. It captures version, config paths, metrics, and timestamps, then recommends simulation and allowlist review before enabling enforcement changes.", + "assertions": [ + "Zero alerts is explicitly treated as ambiguous rather than proof of safety", + "The diagnosis checks file access, labels.type, parser results, and scenario hits", + "Buffered logs and time-window or time-machine behavior are considered", + "The evidence packet includes versions, metrics, configuration context, and timestamps", + "The response recommends simulation or allowlist review before risky enforcement changes" + ] + } + ] +} diff --git a/llms.txt b/llms.txt index 9bfab32..6be67fe 100644 --- a/llms.txt +++ b/llms.txt @@ -22,14 +22,14 @@ - [c4-diagramming](c4-diagramming/SKILL.md): Create and review C4 software-architecture diagrams using Mermaid or Structurizr. Use when teams need a communication-ready system context, container, component, or code-level view, including audience, narrative, hierarchy, labels, legends, accessibility, and uncertainty. Do not use for Mermaid syntax/rendering work without C4 modeling, architecture decisions, or full accessibility conformance reviews. - [capacity-and-cost-engineering](capacity-and-cost-engineering/SKILL.md): Model technical capacity, unit cost, and budget constraints connected to demand, performance, and reliability decisions. Use when projecting capacity from growth forecasts, sizing for peak events, designing cost-aware scaling policies, defining budget thresholds or quota/rate-limit enforcement, running or planning load/soak tests as capacity evidence, resolving SLO-cost tradeoffs, or modeling multi-tenant demand distributions, hot-tenant skew, pooled or siloed headroom, fairness evidence, and tenant-variable unit cost. Do NOT use for financial P&L statements, fundraising scenarios, or SaaS metrics (route to financial-modeling); for infrastructure implementation or cloud-resource provisioning (route to platform-engineering); or for generic cloud-cost tips and universal utilization targets — this skill does not prescribe fixed savings rates or one-size-fits-all thresholds. - [chief-of-staff-methodology](chief-of-staff-methodology/SKILL.md): Prepare accountable executive decisions, information triage, briefing, calendar choices, organizational sensing, and institutional memory without assuming authority or monitoring people. Use when a chief of staff or CoS, executive office, gatekeeping, decision memo, executive briefing, board materials, organizational sensing, team health, institutional memory, calendar triage, meeting audit, strategic time, or attention allocation is requested. -- [cli-builder](cli-builder/SKILL.md): Build or refactor CLI tools designed for AI agent consumption: non-interactive, flag-driven, idempotent, with --json output and --dry-run preview. Use when creating a new script the agent will call, adding agent-friendly flags to an existing tool, or debugging why an agent keeps failing to use your CLI. +- [cli-builder](cli-builder/SKILL.md): Build or refactor agent-facing CLI tools with non-interactive commands, stable --help and --json contracts, idempotent operations, and --dry-run previews. Use for CLI design, agent-friction refactors, output/exit-code debugging, or automation safety. Do not use for GUI/TUI design, conversational tools, MCP server design, or general API architecture; use api-design-and-evolution for the latter. - [cncf-landscape](cncf-landscape/SKILL.md): Use this skill when discovering and comparing cloud-native technologies from the CNCF Landscape for an architecture or engineering decision. Query the live public Landscape API, filter candidates by capability, category, maturity, license, and repository signals, then produce an evidence-backed shortlist with trade-offs, unknowns, and validation steps. Do not use it as a substitute for project documentation, production-readiness testing, legal review, or general architecture methodology. - [color-management](color-management/SKILL.md): Expert-level color management for ICC profiles, working spaces, gamut mapping, and color science. Use when inspecting ICC profiles, converting between color spaces, checking gamut clipping, validating well-behaved working spaces, or troubleshooting color workflow issues with ImageMagick, ArgyllCMS, Exiftool, or LittleCMS. - [conditional-customer-success](conditional-customer-success/SKILL.md): Guide recurring human-relationship practices — success plans, health evidence, renewal and expansion signals, QBRs, handoffs, escalation, and closed-loop Voice of Customer. Do not use this skill for products without accounts, renewals, QBRs, or a customer-success team, including some internal tools, pure transactional products without recurring relationships, and public services without account-based engagement. Load only when the product context includes a recurring human relationship; decline or route away otherwise. - [confluence-cli](confluence-cli/SKILL.md): Interact with Atlassian Confluence from the terminal: list spaces, browse pages, view page content, search with CQL, and create pages. Use when the user mentions Confluence, a space key (e.g. DEV), or asks about documentation, wiki pages, space content, or knowledge base articles. - [crewai](crewai/SKILL.md): Expert skill for role-based multi-agent orchestration with CrewAI. Agents with Role/Goal/Backstory, task design, crew composition (sequential or hierarchical), tool integration, callbacks, and production deployment. Use when orchestrating multi-agent teams or comparing agent frameworks. - [crm](crm/SKILL.md): Operate HubSpot CRM from a terminal or agent: list and search contact records, view deal pipeline stages, and — with explicit confirmation — move deals between stages, backed by a bundled crm-cli script that is read-only by default and gates every stage change behind a --dry-run/--yes confirmation. Use when an agent needs to answer questions about contacts or deals, produce pipeline views, or apply a confirmed stage change. Do not use for building HubSpot apps or workflow automations (that is HubSpot app development), marketing/sequence automation, or other CRMs like Salesforce (that is their own tooling). -- [crowdsec](crowdsec/SKILL.md): Deploy, configure, and manage CrowdSec — the open-source, collaborative IPS/IDPS/WAF. Covers Security Engine setup (Linux, Docker), cscli hub management, remediation components, AppSec WAF, profiles, notifications, blocklists, CTI, and metrics. Use when setting up or troubleshooting CrowdSec. +- [crowdsec](crowdsec/SKILL.md): Deploy, configure, and operate CrowdSec Security Engine, cscli, remediation components, acquisition pipelines, and AppSec WAF. Use for Linux or Docker installation, detection-to-blocking design, incident review, and safe changes. Do not use for generic firewall, Kubernetes, or reverse-proxy design; route those to the named platform skill and use this skill for CrowdSec integration. - [cyberpunk](cyberpunk/SKILL.md): Create or analyze settings, scenes, world operations, and image direction in the literary cyberpunk mode of William Gibson's Sprawl fiction: dense, accreted urban systems; uneven high technology; corporate power; mediated culture; and human-scale survival inside global networks. Use for Gibson-informed creative work, setting design, or visual briefs, not for generic neon cyberpunk, faithful continuation of named canon, or imitation of Gibson's prose. - [daily-life-discovery](daily-life-discovery/SKILL.md): Guide a consent-based conversation that helps a person discover how an AI agent could improve their day-to-day life: routines, friction, attention, decisions, relationships, learning, and small experiments. Use when someone asks for a daily check-in, wants the agent to learn how they work, says "grill me," wants a conversational journal, or asks what an AI could help with. Do not use for therapy, diagnosis, crisis support, covert monitoring, or product requirements interviews. - [data-architect](data-architect/SKILL.md): Use this skill to assess, design, and evolve data architectures, including data platforms, data products, data mesh adoption, event-driven data flows, governance, modeling, and migration decisions. Load it when teams need workload-grounded tradeoffs, ownership and quality agreements, or a current-to-target data architecture. Do not use it for pipeline or platform operations, implementation details, interface contract semantics, SQL tuning, or statistical modeling; route those to data-engineering, platform-engineering, api-design-and-evolution, postgres, or data-scientist.