diff --git a/AGENTS.md b/AGENTS.md index 408bc5a..06433d7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -96,6 +96,7 @@ When the user mentions these keywords, load the corresponding skill: | "brand identity", "brand guidelines", "style guide", "brand card", "brand strategy", "visual identity", "brand documentation", "color palette", "brand book" | [brand-designer](brand-designer/SKILL.md) | || "kanban", "WIP", "cycle time", "flow metrics", "Scrum to Kanban", "multi-portfolio", "throughput", "classes of service" | [kanban-guru](kanban-guru/SKILL.md) | || "langgraph", "multi-agent", "state machine", "graph-based workflow", "LangGraph", "supervisor pattern", "swarm pattern", "agent orchestration", "graph state", "subgraph", "agent routing", "tool-calling loop", "agent loop", "stateful agent", "durable execution", "human in the loop langgraph", "checkpointer", "langgraph persistence" | [langgraph](langgraph/SKILL.md) | +|| "debate", "council", "multi-perspective", "structured debate", "get multiple perspectives", "expert panel", "decision landscape", "what would experts say", "what are we missing", "convergence", "false consensus", "agent-council", "pre-mortem" | [agent-council](agent-council/SKILL.md) | || "skill format", "how do I make a skill", "agentskills.io" | [agent-skills](agent-skills/SKILL.md) | | "last.fm", "scrobble", "music discovery", "listening history", "similar artists", "lastfm", "weekly top artists", "genre charts" | [lastfm](lastfm/SKILL.md) | | "nous", "theia", "hermes brand", "brand identity", "style guide", "mascot", "anime style", "cyber-classical", "color palette reference" | [nous-branding](nous-branding/SKILL.md) | diff --git a/README.md b/README.md index bbc5220..ac2b2ed 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,10 @@ A collection of AI agent skills — reusable workflows, protocols, and knowledge ## Skills +### [agent-council](agent-council/SKILL.md) + +Multi-agent structured debate system — spawn a panel of expert agents to debate any question with convergence-aware iteration. Produces a decision landscape with confidence diagnostics, shared risks, remaining disagreements, and a principal's path narrative. Ships as a pip-installable Python package built on PydanticAI + PydanticGraph. `pip install pydantic-ai && pip install agent-council` and one API key to run your first debate. + ### [agent-skills](agent-skills/SKILL.md) Reference for the Agent Skills open format itself — directory structure, frontmatter schema, naming conventions, and progressive disclosure model. Use this meta-skill when creating or reviewing any other skill in this repository. diff --git a/agent-council/LICENSE b/agent-council/LICENSE new file mode 100644 index 0000000..9745e19 --- /dev/null +++ b/agent-council/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Magnus Hedemark + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/agent-council/README.md b/agent-council/README.md new file mode 100644 index 0000000..f1a48b6 --- /dev/null +++ b/agent-council/README.md @@ -0,0 +1,80 @@ +# Agent Council + +Multi-agent structured debate system — spawn a panel of expert agents to debate any question with convergence-aware iteration. + +```bash +pip install agent-council +agent-council "Should we migrate from SQLite to Postgres?" +``` + +## Quick Start + +```bash +export AGENT_COUNCIL_API_KEY="sk-..." +export AGENT_COUNCIL_MODEL="openai/gpt-4o-mini" + +agent-council "Should we use WebSockets or SSE for real-time notifications?" +``` + +## Features + +- **Structured debate protocol** — compose, premortem, position, cross-examine (iterative), synthesis +- **Convergence-aware iteration** — the protocol measures confidence dispersion and stops when diminishing returns set in, not at a hardcoded round count +- **Typed outputs** — every phase produces validated Pydantic models, consumable as JSON or human-readable markdown +- **Custom personas** — supply your own agent definitions, or let the compose phase generate them from the question +- **Convergence diagnostics** — confidence dispersion, position overlap, argument novelty — surfaced in every synthesis report +- **Cross-platform** — works with any AI harness that supports agentskills.io skills (Claude Code, Cursor, Hermes Agent, OpenHands, etc.) + +## Installation + +```bash +pip install pydantic-ai +pip install agent-council +``` + +Or install from the skill directory: + +```bash +pip install -e /path/to/agent-council/ +``` + +## Usage + +```bash +# Quick debate (3 agents, 1 cross-examine round) +agent-council --mode quick "Should we use Postgres or SQLite?" + +# Standard debate (5 agents, iterative cross-examination) +agent-council "What architecture should we choose for this service?" + +# Deep debate (7 agents, full protocol with assumption mapping) +agent-council --mode deep --agents 7 "Should we migrate to microservices?" + +# With custom personas +agent-council --persona-file personas.json "Evaluate our cloud strategy" + +# JSON output for programmatic consumption +agent-council --json "Which cloud provider should we choose?" +``` + +## Output + +The synthesis report includes: +- **Confidence dispersion table** — agent-by-agent confidence before and after debate +- **Shared risks** — failure modes identified in the pre-mortem (before positional commitment) +- **Shared concerns** — what survived cross-examination as genuine shared risk +- **Genuine disagreements** — positions that remained unresolved after debate +- **Assumptions per position** — what would need to be true for each position to be correct +- **Principal's path** — narrative synthesis of the decision landscape + +## Configuration + +| Env var | Required | Default | Description | +|---------|----------|---------|-------------| +| `AGENT_COUNCIL_API_KEY` | Yes | — | API key for your LLM provider | +| `AGENT_COUNCIL_MODEL` | No | `openai/gpt-4o-mini` | Model string (provider/model) | +| `AGENT_COUNCIL_BASE_URL` | No | Provider default | Custom API endpoint | + +## License + +MIT diff --git a/agent-council/SKILL.md b/agent-council/SKILL.md new file mode 100644 index 0000000..d92f938 --- /dev/null +++ b/agent-council/SKILL.md @@ -0,0 +1,214 @@ +--- +name: agent-council +description: >- + Multi-agent structured debate system. Spawn a panel of expert agents + to debate any question, with convergence-aware iteration and typed + synthesis output. Run via `agent-council` CLI. Compatible with any + AI agent harness that supports agentskills.io skills (Claude Code, + Cursor, Hermes Agent, OpenHands, etc.). +license: MIT +compatibility: Requires Python 3.10+ and pydantic-ai. CLI tool installs via pip. +metadata: + source: https://git.brandyapple.com/magnus/agent-skills/agent-council + spec-version: "1.0" +--- + +# Agent Council + +Spawn a panel of expert agents to debate any question. The council runs a structured protocol — compose, premortem, position, cross-examination (iterative), synthesis — and produces a decision landscape with convergence diagnostics. + +## When to Use + +Invoke the council when any of these apply: + +- The question has genuine tradeoffs with no clear correct answer +- You want multi-perspective analysis to surface hidden assumptions +- A decision would benefit from adversarial collaboration +- You want confidence diagnostics (not just a recommendation) +- The question has high stakes or irreversible consequences + +**Signal phrases:** "Let's get multiple perspectives on this" / "Debate this: X" / "What would experts say about X" / "What are we missing?" + +## Quick Start + +### 1. Install + +```bash +# One-time setup +pip install pydantic-ai +pip install agent-council + +# Or install from this skill directory: +python3 scripts/bootstrap.py +``` + +### 2. Configure + +```bash +export AGENT_COUNCIL_API_KEY="sk-..." +export AGENT_COUNCIL_MODEL="openai/gpt-4o-mini" +``` + +### 3. Run + +```bash +agent-council "Should we use Postgres or SQLite for this service?" +``` + +## Command Reference + +``` +agent-council [OPTIONS] + +Options: + --agents, -n INTEGER Number of agents (3-7, default: 5) + --mode, -m TEXT Debate depth: quick | medium | deep (default: medium) + --persona-file PATH JSON file with custom agent personas + --json Output structured JSON instead of markdown + --verbose, -v Show phase-by-phase progress + --max-rounds INTEGER Max cross-examination rounds (default: 4) + --convergence FLOAT Convergence threshold (default: 0.10) +``` + +### Mode Selection + +| Mode | Agents | Rounds | When to use | +|------|--------|--------|-------------| +| `quick` | 3 | 1 cross-examine round | Low-stakes check, fast answer needed | +| `medium` (default) | 5 | Eval-driven, up to 4 rounds | Standard decisions | +| `deep` | 7 | Eval-driven, up to 4 rounds | High-stakes, hidden assumptions | + +## How It Works + +### Pipeline + +``` +Compose ──► Premortem ──► Position ──► Cross-examine ──► [eval] ──► Synthesis + (1) (parallel) (parallel) (iterative loop) ↑ (1) + + ┌── converged ──────┐ + ├── diminishing_ret │ + eval ───────────┼── genuine_disagr──┼──► Synthesis + └── continue ───────┘ + ↓ + Cross-examine (next round) +``` + +### Phases + +| Phase | What happens | Method | +|-------|-------------|--------| +| **Compose** | A single LLM call generates N expert personas tuned to the question | 1 call | +| **Premortem** | Each agent independently imagines how the decision already failed — bypasses positional commitment bias | N parallel calls | +| **Position** | Each agent forms an independent position, referencing their own premortem | N parallel calls | +| **Cross-examine** | Each agent reads all other positions and responds — concedes, disagrees, updates confidence | N parallel calls per round | +| **Eval** | Convergence detection: measures dispersion, argument novelty, concession rate. Decides whether to loop or stop | Algorithmic | +| **Synthesis** | Collates all phases into a structured decision landscape with LLM-generated narrative | 1 call | + +### Convergence Detection + +The council doesn't use a fixed number of rounds. After each cross-examination round, it measures: + +- **Confidence dispersion** — standard deviation of agent confidence scores. Below threshold = converged. +- **Argument novelty** — new arguments not seen in prior rounds. Near zero = diminishing returns. +- **Concession rate** — points where agents shifted position. Zero + no new arguments = stalled. + +Stopping conditions: +| Condition | Meaning | +|-----------|---------| +| `converged` | Dispersion below threshold, confidence stable. Genuine agreement. | +| `diminishing_returns` | No new arguments or concessions. Nothing more to surface. | +| `genuine_disagreement` | Dispersion widened, positions hardened. Summary of irreducible tension. | +| `max_rounds` | Hard cap reached. Inconclusive — principal must decide. | + +## Bootstrapping + +If `agent-council` is not available on PATH, the invoking agent should run: + +```bash +python3 scripts/bootstrap.py +``` + +This installs the package from the skill directory using the current Python's pip, falling back to pipx. No PyPI dependency for the bootstrap path — the package ships inside the skill directory. + +**If bootstrap fails:** Run one of these manually: +```bash +pip install pydantic-ai +pip install agent-council + +# Or from this directory: +python3 -m pip install -e /path/to/agent-council/ +``` + +## Configuration + +| Env var | Required | Default | Description | +|---------|----------|---------|-------------| +| `AGENT_COUNCIL_API_KEY` | Yes | — | API key for your LLM provider | +| `AGENT_COUNCIL_MODEL` | No | `openai/gpt-4o-mini` | Model string (`provider/model`) | +| `AGENT_COUNCIL_BASE_URL` | No | Provider default | Custom API endpoint (OpenRouter, LiteLLM, etc.) | + +Model strings follow PydanticAI convention: `openai/gpt-4o-mini`, `anthropic/claude-sonnet-4-20250514`, `deepseek/deepseek-v4-flash`, `google/gemini-2.0-flash`. + +## Output + +The synthesis report is a structured decision landscape. In markdown mode it includes: + +1. **Confidence dispersion table** — per-round confidence metrics with diagnostic +2. **Shared risks** — failure modes from the pre-mortem (pre-positional, uncontaminated) +3. **Shared concerns** — what survived cross-examination as genuine shared risk +4. **Remaining disagreements** — positions that did not resolve +5. **Assumptions per position** — what must hold for each position to be valid +6. **Principal's path** — narrative synthesis of the decision landscape + +Use `--json` for programmatic consumption. + +## Architecture Decision + +**Single-model debate:** All agents share one LLM configuration. Diversity comes from persona definitions (system prompts with distinct backgrounds, analytical approaches, biases), not from different model instances. This minimizes setup friction — one API key, one endpoint, predictable cost. + +**Limitation:** All agents share the model's knowledge cutoff and blind spots. The convergence diagnostics include a "possible false consensus" flag when confidence starts high and never shifts. + +## Reference Files + +| File | Load when | +|------|-----------| +| `references/convergence.md` | Understanding the convergence detection algorithm | +| `references/debate-protocol.md` | Deep dive into phase structure and round design | +| `references/configuration.md` | Provider setup, troubleshooting, model strings | + +## Directory Structure + +``` +agent-council/ +├── SKILL.md # This file — skill entry point +├── pyproject.toml # Pip package definition +├── README.md +├── LICENSE # MIT +├── agent_council/ # Python package +│ ├── cli.py # CLI entry point +│ ├── config.py # Env var loading +│ ├── state.py # Typed state + Pydantic models +│ ├── convergence.py # Convergence detection +│ ├── graph.py # Debate graph orchestration +│ └── phases/ +│ ├── compose.py # Persona generation +│ ├── premortem.py # Failure pre-mortem +│ ├── position.py # Initial positions +│ ├── cross_examine.py # Iterative cross-examination +│ └── synthesis.py # Decision landscape +├── scripts/ +│ └── bootstrap.py # First-run installation +├── templates/ +│ └── personas.json # Example custom personas +└── references/ + ├── convergence.md + ├── debate-protocol.md + └── configuration.md +``` + +## Related Skills + +- **langgraph** — for complex state-machine multi-agent orchestration beyond the debate protocol +- **pydanticai** — the underlying framework for type-safe agent definitions +- **spec-driven-development** — for building specs that agent-council can help you evaluate diff --git a/agent-council/agent_council/__init__.py b/agent-council/agent_council/__init__.py new file mode 100644 index 0000000..73ba726 --- /dev/null +++ b/agent-council/agent_council/__init__.py @@ -0,0 +1,3 @@ +"""Agent Council — Multi-agent structured debate system.""" + +__version__ = "0.1.0" diff --git a/agent-council/agent_council/__main__.py b/agent-council/agent_council/__main__.py new file mode 100644 index 0000000..f666882 --- /dev/null +++ b/agent-council/agent_council/__main__.py @@ -0,0 +1,6 @@ +"""__main__.py — enables `python -m agent_council`.""" + +from agent_council.cli import main + +if __name__ == "__main__": + main() diff --git a/agent-council/agent_council/cli.py b/agent-council/agent_council/cli.py new file mode 100644 index 0000000..d0c1dfb --- /dev/null +++ b/agent-council/agent_council/cli.py @@ -0,0 +1,214 @@ +"""CLI entry point for agent-council.""" + +import argparse +import asyncio +import json +import sys + + +def main(): + """Entry point for `agent-council` CLI.""" + parser = argparse.ArgumentParser( + description="Multi-agent structured debate system", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=( + "Examples:\n" + " agent-council \"Should we use Postgres or SQLite?\"\n" + " agent-council --mode quick --agents 3 \"Quick check on this idea\"\n" + " agent-council --json \"Output as machine-readable JSON\"\n" + " agent-council --persona-file personas.json \"Custom agent lineup\"\n\n" + "Environment:\n" + " AGENT_COUNCIL_API_KEY API key (required)\n" + " AGENT_COUNCIL_MODEL Model string (default: openai/gpt-4o-mini)\n" + " AGENT_COUNCIL_BASE_URL Custom API endpoint\n" + ), + ) + parser.add_argument( + "question", + type=str, + help="The question to debate", + ) + parser.add_argument( + "--agents", "-n", + type=int, + default=5, + choices=range(3, 8), + help="Number of debate agents (3-7, default: 5)", + ) + parser.add_argument( + "--mode", "-m", + type=str, + default="medium", + choices=["quick", "medium", "deep"], + help="Debate depth (default: medium)", + ) + parser.add_argument( + "--persona-file", + type=str, + default=None, + help="JSON file with custom agent persona definitions", + ) + parser.add_argument( + "--json", + action="store_true", + help="Output as structured JSON instead of markdown", + ) + parser.add_argument( + "--verbose", "-v", + action="store_true", + help="Show phase-by-phase progress", + ) + parser.add_argument( + "--max-rounds", + type=int, + default=4, + help="Maximum cross-examination rounds (default: 4)", + ) + parser.add_argument( + "--convergence", + type=float, + default=0.10, + help="Convergence threshold for confidence dispersion (default: 0.10)", + ) + + args = parser.parse_args() + + if not args.question.strip(): + print("Error: Question cannot be empty.", file=sys.stderr) + sys.exit(3) + + # Map mode to agent count + agent_map = {"quick": 3, "medium": 5, "deep": 7} + num_agents = args.agents or agent_map.get(args.mode, 5) + + # Import here so CLI help is fast even without pydantic-ai installed + try: + from agent_council.graph import run_debate + except ImportError as e: + print( + f"Error: Could not import agent_council: {e}", + file=sys.stderr, + ) + print( + "Make sure pydantic-ai is installed: pip install pydantic-ai", + file=sys.stderr, + ) + sys.exit(1) + + try: + state = asyncio.run( + run_debate( + question=args.question, + num_agents=num_agents, + mode=args.mode, + max_rounds=args.max_rounds, + convergence_threshold=args.convergence, + verbose=args.verbose, + persona_file=args.persona_file, + ) + ) + except ValueError as e: + print(f"Configuration error: {e}", file=sys.stderr) + sys.exit(1) + except Exception as e: + print(f"Debate failed: {e}", file=sys.stderr) + sys.exit(2) + + synthesis = state.synthesis + if not synthesis: + print("Error: Debate completed but no synthesis was produced.", file=sys.stderr) + sys.exit(1) + + if args.json: + print(synthesis.model_dump_json(indent=2)) + else: + print(format_synthesis_markdown(synthesis)) + + +def format_synthesis_markdown(synthesis) -> str: + """Format synthesis as human-readable markdown.""" + from agent_council.state import Synthesis + + lines = [] + lines.append(f"# Council Synthesis") + lines.append(f"") + lines.append(f"**Question:** {synthesis.question}") + lines.append(f"**Mode:** {synthesis.mode} ({synthesis.num_agents} agents, {synthesis.rounds_completed} rounds)") + lines.append(f"**Stopped because:** {synthesis.stopped_reason}") + lines.append(f"") + + # Confidence dispersion + lines.append(f"## Confidence Dispersion") + lines.append(f"") + lines.append(f"| Round | Mean Confidence | Dispersion | New Args | Concessions |") + lines.append(f"|-------|----------------|------------|----------|-------------|") + for m in synthesis.confidence_history: + lines.append( + f"| {m.round} | {m.mean_confidence:.3f} | {m.dispersion:.3f} | " + f"{m.new_arguments} | {m.concessions_made} |" + ) + lines.append(f"") + lines.append(f"**Final dispersion:** {synthesis.final_dispersion:.3f}") + lines.append(f"**Mean confidence delta:** {synthesis.mean_confidence_delta:+.3f}") + lines.append(f"") + + # Diagnostic + if synthesis.final_dispersion < 0.08: + diag = "Confidence converged — agents reached alignment." + elif synthesis.final_dispersion > 0.15: + diag = "Confidence remained dispersed — genuine disagreement persisted." + else: + diag = "Moderate agreement with meaningful remaining tension." + lines.append(f"> **Diagnostic:** {diag}") + lines.append(f"") + + # Shared risks (from premortem) + if synthesis.shared_risks: + lines.append(f"## Shared Risks (Pre-Mortem)") + lines.append(f"") + for risk in synthesis.shared_risks: + agents = ", ".join(risk.agents_who_flagged) + lines.append(f"- **{risk.severity.upper()}** — {risk.description}") + lines.append(f" *Flagged by: {agents}*") + lines.append(f"") + + # Shared concerns + if synthesis.shared_concerns: + lines.append(f"## Shared Concerns (Confirmed by Debate)") + lines.append(f"") + for concern in synthesis.shared_concerns: + lines.append(f"- {concern}") + lines.append(f"") + + # Disagreements + if synthesis.disagreements: + lines.append(f"## Remaining Disagreements") + lines.append(f"") + for d in synthesis.disagreements: + lines.append(f"- **{d.topic}**") + for agent, pos in d.positions.items(): + lines.append(f" - {agent}: {pos[:120]}") + lines.append(f"") + + # Assumptions + if synthesis.assumptions_per_position: + lines.append(f"## Assumptions per Position") + lines.append(f"") + for agent, assumptions in synthesis.assumptions_per_position.items(): + lines.append(f"- **{agent}**") + for a in assumptions: + lines.append(f" - {a}") + lines.append(f"") + + # Principal's path + if synthesis.principal_path: + lines.append(f"## Principal's Path") + lines.append(f"") + lines.append(synthesis.principal_path) + lines.append(f"") + + return "\n".join(lines) + + +if __name__ == "__main__": + main() diff --git a/agent-council/agent_council/config.py b/agent-council/agent_council/config.py new file mode 100644 index 0000000..1aa369f --- /dev/null +++ b/agent-council/agent_council/config.py @@ -0,0 +1,30 @@ +"""Configuration — env var loading with sensible defaults.""" + +import os + + +def load_config() -> dict: + """Load configuration from environment variables. + + Returns dict with keys: api_key, model, base_url. + Raises ValueError if AGENT_COUNCIL_API_KEY is not set. + """ + api_key = os.environ.get("AGENT_COUNCIL_API_KEY") + model = os.environ.get("AGENT_COUNCIL_MODEL", "openai/gpt-4o-mini") + base_url = os.environ.get("AGENT_COUNCIL_BASE_URL") + + if not api_key: + raise ValueError( + "AGENT_COUNCIL_API_KEY is not set. " + "Set it to your LLM provider's API key:\n" + " export AGENT_COUNCIL_API_KEY='sk-...'\n" + " export AGENT_COUNCIL_MODEL='openai/gpt-4o-mini' # or your model" + ) + + config = { + "api_key": api_key, + "model": model, + } + if base_url: + config["base_url"] = base_url + return config diff --git a/agent-council/agent_council/convergence.py b/agent-council/agent_council/convergence.py new file mode 100644 index 0000000..9eaa151 --- /dev/null +++ b/agent-council/agent_council/convergence.py @@ -0,0 +1,112 @@ +"""Convergence detection — evaluates debate state to decide when to stop.""" + +import math +from agent_council.state import CouncilState, RoundMetrics + + +def compute_round_metrics(state: CouncilState) -> RoundMetrics: + """Compute convergence metrics from the current round's data.""" + if not state.cross_examination_rounds: + return RoundMetrics( + round=state.round_number, + mean_confidence=0.0, + dispersion=0.0, + new_arguments=0, + concessions_made=0, + ) + + current_round = state.cross_examination_rounds[-1] + confidences = [] + concessions = 0 + total_arguments_before = set() + + # Count arguments from all prior rounds for novelty detection + for r in state.cross_examination_rounds[:-1]: + for ce in r.values(): + if ce.remaining_disagreements: + total_arguments_before.update(ce.remaining_disagreements) + if ce.new_evidence_needed: + total_arguments_before.update(ce.new_evidence_needed) + + new_arguments = 0 + for ce in current_round.values(): + if ce.updated_confidence is not None: + confidences.append(ce.updated_confidence) + if ce.concessions: + concessions += len(ce.concessions) + if ce.remaining_disagreements: + for arg in ce.remaining_disagreements: + if arg not in total_arguments_before: + new_arguments += 1 + + mean_conf = sum(confidences) / len(confidences) if confidences else 0.0 + dispersion = ( + math.sqrt(sum((c - mean_conf) ** 2 for c in confidences) / len(confidences)) + if confidences + else 0.0 + ) + + return RoundMetrics( + round=state.round_number, + mean_confidence=round(mean_conf, 3), + dispersion=round(dispersion, 3), + new_arguments=new_arguments, + concessions_made=concessions, + ) + + +def should_stop(state: CouncilState, metrics: RoundMetrics) -> str: + """Evaluate whether the debate should stop. + + Returns one of: + - "converged" — dispersion below threshold, confidence stable + - "diminishing_returns" — nothing new is surfacing + - "genuine_disagreement" — dispersion widened, positions hardened + - "continue" — run another round + """ + # Hard cap + if state.round_number >= state.max_rounds: + return "max_rounds" + + # Need at least 2 rounds to compare + if len(state.cross_examination_rounds) < 2: + return "continue" + + prior = state.cross_examination_rounds[-2] + prior_confs = [ + ce.updated_confidence + for ce in prior.values() + if ce.updated_confidence is not None + ] + current_confs = [ + ce.updated_confidence + for ce in state.cross_examination_rounds[-1].values() + if ce.updated_confidence is not None + ] + + if not prior_confs or not current_confs: + return "continue" + + prior_mean = sum(prior_confs) / len(prior_confs) + current_mean = sum(current_confs) / len(current_confs) + + # Converged: dispersion below threshold + if metrics.dispersion < state.convergence_threshold: + # Still check if anything changed — settled means done + if abs(current_mean - prior_mean) < 0.03: + return "converged" + return "continue" + + # Diminishing returns: no new arguments, no concessions + if metrics.new_arguments == 0 and metrics.concessions_made == 0: + return "diminishing_returns" + + # Genuine disagreement: dispersion widened and no one moved + if ( + metrics.dispersion > state.convergence_threshold * 1.5 + and metrics.concessions_made == 0 + and metrics.new_arguments == 0 + ): + return "genuine_disagreement" + + return "continue" diff --git a/agent-council/agent_council/graph.py b/agent-council/agent_council/graph.py new file mode 100644 index 0000000..63adee6 --- /dev/null +++ b/agent-council/agent_council/graph.py @@ -0,0 +1,121 @@ +"""Graph orchestration — runs the debate protocol as a state machine.""" + +import asyncio +import json +import sys +from agent_council.state import CouncilState +from agent_council.phases.compose import compose_personas +from agent_council.phases.premortem import run_premortems +from agent_council.phases.position import run_positions +from agent_council.phases.cross_examine import run_cross_examination +from agent_council.phases.synthesis import synthesize +from agent_council.convergence import should_stop, compute_round_metrics + + +async def run_debate( + question: str, + num_agents: int = 5, + mode: str = "medium", + max_rounds: int = 4, + convergence_threshold: float = 0.10, + verbose: bool = False, + persona_file: str | None = None, +) -> CouncilState: + """Run the full debate protocol. + + Phases: + 1. Compose — generate/load agent personas + 2. Premortem — each agent envisions failure + 3. Position — each agent forms initial position + 4. Cross-examine — iterative, convergence-checked rounds + 5. Synthesis — produce decision landscape + """ + state = CouncilState( + question=question, + mode=mode, + max_rounds=max_rounds, + convergence_threshold=convergence_threshold, + ) + + # Phase 1: Compose personas + if verbose: + print("Phase 1/5: Composing agent personas...", file=sys.stderr) + + if persona_file: + with open(persona_file) as f: + data = json.load(f) + from agent_council.state import AgentPersona + state.personas = [AgentPersona(**p) for p in data] + else: + state.personas = await compose_personas(question, num_agents) + + if verbose: + for p in state.personas: + print(f" {p.name}: {p.expertise}", file=sys.stderr) + + # Phase 2: Premortem + if verbose: + print("Phase 2/5: Running pre-mortems...", file=sys.stderr) + state.premortems = await run_premortems(question, state.personas) + + # Phase 3: Position + if verbose: + print("Phase 3/5: Forming positions...", file=sys.stderr) + state.positions = await run_positions(question, state.personas, state.premortems) + + if verbose: + for pos in state.positions.values(): + print( + f" {pos.agent_name}: confidence {pos.confidence}", + file=sys.stderr, + ) + + # Phase 4: Iterative cross-examination + if verbose: + print("Phase 4/5: Cross-examination...", file=sys.stderr) + + round_num = 0 + while round_num < max_rounds: + round_num += 1 + state.round_number = round_num + + if verbose: + print(f" Round {round_num}...", file=sys.stderr) + + cross_results = await run_cross_examination( + question, + state.personas, + state.positions, + state.cross_examination_rounds if state.cross_examination_rounds else None, + ) + state.cross_examination_rounds.append(cross_results) + + # Check convergence + metrics = compute_round_metrics(state) + stop_reason = should_stop(state, metrics) + + if verbose: + print( + f" Dispersion: {metrics.dispersion}, " + f"New args: {metrics.new_arguments}, " + f"Concessions: {metrics.concessions_made}", + file=sys.stderr, + ) + print(f" → {stop_reason}", file=sys.stderr) + + if stop_reason != "continue": + # Store the stop reason for synthesis + object.__setattr__(state, "_stopped_reason", stop_reason) + break + + # Phase 5: Synthesis + if verbose: + print("Phase 5/5: Synthesizing...", file=sys.stderr) + + state.synthesis = await synthesize(state) + + # Set the stopped reason + stop_reason = getattr(state, "_stopped_reason", "max_rounds") + state.synthesis.stopped_reason = stop_reason # type: ignore + + return state diff --git a/agent-council/agent_council/phases/__init__.py b/agent-council/agent_council/phases/__init__.py new file mode 100644 index 0000000..83bd6a5 --- /dev/null +++ b/agent-council/agent_council/phases/__init__.py @@ -0,0 +1 @@ +"""Phase init.""" diff --git a/agent-council/agent_council/phases/compose.py b/agent-council/agent_council/phases/compose.py new file mode 100644 index 0000000..3c7bd69 --- /dev/null +++ b/agent-council/agent_council/phases/compose.py @@ -0,0 +1,41 @@ +"""Compose phase — generates agent personas from the question.""" + +from pydantic_ai import Agent +from agent_council.state import AgentPersona +from agent_council.config import load_config + + +async def compose_personas(question: str, num_agents: int = 5) -> list[AgentPersona]: + """Generate debate agent personas tailored to the question. + + Uses an LLM to compose personas with diverse backgrounds, analytical + approaches, and biases. The compose agent is a single LLM call that + outputs a structured list of AgentPersona definitions. + """ + cfg = load_config() + + compose_agent = Agent( + cfg["model"], + output_type=list[AgentPersona], + system_prompt=( + "You are a council composition specialist. Your job is to design " + "expert debating agents for a structured multi-perspective debate.\n\n" + "Critical directive: Prioritize DIVERSITY OF INITIAL POSITION over " + "diversity of expertise. Research shows that a group with four distinct " + "approaches to a problem — none individually correct — outperforms a " + "group with more expertise but shared framing.\n\n" + "For each agent provide: name, one-paragraph career background, specific " + "expertise, analytical approach, and what bias or experience they bring " + f"to THIS question. Design exactly {num_agents} agents.\n\n" + "At least one agent should be structurally skeptical (a light red-team " + "role). At least one agent should approach the problem from a fundamentally " + "different cognitive frame than the others. Design them to create productive " + "friction — real disagreement grounded in real experience, not caricatures." + ), + ) + + result = await compose_agent.run( + f"Design {num_agents} expert debating agents for the question: {question}" + ) + + return result.output diff --git a/agent-council/agent_council/phases/cross_examine.py b/agent-council/agent_council/phases/cross_examine.py new file mode 100644 index 0000000..675890a --- /dev/null +++ b/agent-council/agent_council/phases/cross_examine.py @@ -0,0 +1,88 @@ +"""Cross-examination phase — agents probe each other's positions.""" + +import asyncio +from pydantic_ai import Agent +from agent_council.state import ( + AgentPersona, + Position, + CrossExamination, +) +from agent_council.config import load_config + + +def _format_other_positions( + my_name: str, + positions: dict[str, Position], +) -> str: + """Format other agents' positions for the prompt.""" + lines = [] + for name, pos in positions.items(): + if name == my_name: + continue + lines.append(f"--- {name} ---") + lines.append(f"Stance: {pos.stance}") + lines.append(f"Reasoning: {'; '.join(pos.reasoning)}") + lines.append(f"Confidence: {pos.confidence}") + lines.append(f"Assumptions: {'; '.join(pos.key_assumptions)}") + lines.append("") + return "\n".join(lines) + + +async def run_cross_examination( + question: str, + personas: list[AgentPersona], + positions: dict[str, Position], + prior_rounds: list[dict[str, CrossExamination]] | None = None, +) -> dict[str, CrossExamination]: + """Each agent reads all other positions and responds. + + This is the core convergence mechanism — agents confront alternative + perspectives and either shift, dig in, or identify new evidence needs. + """ + cfg = load_config() + + async def _cross(persona: AgentPersona) -> tuple[str, CrossExamination]: + other_positions = _format_other_positions(persona.name, positions) + round_context = "" + if prior_rounds: + round_context = "\n\nPrevious round context:\n" + for i, rnd in enumerate(prior_rounds): + if persona.name in rnd: + prev = rnd[persona.name] + round_context += f"Round {i + 1} — your reflection: {prev.reflection}\n" + if prev.concessions: + round_context += ( + f" You conceded: {'; '.join(prev.concessions)}\n" + ) + if prev.remaining_disagreements: + round_context += ( + f" Still in dispute: " + f"{'; '.join(prev.remaining_disagreements)}\n" + ) + + system = ( + f"You are {persona.name}.\n" + f"Background: {persona.background}\n" + f"Expertise: {persona.expertise}\n" + f"Approach: {persona.approach}\n" + f"Bias: {persona.bias}\n\n" + "You are in a structured debate. You have read every other agent's " + "position on the question.\n\n" + "Other agents' positions:\n" + f"{other_positions}\n" + f"{round_context}\n\n" + "Respond to what you've read. For each point: concede where the " + "other agent's reasoning is stronger, identify where you still " + "disagree and why, and update your position if warranted. Be " + "specific — do not hedge. If your confidence has changed, say so." + ) + + agent = Agent(cfg["model"], output_type=CrossExamination, system_prompt=system) + result = await agent.run(question) + output = result.output + output.agent_name = persona.name + return persona.name, output + + tasks = [_cross(p) for p in personas] + results = await asyncio.gather(*tasks) + return dict(results) diff --git a/agent-council/agent_council/phases/position.py b/agent-council/agent_council/phases/position.py new file mode 100644 index 0000000..e650298 --- /dev/null +++ b/agent-council/agent_council/phases/position.py @@ -0,0 +1,58 @@ +"""Position phase — each agent forms an independent initial position.""" + +import asyncio +from pydantic_ai import Agent +from agent_council.state import AgentPersona, Position, Premortem +from agent_council.config import load_config + + +async def run_positions( + question: str, + personas: list[AgentPersona], + premortems: dict[str, Premortem], +) -> dict[str, Position]: + """Each agent forms an independent initial position. + + Agents see their own premortem (to maintain continuity) but NOT other + agents' positions or premortems. This ensures independent thought. + """ + cfg = load_config() + + async def _position(persona: AgentPersona) -> tuple[str, Position]: + my_premortem = premortems.get(persona.name) + + system = ( + f"You are {persona.name}.\n" + f"Background: {persona.background}\n" + f"Expertise: {persona.expertise}\n" + f"Approach: {persona.approach}\n" + f"Bias: {persona.bias}\n\n" + "You are in a structured debate. Your task: form your initial " + "position on the question. Be specific about your stance, your " + "reasoning, and what assumptions you're making.\n" + f"Your pre-mortem identified these failure modes:" + ) + if my_premortem: + system += ( + f"\n - Failure scenario: {my_premortem.failure_scenario}\n" + f" - Root causes: {'; '.join(my_premortem.root_causes)}\n" + f" - Warning signals: {'; '.join(my_premortem.early_warning_signals)}\n\n" + "Your position should account for these risks." + ) + else: + system += "\n (none recorded)\n" + + system += ( + "\n\nReturn your position with a confidence score (0-1) and " + "the key assumptions that must hold for your position to be correct." + ) + + agent = Agent(cfg["model"], output_type=Position, system_prompt=system) + result = await agent.run(question) + output = result.output + output.agent_name = persona.name + return persona.name, output + + tasks = [_position(p) for p in personas] + results = await asyncio.gather(*tasks) + return dict(results) diff --git a/agent-council/agent_council/phases/premortem.py b/agent-council/agent_council/phases/premortem.py new file mode 100644 index 0000000..429a994 --- /dev/null +++ b/agent-council/agent_council/phases/premortem.py @@ -0,0 +1,43 @@ +"""Premortem phase — each agent envisions how the decision already failed.""" + +import asyncio +from pydantic_ai import Agent +from agent_council.state import AgentPersona, Premortem +from agent_council.config import load_config + + +async def run_premortems( + question: str, + personas: list[AgentPersona], +) -> dict[str, Premortem]: + """Each agent independently writes a failure scenario. + + Agents do NOT see each other's premortems — this runs before any + positions are formed, bypassing positional commitment bias. + """ + cfg = load_config() + + async def _premortem(persona: AgentPersona) -> tuple[str, Premortem]: + agent = Agent( + cfg["model"], + output_type=Premortem, + system_prompt=( + f"You are {persona.name}.\n" + f"Background: {persona.background}\n" + f"Expertise: {persona.expertise}\n" + f"Approach: {persona.approach}\n" + f"Bias: {persona.bias}\n\n" + "You are in a structured debate. Your first task: write a " + "pre-mortem — imagine it is 6 months in the future and the " + "decision about to be discussed has ALREADY FAILED. Write " + "the history of how it failed. What went wrong? What were the " + "early warning signals nobody heeded? Be specific and draw on " + "your expertise." + ), + ) + result = await agent.run(question) + return persona.name, result.output + + tasks = [_premortem(p) for p in personas] + results = await asyncio.gather(*tasks) + return dict(results) diff --git a/agent-council/agent_council/phases/synthesis.py b/agent-council/agent_council/phases/synthesis.py new file mode 100644 index 0000000..ec80b5b --- /dev/null +++ b/agent-council/agent_council/phases/synthesis.py @@ -0,0 +1,202 @@ +"""Synthesis phase — produces the final decision landscape.""" + +from pydantic_ai import Agent +from agent_council.state import ( + CouncilState, + Position, + Premortem, + CrossExamination, + Synthesis, + RiskVector, + Disagreement, + RoundMetrics, +) +from agent_council.convergence import compute_round_metrics +from agent_council.config import load_config + + +def _collect_risks( + premortems: dict[str, Premortem], + positions: dict[str, Position], + cross_rounds: list[dict[str, CrossExamination]], +) -> list[RiskVector]: + """Collect all risks flagged across phases.""" + risks: list[RiskVector] = [] + + # From premortems (pre-positional) + for p in premortems.values(): + if p.root_causes: + risks.append( + RiskVector( + description="; ".join(p.root_causes[:3]), + agents_who_flagged=[p.agent_name], + severity="medium", + phase_discovered="premortem", + ) + ) + + # From cross-examinations (post-positional) + for rnd in cross_rounds: + for ce in rnd.values(): + if ce.remaining_disagreements: + for d in ce.remaining_disagreements[:2]: + risks.append( + RiskVector( + description=d, + agents_who_flagged=[ce.agent_name], + severity="medium", + phase_discovered="cross_examine", + ) + ) + + return risks + + +async def synthesize(state: CouncilState) -> Synthesis: + """Produce the final synthesis from all phase outputs. + + Combines algorithmic convergence metrics with an LLM-generated + narrative synthesis of the decision landscape. + """ + cfg = load_config() + + # Collect all risks + risks = _collect_risks( + state.premortems, state.positions, state.cross_examination_rounds + ) + + # Build convergence history + history: list[RoundMetrics] = [] + for i in range(len(state.cross_examination_rounds)): + # Temporarily set round_number to replay metrics + state.round_number = i + 1 + metrics = compute_round_metrics(state) + history.append(metrics) + + # Compute final metrics + final_metrics = history[-1] if history else None + first_metrics = history[0] if len(history) > 1 else final_metrics + + # Identify shared concerns from cross-examination + shared_concerns = _extract_shared_concerns(state.cross_examination_rounds) + + # Identify disagreements + disagreements = _extract_disagreements(state) + + # Build assumptions per position + assumptions_per_position = { + name: pos.key_assumptions for name, pos in state.positions.items() + } + + # Generate narrative synthesis via LLM + narrative = await _generate_synthesis_narrative(state, cfg) + + stopped_reason = state.synthesis.stopped_reason if state.synthesis else "max_rounds" + + return Synthesis( + question=state.question, + mode=state.mode, + num_agents=len(state.personas), + rounds_completed=len(state.cross_examination_rounds), + stopped_reason=stopped_reason, # type: ignore + confidence_history=history, + final_dispersion=final_metrics.dispersion if final_metrics else 0.0, + mean_confidence_delta=( + (final_metrics.mean_confidence - first_metrics.mean_confidence) + if final_metrics and first_metrics + else 0.0 + ), + shared_risks=[r for r in risks if r.phase_discovered == "premortem"], + shared_concerns=shared_concerns, + disagreements=disagreements, + assumptions_per_position=assumptions_per_position, + risk_vectors=risks, + principal_path=narrative, + ) + + +def _extract_shared_concerns( + cross_rounds: list[dict[str, CrossExamination]], +) -> list[str]: + """Find concerns raised by multiple agents across rounds.""" + concern_counts: dict[str, int] = {} + for rnd in cross_rounds: + for ce in rnd.values(): + for d in ce.remaining_disagreements: + concern_counts[d] = concern_counts.get(d, 0) + 1 + for c in ce.concessions: + concern_counts[c] = concern_counts.get(c, 0) + 1 + + # Return concerns raised by more than one agent + return [ + concern + for concern, count in sorted( + concern_counts.items(), key=lambda x: -x[1] + ) + if count > 1 + ][:10] + + +def _extract_disagreements(state: CouncilState) -> list[Disagreement]: + """Identify persistent disagreements from the last round.""" + if not state.cross_examination_rounds: + return [] + + last_round = state.cross_examination_rounds[-1] + topic_positions: dict[str, dict[str, str]] = {} + + for ce in last_round.values(): + for d in ce.remaining_disagreements: + if d not in topic_positions: + topic_positions[d] = {} + topic_positions[d][ce.agent_name] = ce.updated_position or "maintains position" + + return [ + Disagreement(topic=topic, positions=positions) + for topic, positions in topic_positions.items() + ][:8] + + +async def _generate_synthesis_narrative( + state: CouncilState, cfg: dict +) -> str: + """Generate a narrative principal's path via LLM.""" + # Build a summary of the debate for the LLM + summary_parts = [f"# Debate: {state.question}\n"] + summary_parts.append(f"Agents: {', '.join(p.name for p in state.personas)}\n") + + summary_parts.append("\n## Positions\n") + for pos in state.positions.values(): + summary_parts.append( + f"- **{pos.agent_name}** (confidence {pos.confidence}): {pos.stance}\n" + ) + + summary_parts.append("\n## Premortem Failure Scenarios\n") + for pm in state.premortems.values(): + summary_parts.append(f"- **{pm.agent_name}**: {pm.failure_scenario[:200]}\n") + + summary_parts.append("\n## Cross-Examination Rounds\n") + for i, rnd in enumerate(state.cross_examination_rounds): + summary_parts.append(f"\n### Round {i + 1}\n") + for ce in rnd.values(): + summary_parts.append(f"- **{ce.agent_name}**: {ce.reflection[:200]}\n") + + debate_summary = "".join(summary_parts) + + agent = Agent( + cfg["model"], + system_prompt=( + "You are a senior decision analyst. You have overseen a structured " + "multi-agent debate on an important question. Your job: synthesize " + "the debate into a clear 'principal's path' — a narrative that " + "presents the decision landscape to someone who must make a call.\n\n" + "Do NOT describe the debate process (rounds, phases, agents). " + "Write as a single analyst presenting their findings. Structure: " + "what's at stake, where the evidence is strongest, where it's weakest, " + "what assumptions each path depends on, and your recommended path " + "forward with associated risks.\n\n" + "Keep it under 500 words. Be direct. No hedging." + ), + ) + result = await agent.run(debate_summary) + return result.output or "" diff --git a/agent-council/agent_council/state.py b/agent-council/agent_council/state.py new file mode 100644 index 0000000..b5f5474 --- /dev/null +++ b/agent-council/agent_council/state.py @@ -0,0 +1,167 @@ +"""Typed state model and phase output schemas.""" + +from dataclasses import dataclass, field +from typing import Literal + +from pydantic import BaseModel, Field + + +# ── Phase output schemas (validated Pydantic models) ── + + +class AgentPersona(BaseModel): + """Profile for a single debate agent.""" + + name: str + background: str = Field(description="One-paragraph career background") + expertise: str = Field(description="Specific domain expertise") + approach: str = Field(description="Analytical approach they bring") + bias: str = Field(description="What experience or bias they bring to THIS question") + + +class Premortem(BaseModel): + """Pre-mortem: agent envisions how the decision already failed.""" + + agent_name: str + failure_scenario: str = Field(description="Narrative of how the decision failed") + root_causes: list[str] = Field(description="What went wrong") + early_warning_signals: list[str] = Field(description="What to watch for") + + +class Position(BaseModel): + """Agent's initial position on the question.""" + + agent_name: str + stance: str = Field(description="Position on the question") + reasoning: list[str] = Field(description="Chain of reasoning") + confidence: float = Field(ge=0, le=1, description="Confidence in this position") + key_assumptions: list[str] = Field(description="Assumptions that must hold") + + +class CrossExamination(BaseModel): + """Agent's response after reading all other positions.""" + + agent_name: str + concessions: list[str] = Field(description="Points where the agent conceded or shifted") + remaining_disagreements: list[str] = Field(description="Points still in dispute") + updated_position: str | None = Field( + default=None, description="Revised position, if changed" + ) + updated_confidence: float | None = Field( + default=None, ge=0, le=1, description="Updated confidence, if changed" + ) + reflection: str = Field( + description="What the agent learned from other perspectives" + ) + new_evidence_needed: list[str] = Field( + default_factory=list, + description="What evidence would close remaining gaps", + ) + + +class RiskVector(BaseModel): + """A risk identified during the debate, with position-relative context.""" + + description: str + agents_who_flagged: list[str] + severity: Literal["low", "medium", "high"] + phase_discovered: Literal["premortem", "position", "cross_examine"] = Field( + description="Which phase first surfaced this risk. " + "Premortem risks are seen BEFORE positional commitment." + ) + + +class RoundMetrics(BaseModel): + """Convergence metrics for a single cross-examination round.""" + + round: int + mean_confidence: float + dispersion: float = Field(description="Standard deviation of agent confidences") + new_arguments: int = Field(description="Arguments not seen in prior rounds") + concessions_made: int + stopped_early: bool = Field( + default=False, + description="True if this round was cut short by convergence detection", + ) + + +class Disagreement(BaseModel): + """A point of genuine disagreement that survived cross-examination.""" + + topic: str + positions: dict[str, str] = Field( + description="Agent name -> summary of their position on this topic" + ) + unresolved: bool = Field( + default=True, + description="Whether this disagreement persisted after all rounds", + ) + + +class Synthesis(BaseModel): + """Structured output of a completed council debate.""" + + # Metadata + question: str + mode: str + num_agents: int + rounds_completed: int + stopped_reason: Literal[ + "converged", + "max_rounds", + "diminishing_returns", + "genuine_disagreement", + ] = Field(description="Why the debate stopped") + + # Convergence diagnostics + confidence_history: list[RoundMetrics] = Field( + description="One entry per cross-examination round" + ) + final_dispersion: float + mean_confidence_delta: float = Field( + description="Change in mean confidence from first to last round" + ) + + # Content: premortem phase (pre-positional) + shared_risks: list[RiskVector] = Field( + description="Risks identified during pre-mortem before any agent " + "formed a position. Compare with shared_concerns to see which " + "worries survived cross-examination." + ) + + # Content: cross-examination phase (post-positional) + shared_concerns: list[str] = Field( + description="Concerns that survived cross-examination and are shared " + "across agents. A risk in shared_risks that also appears here was " + "confirmed by debate. A risk in shared_risks absent here is either " + "resolved or buried by positional commitment." + ) + disagreements: list[Disagreement] + assumptions_per_position: dict[str, list[str]] = Field( + description="Agent name -> assumptions that would need to hold " + "for their position to be correct" + ) + risk_vectors: list[RiskVector] + principal_path: str = Field(description="Narrative synthesis of the decision landscape") + + +# ── Orchestration state (mutable dataclass) ── + + +@dataclass +class CouncilState: + """Mutable state that flows through the debate graph.""" + + question: str + mode: str = "medium" + max_rounds: int = 4 + convergence_threshold: float = 0.10 + + personas: list[AgentPersona] = field(default_factory=list) + premortems: dict[str, Premortem] = field(default_factory=dict) + positions: dict[str, Position] = field(default_factory=dict) + cross_examination_rounds: list[dict[str, CrossExamination]] = field( + default_factory=list + ) + synthesis: Synthesis | None = None + round_number: int = 0 diff --git a/agent-council/pyproject.toml b/agent-council/pyproject.toml new file mode 100644 index 0000000..0c3bb7a --- /dev/null +++ b/agent-council/pyproject.toml @@ -0,0 +1,20 @@ +[project] +name = "agent-council" +version = "0.1.0" +description = "Multi-agent structured debate system — spawn a panel of expert agents to debate any question with convergence-aware iteration" +readme = "README.md" +license = {text = "MIT"} +requires-python = ">=3.10" +dependencies = [ + "pydantic-ai>=1.0.0", +] + +[project.scripts] +agent-council = "agent_council.cli:main" + +[tool.setuptools.packages.find] +include = ["agent_council", "agent_council.*"] + +[build-system] +requires = ["setuptools>=68.0"] +build-backend = "setuptools.build_meta" diff --git a/agent-council/references/configuration.md b/agent-council/references/configuration.md new file mode 100644 index 0000000..1b444c0 --- /dev/null +++ b/agent-council/references/configuration.md @@ -0,0 +1,59 @@ +# Configuration Guide + +## Environment Variables + +| Env var | Required | Default | Description | +|---------|----------|---------|-------------| +| `AGENT_COUNCIL_API_KEY` | Yes | — | API key for your LLM provider | +| `AGENT_COUNCIL_MODEL` | No | `openai/gpt-4o-mini` | Model string in `provider/model` format | +| `AGENT_COUNCIL_BASE_URL` | No | Provider default | Custom API endpoint | + +## Provider Setup + +### OpenAI + +```bash +export AGENT_COUNCIL_API_KEY="sk-..." +export AGENT_COUNCIL_MODEL="openai/gpt-4o-mini" +``` + +### Anthropic + +```bash +export AGENT_COUNCIL_API_KEY="sk-ant-..." +export AGENT_COUNCIL_MODEL="anthropic/claude-sonnet-4-20250514" +``` + +### DeepSeek + +```bash +export AGENT_COUNCIL_API_KEY="sk-..." +export AGENT_COUNCIL_MODEL="deepseek/deepseek-v4-flash" +export AGENT_COUNCIL_BASE_URL="https://api.deepseek.com/v1" +``` + +### OpenRouter + +```bash +export AGENT_COUNCIL_API_KEY="sk-or-..." +export AGENT_COUNCIL_MODEL="openrouter/anthropic/claude-sonnet-4" +export AGENT_COUNCIL_BASE_URL="https://openrouter.ai/api/v1" +``` + +### Local / Ollama + +```bash +export AGENT_COUNCIL_API_KEY="ollama" # or any placeholder +export AGENT_COUNCIL_MODEL="ollama/llama-3.2" +export AGENT_COUNCIL_BASE_URL="http://localhost:11434/v1" +``` + +## Troubleshooting + +| Symptom | Cause | Fix | +|---------|-------|-----| +| `Configuration error: AGENT_COUNCIL_API_KEY is not set` | Missing API key | `export AGENT_COUNCIL_API_KEY="..."` | +| `ImportError: No module named 'pydantic_ai'` | Missing dependency | `pip install pydantic-ai` | +| Model not found | Wrong model string format | Check PydanticAI provider convention: `provider/model-name` | +| Debate hangs or times out | Model too slow for N parallel calls | Reduce agents with `--agents 3`, or use a faster model | +| All agents agree immediately | False consensus (same-model blind spots) | Check synthesis diagnostic; consider richer persona definitions | diff --git a/agent-council/references/convergence.md b/agent-council/references/convergence.md new file mode 100644 index 0000000..10b5ddf --- /dev/null +++ b/agent-council/references/convergence.md @@ -0,0 +1,46 @@ +# Convergence Detection + +The council uses algorithmic convergence detection to decide when to stop debating — not a fixed number of rounds. + +## Metrics + +After each cross-examination round, four metrics are computed: + +| Metric | Calculation | Meaning | +|--------|------------|---------| +| **Mean confidence** | Average of all agents' `updated_confidence` values | Overall conviction level | +| **Dispersion** | Standard deviation of confidence values | Agreement spread — how far apart agents are | +| **New arguments** | `remaining_disagreements` + `new_evidence_needed` not seen in prior rounds | Whether the debate is still surfacing new material | +| **Concessions** | Count of items in `concessions` across all agents | Whether positions are shifting | + +## Decision Logic + +```python +if round >= max_rounds: + stop_reason = "max_rounds" +elif dispersion < threshold and confidence_delta < 0.03: + stop_reason = "converged" +elif new_arguments == 0 and concessions == 0 and rounds > 1: + stop_reason = "diminishing_returns" +elif dispersion > threshold * 1.5 and concessions == 0 and new_arguments == 0: + stop_reason = "genuine_disagreement" +else: + stop_reason = "continue" # run another round +``` + +## Default Thresholds + +| Mode | Default threshold | Max rounds | +|------|------------------|------------| +| quick | 0.15 | 2 | +| medium | 0.10 | 4 | +| deep | 0.08 | 4 | + +## Diagnostic Interpretation + +| Pattern | Meaning | +|---------|---------| +| Mean confidence DROPPED, dispersion WIDENED | Council surfaced genuine doubt — healthy debate | +| Mean confidence ROSE, dispersion NARROWED | Genuine convergence — agents convinced each other | +| Mean confidence STABLE, dispersion NARROWED | False consensus — agents agreed before debating (possible shared blind spots) | +| Mean confidence ROSE, dispersion WIDENED | Polarization — agents became more entrenched in their positions | diff --git a/agent-council/references/debate-protocol.md b/agent-council/references/debate-protocol.md new file mode 100644 index 0000000..8716c01 --- /dev/null +++ b/agent-council/references/debate-protocol.md @@ -0,0 +1,67 @@ +# Debate Protocol + +## Phase Structure + +### Phase 1: Compose + +A single LLM call generates `N` expert personas. The prompt prioritizes **diversity of initial position** over diversity of expertise. Each persona includes: +- Name +- Career background (one paragraph) +- Specific expertise +- Analytical approach +- Bias or experience they bring to the specific question + +At least one agent is structurally skeptical (light red-team). At least one approaches from a fundamentally different cognitive frame. + +### Phase 2: Premortem + +Each agent independently writes how the decision **already failed** — before any positions are formed. This bypasses positional commitment bias. Agents do NOT see each other's premortems. + +The premortem output includes: +- Failure scenario (narrative) +- Root causes +- Early warning signals + +### Phase 3: Position + +Each agent forms an independent position. They see their own premortem (for continuity) but NOT other agents' positions or premortems. + +Position output includes: +- Stance +- Reasoning chain +- Confidence score (0-1) +- Key assumptions + +### Phase 4: Cross-Examination (Iterative) + +Each agent reads all other agents' positions and responds. They see: +- Every other agent's stance, reasoning, confidence, and assumptions +- Their own previous round's reflection, concessions, and remaining disagreements + +Cross-examination output includes: +- Concessions (where the other agent's reasoning was stronger) +- Remaining disagreements (what's still in dispute) +- Updated position (if changed) +- Updated confidence (if changed) +- Reflection on what they learned +- New evidence needed to close gaps + +After each round, convergence detection runs: +- If converged → proceed to synthesis +- If diminishing returns → proceed to synthesis +- If genuine disagreement → proceed to synthesis (with divergence report) +- If more debate needed → run another round (up to max_rounds) + +### Phase 5: Synthesis + +The synthesis combines algorithmic metrics (confidence dispersion, argument novelty) with an LLM-generated narrative. The output preserves the distinction between: +- **Pre-positional risks** (from premortem — uncontaminated by positional commitment) +- **Post-positional concerns** (survived cross-examination — tested against alternatives) + +## Design Principles + +1. **Independent thought first** — agents form positions before seeing others' +2. **Diversity over expertise** — different approaches beat more expertise with shared framing +3. **Pre-mortem before position** — surface failure modes before committing to a stance +4. **Convergence is measured, not assumed** — algorithmic stopping conditions prevent premature or interminable debate +5. **Tension is the output** — the synthesis surfaces genuine disagreement, not forced consensus diff --git a/agent-council/scripts/bootstrap.py b/agent-council/scripts/bootstrap.py new file mode 100644 index 0000000..3fd8521 --- /dev/null +++ b/agent-council/scripts/bootstrap.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python3 +"""Bootstrap script — ensures agent-council CLI is available. + +The SKILL.md instructs agents to run this script if `agent-council` +is not found on PATH. It installs the package from the skill directory +using the current Python's pip, with pipx as a fallback. +""" + +import shutil +import subprocess +import sys +import os + + +def ensure_installed(skill_dir: str | None = None) -> str | None: + """Ensure agent-council CLI is available. Returns path or None.""" + cli_path = shutil.which("agent-council") + if cli_path: + return cli_path + + if skill_dir is None: + skill_dir = os.path.dirname(os.path.abspath(__file__)) + + print("agent-council not found. Installing from skill directory...", file=sys.stderr) + + # Try sys.executable -m pip install (works with any Python + venv) + try: + subprocess.run( + [sys.executable, "-m", "pip", "install", "-e", skill_dir], + check=True, + capture_output=True, + timeout=60, + ) + cli_path = shutil.which("agent-council") + if cli_path: + print(f"Installed. CLI available at: {cli_path}", file=sys.stderr) + return cli_path + except (subprocess.CalledProcessError, subprocess.TimeoutExpired): + pass + + # Fallback: pipx + pipx = shutil.which("pipx") + if pipx: + print("pip install failed, trying pipx...", file=sys.stderr) + try: + subprocess.run([pipx, "install", skill_dir], check=True, timeout=120) + cli_path = shutil.which("agent-council") + if cli_path: + print(f"Installed via pipx at: {cli_path}", file=sys.stderr) + return cli_path + except (subprocess.CalledProcessError, subprocess.TimeoutExpired): + pass + + print( + f"Could not install agent-council automatically.\n" + f"Run one of:\n" + f" {sys.executable} -m pip install -e {skill_dir}\n" + f" pipx install {skill_dir}\n" + f" pip install agent-council", + file=sys.stderr, + ) + return None + + +if __name__ == "__main__": + result = ensure_installed() + sys.exit(0 if result else 1) diff --git a/agent-council/templates/personas.json b/agent-council/templates/personas.json new file mode 100644 index 0000000..d1a2b45 --- /dev/null +++ b/agent-council/templates/personas.json @@ -0,0 +1,26 @@ +{ + "_comment": "Example custom persona file for agent-council. Each entry follows the AgentPersona schema.", + "personas": [ + { + "name": "Dr. Elena Vasquez", + "background": "15 years as a distributed systems engineer at AWS and Google. Led the migration of Google Ads from a monolithic datastore to Spanner. Has seen three major migration projects fail and two succeed.", + "expertise": "Distributed systems, database internals, cloud infrastructure", + "approach": "Data-driven — asks for benchmarks, latency profiles, and failure mode analysis before forming opinions", + "bias": "Strongly favors proven, battle-tested solutions over novel architectures. Skeptical of anything that sounds like premature optimization." + }, + { + "name": "Marcus Chen", + "background": "YC-founder turned CTO. Bootstrapped a SaaS company to $10M ARR on a single Postgres instance. Recently migrated from Postgres to SQLite for their edge deployment and regrets the tooling gap.", + "expertise": "Startup infrastructure, cost-optimization, pragmatic engineering", + "approach": "Start with the simplest thing that could work, add complexity only when proven necessary", + "bias": "Over-indexes on developer experience and operations simplicity. Under-weights long-term scaling needs." + }, + { + "name": "Priya Sharma", + "background": "Database reliability engineer at a fintech unicorn. Manages 200+ Postgres clusters across three regions. Authored internal runbooks on migration rollback strategies.", + "expertise": "Database operations, replication, disaster recovery, performance tuning", + "approach": "Operational-readiness-first — evaluates every proposal by what happens at 3 AM when it breaks", + "bias": "Assumes every system will fail in the most inopportune way. Skeptical of optimistic deployment timelines." + } + ] +}