skills/magnus919_agent-skills
1
0
Fork 0
mirror of https://github.com/magnus919/agent-skills.git synced 2026-09-11 19:47:12 +03:00
Code Issues Packages Projects Releases Wiki Activity

feat: add research-methodology skill (#11)

Browse Source 
* feat: add research-methodology skill

Signed-off-by: Magnus Hedemark <magnus919@pm.me>

* fix: remove nonportable source assumptions

Signed-off-by: Magnus Hedemark <magnus919@pm.me>

---------

Signed-off-by: Magnus Hedemark <magnus919@pm.me>
This commit is contained in:
Magnus Hedemark
2026-07-12 14:37:23 -04:00
committed by GitHub
parent 6fd559e903
commit e863aab851
14 changed files with 792 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+1
View File
@@ -130,6 +130,7 @@ When the user mentions these keywords, load the corresponding skill:
| "growth rate", "weekly growth", "monthly growth", "startup growth", "compound growth", "traction", "are we growing", "growth benchmark", "how fast should we grow", "YC growth", "product-market fit", "acceleration", "growth trajectory" | [yc-weekly-growth-compass](yc-weekly-growth-compass/SKILL.md) |
| "artifact-pyramids", "artifact pyramids" | [artifact-pyramids](artifact-pyramids/SKILL.md) |
| "site-reliability-engineering", "site reliability engineering" | [site-reliability-engineering](site-reliability-engineering/SKILL.md) |
| "research-methodology", "research methodology" | [research-methodology](research-methodology/SKILL.md) |
## Use-When Sections
Every skill description must identify when to load it. Skills with meaningful overlap should also include a `## When not to use` section naming the nearest alternative or prerequisite. Keep these sections trigger-oriented and concise; implementation details belong in references.
README.md
+4
View File
@@ -187,6 +187,10 @@ Build production-grade AI agents and graph-based state machines with PydanticAI
Query, search, and download public datasets from the City of Raleigh Open Data portal. Wraps the ArcGIS REST API to access 170+ datasets — crime reports, food inspections, building permits, bike lanes, parks, zoning, traffic, budgets, and more. No API key needed. Ships a Python CLI with catalog, search, info, query, download, and categories commands.
### [research-methodology](research-methodology/SKILL.md)
Turn an open question into a bounded, evidence-led investigation rather than a plausible-sounding synthesis.
### [site-reliability-engineering](site-reliability-engineering/SKILL.md)
Build practical reliability practices around the work teams actually perform: measurable service objectives, useful alerts, incident response, and learning-oriented follow-up.
research-methodology/README.md
+37
View File
@@ -0,0 +1,37 @@
# Research Methodology
Turn an open question into a bounded, evidence-led investigation rather than a plausible-sounding synthesis.
## Why Install This Skill
Turn an open question into a bounded, evidence-led investigation rather than a plausible-sounding synthesis. It preserves a practical method, local reference material, and reusable templates so an agent can do more than produce a generic answer.
Use it when the work needs a repeatable process and an inspectable result. It is portable across Agent Skills-compatible clients and does not require a profile system or a particular task orchestrator.
## What You Get
| Path | What it provides |
|---|---|
| `SKILL.md` | Trigger conditions, workflow, and guidance for loading deeper resources. |
| `references/` | Reference material: `industry-analysis.md`, `journalistic-research.md`, `research-lifecycle.md`, `source-evaluation.md`, `structured-analytic-techniques.md`, `synthesis-patterns.md`, `technical-verification.md` |
| `assets/` | Assets: `research-brief.md`, `research-log.md` |
## Quick Start
Choose the research track in `SKILL.md`, then start from `assets/research-brief.md` and maintain a research log.
Install or expose this directory using your agent's standard Agent Skills loading mechanism, then ask for work that matches the triggers below.
## Triggers
- Plan, conduct, evaluate, and synthesize rigorous research. Use for journalistic, industry, or technical investigations that need credible evidence and a traceable method.
- Requests involving the method, deliverables, or review process described in `SKILL.md`.
- Work where a reusable template or reference from this skill would reduce avoidable mistakes.
## Requirements
No runtime dependency. Use appropriate retrieval tools and retain source URLs and access dates in the research log.
## Source and maintenance
This skill was extracted from [`magnus919/hermes-profiles`](https://github.com/magnus919/hermes-profiles) at commit [`867a555`](https://github.com/magnus919/hermes-profiles/commit/867a555). The portable methodology was retained; Hermes-specific profile, orchestration, and memory assumptions were removed.
research-methodology/SKILL.md
+64
View File
@@ -0,0 +1,64 @@
---
name: research-methodology
description: Plan, conduct, evaluate, and synthesize rigorous research. Use for journalistic, industry, or technical investigations that need credible evidence and a traceable method.
license: MIT
compatibility: No runtime dependency. Use appropriate retrieval tools and retain source URLs and access dates in the research log.
metadata:
source_repo: https://github.com/magnus919/hermes-profiles
source_commit: 867a555
---
# Research Methodology
Professional research process for a subagent. Three tracks based on the type of research:
- **Journalistic** — investigative pieces, primary source research, source-heavy narrative work
- **Industry analysis** — market research and strategy, signal detection
- **Academic/Comprehensive** — deep systematic research when depth matters most
All three share the same lifecycle (Scope → Gather → Evaluate → Analyze → Synthesize → Report) but differ in evidence standards, speed, and output format.
## The Research Lifecycle
```
SCOPE → GATHER → EVALUATE → ANALYZE → SYNTHESIZE → REPORT
```
## Reference Files
### Tracks
| Track | Reference | When to load |
|-------|-----------|-------------|
| **Journalistic** | `references/journalistic-research.md` | You're researching an investigative piece — primary sources, interviews, documents, series management, pre-publication verification |
| **Industry analysis** | `references/industry-analysis.md` | You're researching an industry analysis piece — signal detection, corporate evidence, competitive intelligence, case study standards |
| **Academic / Comprehensive** | `references/research-lifecycle.md` | You're doing deep systematic research — question scoping, search strategy, inclusion/exclusion criteria |
### Shared Methodology
| Reference | When to load |
|-----------|-------------|
| `references/source-evaluation.md` | You need to judge whether a source is credible — CRAAP test, triangulation, reliability tiers |
| `references/structured-analytic-techniques.md` | You need to evaluate competing explanations — ACH, driving forces, pre-mortem, indicators |
| `references/synthesis-patterns.md` | You need to combine findings from multiple sources into synthesized conclusions |
| `references/technical-verification.md` | You need to test a technical claim by reproducing it — benchmarks, API behavior, configuration |
### Assets
| Asset | What it produces |
|-------|-----------------|
| `assets/research-brief.md` | Structured brief with findings, confidence assessment, evidence table, open questions |
| `assets/research-log.md` | Traceable record of searches, sources, and decisions |
## Pre-Publication Gateway
For any piece that makes factual claims, load the relevant track's verification protocol before reporting back:
- **Journalistic:** 7-step pre-publication protocol from `references/journalistic-research.md`
- **Industry:** 7-step research protocol from `references/industry-analysis.md`
- **Technical:** 5-step reproduction protocol from `references/technical-verification.md`
## Portability
This skill is intentionally host-neutral. Use your agent's normal mechanisms to load the references, templates, and scripts listed here. Do not assume a particular profile system, task orchestrator, memory service, or response-handoff format.
research-methodology/assets/research-brief.md
+70
View File
@@ -0,0 +1,70 @@
# Research Brief
## Quick Reference
- **Research question:**
- **Depth:** Scan / Light / Moderate / Deep
- **Commissioned by:**
- **Date:**
---
## 1. Executive Summary
*One paragraph answering the research question. Someone who reads only this should understand what you found and how confident you are.*
---
## 2. Key Findings
*3-5 synthesized findings. Each with a confidence tag.*
### Finding 1: [Title]
[2-3 sentence synthesis of what the evidence says]
**Confidence:** High / Medium / Low
**Sources:** [Link 1](), [Link 2]()
### Finding 2: [Title]
...
**Confidence:** ...
**Sources:** ...
---
## 3. Evidence Table
| Finding | Supporting sources | Contradicting sources | Key evidence |
|---------|-------------------|-----------------------|--------------|
| Finding 1 | Source A, Source B | Source C | The benchmark showing X |
| Finding 2 | Source D, Source E | — | The case study of Y |
---
## 4. Confidence Assessment
| Dimension | Assessment |
|-----------|-----------|
| **Source quality** | How reliable are the sources? (CRAAP scores) |
| **Convergence** | How many independent sources agree? |
| **Coverage** | Are there gaps in the evidence? |
| **Stability** | Would new information likely change the conclusion? |
**Overall confidence:** High / Medium / Low
---
## 5. Open Questions
*What you still don't know. Every research project should have these.*
- [ ] [Question 1]
- [ ] [Question 2]
---
## 6. Sources
| # | Title | URL | Type | Key contribution |
|---|-------|-----|------|-----------------|
| 1 | | | Paper / Blog / Doc | |
| 2 | | | | |
research-methodology/assets/research-log.md
+31
View File
@@ -0,0 +1,31 @@
# Research Log
A traceable record of every search, source, and decision made during a research engagement.
## Search History
| # | Query | Engine | Date | Result count | Notes |
|---|-------|--------|------|-------------|-------|
| 1 | | Search tool / source name | | | |
| 2 | | | | | |
## Sources Considered
| # | Title | URL | CRAAP score | Decision | Reason |
|---|-------|-----|-------------|----------|--------|
| 1 | | | /25 | Keep / Reject | |
| 2 | | | /25 | Keep / Reject | |
## Citation Chain
*Which sources led to which other sources.*
- Source A → cited Source D, Source E
- Source D → cited Source F
## Key Decisions
| Decision | Rationale | Date |
|----------|-----------|------|
| Included/excluded topic X | | |
| Stopped searching after N sources | | |
| Changed research question | | |
research-methodology/references/industry-analysis.md
+112
View File
@@ -0,0 +1,112 @@
# Industry Analysis Track
For the publication's — analyzing company strategy, market signals, and technology trends with data that enterprise decision-makers will scrutinize.
## The Core Difference From Journalism
Journalism asks "what happened and who's responsible." Industry analysis asks "what does this mean, what's coming next, and what should we do about it." The evidence base is more public, the time pressure is higher, and the conclusion is the point.
## Signal vs. Noise
The fundamental skill of industry analysis is distinguishing what matters from what's loud. Most tech news is noise — a product launch, a funding round, an executive departure. Signal is what changes the decision landscape.
### The Signal Filter
Before investing time in any development, run it through this filter:
| Question | If yes | If no |
|----------|--------|-------|
| Does this change a company's competitive position? | Signal | Noise |
| Does this change the cost structure of a technology? | Signal | Noise |
| Does this change who can build what? | Signal | Noise |
| Does this open a market that was previously closed? | Signal | Noise |
| Does this reveal a strategy the company hasn't announced? | Signal | Noise |
| Is this the first time something has happened? | Signal, investigate | Trend-confirming |
### The "So What" Test
After identifying a development, ask "so what?" three times:
1. **So what?** Salesforce's AI Work Ratio hit 30%. (→ They're changing how they measure productivity)
2. **So what?** That means enterprise AI pricing is about to shift from per-seat to outcome-based. (→ Competitors will follow)
3. **So what?** This changes the TCO calculation for every enterprise evaluating AI tools. (→ That's the story.)
If you can't answer the third "so what," you haven't found the angle yet.
## Corporate Evidence Sources
### Financial Documents
| Document | What it reveals | Where to find it |
|----------|----------------|------------------|
| **10-K / Annual report** | Risk factors, strategy letter, segment financials, competitive landscape | SEC EDGAR, company investor relations |
| **10-Q** | Quarterly trends, updated risk factors, management's discussion | SEC EDGAR |
| **Earnings call transcript** | What leadership emphasizes, how they answer hard questions, what they avoid | Seeking Alpha, Fool.com, company IR |
| **Investor day presentations** | Long-term strategy, TAM estimates, product roadmaps | Company IR site |
| **S-1 (IPO filing)** | Full business model, competitive risks, historical financials | SEC EDGAR |
### Executive Signal Analysis
What executives say is strategic communication. What they *don't* say is often more revealing.
| Signal | What to look for |
|--------|-----------------|
| **Evasive answer to a direct question** | They know something but won't say it on the record — the question hit a nerve |
| **Forward-looking language** | "We see a path to..." vs "This is a priority" — specificity indicates commitment |
| **Who's assigned to what** | An executive moved from a core product to an experimental one tells you where the company's center of gravity is shifting |
| **Rhetorical emphasis** | The same phrase appearing across multiple executives' communications is a coordinated message |
| **What they praise competitors for** | "We admire how [competitor] has done X" often signals an upcoming strategy change |
| **Comparison baselines** | "Faster than the market" is weak. "Faster than our three largest competitors combined" is strong. Which baseline they choose reveals how they measure themselves. |
### The "Follow the Money" Pattern
The most reliable signal in technology analysis:
1. **Where is investment capital flowing?** Follow the money — VC funding, R&D spend, acquisition targets.
2. **Where is talent flowing?** Key hires, departures, executive churn. A mass exodus from one company to another tells you where the center of gravity is shifting.
3. **Where is infrastructure spending going?** Cloud contracts, GPU procurement, data center buildout. Capital deployment reveals strategy more honestly than press releases.
4. **Where is regulatory attention focused?** FTC investigations, EU digital regulations, export controls. Policy follows economic concentration.
## The Case Study Standard
Industry analysis articles often use case studies. The standard for evidence:
| Claim type | Minimum evidence | Strong evidence |
|------------|----------------|-----------------|
| "Company X achieved Y" | Public announcement or earnings mention | Independent verification (user survey, third-party audit, bench test) |
| "X% of organizations do Y" | Industry survey (named firm, linked methodology) | Multiple surveys, same finding, different respondents |
| "Trend X is accelerating" | Two data points showing direction | Three+ data points with consistent trajectory, independent sources |
| "X is more effective than Y" | Company A/B test (must state sample size and duration) | Independent academic study or multi-company meta-analysis |
| "Leadership said Z" | Verbatim quote with source (transcript, event video, blog post) | Quote + context of when/why they said it |
### The "One Company" Rule
A single case study is an anecdote. Two companies with different profiles showing the same pattern is a finding. Five companies across different segments is a trend. Frame accordingly:
| Count | Framing |
|-------|---------|
| 1 company | "At Company X, we observed..." |
| 2-3 companies | "Across early adopters, a consistent pattern..." |
| 4+ companies | "The data across the industry shows..." |
## Competitive Intelligence Ethics
| Do | Don't |
|----|-------|
| Use publicly available information | Misrepresent yourself to get information |
| Analyze published products and marketing | Use non-public pricing decks or internal documents |
| Attend public events and conferences | Ask interview subjects to reveal confidential information |
| Track job postings and patent filings | Pay employees of competitors for information |
| Subscribe to competitors' public newsletters | Use a fake identity to access gated competitor content |
## Industry Analysis Research Protocol
### Standard Depth Piece (Moderate Depth)
1. **Scrape the headline** — what's the development? Identify the core event or data release
2. **Primary source check** — find and read the actual announcement, paper, or filing. Not the news about it
3. **Signal filter** — run through the signal/noise filter above. If it's noise, stop
4. **Context layer** — what has happened in the preceding 6 months that changes how to read this? Load the last 2-3 related sources for trajectory
5. **Competitor reaction scan** — have competitors responded? What does their response (or silence) signal?
6. **The "so what" 3x** — drill to the real implication
7. **Write the brief** — load `assets/research-brief.md` and fill for the publication's format
research-methodology/references/journalistic-research.md
+112
View File
@@ -0,0 +1,112 @@
# Journalistic Research Track
For the publication investigative pieces — source-heavy, narrative-driven work where facts must survive public scrutiny.
## The Core Difference From Academic Research
Academic research assumes all sources are published and findable. Journalistic research assumes the best sources are people, documents that weren't meant to be public, and observations the researcher makes firsthand. The evidence is messier, the verification burden is higher, and the stakes for getting it wrong are immediate.
## Primary Source Handling
### The Hierarchy of Journalistic Evidence
| Tier | Source type | How to treat it |
|------|-------------|-----------------|
| 1 | **Primary documents** — emails, internal memos, court filings, financial statements, code commits, leaked data | Highest weight. Corroborate authenticity, then treat as ground truth for what the document says |
| 2 | **Firsthand testimony** — someone who was there, saw it happen, or did the thing | Named sources are stronger than anonymous. Corroborate specifics against documents when possible |
| 3 | **Contemporary records** — meeting notes, chat logs, recordings made at the time | Stronger than memory. Memory is unreliable within weeks |
| 4 | **Published reporting** — other journalists who covered the story | Cite them, don't re-report their work. If you stand on their reporting, make it visible |
| 5 | **Secondhand accounts** — someone who heard from someone who was there | Weakest tier. Only use if primary sources are unavailable and the chain is short |
### The Named Source Standard
**Always prefer named sources.** A named source who is proven wrong damages your credibility. An anonymous source who is proven right also damages your credibility — because the reader can't verify who said it.
When a source requests anonymity:
1. **Establish why.** Is it fear of retaliation, a non-disclosure agreement, or discomfort with public attribution? The reason determines how much weight to give the information.
2. **Know their identity.** The reader may not know who they are, but you must. Record it. The source must be verifiable to you even if not to the reader.
3. **Corroborate their claims.** An anonymous source making a verifiable claim is useful. An anonymous source making an unverifiable claim is hearsay.
4. **Never grant anonymity to a source you haven't spoken to directly.** Secondhand anonymous sourcing is not sourcing.
5. **Disclose what you can.** "A current employee who spoke on condition of anonymity because they were not authorized to discuss internal matters" is better than "sources say."
### Interview Integrity
- **Record the conversation.** Always ask permission. If they decline, take contemporaneous notes and read critical quotes back during the interview for confirmation.
- **Verify identity.** If you haven't met the person before, verify who they are before treating their information as source material. LinkedIn, corporate website, cross-reference with others.
- **Context matters.** What else was happening when they said this? Under what circumstances did they share this information? A source who is angry may overstate. A source who is afraid may understate.
- **Follow up.** The best quote often comes after you've turned off the recorder — or in a second conversation after they've had time to think.
## Document Investigation
Not all valuable evidence comes from people. Some of the best sources are documents that exist in plain sight.
### Types of Documents Worth Finding
| Document type | What to look for | Where to find it |
|--------------|-----------------|------------------|
| **Git history** | When was a feature added? Who committed it? What was the commit message? Was it reviewed? | GitHub, git log, pull request discussions |
| **Earnings transcripts** | What does leadership actually say about their strategy? Compare with what they do. | SEC.gov, investor relations pages, earnings call transcripts |
| **Job postings** | What skills are they hiring for? What does the job description reveal about priorities or problems? | Company career pages, LinkedIn |
| **Support forums** | What are users actually struggling with? What workarounds have they developed? | GitHub issues, community forums, subreddits |
| **Change logs** | What did the product look like 6 months ago vs now? What features were removed? | Changelog archives, Wayback Machine |
| **Filing/pipeline documents** | SEC filings, patent applications, regulatory submissions | EDGAR, patent databases, FCC filings |
### Reading With Intent
Don't just read a document — interrogate it:
- **Who created this, and why?** Every document has an intended audience and a purpose. A press release is meant to make the company look good. An internal memo is meant to communicate operational direction. A patent application stakes a legal claim.
- **What's missing?** Absences are often more revealing than presences. A security audit that doesn't mention authentication is a finding. A press release that doesn't name the CEO is a signal.
- **What assumptions does it rely on?** "Q4 revenue grew 15% year-over-year" assumes the previous year's Q4 is the right baseline. What if they had an acquisition that quarter?
- **Does the headline match the body?** Press releases, blog posts, and reports often have headlines that overstate the findings in the body. Read past the lede.
## The Three-Source Rule (Journalism Version)
For any factual claim that:
- Accuses someone of wrongdoing
- Relies on a surprising statistic
- Is contested by other sources
- Would be damaging if wrong
**Require three independent sources.** Not three articles citing each other. Three genuinely independent sources — different people, different documents, different methodologies — all pointing to the same fact.
If you can't get three, report the confidence gap: "Only one person with direct knowledge would speak about it, but their account was consistent with internal documents reviewed by [publication]."
## Researching Across a Series
When researching article 3 of a 5-part series, the trap is re-researching what was already established.
### Per-Installment Research Map
For each article in a series:
| Dimension | Done in earlier installment | Needs research for this one |
|-----------|---------------------------|----------------------------|
| **Shared context** | What the reader already knows from parts 1-2 | What new context does this installment need? |
| **Sources to revisit** | Experts who were informative earlier | Is there a new angle that warrants a follow-up conversation? |
| **Claims that need updating** | Facts established in earlier parts | Have any changed since publication? |
| **Recurring characters** | People introduced in earlier parts | Do they appear in this installment? Do they need re-introduction? |
| **Thematic throughline** | What theme has connected the series so far | Does this installment advance it, complicate it, or branch? |
### The Series Brief
Before starting research on a new installment, write a one-paragraph "where we are" summary that answers:
1. What has the series established definitively so far?
2. What open questions remain from earlier installments?
3. What new ground does this installment cover?
4. What sources from earlier installments could speak to this new ground?
This prevents re-researching and keeps each installment building on the last.
## Pre-Publication Verification Protocol
Before any the publication piece with investigative stakes ships:
1. **Read every quote in the draft against the source.** Not from memory. From the recording or notes. Word for word.
2. **Open every link in the draft.** Does the source say what the draft claims? Numbers match? Quotes exact?
3. **Triangulate every surprise claim.** If a finding would change what the reader thinks, it needs at least two sources.
4. **Check recency.** Statistics cited as "current" within 2 years for fast-moving tech, 5 years for slower domains. Flag outdated data explicitly.
5. **Audit for paraphrase drift.** The draft may accurately cite a source but mis-state the strength of the claim. A paper that says "suggests" should not be paraphrased as "proves."
6. **Flag single-source claims.** Any claim that rests on a single source should be called out in the draft. The reader deserves to know the evidence base.
7. **Run the "what if I'm wrong" test.** If every claim in this article turned out to be false, which ones would do the most damage? Verify those hardest.
research-methodology/references/research-lifecycle.md
+84
View File
@@ -0,0 +1,84 @@
# Research Lifecycle
The full arc of a professional research engagement, from question formulation through reporting.
## Phase 1: Scope — Frame the research question
Before gathering a single source, define what you're looking for and why.
### The Research Brief
Every research project starts with a brief. Load `assets/research-brief.md` and fill it out before proceeding. The brief answers:
- **Core question:** What exactly are we trying to find out? Frame as a single, falsifiable question.
- **Scope boundaries:** What's in and what's out. Define inclusion/exclusion criteria before searching.
- **Depth required:** Quick scan (2-3 sources for an overview) vs comprehensive (exhaustive on a narrow question) vs deep-dive (multi-angle on a complex question)
- **Decision context:** Who needs this information and what will they do with it? A product decision needs different evidence than a research paper.
### Question Types
| Question type | What you're looking for | Example |
|--------------|------------------------|---------|
| **Descriptive** | What's happening? | "What architectures are used for local LLM inference on consumer GPUs?" |
| **Comparative** | How does X compare to Y? | "How does Qwen3.6 compare to Gemma 4 for tool calling?" |
| **Causal** | What drives X? | "Why do MoE models have lower inference latency at high batch sizes?" |
| **Evaluative** | Is X effective? | "What evidence exists that RAG improves accuracy over zero-shot?" |
| **Gap-finding** | What's not known? | "What hasn't been published about neuromorphic edge deployment?" |
### Inclusion/Exclusion Criteria
Before searching, define:
- **Date range:** How recent must sources be?
- **Source type:** Peer-reviewed, industry reports, blog posts, documentation?
- **Authority threshold:** What makes a source credible enough?
- **Language:** English only, or other languages?
- **Duplication rule:** Multiple sources reporting the same finding — count as one or many?
## Phase 2: Gather — Systematic source discovery
### Search Strategy
1. **Start broad, then narrow.** First query should be broad enough to map the territory. Subsequent queries narrow based on what you found.
2. **Use multiple search angles.** Don't rely on one query. Search by: keyword, author/institution, tool name, problem statement, related concept.
3. **Citation chaining.** From each promising source:
- **Backward:** Follow the citations/bibliography to find the sources the author relied on
- **Forward:** Search for papers that cite this source (Google Scholar "cited by")
4. **Source diversity.** Don't rely on one type of source. Mix:
- Primary research (papers, technical reports)
- Grey literature (blog posts, documentation, forum discussions)
- Expert commentary (industry analysis, conference talks)
- Empirical data (benchmarks, datasets, reproducible experiments)
### For Each Source Found, Record
| Field | Purpose |
|-------|---------|
| Title + URL | Find it again |
| Author/source | Credibility assessment |
| Date | Recency check |
| Key claims | What it says that's relevant |
| Supporting evidence | What backs the claims |
| Gaps/limitations | What it doesn't say |
| Connection to brief | How it answers the research question |
## Phase 6: Report — Structure the findings
A research brief should have this structure (see `assets/research-brief.md`):
1. **Executive summary** — One-paragraph answer to the research question
2. **Key findings** — 3-5 synthesized findings with confidence levels
3. **Evidence table** — Sources mapped to findings
4. **Confidence assessment** — What's solid, what's uncertain, what's missing
5. **Open questions** — What you still don't know
6. **Sources** — Full citations with URLs
## Decision Contexts
Different depths of research for different needs:
| Context | Depth | Sources | Time |
|---------|-------|---------|------|
| Quick answer for a decision | Scan | 2-3 targeted sources | Minutes |
| Briefing for a conversation | Light | 5-8 sources, 2 angles | 1 hour |
| Support for a recommendation | Moderate | 10-15 sources, 3+ angles | Half day |
| Foundation for a publication | Deep | 20+ sources, exhaustive | Days |
research-methodology/references/source-evaluation.md
+61
View File
@@ -0,0 +1,61 @@
# Source Evaluation
How to judge whether a source is credible, relevant, and useful before committing to deeper work with it.
## The CRAAP Test
The most widely used framework for evaluating source credibility. Each dimension scored 1 (poor) to 5 (excellent).
| Dimension | What to ask | Low (1-2) | High (4-5) |
|-----------|-------------|-----------|------------|
| **Currency** | When was this published? Has it been updated? Is the field moving fast enough that age matters? | >5 years old in a fast-moving field | Recent, or classic/stable reference |
| **Relevance** | Does this directly address the research question? Who is the intended audience? | Tangential, or wrong audience | Directly addresses the brief |
| **Authority** | Who wrote this? What are their credentials? Is the publisher reputable? | Unknown author, non-reputable publisher | Known expert, peer-reviewed, institutional affiliation |
| **Accuracy** | Is the information supported by evidence? Can it be verified elsewhere? | No sources, claims unsupported | Cited, verifiable, consistent with other sources |
| **Purpose** | Why does this source exist? To inform, persuade, sell, entertain? Is there bias? | Clear bias, advocacy, or commercial intent | Educational or informational, bias acknowledged |
**Thresholds:**
- **18-25:** High-quality source, use confidently
- **13-17:** Adequate, corroborate with another source
- **Below 13:** Low quality, use with caution or discard
## Triangulation
A single source is never enough. Confidence comes from convergence across multiple independent sources.
### Types of Triangulation
| Type | What it means | How to use |
|------|--------------|------------|
| **Source triangulation** | Multiple sources saying the same thing | If 3+ independent sources agree, confidence is high |
| **Methodological triangulation** | Different methods producing the same conclusion | A paper + a benchmark + a case study all pointing the same way |
| **Investigator triangulation** | Different researchers/teams reaching the same conclusion | Multiple labs replicating a finding |
| **Theory triangulation** | Different theoretical frameworks predicting the same outcome | Multiple lenses converging |
### The Triangulation Rule
- **3+ independent sources agree:** Treat as high confidence
- **2 sources agree, 1 contradicts:** Investigate the contradiction — it may be the more interesting finding
- **Sources disagree without clear resolution:** Report the disagreement, don't force consensus
- **Single source on a key claim:** Flag as "single source" in the brief, not as established fact
## Red Flags That Warrant Caution
| Red flag | What to do |
|----------|------------|
| Source is the sole origin of a claim | Corroborate before trusting |
| Source has a clear commercial interest | Discount claims that align with their product |
| Source cites itself or a closed loop | Low credibility for external claims |
| Source makes extraordinary claims without extraordinary evidence | Require higher corroboration standard |
| Source is anonymous or unverifiable | Use only as directional signal |
| Source is from a known advocacy organization | Understand the bias, still use if evidence is sound |
## Source Types by Reliability
| Tier | Source type | Trust |
|------|-------------|-------|
| 1 | Peer-reviewed research, official documentation, primary sources | High |
| 2 | Industry reports, technical blogs by known practitioners, conference talks | Medium-High |
| 3 | News articles, community discussions, forum posts | Medium |
| 4 | Social media, anonymous posts, marketing content | Low-Directional |
| 5 | Opinion without evidence, satire, known unreliable sources | Discard |
research-methodology/references/source-index.md
+6
View File
@@ -0,0 +1,6 @@
# Source index
- **Source repository:** https://github.com/magnus919/hermes-profiles
- **Inspected commit:** `867a555`
- **Imported source directory:** `research-methodology`
- **Porting boundary:** Retained portable methodology, templates, scripts, and references. Removed or generalized Hermes profile, task-orchestration, memory, and rigid response-handoff assumptions.
research-methodology/references/structured-analytic-techniques.md
+72
View File
@@ -0,0 +1,72 @@
# Structured Analytic Techniques
Techniques adapted from intelligence analysis to systematically evaluate evidence, challenge assumptions, and avoid cognitive bias.
## Analysis of Competing Hypotheses (ACH)
Use when you have multiple possible explanations for the same evidence and need to decide which is most credible.
### Step-by-Step
1. **Identify all hypotheses.** Brainstorm every plausible explanation for what you're seeing. Include the null hypothesis (nothing unusual is happening). Do NOT pick a favorite yet.
2. **List evidence and assumptions.** For each hypothesis, note what would support it, what would contradict it, and what assumptions it requires.
3. **Build a matrix.** Rows = evidence items. Columns = hypotheses. For each cell, mark:
- **CC** (consistent): This evidence supports this hypothesis
- **IC** (inconsistent): This evidence contradicts this hypothesis
- **N/A** (not applicable): This evidence doesn't bear on this hypothesis
4. **Work across, not down.** The most important step. For each piece of evidence, evaluate it against ALL hypotheses before moving to the next piece. This prevents confirmation bias toward a single hypothesis.
5. **Count inconsistencies.** The hypothesis with the FEWEST inconsistencies is the most likely — NOT the one with the most supporting evidence. ACH is a refutation tool, not a confirmation tool.
6. **Test sensitivity.** What if a key piece of evidence is wrong? If removing it changes your conclusion, that evidence is a linchpin — verify it.
7. **Report.** Present the conclusion, the rejected alternatives, and the linchpin evidence that drove the decision.
### When to Use ACH
| Good for | Not good for |
|----------|-------------|
| Competing technical explanations | Single-hypothesis verification |
| Evaluating competing vendor claims | Exploratory research |
| Root cause analysis | Routine fact-gathering |
| Contradictory evidence sets | Simple yes/no questions |
## Driving Forces Analysis
Use to understand what's shaping a trend, market, or technology trajectory.
1. **List driving forces.** What factors are pushing in one direction? (Technology advances, regulation, market demand, cost curves)
2. **List restraining forces.** What's holding back change? (Incumbent lock-in, technical limitations, talent gaps, infrastructure debt)
3. **Which forces are accelerating?** Are drivers getting stronger or weaker?
4. **What would change the balance?** What event or discovery would shift the equilibrium?
5. **Two scenarios.** If drivers win → what happens? If restrainers hold → what happens?
## Pre-Mortem Analysis
Use before committing to a research conclusion to identify what could be wrong.
1. **Assume the conclusion is wrong.** Imagine it's six months from now and your research finding turned out to be completely incorrect.
2. **Write the failure story.** What happened? What evidence misled you? What assumptions were wrong? What did you miss?
3. **Identify failure modes.** Which specific evidence items, assumptions, or reasoning steps are most vulnerable?
4. **Harden the analysis.** For each failure mode: what additional evidence would rule it out? What alternative explanation would cover it?
## Indicator / Validator Framework
Use to track whether an ongoing development is trending toward or away from a predicted outcome.
1. **Define observable indicators.** What would you see if the prediction is correct? What would you see if it's wrong?
2. **Assign diagnostic value.** Some indicators are stronger than others. An indicator that would exist ONLY under one scenario is highly diagnostic.
3. **Track over time.** Indicators don't fire all at once. Track which are appearing, which haven't, and which are contradictory.
4. **Update confidence.** As evidence accumulates, adjust your confidence in each scenario.
## Linchpin Analysis
Use to identify which single element your entire conclusion rests on.
1. **Trace the reasoning chain.** Conclusion → supporting evidence → foundational assumptions.
2. **Find the linchpin.** Which assumption or evidence item, if wrong, would collapse the entire conclusion?
3. **Test that specific element.** Don't test random alternatives. Test the linchpin.
4. **Report linchpin confidence separately.** "I'm confident in the conclusion IF [linchpin] holds. Here's what would change if it doesn't."
research-methodology/references/synthesis-patterns.md
+59
View File
@@ -0,0 +1,59 @@
# Synthesis Patterns
Methods for combining findings from multiple sources into something more valuable than any single source.
## Thematic Synthesis
Best for: Combining qualitative findings, expert opinions, and case studies.
1. **Extract claims.** From each source, pull the specific claims relevant to your research question.
2. **Find patterns.** Group claims that point in the same direction. What themes emerge?
3. **Name each theme.** A good theme name is specific enough to be meaningful, broad enough to contain related claims.
4. **Support each theme.** List which sources support it, which contradict it, and which are neutral.
5. **Identify convergent and divergent themes.** Where do sources agree? Where do they disagree?
### Convergence Signals
| Signal | What it means |
|--------|--------------|
| 3+ independent sources agree | High confidence in the finding |
| Sources from different fields converge | Very high confidence — cross-validation across disciplines |
| Different methodologies produce same result | Method-independent finding |
| Even critics concede the point | Defensible claim |
### Divergence Signals
| Signal | What it means |
|--------|--------------|
| Sources disagree without clear pattern | Territory is unsettled — report both sides |
| Disagreement correlates with methodology | Methods may be driving results |
| Disagreement correlates with funding source | Conflict of interest may be a factor |
| Single source contradicts consensus | Investigate deeper — the outlier may be wrong or may have found something others missed |
## Lines of Argument
Best for: Building a case for or against a position.
1. **State the conclusion.** What are you trying to support?
2. **Build independent lines.** Each line of argument is a separate chain of reasoning that supports the conclusion. They should be as independent as possible.
3. **Test each line.** Does each line stand on its own evidence? Or do they share assumptions?
4. **Weight the lines.** Some lines are stronger than others. Score each: strong anchoring evidence, moderate supporting evidence, or weak circumstantial evidence.
5. **Cross-line consistency.** If multiple independent lines all point to the same conclusion, confidence is much higher than if they share a common assumption.
## Constant Comparison
Best for: Iterative research where findings emerge gradually.
1. **Start with the first source.** Extract initial themes.
2. **Compare each new source.** Does it confirm existing themes? Add new ones? Contradict?
3. **Update themes.** Refine, split, or merge themes as evidence accumulates.
4. **Test saturation.** Are new sources adding new themes or just confirming existing ones? When new sources stop adding new themes, you've reached saturation.
## The Convergence-Divergence Matrix
Map findings along two axes:
| | Convergent (sources agree) | Divergent (sources disagree) |
|---|---|---|
| **Well-evidenced** (many sources) | ⭐ High-confidence finding — report as solid | ⚠️ Active debate — report both sides with evidence weights |
| **Thinly-evidenced** (few sources) | 📌 Tentative finding — flag as provisional | 🔍 Gap — identify as area needing more research |
research-methodology/references/technical-verification.md
+79
View File
@@ -0,0 +1,79 @@
# Technical Verification Track
For claims that can be tested by doing — running code, querying APIs, checking benchmarks, measuring yourself. The most credible evidence is evidence you can reproduce.
## The Core Principle
If a claim rests on a number, benchmark, or observable behavior, and you have the tools to test it, do not rely on the source's claim. Test it yourself. Your own measurement, properly conducted, is a tier-1 source.
## What to Verify by Testing
| Claim type | Test method | Tool |
|------------|-------------|------|
| **Performance numbers** (latency, throughput, VRAM usage) | Reproduce the benchmark with the same parameters | Terminal, Python scripts |
| **API behavior** ("the API returns X when Y") | Call the API with the documented parameters | curl, Python httpx |
| **Model outputs** ("Model A beats Model B on X") | Run the same prompt through both models | llama.cpp, OpenRouter API |
| **Configuration claims** ("Set flag X for best results") | Try it with and without the flag, compare | Terminal, A/B testing |
| **Memory/disk usage** ("This uses less than X") | Build the system, measure actual usage | `du`, `ps`, `nvidia-smi`, `htop` |
| **Availability claims** ("The protocol supports Y") | Read the spec, then try to do it | Source code, protocol docs, actual implementation |
| **Compatibility claims** ("Works on macOS and Linux") | Test on both platforms or verify per-platform CI results | CI logs, Docker |
## The Reproduction Standard
### Step 1: Read the claim carefully
What, exactly, does the source claim? Write down the specific numbers, flags, parameters, and conditions.
### Step 2: Replicate the conditions
Use the same:
- Model version / software version
- Hardware (or comparable)
- Configuration flags
- Input data (or equivalent)
- Measurement methodology
If the source doesn't specify conditions fully, note what's missing. A benchmark that doesn't specify GPU driver version, CUDA version, or `nvidia-smi` output is incomplete.
### Step 3: Run the test
Run it once and observe. Then run it again. Then a third time. Variability across runs is itself data.
### Step 4: Compare results
| Situation | What it means |
|-----------|--------------|
| Your result matches the claim within expected variance | Claim verified — high confidence |
| Your result differs significantly | Either the claim is wrong, or your conditions differ. Check conditions, then report the discrepancy |
| You can't reproduce at all | Claim is unverifiable with available resources. Flag it |
| Your result is better than the claim | Interesting — may mean setup differences, or the claim was conservative |
### Step 5: Document the reproduction
```
## Verification
- Claim tested: [exact claim from source]
- My results: [numbers]
- Conditions: [hardware, software, flags, methodology]
- Variance across runs: [min/max/mean across N runs]
- Verdict: Verified / Partially supported / Contradicted / Unverifiable
- Notes: [any caveats about the test conditions]
```
## When Testing Isn't Feasible
Some claims can't be tested with available resources (requires $10K of cloud credits, proprietary hardware, or access to a system you don't have). In these cases:
1. **Find independent reproductions.** Has someone else tested the same claim? Look for replication studies, community benchmarks, or forum discussions.
2. **Read the methodology critically.** If you can't test it yourself, audit the testing methodology. Was the sample size adequate? Were confounding variables controlled? Was there a conflict of interest?
3. **Flag untested claims in the draft.** "This benchmark was conducted by the vendor and has not been independently verified" is honest and keeps you protected.
## The "I Built It" Standard
For technical tutorials and walkthroughs (like "Running a 35B MoE Model on a 16GB Consumer GPU"):
- Every configuration flag in the article must have been tested by the author
- Every command in the article must produce the stated output
- Every screenshot or terminal output must be from the author's own system
- No "should work" — only "worked for me under these conditions"
- If a configuration didn't work, say so and explain why
This standard distinguishes evidence-led technical writing from generic tutorials. The mistakes and dead ends are often the value.