magnus919_agent-skills/research-methodology/references/technical-verification.md

Technical Verification Track

For claims that can be tested by doing — running code, querying APIs, checking benchmarks, measuring yourself. The most credible evidence is evidence you can reproduce.

The Core Principle

If a claim rests on a number, benchmark, or observable behavior, and you have the tools to test it, do not rely on the source's claim. Test it yourself. Your own measurement, properly conducted, is a tier-1 source.

What to Verify by Testing

Claim type	Test method	Tool
Performance numbers (latency, throughput, VRAM usage)	Reproduce the benchmark with the same parameters	Terminal, Python scripts
API behavior ("the API returns X when Y")	Call the API with the documented parameters	curl, Python httpx
Model outputs ("Model A beats Model B on X")	Run the same prompt through both models	llama.cpp, OpenRouter API
Configuration claims ("Set flag X for best results")	Try it with and without the flag, compare	Terminal, A/B testing
Memory/disk usage ("This uses less than X")	Build the system, measure actual usage	du, ps, nvidia-smi, htop
Availability claims ("The protocol supports Y")	Read the spec, then try to do it	Source code, protocol docs, actual implementation
Compatibility claims ("Works on macOS and Linux")	Test on both platforms or verify per-platform CI results	CI logs, Docker

The Reproduction Standard

Step 1: Read the claim carefully

What, exactly, does the source claim? Write down the specific numbers, flags, parameters, and conditions.

Step 2: Replicate the conditions

Use the same:

• Model version / software version
• Hardware (or comparable)
• Configuration flags
• Input data (or equivalent)
• Measurement methodology

If the source doesn't specify conditions fully, note what's missing. A benchmark that doesn't specify GPU driver version, CUDA version, or nvidia-smi output is incomplete.

Step 3: Run the test

Run it once and observe. Then run it again. Then a third time. Variability across runs is itself data.

Step 4: Compare results

Situation	What it means
Your result matches the claim within expected variance	Claim verified — high confidence
Your result differs significantly	Either the claim is wrong, or your conditions differ. Check conditions, then report the discrepancy
You can't reproduce at all	Claim is unverifiable with available resources. Flag it
Your result is better than the claim	Interesting — may mean setup differences, or the claim was conservative

Step 5: Document the reproduction

## Verification

- Claim tested: [exact claim from source]
- My results: [numbers]
- Conditions: [hardware, software, flags, methodology]
- Variance across runs: [min/max/mean across N runs]
- Verdict: Verified / Partially supported / Contradicted / Unverifiable
- Notes: [any caveats about the test conditions]

When Testing Isn't Feasible

Some claims can't be tested with available resources (requires $10K of cloud credits, proprietary hardware, or access to a system you don't have). In these cases:

1. Find independent reproductions. Has someone else tested the same claim? Look for replication studies, community benchmarks, or forum discussions.
2. Read the methodology critically. If you can't test it yourself, audit the testing methodology. Was the sample size adequate? Were confounding variables controlled? Was there a conflict of interest?
3. Flag untested claims in the draft. "This benchmark was conducted by the vendor and has not been independently verified" is honest and keeps you protected.

The "I Built It" Standard

For technical tutorials and walkthroughs (like "Running a 35B MoE Model on a 16GB Consumer GPU"):

• Every configuration flag in the article must have been tested by the author
• Every command in the article must produce the stated output
• Every screenshot or terminal output must be from the author's own system
• No "should work" — only "worked for me under these conditions"
• If a configuration didn't work, say so and explain why

This standard distinguishes evidence-led technical writing from generic tutorials. The mistakes and dead ends are often the value.