magnus919_agent-skills/binary-analysis/references/evidence-and-confidence.md

Evidence & Confidence

How to build evidence-backed arguments from CLI output. Load this when writing reports, presenting findings, or when the user challenges the certainty of your conclusions. This reference defines the evidence hierarchy, confidence scoring methodology, and the hard boundary between deterministic CLI evidence and agent inference.

The Evidence Boundary

The single most important concept in this skill:

Source	Nature	Scope	Label
CLI data fields	Deterministic facts	What the backend observed	"CLI evidence"
CLI diagnostics	Limitations and caveats	What the backend could NOT determine	"CLI diagnostic"
Agent synthesis	Interpretive conclusions	What the observations mean together	"Agent assessment"

Never blur these categories. Do not present an agent assessment as if it came from the CLI. Do not cite a CLI diagnostic as if it were positive evidence. The user (human or downstream agent) must always know which layer produced each claim.

Evidence Hierarchy

Evidence is ranked from strongest to weakest. Cite from the strongest available level.

Tier 1: Deterministic Observations

Source: binary triage → data.observations[], or any CLI data field.

Properties:

• Directly measured or enumerated by the backend.
• No confidence field (it is a fact, not an interpretation).
• Reproducible: the same binary + same backend = same result.

Examples:

• "Section .text is 4096 bytes, flags: rx"
• "Function main at 0x401000, size 384 bytes"
• "Import: kernel32.dll!CreateFileW"
• "String at 0x403000: 'C:\Users\Public\payload.exe'"

How to cite: Present as an unqualified fact. "The binary imports CreateFileW from kernel32.dll."

Tier 2: Rule-Derived Heuristics (HIGH confidence)

Source: binary triage → data.heuristics[] with confidence: "HIGH". Also binary suspicious-apis matches with confidence: "HIGH".

Properties:

• Rule engine matched a pattern with strong signal-to-noise ratio.
• HIGH confidence means the pattern is unambiguous given the available evidence.
• Still an interpretation — not a direct observation.

Examples:

• "process-injection (confidence: HIGH) based on VirtualAlloc + WriteProcessMemory + CreateRemoteThread chain"
• "network-communication (confidence: HIGH) based on socket, connect, send, recv imports"

How to cite: Present as a strong indicator with the method. "The API combination suggests process injection capability (confidence: HIGH, based on the VirtualAlloc → WriteProcessMemory → CreateRemoteThread import chain)."

Tier 3: Rule-Derived Heuristics (MEDIUM confidence)

Source: binary triage → data.heuristics[] with confidence: "MEDIUM". Also binary capability-map entries at MEDIUM.

Properties:

• Rule engine matched a pattern but with lower specificity.
• Could be a false positive or the evidence is ambiguous.
• Common with capability assessments based on limited evidence.

Examples:

• "credential-access (confidence: MEDIUM) based on CredEnumerateW import"
• "cryptography (confidence: MEDIUM) based on CryptEncrypt import"

How to cite: Present with explicit qualification. "The binary imports CredEnumerateW, which is consistent with credential enumeration (confidence: MEDIUM). This API is also used by legitimate credential managers."

Tier 4: Rule-Derived Heuristics (LOW confidence)

Source: binary triage → data.heuristics[] with confidence: "LOW". Also binary capability-map entries at LOW.

Properties:

• Weak signal. The pattern is ambiguous or the evidence is thin.
• May be noise. Use only to guide further investigation, not as a conclusion.

Examples:

• "anti-debugging (confidence: LOW) based on a single IsDebuggerPresent import with no supporting evidence"

How to cite: Present with strong caveat. "One anti-debugging API (IsDebuggerPresent) was detected, but with LOW confidence. This is common in many legitimate applications and may be a compiler default. It is not evidence of malicious intent on its own."

Tier 5: Unknowns

Source: binary triage → data.unknowns[], or any null/missing field in CLI output.

Properties:

• The backend could not determine this information.
• Explicit gaps — not failures.

Examples:

• "Indirect call target at 0x402080 could not be resolved"
• "GetProcAddress at 0x403c10: runtime-resolved APIs unknown"

How to cite: Present as an open question. "Seven functions are called indirectly and their targets could not be resolved by static analysis. The dynamic behavior of these call sites is unknown."

Tier 6: Agent Inferences

Source: Your own synthesis. Not in any CLI output.

Properties:

• Your interpretation of multiple pieces of evidence.
• May be correct or incorrect. Cannot be verified by the CLI alone.
• Must be clearly labeled as an agent inference.

Examples:

• "The combination of process injection and credential access APIs suggests this is a credential harvesting tool."
• "The function at 0x402000 appears to be a custom XOR decryption routine based on the loop pattern and XOR constant."

How to cite: Always label explicitly. "Based on the combination of X and Y, the agent assesses that Z. This inference has not been verified by dynamic analysis."

Confidence Scoring Methodology

The CLI uses these confidence levels:

Level	Meaning	When Applied by Rules
HIGH	Rule matched with strong, unambiguous evidence	Multiple corroborating indicators, no contradicting evidence
MEDIUM	Rule matched with reasonable but not conclusive evidence	Single strong indicator or multiple weak ones
LOW	Rule matched with weak or ambiguous evidence	Single weak indicator, or pattern known to produce false positives
UNKNOWN	Rule could not determine	Insufficient evidence to evaluate the rule

How Rules Determine Confidence

Rules in the rule engine combine:

• Evidence count: How many supporting indicators were found.
• Evidence strength: How specific each indicator is to the rule.
• Contradicting evidence: Whether any indicators point away from the rule.
• False positive rate: Historical (or conservatively estimated) noise level for this pattern.

For example, the process-injection rule:

• 3 APIs (VirtualAlloc, WriteProcessMemory, CreateRemoteThread) → HIGH
• 2 APIs (VirtualAlloc, WriteProcessMemory) → MEDIUM
• 1 API (CreateRemoteThread alone) → LOW
• Contradicting: binary also imports legitimate IPC APIs → confidence reduced

What Confidence Does NOT Mean

• HIGH confidence does NOT mean the binary is malicious. It means the pattern is unambiguous.
• LOW confidence does NOT mean the binary is benign. It means the evidence is insufficient.
• UNKNOWN does NOT mean there's nothing there. It means the rules couldn't evaluate.

Building an Evidence-Backed Argument

When presenting findings, follow this structure:

1. State the Evidence (Deterministic)

"Here is what the CLI found." List observations from data blocks. Do not interpret.

2. State the Heuristics (Rule-Derived)

"The rule engine identified these patterns." List heuristics with confidence levels and the evidence that triggered them.

3. State the Unknowns (Gaps)

"The backend could not determine the following." List unknowns and their addresses.

4. State Your Assessment (Agent Inference)

"Based on the above, I assess that..." Clearly separate this from the CLI evidence. Use qualifying language:

Strength	Language
Strong	"The evidence shows", "The analysis confirms"
Moderate	"The evidence suggests", "This is consistent with"
Weak	"It is possible that", "One interpretation is"
Speculative	"The agent speculates that", "Without dynamic analysis, one cannot confirm"

5. Disclose Limitations

"What we cannot determine from static analysis alone." List:

• Indirect call targets
• Runtime-resolved imports
• Encrypted/obfuscated regions
• Missing debug symbols
• Timeouts or partial results
• Backend capability limitations

Common Evidence Pitfalls

Pitfall 1: Confirming the Consequent

Wrong: "The binary imports CreateRemoteThread, therefore it performs process injection."

Right: "The binary imports CreateRemoteThread (CLI evidence). The rule engine flags this as process-injection with MEDIUM confidence (CLI heuristic). The agent notes that import presence alone does not confirm the API is called or with what parameters."

Pitfall 2: Overclaiming from Capability Map

Wrong: "The binary has networking capability, so it exfiltrates data."

Right: "The capability map suggests networking capability (confidence: MEDIUM, based on WinHTTP imports). This means the binary CAN communicate over HTTP. Whether it DOES, and what data it sends, cannot be determined by static analysis alone."

Pitfall 3: Hiding Partial Results

Wrong: Presenting findings without mentioning that 40% of functions timed out during decompilation.

Right: "Function analysis completed for 60 of 100 functions before a timeout. The following findings are based on the 60 functions that completed. The remaining 40 functions (listed in diagnostics) were not analyzed."

Pitfall 4: Presenting Inferences as Facts

Wrong: "This is a ransomware binary."

Right: "The binary imports cryptographic APIs (CryptEncrypt, CryptAcquireContext) and file enumeration APIs (FindFirstFileW, FindNextFileW). The agent assesses this combination is consistent with ransomware behavior, but ransomware cannot be confirmed without dynamic analysis showing actual file encryption."

Evidence Quality Checklist

Before presenting findings, verify:

• Every factual claim is traceable to a specific CLI data field.
• Agent assessments are explicitly separated from CLI evidence.
• Confidence levels are cited for every heuristic claim.
• Unknowns and limitations are disclosed, not buried.
• No claim of certainty where the CLI reports partial results or LOW confidence.
• The binary SHA-256 is included (every claim ties back to a specific binary).
• Provenance fields (adapter, backend, version) are available for reproducibility.

Reporting Evidence

When generating a formal report with binary export-report, the report structure already separates evidence categories. See reporting.md for report generation.

For informal presentations (terminal output, chat responses), use the structure from the SKILL.md:

## CLI Evidence (deterministic)
- <observation 1>
- <observation 2>

## CLI Heuristics (rule-derived)
- <heuristic 1> (confidence: HIGH, evidence: ...)
- <heuristic 2> (confidence: MEDIUM, evidence: ...)

## Unknowns
- <unknown 1> at <address>
- <unknown 2> at <address>

## Agent Assessment
<your synthesis, clearly labeled as inference>

## Limitations
<what static analysis could not determine>