feat(ffmpeg): add target compatibility verification (#482)

This commit is contained in:
Magnus Hedemark
2026-09-08 09:31:41 -04:00
committed by GitHub
parent a411903657
commit b4669acf4c
8 changed files with 456 additions and 0 deletions
+11
View File
@@ -51,6 +51,7 @@ This skill provides a repeatable intake-to-acceptance workflow. It separates tec
| `templates/podcast-edit-plan.md` | Mechanical, signal-processing, and editorial audio plan |
| `templates/media-acceptance-report.md` | Criterion-by-criterion evidence and release verdict |
| `templates/media-acceptance-contract.json` | Parseable stream, format, evidence, loudness, and downstream criteria |
| `templates/target-compatibility-manifest.json` | Named consumer, sourced requirements, technical limits, and authorized target lane |
| `templates/research-experiment-record.md` | Versioned, reproducible command experiment record |
### Existing automation and evals
@@ -67,6 +68,7 @@ This skill provides a repeatable intake-to-acceptance workflow. It separates tec
| `scripts/render-edl` | Validate single- or multi-source EDLs and emit non-executing concat-filter or concat-demuxer plans |
| `scripts/audio-inspect` | Bounded silence, loudness, peak/clipping, transcript-candidate, and podcast-plan evidence |
| `scripts/media-verify` | Evaluate output probe and review evidence against a declared acceptance contract |
| `scripts/target-compatibility` | Separate technical probe conformance from one named player's/editor's/host's result |
| `scripts/editorial-workflow-example` | Generate synthetic audio/video and exercise the complete workflow with durable evidence |
| `scripts/generate-media-fixtures` | Generate a bounded sanitized fixture battery and versioned evidence manifest |
| `evals/evals.json` | Output-quality cases for core FFmpeg, media evidence, video, podcast, EDL, safety, and acceptance behavior |
@@ -142,6 +144,15 @@ ffmpeg/scripts/vision-review-handoff private.mov --asset-id asset-017 \
The manifest covers only its listed samples. An authorized reviewer must add attributed observations before `import-vision-review` can link them to an EDL; sparse frames never prove absence throughout a video.
For destination-specific delivery, evaluate a sourced target contract and preserve target evidence separately:
```sh
ffmpeg/scripts/target-compatibility target.json output-probe.json \
--target-evidence target-result.json --json
```
The verdict applies only to the named target and version. A local FFmpeg decode pass does not substitute for import, playback, or ingest evidence from that consumer.
## Triggers
Load this skill for:
+3
View File
@@ -93,6 +93,7 @@ For shipped helper workflows, run the helper from the skill root with explicit o
- `templates/podcast-edit-plan.md` — mechanical, signal, and editorial audio decisions
- `templates/media-acceptance-report.md` — layered verification and criterion verdicts
- `templates/media-acceptance-contract.json` — machine-readable stream, format, evidence, loudness, and downstream requirements
- `templates/target-compatibility-manifest.json` — one named target, sourced requirements, technical constraints, and downstream lane
- `templates/research-experiment-record.md` — reproducible version/command/result record
Run `scripts/editorial-workflow-example` in a new or empty task-local directory when a reproducible synthetic integration proof is required. Its `PASS_WITH_UNVERIFIED_BOUNDARIES` result is deliberately narrower than editorial or destination acceptance.
@@ -105,6 +106,8 @@ Use `scripts/audio-inspect` for bounded silence, EBU R128, peak/clipping, and tr
Use `scripts/media-verify` with a declared acceptance contract, output FFprobe JSON, and optional evidence JSON. It reports every criterion independently as `PASS`, `FAIL`, `BLOCKED`, `UNVERIFIED`, or `NOT_APPLICABLE`; only a report with no failed or missing required evidence is an overall pass.
Use `scripts/target-compatibility` when acceptance names a real player, editor, host, archive, or service. Keep sourced technical requirements and local probe results separate from evidence produced by that exact consumer; a pass applies only to the named target/version.
Use `scripts/vision-review-handoff` to prepare bounded, privacy-safe frame packets for an authorized human or vision reviewer. Import only attributed reviewed observations with `scripts/import-vision-review`; treat proposed editorial consequences as evidence for review, never automatic decisions.
Copy a template into the task workspace and replace its placeholder/example values. Do not put private paths, media, transcripts, or review evidence in the public skill repository.
+13
View File
@@ -170,6 +170,19 @@
"links reviewed observations to named EDL events as evidence rather than automatic truth"
],
"case_set": "release"
},
{
"id": "named-target-compatibility",
"prompt": "Will this MP4 work everywhere? I have its FFprobe output and a successful local decode, and I need it accepted by a named editor whose documented limits are attached.",
"expected_output": "Build a versioned manifest for that one editor from official or explicitly observed requirements, evaluate probe-visible codec/profile, dimensions/rates, mapping, subtitles/metadata, duration, and size separately, and require evidence from importing or playing the exact artifact in the named editor. Report missing access or evidence as BLOCKED or UNVERIFIED and do not generalize the result.",
"assertions": [
"links each target requirement to official documentation or labeled observed behavior with an access date",
"checks codec profile, dimensions, rates, stream order, subtitles, metadata, duration, and size",
"separates local probe and decode evidence from the named target result",
"uses PASS, FAIL, BLOCKED, or UNVERIFIED for the exact target lane",
"does not claim compatibility with every platform and leaves upload operations to the owning platform skill"
],
"case_set": "release"
}
]
}
@@ -63,6 +63,19 @@ The contract can require exact stream order/counts, codecs, dimensions, pixel fo
The verifier emits one record per criterion with expected and observed values, evidence locator, boundary, reason, and one of `PASS`, `FAIL`, `BLOCKED`, `UNVERIFIED`, or `NOT_APPLICABLE`. A required stream that is absent fails; a required field or review artifact that cannot be observed remains unverified. The overall verdict follows the strongest unresolved state: `FAIL`, then `BLOCKED`, then `UNVERIFIED`, otherwise `PASS`.
## Named-target compatibility
For a real player, editor, host, archive, or service, copy `templates/target-compatibility-manifest.json` and retain requirements provenance separately from the output probe. Every requirement source needs an official URL or an observed-behavior locator, access date, and the exact claim it supports. Run:
```sh
scripts/target-compatibility TARGET.json OUTPUT-PROBE.json \
--target-evidence TARGET-EVIDENCE.json --json
```
The helper evaluates codec/profile, dimensions, pixel format, rates, audio layout, stream order, subtitles, chapters, metadata, duration, and size at the technical boundary. It reports import/playback/ingest evidence for exactly one named target in a separate result. Missing target evidence is `UNVERIFIED`; an explicitly unavailable authorized environment is `BLOCKED`; neither local decoding nor a pass in one consumer is generalized to another consumer.
A concrete FFplay lane may use the [official FFplay documentation](https://ffmpeg.org/ffplay.html) and record the installed version, exact artifact digest, invocation, interactive audio/video/subtitle checks, warnings, and result. The repository's automated tests are headless and do not exercise an authorized display/audio session, so that real playback lane remains explicitly unavailable in CI. Tests instead verify the contract mechanics against a recorded named-target evidence fixture. Platform upload/API actions remain in the owning platform skill.
Minimize reports before sharing: remove private paths, personal names, account identifiers, unnecessary transcript excerpts, and embedded metadata.
## Evidence and heuristic boundary
+220
View File
@@ -0,0 +1,220 @@
#!/usr/bin/env python3
"""Evaluate technical conformance separately from one named target consumer."""
from __future__ import annotations
import argparse
import json
import subprocess
import sys
import tempfile
from pathlib import Path
from typing import Any
VERDICTS = {"PASS", "FAIL", "BLOCKED", "UNVERIFIED"}
class ContractError(Exception):
pass
def load(path: str, label: str) -> dict[str, Any]:
try:
value = json.loads(Path(path).read_text())
except (OSError, json.JSONDecodeError) as exc:
raise ContractError(f"could not load {label}: {exc}") from exc
if not isinstance(value, dict):
raise ContractError(f"{label} root must be an object")
return value
def target_sources(target: dict[str, Any]) -> None:
basis = target.get("requirement_basis")
if basis not in {"official_documentation", "observed_behavior", "mixed"}:
raise ContractError(
"target requirement_basis must be official_documentation, observed_behavior, or mixed"
)
sources = target.get("sources")
if not isinstance(sources, list) or not sources:
raise ContractError("target requirements need at least one source record")
for source in sources:
if (
not isinstance(source, dict)
or not source.get("locator")
or not source.get("accessed_at")
or not source.get("claim")
):
raise ContractError("each target source needs locator, accessed_at, and claim")
if source.get("basis") not in {"official_documentation", "observed_behavior"}:
raise ContractError("each target source needs an explicit evidence basis")
def overall(*verdicts: str) -> str:
for verdict in ("FAIL", "BLOCKED", "UNVERIFIED"):
if verdict in verdicts:
return verdict
return "PASS"
def run_technical(contract: dict[str, Any], probe_path: str) -> dict[str, Any]:
isolated = dict(contract)
isolated["downstream"] = {}
with tempfile.NamedTemporaryFile(mode="w", suffix=".json") as temporary:
json.dump(isolated, temporary)
temporary.flush()
command = [
sys.executable,
str(Path(__file__).with_name("media-verify")),
temporary.name,
probe_path,
"--json",
]
result = subprocess.run(command, capture_output=True, text=True, check=False)
try:
report = json.loads(result.stdout)
except json.JSONDecodeError as exc:
raise ContractError("technical verifier did not return JSON") from exc
if result.returncode == 2:
raise ContractError(str(report.get("error", "invalid technical contract")))
report["boundary"] = "local_ffprobe_and_declared_technical_requirements"
return report
def add_maximum(criteria: list[dict[str, Any]], name: str, observed: Any, maximum: Any) -> str:
try:
actual = float(observed)
limit = float(maximum)
except (TypeError, ValueError):
verdict = "UNVERIFIED"
reason = "probe value or maximum is absent or nonnumeric"
else:
verdict = "PASS" if actual <= limit else "FAIL"
reason = None if verdict == "PASS" else "observed value exceeds target maximum"
criteria.append(
{
"criterion": name,
"boundary": "target_technical_policy",
"verdict": verdict,
"expected": {"maximum": maximum},
"observed": observed,
"evidence": "output probe",
"reason": reason,
}
)
return verdict
def downstream_result(
target: dict[str, Any], lane: dict[str, Any], evidence: dict[str, Any]
) -> dict[str, Any]:
target_id = target["id"]
result = evidence.get("target_consumer")
if isinstance(result, dict):
verdict = result.get("status")
if verdict not in VERDICTS:
verdict = "UNVERIFIED"
reason = "target-consumer status is missing or invalid"
elif result.get("target_id") != target_id:
verdict = "FAIL"
reason = "target-consumer evidence names a different target"
elif not result.get("artifact") or not result.get("method"):
verdict = "UNVERIFIED"
reason = "target-consumer evidence needs artifact identity and method"
else:
reason = result.get("reason")
return {
"target_id": target_id,
"target_version": result.get("target_version"),
"verdict": verdict,
"method": result.get("method"),
"artifact": result.get("artifact"),
"warnings": result.get("warnings", []),
"reason": reason,
"boundary": "named_target_consumer",
}
unavailable = lane.get("unavailable_reason")
return {
"target_id": target_id,
"target_version": None,
"verdict": "BLOCKED" if unavailable else "UNVERIFIED",
"method": lane.get("method"),
"artifact": None,
"warnings": [],
"reason": unavailable or "no evidence from the named target consumer was supplied",
"boundary": "named_target_consumer",
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("manifest")
parser.add_argument("output_probe")
parser.add_argument("--target-evidence")
parser.add_argument("--json", action="store_true")
args = parser.parse_args()
try:
manifest = load(args.manifest, "compatibility manifest")
probe = load(args.output_probe, "output probe")
evidence = load(args.target_evidence, "target evidence") if args.target_evidence else {}
if manifest.get("schema_version") != 1:
raise ContractError("compatibility manifest schema_version must be 1")
target = manifest.get("target")
contract = manifest.get("technical_requirements")
lane = manifest.get("target_lane")
if not isinstance(target, dict) or not target.get("id") or not target.get("name"):
raise ContractError("target needs id and name")
if not isinstance(contract, dict) or not isinstance(lane, dict):
raise ContractError("technical_requirements and target_lane must be objects")
target_sources(target)
technical = run_technical(contract, args.output_probe)
limits = manifest.get("target_limits", {})
if not isinstance(limits, dict):
raise ContractError("target_limits must be an object")
limit_verdicts: list[str] = []
if "maximum_duration_seconds" in limits:
limit_verdicts.append(
add_maximum(
technical["criteria"],
"target_maximum_duration",
probe.get("format", {}).get("duration"),
limits["maximum_duration_seconds"],
)
)
if "maximum_file_size_bytes" in limits:
limit_verdicts.append(
add_maximum(
technical["criteria"],
"target_maximum_file_size",
probe.get("format", {}).get("size"),
limits["maximum_file_size_bytes"],
)
)
technical["overall_verdict"] = overall(technical["overall_verdict"], *limit_verdicts)
target_result = downstream_result(target, lane, evidence)
verdict = overall(technical["overall_verdict"], target_result["verdict"])
payload = {
"ok": verdict == "PASS",
"schema_version": 1,
"target": target,
"requirements_provenance": target["sources"],
"technical_probe_result": technical,
"target_consumer_result": target_result,
"overall_verdict": verdict,
"boundary_statement": f"This result applies only to target {target['id']}; local FFprobe/FFmpeg success and this target result do not establish compatibility with any other player, editor, host, archive, or service.",
"platform_operation": "not_performed",
}
print(
json.dumps(payload, sort_keys=True, separators=(",", ":"))
if args.json
else json.dumps(payload, indent=2, sort_keys=True)
)
return 0 if verdict == "PASS" else 1
except ContractError as exc:
print(
json.dumps({"ok": False, "status": "INVALID_INPUT", "error": str(exc)}, sort_keys=True)
)
return 2
if __name__ == "__main__":
raise SystemExit(main())
+159
View File
@@ -1006,3 +1006,162 @@ def test_generate_media_fixtures_refuses_nonempty_workspace(tmp_path: Path) -> N
assert result.returncode == 2
assert "workspace must be absent or empty" in json.loads(result.stdout)["error"]
assert marker.read_text() == "preserve"
def compatibility_manifest() -> dict[str, object]:
return {
"schema_version": 1,
"target": {
"id": "reference-editor-3.2",
"name": "Reference Editor 3.2",
"requirement_basis": "mixed",
"sources": [
{
"basis": "official_documentation",
"locator": "https://vendor.example/3.2/media",
"accessed_at": "2026-09-08",
"claim": "H.264 High/AAC MP4 import requirements",
},
{
"basis": "observed_behavior",
"locator": "test-run:reference-editor-3.2:fixture-7",
"accessed_at": "2026-09-08",
"claim": "Exact fixture imported and played in the named build",
},
],
},
"technical_requirements": {
"schema_version": 1,
"required_streams": [
{
"type": "video",
"codec_name": "h264",
"profile": "High",
"width": 1920,
"height": 1080,
"pix_fmt": "yuv420p",
"avg_frame_rate": "30/1",
},
{
"type": "audio",
"codec_name": "aac",
"sample_rate": "48000",
"channels": 2,
"channel_layout": "stereo",
},
{"type": "subtitle", "codec_name": "mov_text"},
],
"stream_order": ["video", "audio", "subtitle"],
"forbidden_stream_types": ["data", "attachment"],
"format": {"format_name": "mov,mp4,m4a,3gp,3g2,mj2"},
"chapters": {"count": 0},
"metadata": {"required": {"title": "Delivery"}, "forbidden": ["comment"]},
"evidence": {
"decode": "optional",
"visual_review": "optional",
"audio_review": "optional",
},
},
"target_limits": {"maximum_duration_seconds": 60, "maximum_file_size_bytes": 5000000},
"target_lane": {"method": "import and playback", "authorization_required": True},
}
def compatibility_probe() -> dict[str, object]:
return {
"streams": [
{
"index": 0,
"codec_type": "video",
"codec_name": "h264",
"profile": "High",
"width": 1920,
"height": 1080,
"pix_fmt": "yuv420p",
"avg_frame_rate": "30/1",
},
{
"index": 1,
"codec_type": "audio",
"codec_name": "aac",
"sample_rate": "48000",
"channels": 2,
"channel_layout": "stereo",
},
{"index": 2, "codec_type": "subtitle", "codec_name": "mov_text"},
],
"format": {
"format_name": "mov,mp4,m4a,3gp,3g2,mj2",
"duration": "30.0",
"size": "4000000",
"tags": {"title": "Delivery"},
},
"chapters": [],
}
def test_target_compatibility_separates_probe_from_named_consumer(tmp_path: Path) -> None:
manifest = write_json(tmp_path / "target.json", compatibility_manifest())
output_probe = write_json(tmp_path / "probe.json", compatibility_probe())
target_evidence = write_json(
tmp_path / "target-evidence.json",
{
"target_consumer": {
"status": "PASS",
"target_id": "reference-editor-3.2",
"target_version": "3.2.1",
"artifact": "sha256:fixture-7",
"method": "import, timeline playback, and subtitle toggle",
"warnings": [],
}
},
)
result = run_script(
"target-compatibility",
str(manifest),
str(output_probe),
"--target-evidence",
str(target_evidence),
"--json",
)
assert result.returncode == 0, result.stdout + result.stderr
report = json.loads(result.stdout)
assert report["overall_verdict"] == "PASS"
assert report["technical_probe_result"]["overall_verdict"] == "PASS"
assert report["target_consumer_result"]["verdict"] == "PASS"
criteria = {item["criterion"]: item for item in report["technical_probe_result"]["criteria"]}
for expected in (
"video_0_profile",
"video_0_width",
"video_0_avg_frame_rate",
"stream_order",
"subtitle_0_codec_name",
"metadata_title",
"target_maximum_duration",
"target_maximum_file_size",
):
assert criteria[expected]["verdict"] == "PASS"
assert "only to target reference-editor-3.2" in report["boundary_statement"]
def test_target_compatibility_keeps_unavailable_lane_separate(tmp_path: Path) -> None:
manifest = compatibility_manifest()
manifest["target_lane"]["unavailable_reason"] = "headless CI has no authorized editor session"
probe_document = compatibility_probe()
probe_document["streams"][0]["profile"] = "Main"
probe_document["format"]["size"] = "6000000"
target_path = write_json(tmp_path / "target.json", manifest)
probe_path = write_json(tmp_path / "probe.json", probe_document)
result = run_script("target-compatibility", str(target_path), str(probe_path), "--json")
assert result.returncode == 1
report = json.loads(result.stdout)
assert report["technical_probe_result"]["overall_verdict"] == "FAIL"
assert report["target_consumer_result"]["verdict"] == "BLOCKED"
assert (
report["target_consumer_result"]["reason"] == "headless CI has no authorized editor session"
)
assert report["overall_verdict"] == "FAIL"
@@ -16,6 +16,8 @@
| Audio levels/listening | | | editorial | PASS/FAIL/BLOCKED/UNVERIFIED |
| Downstream consumer | | | downstream | PASS/FAIL/BLOCKED/UNVERIFIED/NOT_APPLICABLE |
Keep named-target evidence in a separate target-consumer record: target ID/version, exact artifact digest, method, warnings, requirements sources/access dates, and result. A local probe/decode pass is not the target result, and one target result does not establish universal compatibility.
## Unverified Boundaries
-
@@ -0,0 +1,35 @@
{
"schema_version": 1,
"target": {
"id": "named-target-and-version",
"name": "Named player, editor, host, archive, or service",
"requirement_basis": "official_documentation",
"sources": [
{
"basis": "official_documentation",
"locator": "https://official.example/media-requirements",
"accessed_at": "YYYY-MM-DD",
"claim": "Exact requirement supported by this source"
}
]
},
"technical_requirements": {
"schema_version": 1,
"required_streams": [
{"type": "video", "codec_name": "h264", "profile": "High", "width": 1920, "height": 1080, "pix_fmt": "yuv420p", "avg_frame_rate": "30000/1001", "tolerances": {"avg_frame_rate": 0.001}},
{"type": "audio", "codec_name": "aac", "sample_rate": "48000", "channels": 2, "channel_layout": "stereo"}
],
"stream_order": ["video", "audio"],
"forbidden_stream_types": ["data", "attachment"],
"format": {"format_name": "mov,mp4,m4a,3gp,3g2,mj2"},
"chapters": {"count": 0},
"metadata": {"required": {}, "forbidden": ["comment"]},
"evidence": {"decode": "optional", "visual_review": "optional", "audio_review": "optional"}
},
"target_limits": {"maximum_duration_seconds": 3600, "maximum_file_size_bytes": 1000000000},
"target_lane": {
"method": "Import, play, validate, or ingest the exact artifact in the named target",
"authorization_required": true,
"unavailable_reason": "Remove this field only when an authorized target environment is available"
}
}