magnus919_agent-skills/privacy-engineering/evals/evals.json

{
  "schema_version": 1,
  "skill_name": "privacy-engineering",
  "evals": [
    {
      "id": "analytics-telemetry-privacy",
      "prompt": "We are adding product analytics to our consumer finance app. We want to track feature usage, user flows, and conversion funnels. The app handles financial account data (balances, transactions, account numbers). Our product team wants a full tracking plan including page views, button clicks, and feature interaction events. What privacy requirements must we define before instrumentation begins, and what acceptance criteria should gate the analytics launch?",
      "expected_output": "A privacy requirements definition that addresses analytics telemetry with specific, verifiable criteria. The response: (1) defines a consent boundary — pre-consent events limited to strictly necessary (e.g., session-start, error) with no financial data properties; post-consent events may include feature usage but must exclude financial account data (balances, account numbers, transaction details) under all consent states; (2) specifies minimization — no speculative properties, each event property tied to a declared measurement purpose; (3) defines an aggregation threshold (minimum cohort size) below which metrics are not reported, to prevent re-identification; (4) specifies raw event retention period and deletion behavior when a user closes their account or revokes consent; (5) defines verifiable acceptance criteria such as 'financial account data (balances, account numbers, transaction amounts) does not appear in any analytics event payload, verified by schema inspection and event-stream sampling'; (6) routes any question about whether a specific financial regulation applies to legal-strategy. The response explicitly states that financial data must not appear in analytics under any consent state — not just 'be careful with it.'",
      "assertions": [
        "The response defines a consent boundary with distinct pre-consent, post-consent, and never-measured categories",
        "Financial account data (balances, account numbers, transactions) is classified as never-measured — not conditional on consent",
        "The response defines an aggregation threshold to prevent re-identification from small cohorts",
        "The response specifies raw event retention period and deletion behavior for account closure and consent revocation",
        "At least one acceptance criterion is verifiable with a specific verification method (e.g., schema inspection, event-stream sampling) and a pass/fail condition",
        "The response does not recommend collecting financial data in analytics under any circumstances"
      ]
    },
    {
      "id": "agent-traces-privacy",
      "prompt": "Our customer-support AI agent handles user conversations that include PII (names, email addresses, order numbers, shipping addresses). The agent uses an LLM and tools to look up orders, process returns, and update account details. We need to store agent traces for debugging and evaluation. The traces include the full LLM conversation (user messages, system prompt, assistant responses) and tool-call arguments and responses. What privacy requirements must we define for these traces, and what acceptance criteria should gate trace storage?",
      "expected_output": "A privacy requirements definition for agent traces (LLM conversation logs, tool-call history) with specific, verifiable criteria. The response: (1) identifies that agent traces contain PII — user messages include names, emails, addresses, order numbers; tool-call arguments and responses include account data; system prompts may include context with PII; (2) defines minimization requirements — strip prompt content and tool arguments before storage unless each field is justified by a specific purpose (debugging, eval); (3) requires redaction at collection time (before traces leave the execution boundary), not post-storage filtering; (4) defines trace retention period tied to purpose (e.g., 72 hours for debugging, longer for consented eval datasets); (5) requires that traces be deletable by user ID and that deletion be verifiable; (6) defines consent: user-facing agents must obtain consent before trace collection; consent revocation must stop future collection and trigger deletion; (7) produces verifiable acceptance criteria such as 'PII fields (name, email, address) are redacted from trace payloads before storage, verified by sampling 100 traces and confirming no PII in stored payload.' The response explicitly distinguishes between redaction (pre-storage) and filtering (post-storage) and requires the former.",
      "assertions": [
        "The response identifies that agent traces contain PII in user messages, tool arguments, tool responses, and system prompts",
        "The response requires redaction at collection time (before traces leave the execution boundary), not post-storage filtering",
        "The response defines a trace retention period tied to a specific purpose (debugging, evaluation) with a specific duration",
        "The response requires traces to be deletable by user ID with verifiable deletion",
        "The response addresses consent for trace collection in user-facing agents and consent revocation behavior",
        "At least one acceptance criterion is verifiable with a specific verification method (e.g., trace sampling) and a pass/fail condition"
      ]
    },
    {
      "id": "multi-tenant-data-isolation",
      "prompt": "Our B2B SaaS platform hosts data for multiple enterprise customers in a shared database (tenant_id column on every table). A new feature allows customer admins to run custom reports with a query builder. We must ensure that Tenant-A admins can never see Tenant-B data through the report builder, directly or indirectly (e.g., through aggregate queries, error messages, or timing side channels). The query builder generates SQL dynamically. Define the privacy acceptance criteria and verification approach for tenant isolation.",
      "expected_output": "A privacy engineering response focused on tenant isolation with verifiable acceptance criteria. The response: (1) defines the tenant isolation requirement as a hard boundary — Tenant-A queries must never return Tenant-B data, full stop; (2) identifies attack vectors beyond simple WHERE clauses: aggregate queries (COUNT, AVG across tenants), error messages that leak tenant-B metadata, timing side channels, query-plan inspection; (3) defines acceptance criteria such as 'every generated SQL query includes a tenant_id filter applied at the query level (not application-level post-filter), verified by SQL audit log inspection'; 'cross-tenant aggregate queries are either prevented (query rejected) or scoped to the authenticated tenant only, verified by executing cross-tenant aggregate attempts and confirming zero Tenant-B rows in results'; 'error messages do not disclose other-tenant data or metadata, verified by fault-injection testing'; (4) specifies verification: integration tests that authenticate as Tenant-A and attempt to access Tenant-B data through every available query path, expecting zero cross-tenant data in any response; (5) defines the isolation mechanism as a database-level enforcement (row-level security, tenant-scoped views, or middleware-enforced tenant filtering), not application-level post-filtering. The response explicitly states that application-level post-filtering (retrieve all rows, then filter in app code) is insufficient for tenant isolation.",
      "assertions": [
        "The response defines tenant isolation as a hard boundary with a 'zero cross-tenant data' pass/fail condition",
        "The response identifies attack vectors beyond direct SELECT: aggregate queries, error-message leakage, timing side channels",
        "The response requires tenant filtering at the database or query level, not application-level post-filtering",
        "The response defines verifiable acceptance criteria with specific verification methods (SQL audit log, fault-injection, integration tests)",
        "The response specifies that error messages must not disclose other-tenant data or metadata",
        "At least one acceptance criterion includes a measurable pass/fail condition directly testable through automated verification"
      ]
    },
    {
      "id": "deletion-revocation-verification",
      "prompt": "Our social media platform allows users to delete their accounts. Our privacy policy states: 'When you delete your account, your data is removed from our systems within 30 days.' Data lives in: PostgreSQL (user profiles, posts), Elasticsearch (search index), S3 (uploaded photos), Redis (session cache), Kafka (event stream with 7-day retention), nightly PostgreSQL backups (retained 90 days), and a BigQuery analytics warehouse (derived tables). We have never verified that deletion actually works end-to-end. Design a retention/deletion verification plan with measurable success conditions.",
      "expected_output": "A retention/deletion verification plan with measurable, testable success conditions — not policy prose. The response: (1) identifies every store in the data inventory (PostgreSQL, Elasticsearch, S3, Redis, Kafka, backups, BigQuery) and defines a deletion SLA per store: primary stores (PostgreSQL, Elasticsearch, S3) within 72 hours of verified account closure; cache (Redis) within 1 hour; event stream (Kafka) aged out per 7-day retention; backups within 30 days (aged out of rotation); analytics warehouse derived tables within 7 days; (2) defines a verification procedure: pre-verification baseline (record row counts, sample data), trigger deletion, wait for cascading deletions, execute post-verification audit per store with specific queries; (3) defines measurable success conditions such as 'user PII row count = 0 in PostgreSQL after 72h, verified by SELECT COUNT(*)' and 'user photos = 0 objects in S3 bucket after 72h, verified by bucket listing'; (4) identifies that the 30-day privacy-policy statement is ambiguous — 'removed from our systems' must be decomposed into per-store SLAs; (5) requires exercise evidence: the plan must be executed against a test account, not only documented; (6) addresses consent revocation: if a user revokes consent (but doesn't delete the account), what data stops being collected and what existing data is deleted? The response explicitly states that a policy statement without a verification plan is not an engineering artifact.",
      "assertions": [
        "The response produces a verification plan with per-store deletion SLAs, not a single '30 days' policy restatement",
        "Every store in the data inventory (PostgreSQL, Elasticsearch, S3, Redis, Kafka, backups, BigQuery) has a defined deletion SLA and verification method",
        "The response identifies that 'removed from our systems within 30 days' is ambiguous and decomposes it into per-store SLAs",
        "The response defines measurable success conditions with specific queries (e.g., SELECT COUNT(*), bucket listing) and numeric pass/fail thresholds",
        "The response requires exercise evidence — the plan must be executed against a test account, not only documented",
        "The response addresses consent revocation behavior distinct from account deletion",
        "The response explicitly states that a policy statement without a verification plan is not an engineering artifact"
      ]
    },
    {
      "id": "residency-constraint-engineering",
      "prompt": "Our application serves users in the EU and the US. We store user data in AWS us-east-1 (primary region) with a read replica in eu-west-1 for latency. The EU read replica contains full copies of all user data, including EU user PII. Our legal team has advised that EU user PII must stay within the EU. However, we also have a nightly backup pipeline that copies all data (both regions) to S3 in us-east-1, and our analytics pipeline (BigQuery, US multi-region) ingests all user events including PII fields. Define the residency requirements and acceptance criteria, and identify the gaps.",
      "expected_output": "A residency constraint engineering analysis that identifies specific gaps and defines verifiable acceptance criteria. The response: (1) identifies three residency violations: (a) EU user PII in the us-east-1 primary database — the primary is the source of truth and EU user data must not reside there; (b) backups of EU user data stored in us-east-1 S3 — backup residency must match data residency; (c) EU user PII in BigQuery US multi-region — analytics must either exclude EU user PII fields or use an EU-region analytics store; (2) defines the target state: EU user PII stored only in EU regions (primary in eu-west-1 or equivalent), read replicas for EU users in EU regions, backups in EU regions, analytics either with PII stripped before US ingestion or in EU-region store; (3) defines verifiable acceptance criteria such as 'EU user PII rows exist only in EU-region database instances, verified by infrastructure audit of database instance regions and row-level residency tagging'; 'EU user PII does not appear in us-east-1 S3 backup manifests, verified by backup manifest inspection'; 'EU user PII fields (name, email, address) are absent from BigQuery US multi-region, verified by schema inspection and column-level audit'; (4) identifies that the read replica in eu-west-1 is actually residency-compliant but the primary in us-east-1 is not — the replication direction is backwards for EU residency; (5) does not provide legal interpretation of 'must stay within the EU' — it accepts the legal team's determination as input and engineers against it. The response routes any question about whether a specific transfer mechanism (SCCs, BCRs) would permit the us-east-1 storage to legal-strategy.",
      "assertions": [
        "The response identifies at least three specific residency violations (us-east-1 primary, us-east-1 backups, BigQuery US multi-region)",
        "The response identifies that the replication direction is backwards — having an EU read replica does not satisfy EU residency when the primary is in the US",
        "The response defines verifiable acceptance criteria with specific verification methods (infrastructure audit, backup manifest inspection, schema inspection)",
        "The response does not provide legal interpretation of 'must stay within the EU' — it accepts the legal team's determination as input",
        "At least one acceptance criterion includes a measurable pass/fail condition for residency compliance",
        "The response does not recommend 'just use SCCs' or interpret transfer mechanisms — it routes legal questions to legal-strategy"
      ]
    },
    {
      "id": "jurisdiction-escalation-legal-review",
      "prompt": "Our company is based in the US and we are launching in Brazil. Our legal team has not yet provided guidance on the LGPD (Brazil's data protection law). The product team is asking us to define the privacy requirements for the Brazil launch — data collection, retention, deletion, consent, and data subject rights. We have our existing GDPR-based privacy engineering artifacts. The product manager says 'LGPD is basically the same as GDPR, just use those requirements.' How should we approach this?",
      "expected_output": "A response that explicitly refuses to assume LGPD is equivalent to GDPR and escalates the jurisdiction-specific interpretation to legal counsel. The response: (1) rejects the 'LGPD is basically GDPR' assumption — while LGPD and GDPR share principles, they differ in specifics (legal bases, DPO requirements, breach notification timelines, cross-border transfer rules, enforcement); (2) identifies that the privacy engineering artifacts (data-lifecycle records, acceptance criteria, deletion verification plans) can be prepared as templates — the structure is reusable, but the specific requirements (retention periods, consent conditions, deletion timelines, legal bases) must be populated by legal-strategy after LGPD-specific interpretation; (3) defines what privacy engineering CAN do now: map data flows for the Brazil launch, inventory data categories, prepare template acceptance criteria with placeholder values for legal to fill, design the deletion verification plan structure; (4) defines what privacy engineering MUST NOT do: assert that GDPR compliance equals LGPD compliance, define retention periods or consent conditions for LGPD without legal input, claim that existing GDPR artifacts satisfy LGPD requirements; (5) escalates to legal-strategy with specific questions: what are the LGPD legal bases applicable to our processing? What are the LGPD data subject rights and response timelines? What are the LGPD cross-border transfer requirements? Are there sector-specific requirements for our industry? (6) routes the jurisdiction-specific interpretation to legal-strategy explicitly — this is the core test of the escalation boundary. The response explicitly states that this skill does not provide legal advice and does not determine whether one jurisdiction's law is equivalent to another's. Claims are scoped to the harness, model, fixtures, and revision under test.",
      "assertions": [
        "The response explicitly refuses to assume LGPD is equivalent to GDPR and rejects 'just use GDPR requirements'",
        "The response identifies specific privacy engineering artifacts that CAN be prepared as templates vs. what requires legal input",
        "The response explicitly escalates jurisdiction-specific interpretation to legal-strategy with specific, answerable questions",
        "The response does NOT assert that GDPR compliance equals LGPD compliance",
        "The response does NOT define LGPD-specific retention periods, consent conditions, or legal bases",
        "The response states that this skill does not provide legal advice and does not determine regulatory equivalence"
      ]
    }
  ]
}