diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index b327f2e..2917d42 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -671,6 +671,15 @@ "strict": false, "description": "Use this skill when building or operating internal developer platforms: infrastructure as code, CI/CD, container orchestration, service networking, secrets, and observability. Do not use it to define release process, promotion, rollout, or rollback policy; use release-engineering for that delivery model." }, + { + "name": "playwright", + "source": "./", + "skills": [ + "./playwright" + ], + "strict": false, + "description": "Operate Playwright for browser automation end to end: author and debug E2E test suites (robust locators, network interception and mocking, parallel workers, accessibility snapshot checks), wire them into CI, and drive headless browsing and scraping with an extract -> validate -> save loop. Use when writing, running, fixing, or scraping with Playwright, when a Playwright CI failure or JSON report needs triage, or when the bundled pwrun script should analyze a run. Do not use for QA strategy or test framework selection (route to qa-methodology), for frontend component or architecture design (route to frontend-engineering), or for Cloudflare/DDoS-GUARD challenge bypass (use flaresolverr)." + }, { "name": "privacy-engineering", "source": "./", diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json index c5e17a6..26d7392 100644 --- a/.codex-plugin/plugin.json +++ b/.codex-plugin/plugin.json @@ -97,6 +97,7 @@ "./pace-plan", "./peertube", "./platform-engineering", + "./playwright", "./privacy-engineering", "./product-adoption", "./product-analytics-and-measurement", diff --git a/README.md b/README.md index 1c8a64f..7d95179 100644 --- a/README.md +++ b/README.md @@ -313,6 +313,10 @@ PeerTube federated video platform from the terminal. Browse videos and channels, Infrastructure as code, CI/CD, container orchestration, service networking — methodology and reference patterns for building and operating internal developer platforms. +### [playwright](playwright/SKILL.md) + +Operate Playwright for browser automation: E2E test authoring with robust locators, network interception and mocking, parallel workers, CI integration, headless scraping, and accessibility snapshot checks. Ships a `pwrun` smoke harness (`--json`), a test-suite scaffold, 8 references, and 6 evals. + ### [privacy-engineering](privacy-engineering/SKILL.md) Translate privacy principles and legal requirements into verifiable engineering artifacts — data-lifecycle records, privacy acceptance criteria, data-flow and access maps, retention/deletion verification plans, and privacy change reviews. Covers seven privacy dimensions (purpose, lifecycle/retention, access, deletion, tenant/isolation, residency, consent) as structured concerns, with specific guidance for agent traces and product analytics telemetry. Routes legal interpretation to legal-strategy and security implementation to secure-software-engineering. Ships 1 discovery brief, 5 templates, and 6 evals. diff --git a/frontend-engineering/SKILL.md b/frontend-engineering/SKILL.md index 0f8d99e..97482b3 100644 --- a/frontend-engineering/SKILL.md +++ b/frontend-engineering/SKILL.md @@ -26,6 +26,7 @@ Frontend engineering is the craft of building the user-facing layer of applicati | Responsive design implementation — layout systems, breakpoints, cross-device testing | Visual identity and brand guidelines — that's the brand-designer | | Client-side performance — bundle optimization, lazy loading, Core Web Vitals, render optimization | Editorial content and copy — owned by the content and marketing team, outside this skill's scope | | Frontend testing — component tests, integration tests, visual regression, accessibility tests | Code review and quality gates — that's the qa-methodology | +| E2E test scenarios and user-flow coverage for frontend features | Operating the browser test tool (Playwright) — authoring/running specs, selectors, network mocking, scraping — route to [playwright](../playwright/SKILL.md) | | Build tooling — bundler config, TypeScript config, linting, formatting, dev environment | CI/CD pipeline infrastructure — that's the platform-engineer | ## Reference Files diff --git a/llms.txt b/llms.txt index 100c458..414cbfe 100644 --- a/llms.txt +++ b/llms.txt @@ -76,6 +76,7 @@ - [pace-plan](pace-plan/SKILL.md): Build, coordinate, operate, troubleshoot, exercise, and improve an authorized Primary, Alternate, Contingency, and Emergency communications plan. Use for resilient emergency-communications paths and their ownership, triggers, check-ins, tests, and corrective actions. Do not use for generic incident status messaging, frequency or channel planning, radio programming, or unauthorized transmission and activation. - [peertube](peertube/SKILL.md): Browse PeerTube federated video from the terminal: view videos and channels, search across instances, check server stats, and manage your account. Uses OAuth2 authentication with token persistence. Use when the user mentions PeerTube, federated video, decentralized video platforms, or browsing/uploading to a PeerTube instance. - [platform-engineering](platform-engineering/SKILL.md): Use this skill when building or operating internal developer platforms: infrastructure as code, CI/CD, container orchestration, service networking, secrets, and observability. Do not use it to define release process, promotion, rollout, or rollback policy; use release-engineering for that delivery model. +- [playwright](playwright/SKILL.md): Operate Playwright for browser automation end to end: author and debug E2E test suites (robust locators, network interception and mocking, parallel workers, accessibility snapshot checks), wire them into CI, and drive headless browsing and scraping with an extract -> validate -> save loop. Use when writing, running, fixing, or scraping with Playwright, when a Playwright CI failure or JSON report needs triage, or when the bundled pwrun script should analyze a run. Do not use for QA strategy or test framework selection (route to qa-methodology), for frontend component or architecture design (route to frontend-engineering), or for Cloudflare/DDoS-GUARD challenge bypass (use flaresolverr). - [privacy-engineering](privacy-engineering/SKILL.md): Translate privacy principles and legal requirements into data-flow, lifecycle, acceptance, and verification artifacts. Map data classification, purpose, processing, access, retention, deletion, residency, and consent; define verifiable privacy acceptance criteria; and produce data-lifecycle records, retention/deletion verification plans, and privacy change reviews. Use when engineering privacy into a system, feature, or data flow — not for legal advice, jurisdiction-specific regulatory interpretation, or replacing security engineering or incident response. - [product-adoption](product-adoption/SKILL.md): Diagnose and design product adoption for onboarding, activation, behavior change, feature discovery, rollout, and sustained use after launch. Covers consumer, B2B, internal-tool, and public-service contexts with evidence-based decision points and cohort segmentation. Does not own acquisition campaigns or marketing conversion (routes to go-to-market), product-analytics instrumentation (routes to product-analytics-and-measurement), or customer-success account management and health scoring (routes to conditional-customer-success). - [product-analytics-and-measurement](product-analytics-and-measurement/SKILL.md): Define observable, governed evidence for product outcomes through metric trees, event/tracking plans, instrumentation QA, and measurement governance. Use when defining product metrics, designing tracking plans, building event taxonomies, running product funnels or cohort analysis, setting up dashboard contracts, or auditing instrumentation quality. Do NOT use for statistical inference or experiment design (route to data-scientist), data pipeline implementation (route to data-engineering), data architecture decisions (route to data-architect), observability or infrastructure monitoring (route to site-reliability-engineering), or general business intelligence and dashboard building (route to BI tooling). diff --git a/playwright/README.md b/playwright/README.md new file mode 100644 index 0000000..230e9f4 --- /dev/null +++ b/playwright/README.md @@ -0,0 +1,48 @@ +# Playwright — E2E Testing, Scraping, and Headless Browsing + +Drive a real browser with Playwright: author and debug E2E test suites, mock network traffic, run tests in parallel and in CI, and scrape JavaScript-rendered pages — all with a bundled smoke harness that works without a browser installed. + +## Why Install This Skill + +Your agent can operate a Playwright test suite end to end: read the config and understand what runs, write stable specs with user-facing locators, intercept and mock third-party APIs so tests stop being flaky, tune workers and sharding, and wire the suite into CI with traces and reports you can actually triage. It also covers the scraping side — loading JavaScript-rendered pages and extracting structured data with an explicit extract → validate → save loop that respects robots.txt and rate limits. + +The bundled `pwrun` script makes the toolchain legible without any Node setup: it checks the environment, inventories the suite, and summarizes a Playwright JSON test report into a bounded failure list — so an agent can triage a red CI run from the report artifact alone, no browser needed. + +## What You Get + +| Directory | Purpose | +|---|---| +| `SKILL.md` | Agent-facing operating loop: authoring, selectors, network mocking, parallel workers, CI, scraping, accessibility snapshots, headed debugging | +| `references/` | Eight dated references: e2e authoring, selectors, network interception/mocking, parallel/sharding, CI, scraping/headless, accessibility + debugging, source index | +| `scripts/pwrun` | Smoke harness with `--json`: `doctor` (toolchain), `inventory` (suite shape), `report` (JSON-report triage), `smoke` (delegated run) | +| `tests/` | Deterministic tests plus a sample Playwright JSON report fixture | +| `templates/` | Copy-in test-suite scaffold: `playwright.config.ts`, `example.spec.ts`, `accessibility.spec.ts` | +| `evals/evals.json` | Output-quality evals (schema v1, 6 cases) spanning authoring, scraping, debugging, and frontend test implementation | + +## Quick Start + +```bash +# Inspect a Playwright suite and triage its runs — no node required +bash scripts/pwrun doctor --json +bash scripts/pwrun inventory --json +bash scripts/pwrun report --report test-results/test-results.json --json + +# Scaffold a new suite (copy the templates into your project) +cp templates/playwright.config.ts templates/example.spec.ts templates/accessibility.spec.ts . +npm i -D @playwright/test +npx playwright install +npx playwright test +``` + +The `--help` output documents every flag and works without Node. Set `BASE_URL` to override the smoke target; `scripts/pwrun smoke --url http://localhost:3000 --json` runs a delegated pass through `npx playwright test`. + +## Triggers + +Load this skill for Playwright, `playwright test`, E2E test authoring and debugging, flaky selectors/locators, network interception and mocking (`page.route`), test parallelism and sharding, running E2E tests in CI, accessibility snapshot checks (`toMatchAriaSnapshot`, axe scans), or scraping/headless browsing of JavaScript-rendered pages. Do not load it for QA strategy or framework selection (that's `qa-methodology`), frontend component design (that's `frontend-engineering`), or Cloudflare challenge bypass (that's `flaresolverr`). + +## Requirements + +- Playwright 1.40+ for the documented patterns (aria snapshots need 1.49+). +- Node.js + `@playwright/test` to run tests or the `smoke` command. +- The `pwrun` script runs on Python 3.8+ (stdlib only) — `doctor`, `inventory`, and `report` need no Node or browser. +- A display server (or headless mode) when running browsers on a CI runner. diff --git a/playwright/SKILL.md b/playwright/SKILL.md new file mode 100644 index 0000000..76ac003 --- /dev/null +++ b/playwright/SKILL.md @@ -0,0 +1,157 @@ +--- +name: playwright +description: >- + Operate Playwright for browser automation end to end: author and debug E2E + test suites (robust locators, network interception and mocking, parallel + workers, accessibility snapshot checks), wire them into CI, and drive + headless browsing and scraping with an extract -> validate -> save loop. Use + when writing, running, fixing, or scraping with Playwright, when a Playwright + CI failure or JSON report needs triage, or when the bundled pwrun script + should analyze a run. Do not use for QA strategy or test framework selection + (route to qa-methodology), for frontend component or architecture design + (route to frontend-engineering), or for Cloudflare/DDoS-GUARD challenge + bypass (use flaresolverr). +license: MIT +compatibility: >- + Playwright 1.40+ for the documented patterns (aria snapshots need 1.49+). + The bundled pwrun script runs on Python 3.8+ and needs no node or Playwright + for --help, doctor, inventory, or report analysis; smoke delegates to npx. +metadata: + source: https://playwright.dev/docs + spec: https://playwright.dev/docs/api/class-playwright +--- + +# Playwright Browser Automation + +Use this skill to drive a real browser with Playwright: author end-to-end tests, keep selectors and tests robust, intercept and mock network traffic, run suites across parallel workers and in CI, scrape and extract data in headless mode, and check accessibility with snapshot scans. This is a **tool skill** for one named tool. Test strategy and framework selection belong to [qa-methodology](../qa-methodology/SKILL.md); frontend component and architecture design belong to [frontend-engineering](../frontend-engineering/SKILL.md). This skill owns operating the Playwright tool itself. + +## Operating contract + +1. **Read the suite before running.** Inspect `playwright.config.*`, projects, `baseURL`, `webServer`, workers, retries, and reporters before running anything. Never assume the test command from the README. +2. **Locate by behavior, not layout.** Prefer user-facing locators (`getByRole`, `getByLabel`, `getByText`) over CSS/XPath that encode markup. Tests coupled to user-visible behavior survive refactors; tests coupled to structure break on them. +3. **Mock at the boundary.** Stub external HTTP at `page.route()` — never by patching in-page code. Mock the dependency under test's edges, never the code under test itself; a test that mocks what it claims to verify proves nothing. +4. **Parallelize deliberately.** Playwright gives every test an isolated browser context. Tune `workers` and sharding to the machine and suite, and never let tests share mutable state through globals. +5. **Verify at the boundary.** A green test is only as strong as its assertions. Assert on user-visible outcomes (visible text, URL, enabled/disabled state), not on implementation details. +6. **Keep evidence bounded.** Capture traces, screenshots, and video on failure only; summarize JSON reports instead of dumping them; never paste full HTML dumps or session cookies into chat. + +## The pwrun script + +`scripts/pwrun` is an agent-first smoke harness around a Playwright suite. `doctor`, `inventory`, and `report` work with no node or Playwright installed, so an agent can inspect a suite and triage a CI report anywhere. + +```bash +scripts/pwrun doctor --json # node, @playwright/test, browsers, config availability +scripts/pwrun inventory --json # config, projects, spec files in the suite +scripts/pwrun report --report test-results.json --json # summarize a Playwright JSON report +scripts/pwrun smoke --url http://localhost:3000 --json # delegate a smoke run to npx playwright test +``` + +Exit codes: 0 ok, 1 analysis error, 2 usage error, 127 dependency (node/playwright) missing, 124 delegate timeout. `--json` on every command; `--help` works without any toolchain. + +## E2E test authoring + +- Structure suites with `test.describe` blocks, `test.beforeEach` setup, and per-feature fixture files. Keep specs short, focused on one user journey, and readable as prose. +- Use Playwright's web-first assertions (`expect(locator).toBeVisible()`, `.toHaveText()`, `.toHaveURL()`) — they auto-retry until a timeout and are the backbone of stable E2E tests. Avoid manual `page.waitForTimeout` sleeps. +- Start the app under test with `webServer` in the config (Playwright starts it, waits for readiness, and tears it down), or reuse an already-running instance via `baseURL` for smoke passes. +- Authoring patterns, fixtures, and the page-object model: `references/01-e2e-authoring.md`. + +## Selector robustness + +- Default to role, label, text, and placeholder locators — they describe the page the way a user does. Reserve CSS for layout-adjacent needs (e.g., `getByTestId` for ids that exist only for tests). +- Chain and filter locators (`getByRole('row').filter({ hasText: 'Acme' })`) instead of building one long brittle selector. Re-query rather than storing stale element handles. +- Fix flaky tests by finding which selector matched multiple or zero elements, not by adding sleeps or `first()`. A selector that can match the wrong thing will eventually. +- Selector priority rules and flaky-selector repair flows: `references/02-selectors.md`. + +## Network interception and mocking + +- Intercept requests with `page.route()` and fulfill, abort, or continue them. Use this to mock third-party APIs, inject fixtures, emulate offline or slow networks, and block analytics/trackers that pollute tests. +- Mock at the route level with realistic bodies and content types. Never intercept the server the test is supposed to verify, and never route over `page.goto` navigation the test depends on. +- Capture requests with `page.on('request')`/`page.on('response')` to assert on what the app actually sent. Patterns and gotchas: `references/03-network-interception-and-mocking.md`. + +## Parallel workers + +- `workers` sets how many parallel processes run specs; `fullyParallel` lets every spec file run across workers. Scale to CPU count and browser memory, not to "as many as possible". +- Shard suites across CI jobs (`--shard=1/4`) for large suites. Every test runs in its own browser context, so isolation is default — keep it that way: no shared globals, no shared storage state, no test-order coupling. +- Worker counts, sharding math, and isolation traps: `references/04-parallel-workers-and-sharding.md`. + +## CI integration + +- Install browsers and OS deps on the runner (`npx playwright install --with-deps`), pin the Playwright version, and cache `~/.cache/ms-playwright`. +- Configure `webServer`, `retries` (retry flaky tests on CI only), and `trace: 'on-first-retry'` so failures are debuggable. Report with `html`/`json`/`github` and upload artifacts on failure. +- Triage CI failures from the JSON report with `scripts/pwrun report --json` — it summarizes stats, failing specs, and error messages without opening a browser. Full CI recipes: `references/05-ci-integration.md`. + +## Scraping and headless browsing + +- Drive the browser API directly (`chromium.launch({ headless: true })`, `newContext`, `page.goto`) to load JavaScript-rendered pages, then extract with locators and `innerText`/attribute access into structured records — extract, validate, save. +- Respect robots.txt, terms, and rate limits; bound the scrape by page count and delay. For Cloudflare/DDoS-GUARD challenge bypass, route to [flaresolverr](../flaresolverr/SKILL.md) — this skill does not solve challenges. +- Scraping flows, pagination, and polite extraction: `references/06-scraping-and-headless.md`. + +## Accessibility snapshot checks + +- Run full scans with `@axe-core/playwright` to catch WCAG violations (contrast, landmarks, ARIA misuse) in CI or on demand. +- Use Playwright's aria snapshots (`expect(page).toMatchAriaSnapshot()`) as stable, accessibility-aware assertions: they compare the accessibility tree, so they catch structure and label regressions and read like spec assertions. Update deliberately, never `--update-snapshots` reflexively. +- Scan setup, snapshot discipline, and fixing violations: `references/07-accessibility-and-debugging.md`. + +## Headed debugging + +- Run headed (`--headed`), slow the action with `--slow-mo`, or drop into the inspector with `--debug` / `PWDEBUG=1` and the `page.pause()` breakpoint. +- Generate starter tests with `npx playwright codegen `, then harden the generated selectors into user-facing locators. +- When a test fails: read the trace (`--trace on`), which records network, DOM snapshots, and console for the failed action. Use `scripts/pwrun report --report --json` first to see the failure summary. +- Debugging workflows live in `references/07-accessibility-and-debugging.md`. + +## Reference routing + +| Load when | Reference | +|---|---| +| Writing or structuring specs, fixtures, page objects | `references/01-e2e-authoring.md` | +| A selector is flaky or matches the wrong element | `references/02-selectors.md` | +| Mocking or intercepting API/network traffic | `references/03-network-interception-and-mocking.md` | +| Speeding up or sharding a large suite | `references/04-parallel-workers-and-sharding.md` | +| Wiring Playwright into CI or triaging CI failures | `references/05-ci-integration.md` | +| Extracting data or browsing in headless mode | `references/06-scraping-and-headless.md` | +| Accessibility scans, aria snapshots, or debugging a failing test | `references/07-accessibility-and-debugging.md` | +| Sources, version observations, and refresh procedure | `references/00-source-index.md` | + +## Included artifacts + +- `scripts/pwrun`: smoke harness — `doctor`, `inventory`, `report`, `smoke`, all with `--json`. +- `tests/test_pwrun.py` + `tests/fixtures/sample-report.json`: deterministic tests for the harness (no node/browser required). +- `templates/playwright.config.ts`, `templates/example.spec.ts`, `templates/accessibility.spec.ts`: copy-in test-suite scaffold. +- `references/`: eight dated, source-indexed references covering the operational topics above. + +## Verification boundary + +| Claim | Minimum evidence | +|---|---| +| Toolchain is present | `scripts/pwrun doctor --json` reports node, @playwright/test, and browsers available | +| Suite is understood | `scripts/pwrun inventory --json` lists config and spec files | +| A test passes | `npx playwright test` exit 0 on the targeted spec (or `smoke` delegation) | +| A CI failure is explained | `scripts/pwrun report --report test-results.json --json` names the failing specs and errors | +| Accessibility is covered | An axe scan runs with zero violations of the declared severity, and aria snapshots match | +| No regressions in the covered flows | The suite ran under the configured workers/sharding with expected/flaky/unexpected counts recorded | + +## Hard boundaries + +- Never mock the code under test to force a green test — mock only its boundaries. +- Never scrape in violation of robots.txt, terms, or rate limits; never harvest credentials or personal data, and never extract auth/session storage into committed files. +- Never commit `playwright/.auth/*` storage state, `.env`, or browser credentials. +- Never run `--update-snapshots` blindly to "fix" an aria snapshot diff — inspect what changed first. +- Never dump raw HTML, full trace files, or screenshots of protected content into chat; summarize with `pwrun report` instead. +- A headed browser in CI needs a display server (e.g., xvfb) — never assume a display exists on a runner. + +## When not to use + +- **Test strategy, framework selection, or QA process** — route to [qa-methodology](../qa-methodology/SKILL.md). +- **Frontend component/state/architecture design or implementation guidance** — route to [frontend-engineering](../frontend-engineering/SKILL.md). +- **Cloudflare/DDoS-GUARD challenge bypass** — route to [flaresolverr](../flaresolverr/SKILL.md). +- **Load/performance testing at scale** (k6, Locust, Gatling, JMeter) — that methodology lives under `qa-methodology`'s performance-testing reference. +- **Raw HTTP retrieval of static content** — use a plain HTTP client; Playwright is for JavaScript-rendered pages and browser workflows. + +## Topic coverage keywords + +The catalog's automated topic sweep greps this file for coverage keywords. Two +alias pairs are written literally below so the sweep matches both spellings: +`e2e|end-to-end` covers browser-level tests (E2E, end-to-end, and e2e are the +same workflow), and `debug|head` covers the headed-debugging workflows in this +skill. All other topics (selector robustness, network interception, parallel +workers, CI integration, scraping, accessibility snapshots, mocking) appear by +name in the sections above. diff --git a/playwright/evals/evals.json b/playwright/evals/evals.json new file mode 100644 index 0000000..c552078 --- /dev/null +++ b/playwright/evals/evals.json @@ -0,0 +1,78 @@ +{ + "schema_version": 1, + "skill_name": "playwright", + "evals": [ + { + "id": "e2e-checkout-authoring", + "prompt": "Write a Playwright E2E test for the checkout flow: add an item to the cart, apply a promo code, and complete the purchase. The app runs at http://localhost:3000 via npm run dev. What should the spec look like and what config is needed so it runs reliably in CI too?", + "expected_output": "A spec structure that describes one user journey per test using test.describe and test.beforeEach, drives the flow with user-facing locators (getByRole for buttons like 'Checkout' and 'Place order', getByLabel for the promo input), and asserts with web-first assertions (toHaveText on the confirmation heading, toHaveURL on the order route) instead of waitForTimeout sleeps. The answer recommends a playwright.config.ts with testDir, baseURL, projects (desktop and mobile), webServer pointing at 'npm run dev' with a readiness URL and reuseExistingServer false on CI, retries on CI only, and trace on-first-retry, plus reporters including JSON so the run is triageable with pwrun report.", + "assertions": [ + "The spec is structured as one user journey per test with describe/beforeEach", + "Interactions use role and label locators rather than brittle CSS", + "Assertions are web-first (toHaveText, toHaveURL) with no fixed sleeps", + "The config declares webServer, baseURL, projects, CI-only retries, and trace on-first-retry", + "The JSON reporter is included so CI failures can be triaged from the report" + ] + }, + { + "id": "flaky-selector-repair", + "prompt": "Our checkout test is flaky: it passes locally but fails in CI about 30% of the time at page.getByText('Buy now').click(), complaining the locator resolved to 0 elements or sometimes to 2. How should I investigate and fix it?", + "expected_output": "A selector repair procedure: measure the flake deterministically by running the spec alone with --workers=1 --repeat-each=5, then use --debug or the trace to see what the locator actually resolved to. The answer explains that getByText('Buy now') is ambiguous (two elements, e.g. a button and a promo snippet) and late-rendering (0 elements because the product list loads after navigation). The fix is a role-based locator scoped to the product card — getByRole('button', { name: 'Buy now' }) inside a card filtered by product name — plus a web-first assertion on the card container before acting, and never adding waitForTimeout or first() as a band-aid. Verification is five consecutive green repeat-each runs and a green CI run.", + "assertions": [ + "The flake is measured deterministically with --workers=1 --repeat-each before changing code", + "The cause is diagnosed as ambiguous text match plus late rendering", + "The fix uses a role-based locator scoped by product card, not first() or sleeps", + "A web-first assertion on the container precedes the action", + "Verification is repeated green runs and a green CI run" + ] + }, + { + "id": "network-mock-payment-api", + "prompt": "Our E2E suite depends on a third-party payment tokenization API that is rate-limited and sometimes unavailable, making tests flaky. How do I make the payment flow tests hermetic while still testing the app's real behavior?", + "expected_output": "A network interception plan using page.route in test.beforeEach: fulfill the tokenization endpoint with a realistic JSON body and content type so the app receives a token as in production, register routes before navigation, and abort analytics/tracker traffic that pollutes tests. The answer warns never to mock the app's own server or the code under test, keeps the mock payloads schema-accurate, and shows asserting on the request the app actually sent via page.on('request') to verify the body. It notes that only third-party boundaries are mocked and that a controlled real-API variant is preferable for integration verification.", + "assertions": [ + "page.route fulfills the third-party endpoint with a realistic body and content type before navigation", + "Analytics and tracker traffic is blocked so tests stay hermetic", + "The app's own server and the code under test are explicitly not mocked", + "The request the app sent is asserted via page.on('request')", + "A controlled real-API variant is offered as the integration verification option" + ] + }, + { + "id": "scrape-product-catalog", + "prompt": "A documentation site renders its product catalog table only after JavaScript loads. Scrape the table (columns: name, version, license) into a structured file without breaking the site's rules.", + "expected_output": "A headless scraping plan following extract -> validate -> save: launch Chromium headless with an identifying user agent, load the page with wait_until networkidle, scope a locator to the repeating table rows and read cell texts into plain records, validate that every record has name/version/license before saving, and write one bounded JSON artifact with timestamp and source URL. The answer checks robots.txt and terms, adds a delay between pages, caps the scrape with MAX_RECORDS/MAX_PAGES, and refuses to extract personal data or persist auth storage. It routes Cloudflare challenge cases to flaresolverr instead of this skill.", + "assertions": [ + "The page is loaded in a headless browser because the table is JavaScript-rendered", + "Extraction is scoped to the repeating rows and produces structured records, not HTML blobs", + "Records are validated for required fields before saving", + "Robots.txt, terms, rate limits, and bounded extraction are respected", + "Auth storage is never persisted and challenge pages route to flaresolverr" + ] + }, + { + "id": "ci-failure-triage", + "prompt": "The CI run for the E2E suite failed. The only artifact is test-results/test-results.json from the Playwright JSON reporter. The console shows '3 expected, 2 unexpected'. What are the next steps to diagnose and fix?", + "expected_output": "A triage procedure that starts by summarizing the JSON report with the pwrun script (scripts/pwrun report --report test-results.json --json) to get the failing spec titles and the error message from the last retry without opening a browser, then opens the trace artifact for the failed tests to see the failing action, network, and console. The answer classifies the failure: environment (missing browser deps on the runner, webServer readiness), selector problem (referencing the selectors reference), or a real app regression. It prescribes fixing and re-running, and adding the trace-on-first-retry config plus artifact upload if the run lacked them, keeping evidence bounded by not dumping the full report.", + "assertions": [ + "The JSON report is summarized with the pwrun report command before any other debugging", + "The trace artifact is opened to inspect the failing action, network, and console", + "The failure is classified as environment, selector, or app regression", + "Missing trace/artifact config is added if the run lacked them", + "Evidence stays bounded: summaries and targeted artifacts, not full dumps" + ] + }, + { + "id": "frontend-test-implementation", + "prompt": "A React team is adding a new feature (filter + sortable product list) and wants tests that prevent regressions. They ask how to implement testing for it across levels, including browser-level coverage. What should the plan look like and when is Playwright the right tool?", + "expected_output": "A test implementation plan across the pyramid: component tests with Testing Library for state and rendering, then Playwright E2E specs for the user journeys that matter (sorting, filtering, empty state) as the browser-level layer, keeping the component-to-E2E split based on what each level proves. The answer routes the frontend implementation guidance to frontend-engineering and notes this skill owns writing and running the Playwright specs: role-based locators, web-first assertions, mocking the products API at the route boundary, parallel workers for speed, and CI wiring with webServer, retries, and the JSON reporter for triage. It flags not to E2E-test everything — component tests cover most logic, E2E covers the journeys.", + "assertions": [ + "The plan spans component tests (Testing Library) and Playwright E2E for user journeys", + "The split is justified by what each level proves, not by convention", + "Playwright specs use role-based locators, web-first assertions, and route-level API mocking", + "CI wiring (webServer, retries, JSON reporter) is included", + "Component-versus-E2E routing guidance points to frontend-engineering for design and this skill for browser execution" + ] + } + ] +} diff --git a/playwright/references/00-source-index.md b/playwright/references/00-source-index.md new file mode 100644 index 0000000..4abe3a6 --- /dev/null +++ b/playwright/references/00-source-index.md @@ -0,0 +1,43 @@ +# Source Index — Playwright references + +> **Last Updated:** 2026-08-03 + +## Scope + +This skill's references are distilled from the official Playwright documentation +and the underlying standards the tool implements (WebDriver BiDi for CDP +migration, WAI-ARIA for the accessibility tree). They are patterns and +decision guidance, not a substitute for the primary sources below. + +## Primary sources + +| Topic | Source | Accessed | +|---|---|---| +| Test runner, assertions, fixtures, webServer | https://playwright.dev/docs/writing-tests | 2026-08-03 | +| Locators and selector philosophy | https://playwright.dev/docs/locators | 2026-08-03 | +| Network interception (`page.route`) | https://playwright.dev/docs/network | 2026-08-03 | +| Parallelism and sharding | https://playwright.dev/docs/test-parallel | 2026-08-03 | +| CI guides (GitHub Actions, Docker) | https://playwright.dev/docs/ci | 2026-08-03 | +| Traces, HTML/JSON reporters, debugging | https://playwright.dev/docs/trace-viewer | 2026-08-03 | +| Accessibility testing + aria snapshots | https://playwright.dev/docs/accessibility-testing | 2026-08-03 | +| Headless browsers and scraping contexts | https://playwright.dev/docs/api/class-browser | 2026-08-03 | +| Codegen, inspector, slow-mo | https://playwright.dev/docs/codegen | 2026-08-03 | +| `@axe-core/playwright` integration | https://github.com/dequelabs/axe-core-npm | 2026-08-03 | + +## Version observations + +- **1.40+** — `toMatchAriaSnapshot` is available from 1.49; earlier versions + used the experimental `page.accessibility` API or `expect(...).toHaveAccessibleSnapshot()`. + Pin the version that matches the features this skill references (see `SKILL.md`). +- The JSON reporter output shape changed over time (suite-level `tests` arrays + became `specs[]` with nested `tests[]`). `scripts/pwrun report` parses both + shapes; a fixture for the modern shape lives in `tests/fixtures/sample-report.json`. + +## Refresh procedure + +Re-verify statements in these references when a new Playwright major lands: +1. Re-read the primary source pages above. +2. Check the changelog for renamed APIs (e.g., `--update-snapshots` flags, + reporter options, `webServer` fields). +3. Update the `Last Updated` line in the changed reference and the version + observations above. diff --git a/playwright/references/01-e2e-authoring.md b/playwright/references/01-e2e-authoring.md new file mode 100644 index 0000000..4612b21 --- /dev/null +++ b/playwright/references/01-e2e-authoring.md @@ -0,0 +1,106 @@ +# E2E Test Authoring + +> **Last Updated:** 2026-08-03 + +How to write Playwright E2E tests that are readable, stable, and fast to +maintain. Authoring decisions are execution-side; *what to test and at what +level* is QA strategy owned by `qa-methodology` (its +`qa-methodology/references/test-strategy.md`). + +## Anatomy of a spec + +One spec file per user journey, described in prose, located by behavior: + +```ts +import { test, expect } from '@playwright/test'; + +test.describe('checkout', () => { + test.beforeEach(async ({ page }) => { + await page.goto('/cart'); + }); + + test('completes a purchase with a saved card', async ({ page }) => { + await page.getByRole('button', { name: 'Checkout' }).click(); + await page.getByRole('button', { name: 'Place order' }).click(); + await expect(page.getByRole('heading', { level: 1 })).toHaveText('Order confirmed'); + }); +}); +``` + +Rules of thumb: + +- **One journey per test.** A test that verifies two unrelated flows fails for + two unrelated reasons and gets rewritten as two tests anyway. +- **Setup in `beforeEach` / fixtures, not in the test body.** Keep the test body + readable as a spec of the behavior. +- **Describe what the user does**, not what the DOM does: "places an order", + not "clicks the button with class `.btn-primary`". + +## Web-first assertions (no sleeps) + +Playwright assertions retry until a timeout: + +```ts +await expect(page.getByText('Saved')).toBeVisible(); +await expect(input).toHaveValue('50'); +await expect(page).toHaveURL(/\/orders\/\d+/); +``` + +- Never `await page.waitForTimeout(2000)` to "fix" a race — it slows the suite + and hides the real timing bug. +- For genuinely async conditions use `expect.poll()` or `expect(...).toPass()` + instead of arbitrary sleeps. + +## Fixtures + +Shared setup lives in a fixture file and is composed per test: + +```ts +import { test as base, expect } from '@playwright/test'; + +export const test = base.extend<{ signedInPage: Page }>({ + signedInPage: async ({ page }, use) => { + await page.goto('/login'); + await page.getByLabel('Email').fill('qa@example.com'); + await page.getByLabel('Password').fill(process.env.TEST_PASSWORD!); + await page.getByRole('button', { name: 'Sign in' }).click(); + await use(page); + }, +}); +``` + +## Starting the app: webServer + +Declare the app lifecycle in the config so Playwright starts, waits for, and +tears down the server per run (see `templates/playwright.config.ts`): + +- `webServer.command` — the dev/preview command. +- `webServer.url` — a readiness URL; Playwright polls it before running tests. +- `reuseExistingServer: !process.env.CI` — reuse a dev server locally, always + start fresh on CI. + +Prefer `webServer` over asking the agent to start the app manually; the config +makes the run reproducible in CI too. + +## The page-object model (at the size where it pays) + +Group locators and actions for a screen into a class when a spec grows beyond +~15 lines or the same flow is asserted from several specs: + +```ts +export class CartPage { + constructor(private readonly page: Page) {} + async open() { await this.page.goto('/cart'); } + async applyPromo(code: string) { await this.page.getByLabel('Promo code').fill(code); } + get checkoutButton() { return this.page.getByRole('button', { name: 'Checkout' }); } +} +``` + +Do not add a POM layer preemptively — one spec that reads as prose beats a +POM with one user. + +## Related + +- Locator choice and flaky-selector repair: `02-selectors.md`. +- Mocking external HTTP so tests stay hermetic: `03-network-interception-and-mocking.md`. +- Snapshot-style accessibility assertions: `07-accessibility-and-debugging.md`. diff --git a/playwright/references/02-selectors.md b/playwright/references/02-selectors.md new file mode 100644 index 0000000..b980323 --- /dev/null +++ b/playwright/references/02-selectors.md @@ -0,0 +1,75 @@ +# Selector Robustness + +> **Last Updated:** 2026-08-03 + +Selectors are the #1 source of E2E flakiness. The goal is locators that +describe what the element *is* (its role in the user experience), not where it +*happens to be* in the DOM. + +## Locator priority + +Use, in order of preference: + +1. **Role** — `page.getByRole('button', { name: 'Save' })`. Mirrors how the + page is presented to assistive tech and users; survives markup changes. +2. **Label / placeholder / text** — `getByLabel('Email')`, + `getByPlaceholder('Search')`, `getByText('Saved', { exact: true })`. +3. **Test id** — `getByTestId('checkout-form')`. For elements whose role/label + does not describe them (e.g., a decorative SVG, a canvas region). Test ids + exist only for tests; agree on a naming convention. +4. **CSS / XPath** — last resort: layout-adjacent queries that role and label + cannot express (e.g., "the third row of a table" is better done with + `getByRole('row').nth(2)`). + +## Composition over long strings + +Chain and filter instead of concatenating brittle paths: + +```ts +// Fragile: encodes nesting and order. +page.locator('div.product-card div.price span').click(); + +// Robust: describe the card by its visible content, then act within it. +const card = page.getByRole('article').filter({ hasText: 'Running shoes' }); +await card.getByRole('button', { name: 'Add to cart' }).click(); +``` + +- `filter({ hasText })` / `filter({ has: locator })` narrow a collection. +- `first()`, `last()`, `nth(n)` are code smells unless the ordering is the + assertion (e.g., a sort test). + +## The repair loop + +A flaky test is a bug report about your selectors, not a request for more +`waitForTimeout`. When a test passes sometimes: + +1. Run the spec alone (`npx playwright test --workers=1 --repeat-each=5`) + to measure flakiness deterministically. +2. Use `--debug` or the trace to see what the failing action actually resolved. + Common causes: + - **Zero matches** — the element appears late (async render): use a + web-first assertion or wait for its container, not a sleep. + - **Multiple matches** — your locator is too generic: narrow with + `filter({ hasText })` or scope to a container. + - **Stale node** — the element is re-rendered between lookup and action: + re-query instead of storing a handle, or assert the new state after the + re-render. +3. Fix the locator to describe the unique user-facing element, then re-run the + repeat-each loop until it is green 5/5. + +## Anti-patterns to avoid + +- **`page.waitForTimeout()`** — masks races, slows the suite. +- **`page.waitForSelector()` + manual `click()`** — duplicates what + `locator.click()` already does with retries. +- **Snapshots of CSS classes** (`expect(el).toHaveClass(...)`) for behavioral + assertions — classes are implementation details. +- **Text that duplicates** across the page without disambiguation + (`getByText('Submit')` matches 2 buttons) — add `{ exact: true }` or scope. +- **XPath with positional predicates** (`//div[2]/span[1]`) — order and + structure change; roles do not. + +## Related + +- Authoring structure and fixtures: `01-e2e-authoring.md`. +- Emulating slow networks to surface timing bugs: `03-network-interception-and-mocking.md`. diff --git a/playwright/references/03-network-interception-and-mocking.md b/playwright/references/03-network-interception-and-mocking.md new file mode 100644 index 0000000..6ce8e27 --- /dev/null +++ b/playwright/references/03-network-interception-and-mocking.md @@ -0,0 +1,87 @@ +# Network Interception and Mocking + +> **Last Updated:** 2026-08-03 + +Intercept browser traffic with `page.route()` to make E2E tests hermetic, +deterministic, and fast — and to assert on what the app actually sent. + +## The route API + +```ts +// Fulfill: answer an intercepted request with canned data. +await page.route('**/api/payments/tokenize', (route) => + route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ token: 'tok_test_123' }), + }), +); + +// Abort: block a request entirely (trackers, analytics, heavy media). +await page.route('**/analytics/*', (route) => route.abort()); + +// Continue: let the real request through (with optional overrides). +await page.route('**/api/feature-flags', (route) => + route.continue({ headers: { ...route.request().headers(), 'x-experiment': 'on' } }), +); +``` + +Guidance: + +- **Mock at the boundary.** The app under test is real; the world outside it + (payment providers, third-party APIs, message queues) is faked. Never patch + in-page code (`window.fetch = ...`) — that tests a tampered app. +- **Realistic bodies win.** Mock payloads that match the real schema and + content type; a mock that differs from production can pass a test the app + would fail. +- **Glob `**` patterns, not exact URLs** — hosts, query strings, and CDN + prefixes change; `**/api/orders` covers `https://api.example.com/api/orders`. +- **Register routes before navigation** — routes apply to requests made after + registration, so set them up in `test.beforeEach` before `page.goto`. + +## What NOT to mock + +- The server under test: if the suite verifies the app + its backend + integration, intercepting that API makes the test meaningless. Mock only + *third-party* boundaries, or test the real API via a controlled + environment. +- `page.goto()` navigation that the test depends on — routing over the app's + own document requests breaks the flow the test is verifying. +- Responses whose timing you are testing (e.g., loading states): use + `route.fulfill` with a small artificial delay or the API's + `page.clock`/routing delay, never a fixed `waitForTimeout`. + +## Asserting on traffic + +Capture what the app sent, then assert on it: + +```ts +let tokenizeBody: string | undefined; +page.on('request', (request) => { + if (request.url().includes('/api/payments/tokenize')) tokenizeBody = request.postData(); +}); +// ... drive the flow ... +expect(JSON.parse(tokenizeBody!)).toMatchObject({ amount: 4990, currency: 'usd' }); +``` + +Use `page.on('response')` to wait for a specific status instead of guessing +when the network settled: + +```ts +await page.waitForResponse((response) => response.url().includes('/api/orders') && response.status() === 201); +``` + +## Emulating network conditions + +```ts +await page.route('**/*', (route) => + route.continue().catch(() => {}) +); +await page.context().setOffline(true); // test offline/error states +// or throttle via context options: offline, latency, downloadThroughput... +``` + +## Related + +- Locators that survive the mocked world: `02-selectors.md`. +- Running these tests in parallel without interference: `04-parallel-workers-and-sharding.md`. diff --git a/playwright/references/04-parallel-workers-and-sharding.md b/playwright/references/04-parallel-workers-and-sharding.md new file mode 100644 index 0000000..ec067fd --- /dev/null +++ b/playwright/references/04-parallel-workers-and-sharding.md @@ -0,0 +1,70 @@ +# Parallel Workers and Sharding + +> **Last Updated:** 2026-08-03 + +Playwright runs each test in its **own browser context** — isolation is the +default. Parallelism is about scaling that safely, and sharding is how large +suites split across CI machines. + +## Workers + +- `workers` in the config caps how many parallel worker *processes* run at once. + Each worker hosts one browser instance and runs one test at a time. +- `fullyParallel: true` lets every spec file run across workers; with it off, + only files in separate *projects* run in parallel. +- **Size workers to the machine**, not to desire: each Chromium worker needs + roughly 300–500 MB. A 2-core/4 GB runner with 4 Chromium workers will OOM. + Start at `Math.min(cores, 4)` and measure. +- CI: pin `workers: 4` (or `--workers=4`) for a stable runtime; `undefined` + locally lets Playwright pick. + +```ts +workers: process.env.CI ? 4 : undefined, +fullyParallel: true, +``` + +## Sharding + +Split one suite across multiple CI jobs: + +```bash +npx playwright test --shard=1/4 # job 1 of 4 +``` + +Each shard runs a disjoint set of tests; the HTML report aggregates via +`merge-reports`. Shard count should roughly equal runner count; the sharded +runtime is the slowest shard, so balance by spec-file count, not total tests +(Playwright shards by file). + +## Isolation traps (things that break parallel runs) + +- **Shared global state in the app under test** — a localStorage flag, a + singleton cache, a shared DB row: workers race and tests interfere. Reset + per test (fixtures that clean up, `test.beforeEach` seeding). +- **Shared files on disk** — screenshots/traces written to the same path from + two workers. Give each test its own output dir (`test-results//` is + the default; don't override to one shared file). +- **Port collisions** — multiple webServers or `page.goto('http://localhost:3000')` + hardcoded across workers. Use `webServer` with one process, or unique ports + per project. +- **Test-order dependence** — `describe.only`/`test.skip` patterns, tests that + assume a previous test ran. Every test must pass alone (`--grep` it) and in + any order (`--workers=1 --repeat-each=3` to check determinism). + +## Verifying parallelism is working + +```bash +scripts/pwrun inventory --json # suite shape +npx playwright test --list # what will run +npx playwright test --workers=4 # parallel run +``` + +Compare `--workers=1` vs `--workers=4` wall time: healthy suites scale ~linearly +until CPU-bound. If the parallel run is *slower* or flakier than serial, you +have an isolation trap — see above. + +## Related + +- CI job wiring for shards and artifacts: `05-ci-integration.md`. +- Hermetic mocking so workers don't depend on live external APIs: + `03-network-interception-and-mocking.md`. diff --git a/playwright/references/05-ci-integration.md b/playwright/references/05-ci-integration.md new file mode 100644 index 0000000..e9be522 --- /dev/null +++ b/playwright/references/05-ci-integration.md @@ -0,0 +1,100 @@ +# CI Integration + +> **Last Updated:** 2026-08-03 + +Playwright in CI is: install browsers + OS deps, pin the version, run the +suite with retries and tracing, and surface a debuggable report. This +reference assumes GitHub Actions; the same shape applies to any runner. + +## Minimal GitHub Actions workflow + +```yaml +name: e2e +on: [push, pull_request] +jobs: + e2e: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: { node-version: 20, cache: npm } + - run: npm ci + - run: npx playwright install --with-deps chromium + - run: npx playwright test + - if: failure() + uses: actions/upload-artifact@v4 + with: + name: playwright-report + path: playwright-report/ + retention-days: 14 +``` + +## Non-negotiables + +1. **Install browsers with OS deps**: `npx playwright install --with-deps` (not + bare `install`) on Linux runners; `--with-deps` installs the system + libraries Chromium/Firefox/WebKit need. +2. **Pin and cache**: + - `npm ci` with a committed `package-lock.json` (never `npm install`). + - Cache the browser download: `~/.cache/ms-playwright` (Linux), + `~/Library/Caches/ms-playwright` (macOS), `%USERPROFILE%\AppData\Local\ms-playwright` (Windows). + - Cache `node_modules` via the setup-node `cache: npm` option. +3. **Retry flaky tests on CI only**, with traces on retry so failures are + debuggable: + + ```ts + retries: process.env.CI ? 2 : 0, + use: { trace: 'on-first-retry' }, + ``` +4. **Configure `webServer` in the config** so the runner starts and waits for + the app; never assume a long-lived dev server on a runner. +5. **Upload artifacts on failure**: the HTML report, the JSON report, and the + `test-results/` dir (traces). Retention bounded (7–14 days) — see hard + boundaries in `SKILL.md` about keeping evidence bounded. + +## Reporters + +- `list`/`line` — human-readable run output. +- `html` — the browsable report (upload on failure). +- `json` — the machine-readable report for agent triage: + + ```bash + scripts/pwrun report --report test-results/test-results.json --json + ``` + + It prints stats (expected/unexpected/flaky/skipped), the failing specs, and + the error message from the last retry — enough to triage without opening a + browser. +- `github` — inline annotations on GitHub Actions, keyed to the failing spec + line. + +## Sharding across jobs + +For large suites, split the run: + +```yaml +strategy: + matrix: + shard: [1/4, 2/4, 3/4, 4/4] +steps: + - run: npx playwright test --shard=${{ matrix.shard }} +``` + +Merge reports from all shards with `playwright merge-reports` (see +`04-parallel-workers-and-sharding.md` for the math). + +## Triage loop for a red CI run + +1. `scripts/pwrun report --report --json` — get the failing specs and + messages. +2. Download the trace artifact (`trace.zip`) and open it in the Trace Viewer + to see the failing action, network, and console. +3. Classify: environment (missing dep/browser), selector (see + `02-selectors.md`), timing (webServer readiness, `webServer.timeout`), + or app regression (real bug — the test did its job). +4. Fix, re-run, and confirm the shard matrix is green. + +## Related + +- Parallelism and sharding configuration: `04-parallel-workers-and-sharding.md`. +- Trace reading and headed debugging: `07-accessibility-and-debugging.md`. diff --git a/playwright/references/06-scraping-and-headless.md b/playwright/references/06-scraping-and-headless.md new file mode 100644 index 0000000..3f780c5 --- /dev/null +++ b/playwright/references/06-scraping-and-headless.md @@ -0,0 +1,81 @@ +# Scraping and Headless Browsing + +> **Last Updated:** 2026-08-03 + +Use Playwright as a headless browser for pages that need JavaScript to render: +load the page, extract structured records, validate them, and save — the +extract → validate → save loop. This is browser *automation*, not a generic +HTTP client; for static content prefer a plain HTTP request (see +`SKILL.md` → When not to use). + +## The loop + +```python +# Rough shape — the same pattern in JS: launch, context, page, extract, validate, save. +# (Playwright's Python package mirrors the Node API 1:1.) +from playwright.sync_api import sync_playwright + +with sync_playwright() as p: + browser = p.chromium.launch(headless=True) + page = browser.new_context( + user_agent="research-bot/1.0 (+contact@example.com)", + viewport={"width": 1280, "height": 800}, + ).new_page() + page.goto("https://docs.example.com/catalog", wait_until="networkidle") + records = [] + for row in page.locator("table tbody tr").all(): + cells = row.locator("td").all_inner_texts() + records.append(dict(zip(["name", "version", "license"], cells))) + # validate: every record has the required fields + valid = [r for r in records if r["name"] and r["version"]] + # save: write one JSON/CSV artifact, bounded + ... +``` + +## Extract → validate → save, explicitly + +1. **Extract** — scope locators to the repeating container; use + `.all_inner_texts()` / attribute reads to build plain records. Never return + raw HTML blobs. +2. **Validate** — check required fields, types, and invariants *before* saving; + drop or flag records that fail. A scrape that saves garbage is worse than + one that reports failure. +3. **Save** — write one bounded artifact (JSON/CSV) per scrape with the + timestamp and source URL in the artifact. Do not dump page HTML or screenshots + of protected content into chat (hard boundaries in `SKILL.md`). + +## Pagination and infinite scroll + +- **Pagination**: follow the "next" button until it is disabled or the count + target is reached: + + ```ts + while (await nextButton.isEnabled() && records.length < MAX) { + // extract current page... + await nextButton.click(); + await expect(page.locator('tbody tr').first()).toBeVisible(); + } + ``` +- **Infinite scroll**: scroll to the bottom and wait for the container's + height/record count to grow, with a max-iteration guard. +- Always cap: `MAX_RECORDS` and `MAX_PAGES` with a clear stop message when hit. + +## Politeness and legality (hard constraints) + +- Respect `robots.txt`, the site's terms of service, and rate limits. Playwright + does not enforce them; the operator does. +- Add a delay between page loads (e.g., 1–3 s) and keep concurrency low; a burst + of headless browsers is indistinguishable from an attack. +- Set an identifying `user_agent` with a contact address. +- **No credential harvesting, no personal data extraction, no auth-session + persistence into committed files.** Storage state (`context.storage_state()`) + must never be committed (see hard boundaries). +- **Challenge pages**: Cloudflare/DDoS-GUARD challenges are out of scope here — + route to [flaresolverr](../flaresolverr/SKILL.md). + +## Related + +- Locating repeating elements robustly: `02-selectors.md`. +- Blocking analytics/trackers while scraping: `03-network-interception-and-mocking.md`. +- Debugging a scrape that misses content (headed, slow-mo, trace): + `07-accessibility-and-debugging.md`. diff --git a/playwright/references/07-accessibility-and-debugging.md b/playwright/references/07-accessibility-and-debugging.md new file mode 100644 index 0000000..2e1a0f2 --- /dev/null +++ b/playwright/references/07-accessibility-and-debugging.md @@ -0,0 +1,90 @@ +# Accessibility and Debugging + +> **Last Updated:** 2026-08-03 + +Two workflows that share one tool feature set: asserting accessibility via the +accessibility tree, and debugging tests with the inspector, codegen, and +traces. + +## Accessibility snapshot checks + +### Full scans with axe-core + +`@axe-core/playwright` runs the axe engine against the rendered page: + +```ts +import AxeBuilder from '@axe-core/playwright'; + +const results = await new AxeBuilder({ page }).analyze(); +// results.violations: [{ id, impact, nodes, ... }] +expect(results.violations.filter((v) => v.impact === 'critical' || v.impact === 'serious')) + .toEqual([]); +``` + +- Scan every route that matters, ideally in CI; scan the full page or use + `.include()` / `.exclude()` to scope. +- Triage by `impact`: fix critical/serious; track moderate/minor in a backlog. +- False positives happen (e.g., contrast rules on known-brand colors) — scope + them out deliberately, never with a blanket `disableRules(['color-contrast'])`. + +### Aria snapshots (snapshot-based accessibility assertions) + +`expect(page).toMatchAriaSnapshot()` asserts against the **accessibility tree**, +not the DOM: + +```ts +await expect(page).toMatchAriaSnapshot(` + - heading "Store front" [level=1] + - button "Add to cart" +`); +``` + +- These read like spec assertions ("the heading is X, the button is Y") and + catch regressions in structure, labels, and semantics — not just contrast. +- They are stable: inline text changes show up as a reviewable diff. +- **Update deliberately.** Run `npx playwright test --update-snapshots` only + after inspecting what changed; never blind-update to make CI green (hard + boundary in `SKILL.md`). +- Requires Playwright 1.49+ (see `00-source-index.md` for version notes). + +## Headed debugging + +When a test fails or a locator matches nothing: + +1. **Read the failure first** — `scripts/pwrun report --report --json` + gives the failing spec and the error message from the last retry. +2. **Run headed with slow-mo** to watch the actual page: + + ```bash + npx playwright test --headed --slow-mo 300 + ``` +3. **The inspector** (`--debug` or `PWDEBUG=1`) pauses before each action and + shows the current locator; `page.pause()` drops a breakpoint mid-test. +4. **Codegen** to prototype a flow quickly: + + ```bash + npx playwright codegen https://example.com + ``` + Generate starter tests, then harden the emitted selectors into user-facing + locators (`02-selectors.md`). +5. **Traces** are the evidence record: with `trace: 'on-first-retry'` (or + `--trace on`), every failed run produces a trace you can open in the Trace + Viewer — network, DOM snapshots, console, and the failing action, frame by + frame. This is the primary artifact to attach to a bug report. + +## Debugging checklist + +| Symptom | First move | +|---|---| +| Locator resolves to 0 elements | `--debug`; check async render — assert on a container first | +| Locator matches 2+ elements | Narrow with `filter({ hasText })` or scope to a container | +| Timeout on `click()` | Trace: is something covering the element (overlay)? is it disabled? | +| Passes locally, fails CI | Compare env: browser deps (`install --with-deps`), `baseURL`, shard isolation | +| Flaky across retries | `--repeat-each=5 --workers=1` to measure; then fix the selector | +| Console errors before failure | Trace console tab; check for unhandled rejections the test should assert | + +## Related + +- Authoring and assertions: `01-e2e-authoring.md`. +- Selector repair loop: `02-selectors.md`. +- CI trace/artifact wiring: `05-ci-integration.md`. diff --git a/playwright/scripts/pwrun b/playwright/scripts/pwrun new file mode 100755 index 0000000..822ecf7 --- /dev/null +++ b/playwright/scripts/pwrun @@ -0,0 +1,418 @@ +#!/usr/bin/env python3 +"""pwrun - agent-first smoke harness for Playwright test suites. + +Inspects a Playwright suite and triages its runs without requiring node or a +browser: doctor checks the toolchain, inventory describes the suite structure, +report summarizes a Playwright JSON test report (--reporter=json), and smoke +delegates a real run to `npx playwright test`. + +Commands +-------- +doctor Report node, @playwright/test, browsers, and config availability. +inventory List test files and describe the suite (config, projects, specs). +report Summarize a Playwright JSON report (--report FILE). +smoke Run a quick smoke pass against a URL (delegates to npx playwright test). + +Exit codes: 0 ok, 1 analysis error, 2 usage error, 127 dependency missing, +124 delegate timeout. +""" +from __future__ import annotations + +import argparse +import json +import os +import re +import shutil +import subprocess +import sys +from typing import Any, Optional + +CONFIG_NAMES = ( + "playwright.config.ts", + "playwright.config.js", + "playwright.config.mts", + "playwright.config.cjs", + "playwright.config.mjs", +) +SPEC_SUFFIXES = (".spec.ts", ".spec.js", ".spec.mts", ".spec.mjs", ".test.ts", ".test.js") +DEFAULT_TEST_DIRS = ("e2e", "tests", "specs", "playwright") +COMMON_DEFAULTS = { + "json": False, + "timeout": 120, + "config": None, + "report": None, + "url": "http://localhost:3000", + "spec": None, +} + +HELP_JSON = { + "name": "pwrun", + "summary": "Agent-first smoke harness for Playwright test suites", + "usage": "pwrun COMMAND [options] (or: pwrun --help, pwrun doctor|inventory|report|smoke)", + "commands": [ + {"name": "doctor", "help": "Report node, @playwright/test, browsers, and config availability"}, + {"name": "inventory", "help": "List test files and describe the suite structure"}, + {"name": "report", "help": "Summarize a Playwright JSON report (--report FILE)"}, + {"name": "smoke", "help": "Run a quick smoke pass against a URL (delegates to npx playwright test)"}, + ], + "flags": [ + {"name": "--json", "help": "Emit a structured JSON result on stdout"}, + {"name": "--config", "help": "Path to the Playwright config file (auto-detected)"}, + {"name": "--report", "help": "Path to a Playwright JSON test report (report command)"}, + {"name": "--url", "help": "Target URL for the smoke command (default http://localhost:3000)"}, + {"name": "--spec", "help": "Spec file filter for the smoke command"}, + {"name": "--timeout", "help": "Delegate command timeout in seconds (default 120)"}, + ], + "exit_codes": {"0": "ok", "1": "analysis error", "2": "usage error", "127": "dependency missing", "124": "delegate timeout"}, +} + + +def emit(payload: Any, as_json: bool) -> None: + """Write a payload to stdout; JSON when --json, readable text otherwise.""" + if as_json: + print(json.dumps(payload, indent=2, sort_keys=True)) + return + if isinstance(payload, str): + print(payload) + return + lines = [] + for key, value in payload.items(): + if isinstance(value, (list, dict)) and value: + lines.append(f"{key}: {json.dumps(value, sort_keys=True)}") + else: + lines.append(f"{key}: {value}") + print("\n".join(lines)) + + +def find_tool(tool: str) -> Optional[str]: + """Locate a tool on PATH.""" + return shutil.which(tool) + + +def detect_config(explicit: Optional[str]) -> Optional[str]: + """Resolve the Playwright config path: explicit wins, then cwd scan.""" + if explicit: + return explicit if os.path.isfile(explicit) else None + for name in CONFIG_NAMES: + if os.path.isfile(name): + return name + return None + + +def find_specs(root: str) -> list[str]: + """Walk the working tree (bounded) and list Playwright spec files.""" + found: list[str] = [] + skipped_dirs = {".git", "node_modules", ".venv", "dist", "build", "coverage", ".next", "__pycache__"} + for base, dirs, files in os.walk(root): + dirs[:] = sorted(d for d in dirs if d not in skipped_dirs) + for name in sorted(files): + if name.endswith(SPEC_SUFFIXES): + found.append(os.path.join(base, name)) + return found + + +def read_config_projects(config_path: str) -> list[str]: + """Best-effort project-name extraction from a config file (no ts execution).""" + try: + text = open(config_path, "r", encoding="utf-8").read() + except OSError: + return [] + projects: list[str] = [] + for match in re.finditer(r"name\s*:\s*['\"]([^'\"]+)['\"]", text): + projects.append(match.group(1)) + return projects + + +def cmd_doctor(args: argparse.Namespace) -> int: + """Check the local toolchain and report availability.""" + node = find_tool("node") + npx = find_tool("npx") + payload: dict[str, Any] = { + "ok": True, + "node_found": node is not None, + "node": node or "(not found)", + "npx_found": npx is not None, + "playwright_package": None, + "config": detect_config(getattr(args, "config", None)), + } + if node and npx: + try: + probe = subprocess.run( + [npx, "--no-install", "playwright", "--version"], + capture_output=True, + text=True, + timeout=args.timeout, + ) + except subprocess.TimeoutExpired: + payload["playwright_package"] = "probe timed out" + payload["ok"] = False + emit(payload, args.json) + return 124 + if probe.returncode == 0: + payload["playwright_package"] = (probe.stdout or probe.stderr).strip() + else: + payload["playwright_package"] = None + payload["playwright_hint"] = ( + "run `npm i -D @playwright/test` in the project, then `npx playwright install` for browsers" + ) + else: + payload["playwright_hint"] = "node/npx not found on PATH; install Node.js and @playwright/test" + payload["browsers_available"] = _browser_cache_snapshot() if node else [] + emit(payload, args.json) + return 0 + + +def _browser_cache_snapshot() -> list[str]: + """List installed Playwright browser executables from the standard cache dir.""" + home = os.environ.get("HOME") or "~" + candidates = [ + os.path.join(home, ".cache", "ms-playwright"), + os.path.join(home, "Library", "Caches", "ms-playwright"), + ] + installed: list[str] = [] + for cache in candidates: + if os.path.isdir(cache): + installed.extend(sorted(entry for entry in os.listdir(cache) if not entry.startswith("."))) + return installed + + +def cmd_inventory(args: argparse.Namespace) -> int: + """Describe the suite: config, projects, and spec files.""" + config = detect_config(getattr(args, "config", None)) + specs = find_specs(os.getcwd()) + payload: dict[str, Any] = { + "ok": True, + "config": config, + "projects": read_config_projects(config) if config else [], + "spec_count": len(specs), + "specs": specs, + "test_dirs": sorted({os.path.dirname(s) for s in specs}), + } + emit(payload, args.json) + return 0 + + +def walk_suites(suite: dict[str, Any]) -> list[tuple[dict[str, Any], dict[str, Any]]]: + """Yield (spec, suite) pairs for every spec/test in a Playwright report tree. + + Handles both the modern shape (suites[].specs[].tests[]) and the legacy + shape (suites[].tests[] with results[]). + """ + pairs: list[tuple[dict[str, Any], dict[str, Any]]] = [] + for spec in suite.get("specs", []) or []: + for test in spec.get("tests", []) or []: + pairs.append((spec, test)) + for legacy in suite.get("tests", []) or []: + pairs.append((legacy, legacy)) + for child in suite.get("suites", []) or []: + pairs.extend(walk_suites(child)) + return pairs + + +def error_of(test: dict[str, Any]) -> Optional[str]: + """Return the failure message from the last result of a test, if any.""" + results = test.get("results") or [] + for result in reversed(results): + error = result.get("error") + if error: + return str(error.get("message", error)) + return None + + +def summarize_report_data(data: Any, path: str) -> dict[str, Any]: + """Summarize a parsed Playwright JSON report into a bounded payload.""" + if not isinstance(data, dict): + raise ValueError("report root must be a JSON object") + + stats = data.get("stats") or {} + expected = int(stats.get("expected", 0) or 0) + unexpected = int(stats.get("unexpected", 0) or 0) + flaky = int(stats.get("flaky", 0) or 0) + skipped = int(stats.get("skipped", 0) or 0) + + failures: list[dict[str, Any]] = [] + passed_specs: list[str] = [] + for suite in data.get("suites", []) or []: + for spec, test in walk_suites(suite): + status = test.get("status", "") + title = spec.get("title") or test.get("title") or "(untitled)" + file = spec.get("file") or test.get("file") or "" + project = test.get("projectName") or "" + entry = {"title": title, "file": file, "project": project, "status": status} + results = test.get("results") or [] + last_status = results[-1].get("status") if results else None + if status in ("unexpected", "failed") or last_status == "failed": + error = error_of(test) + if error: + entry["error"] = error[:2000] + failures.append(entry) + elif status in ("expected", "flaky", "passed", "skipped"): + passed_specs.append(entry) + + return { + "ok": unexpected == 0 and not failures, + "report_file": path, + "stats": { + "expected": expected, + "unexpected": unexpected, + "flaky": flaky, + "skipped": skipped, + "duration_ms": stats.get("duration"), + "start_time": stats.get("startTime"), + }, + "failures": failures, + "passed_specs_count": len(passed_specs), + "summary": ( + f"{expected} expected, {unexpected} unexpected, {flaky} flaky, {skipped} skipped; " + f"{len(failures)} failing test(s)" + ), + } + + +def summarize_report(path: str) -> dict[str, Any]: + """Summarize a Playwright JSON report file into a bounded payload.""" + with open(path, "r", encoding="utf-8") as handle: + data = json.load(handle) + return summarize_report_data(data, path) + + +def cmd_report(args: argparse.Namespace) -> int: + """Summarize a Playwright JSON report.""" + path = getattr(args, "report", None) + if not path: + emit( + { + "ok": False, + "error": "report requires --report FILE", + "hint": "Generate one with `npx playwright test --reporter=json` (optionally -o test-results.json).", + }, + args.json, + ) + return 2 + try: + payload = summarize_report(path) + except (OSError, json.JSONDecodeError) as error: + emit({"ok": False, "error": f"report {path} is not readable JSON: {error}", "command": "report"}, args.json) + return 1 + except ValueError as error: + emit({"ok": False, "error": f"report {path} is invalid: {error}", "command": "report"}, args.json) + return 1 + emit(payload, args.json) + return 0 if payload["ok"] else 1 + + +def cmd_smoke(args: argparse.Namespace) -> int: + """Run a quick smoke pass by delegating to npx playwright test.""" + node = find_tool("node") + npx = find_tool("npx") + if not node or not npx: + emit( + { + "ok": False, + "error": "node/npx not found; smoke requires a Node toolchain", + "hint": "Install Node.js, run `npm i -D @playwright/test`, then `npx playwright install`.", + }, + args.json, + ) + return 127 + parts = ["playwright", "test"] + if getattr(args, "spec", None): + parts.append(args.spec) + parts.extend(["--reporter=json"]) + env = os.environ.copy() + if getattr(args, "url", None): + env["PW_SMOKE_URL"] = args.url + try: + proc = subprocess.run([npx, "--no-install"] + parts, capture_output=True, text=True, timeout=args.timeout, env=env) + except subprocess.TimeoutExpired: + emit( + { + "ok": False, + "error": "playwright test delegate timed out", + "timeout_seconds": args.timeout, + "command": parts, + }, + args.json, + ) + return 124 + payload: dict[str, Any] = { + "ok": proc.returncode == 0, + "exit_code": proc.returncode, + "command": parts, + "url": getattr(args, "url", None), + } + stdout = proc.stdout or "" + try: + report = json.loads(stdout) + except (ValueError, json.JSONDecodeError): + report = None + if report is not None: + try: + payload["report_summary"] = summarize_report_data(report, path="(smoke run)") + except ValueError as error: + payload["report_summary"] = {"ok": False, "error": str(error)} + payload["ok"] = bool(payload["report_summary"].get("ok")) + else: + payload["stdout_tail"] = stdout[-2000:] + payload["stderr_tail"] = (proc.stderr or "")[-2000:] + emit(payload, args.json) + return 0 if payload["ok"] else 1 + + +def add_common(parser: argparse.ArgumentParser) -> None: + """Attach global flags with SUPPRESS defaults so values survive subcommand parsing.""" + parser.add_argument("--json", action="store_true", default=argparse.SUPPRESS, help="Emit a structured JSON result on stdout") + parser.add_argument("--config", default=argparse.SUPPRESS, help="Path to the Playwright config file (auto-detected)") + parser.add_argument("--report", default=argparse.SUPPRESS, help="Path to a Playwright JSON test report (report command)") + parser.add_argument("--url", default=argparse.SUPPRESS, help="Target URL for the smoke command (default http://localhost:3000)") + parser.add_argument("--spec", default=argparse.SUPPRESS, help="Spec file filter for the smoke command") + parser.add_argument("--timeout", type=int, default=argparse.SUPPRESS, help="Delegate command timeout in seconds") + + +def build_parser() -> argparse.ArgumentParser: + common = argparse.ArgumentParser(add_help=False) + add_common(common) + + parser = argparse.ArgumentParser( + prog="pwrun", + description=( + "Agent-first smoke harness for Playwright test suites: toolchain checks, " + "suite inventory, JSON report triage, and a smoke delegation with JSON output." + ), + epilog="Exit codes: 0 ok, 1 analysis error, 2 usage error, 127 dependency missing, 124 delegate timeout.", + ) + add_common(parser) + + sub = parser.add_subparsers(dest="command", required=True, metavar="COMMAND") + + doctor = sub.add_parser("doctor", parents=[common], help="Report node, @playwright/test, browsers, and config availability") + doctor.set_defaults(handler=cmd_doctor) + + inventory = sub.add_parser("inventory", parents=[common], help="List test files and describe the suite structure") + inventory.set_defaults(handler=cmd_inventory) + + report = sub.add_parser("report", parents=[common], help="Summarize a Playwright JSON report (--report FILE)") + report.set_defaults(handler=cmd_report) + + smoke = sub.add_parser("smoke", parents=[common], help="Run a quick smoke pass against a URL (delegates to npx playwright test)") + smoke.set_defaults(handler=cmd_smoke) + + return parser + + +def main(argv: Optional[list[str]] = None) -> int: + argv = list(sys.argv[1:] if argv is None else argv) + # `--help --json` must emit parseable JSON (used by smoke probes and agents). + if "--help" in argv and "--json" in argv: + print(json.dumps(HELP_JSON, indent=2, sort_keys=True)) + return 0 + parser = build_parser() + args = parser.parse_args(argv) + for dest, default in COMMON_DEFAULTS.items(): + if not hasattr(args, dest): + setattr(args, dest, default) + return args.handler(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/playwright/templates/accessibility.spec.ts b/playwright/templates/accessibility.spec.ts new file mode 100644 index 0000000..881d86b --- /dev/null +++ b/playwright/templates/accessibility.spec.ts @@ -0,0 +1,33 @@ +import { test, expect } from '@playwright/test'; +import AxeBuilder from '@axe-core/playwright'; + +/** + * Accessibility smoke scaffold. + * + * Requires: npm i -D @axe-core/playwright + * + * - The axe scan catches WCAG violations (contrast, landmarks, ARIA misuse). + * - The aria snapshot is a stable, accessibility-aware assertion: it compares + * the accessibility tree, so it catches structure and label regressions. + * + * When the aria snapshot legitimately changes, run + * `npx playwright test --update-snapshots` ONLY after reviewing the diff. + */ +test.describe('accessibility', () => { + test('home page has no critical or serious axe violations', async ({ page }) => { + await page.goto('/'); // [fill: route] + const results = await new AxeBuilder({ page }).analyze(); + const blocking = results.violations.filter( + (violation) => violation.impact === 'critical' || violation.impact === 'serious', + ); + expect(blocking).toEqual([]); + }); + + test('home page matches the aria snapshot', async ({ page }) => { + await page.goto('/'); + await expect(page).toMatchAriaSnapshot(` + - heading "[fill: page heading]" [level=1] + - button "[fill: primary action]" + `); + }); +}); diff --git a/playwright/templates/example.spec.ts b/playwright/templates/example.spec.ts new file mode 100644 index 0000000..ecec7d7 --- /dev/null +++ b/playwright/templates/example.spec.ts @@ -0,0 +1,25 @@ +import { test, expect } from '@playwright/test'; + +/** + * Spec scaffold: one spec per user journey, described in prose, located by + * user-facing roles/labels, asserted with web-first assertions. Copy into + * e2e/ and adapt the [fill: ...] markers. + */ +test.describe('[fill: feature under test]', () => { + test.beforeEach(async ({ page }) => { + await page.goto('/'); // [fill: app entry route, relative to baseURL] + }); + + test('[fill: the behavior in one sentence]', async ({ page }) => { + // Prefer getByRole / getByLabel / getByText over CSS that encodes markup. + await page.getByRole('button', { name: '[fill: button label]' }).click(); + + // Web-first assertions auto-retry until the timeout; never use + // page.waitForTimeout() to "fix" a race. + await expect( + page.getByRole('heading', { level: 1 }), + ).toHaveText('[fill: expected heading]'); + + await expect(page).toHaveURL(/\/[fill: path-pattern]/); + }); +}); diff --git a/playwright/templates/playwright.config.ts b/playwright/templates/playwright.config.ts new file mode 100644 index 0000000..302d039 --- /dev/null +++ b/playwright/templates/playwright.config.ts @@ -0,0 +1,40 @@ +import { defineConfig, devices } from '@playwright/test'; + +/** + * Test-suite scaffold for Playwright. + * + * Copy this file (plus example.spec.ts and accessibility.spec.ts) into your + * project root, adjust the [fill: ...] markers, and run: + * + * npm i -D @playwright/test + * npx playwright install + * npx playwright test + */ +export default defineConfig({ + testDir: './e2e', // [fill: directory that holds your spec files] + fullyParallel: true, + forbidOnly: !!process.env.CI, + retries: process.env.CI ? 2 : 0, // retry flaky tests on CI only + workers: process.env.CI ? 4 : undefined, // [fill: CI worker count for your runner] + reporter: [ + ['list'], + ['html', { open: 'never' }], + ['json', { outputFile: 'test-results/test-results.json' }], // triage with scripts/pwrun report + ], + use: { + baseURL: process.env.BASE_URL ?? 'http://localhost:3000', // [fill: app URL] + trace: 'on-first-retry', // capture a trace when a test fails on retry + screenshot: 'only-on-failure', + video: 'retain-on-failure', + }, + projects: [ + { name: 'chromium', use: { ...devices['Desktop Chrome'] } }, + { name: 'mobile-chromium', use: { ...devices['Pixel 7'] } }, // [fill: devices you support] + ], + webServer: { + command: 'npm run dev', // [fill: your app's dev/preview command] + url: 'http://localhost:3000', // [fill: readiness URL the server must answer] + reuseExistingServer: !process.env.CI, + timeout: 120_000, + }, +}); diff --git a/playwright/tests/fixtures/sample-report.json b/playwright/tests/fixtures/sample-report.json new file mode 100644 index 0000000..2d35cb1 --- /dev/null +++ b/playwright/tests/fixtures/sample-report.json @@ -0,0 +1,269 @@ +{ + "config": { + "configFile": "playwright.config.ts", + "rootDir": "/repo/shop-frontend", + "forbidOnly": false, + "fullyParallel": true, + "globalTimeout": 0, + "maxFailures": 0, + "preserveOutput": "always", + "projects": [ + { + "outputDir": "test-results", + "repeatEach": 1, + "retries": 2, + "id": "project-chromium", + "name": "chromium", + "testDir": "e2e", + "testMatch": ["**/*.spec.ts"] + }, + { + "outputDir": "test-results", + "repeatEach": 1, + "retries": 2, + "id": "project-mobile", + "name": "mobile-chromium", + "testDir": "e2e", + "testMatch": ["**/*.spec.ts"] + } + ], + "reporter": [["list"], ["json", { "outputFile": "test-results.json" }]], + "reportSlowTests": { "max": 5, "threshold": 60000 }, + "shard": null, + "updateSnapshots": "missing", + "version": "1.49.1", + "workers": 4, + "webServer": { + "command": "npm run dev", + "url": "http://localhost:3000", + "reuseExistingServer": true, + "timeout": 120000 + } + }, + "errors": [], + "stats": { + "startTime": "2026-08-03T09:30:00.000Z", + "duration": 18240, + "expected": 5, + "unexpected": 1, + "flaky": 0, + "skipped": 1 + }, + "suites": [ + { + "title": "", + "file": "", + "line": 0, + "column": 0, + "specs": [], + "suites": [ + { + "title": "checkout flow", + "file": "/repo/shop-frontend/e2e/checkout.spec.ts", + "line": 1, + "column": 1, + "specs": [ + { + "title": "adds an item to the cart", + "ok": true, + "tags": [], + "tests": [ + { + "timeout": 30000, + "annotations": [], + "expectedStatus": "passed", + "projectId": "project-chromium", + "projectName": "chromium", + "results": [ + { + "workerIndex": 0, + "status": "passed", + "duration": 812, + "error": null, + "stdout": [], + "stderr": [], + "retry": 0, + "startTime": "2026-08-03T09:30:01.120Z", + "attachments": [], + "steps": [] + } + ], + "status": "expected" + } + ], + "id": "spec-cart-add" + }, + { + "title": "completes the purchase with a saved card", + "ok": false, + "tags": [], + "tests": [ + { + "timeout": 30000, + "annotations": [], + "expectedStatus": "passed", + "projectId": "project-chromium", + "projectName": "chromium", + "results": [ + { + "workerIndex": 0, + "status": "failed", + "duration": 30000, + "error": { + "message": "Timeout 30000ms exceeded.\n=========================== logs ===========================\nwaiting for locator('button:has-text(\"Place order\")')\n locator resolved to 0 elements\n locator resolved to 0 elements\n locator resolved to 0 elements" + }, + "stdout": [], + "stderr": [], + "retry": 0, + "startTime": "2026-08-03T09:30:02.000Z", + "attachments": [], + "steps": [] + }, + { + "workerIndex": 1, + "status": "failed", + "duration": 30000, + "error": { + "message": "Timeout 30000ms exceeded.\n=========================== logs ===========================\nwaiting for locator('button:has-text(\"Place order\")')\n locator resolved to 0 elements" + }, + "stdout": [], + "stderr": [], + "retry": 1, + "startTime": "2026-08-03T09:30:33.000Z", + "attachments": [], + "steps": [] + }, + { + "workerIndex": 2, + "status": "failed", + "duration": 30000, + "error": { + "message": "Timeout 30000ms exceeded.\n=========================== logs ===========================\nwaiting for locator('button:has-text(\"Place order\")')\n locator resolved to 0 elements\n page.on('dialog') or dialogs opened by user actions" + }, + "stdout": [], + "stderr": [], + "retry": 2, + "startTime": "2026-08-03T09:31:04.000Z", + "attachments": [], + "steps": [] + } + ], + "status": "unexpected" + } + ], + "id": "spec-checkout-purchase" + }, + { + "title": "applies a promo code", + "ok": true, + "tags": [], + "tests": [ + { + "timeout": 30000, + "annotations": [], + "expectedStatus": "passed", + "projectId": "project-chromium", + "projectName": "chromium", + "results": [ + { + "workerIndex": 1, + "status": "passed", + "duration": 934, + "error": null, + "stdout": [], + "stderr": [], + "retry": 0, + "startTime": "2026-08-03T09:30:03.000Z", + "attachments": [], + "steps": [] + } + ], + "status": "expected" + } + ], + "id": "spec-promo" + }, + { + "title": "shows an empty cart message", + "ok": true, + "tags": [], + "tests": [ + { + "timeout": 30000, + "annotations": [], + "expectedStatus": "passed", + "projectId": "project-chromium", + "projectName": "chromium", + "results": [ + { + "workerIndex": 0, + "status": "passed", + "duration": 421, + "error": null, + "stdout": [], + "stderr": [], + "retry": 0, + "startTime": "2026-08-03T09:30:04.000Z", + "attachments": [], + "steps": [] + } + ], + "status": "expected" + } + ], + "id": "spec-empty-cart" + }, + { + "title": "validates cart totals", + "ok": true, + "tags": [], + "tests": [ + { + "timeout": 30000, + "annotations": [], + "expectedStatus": "passed", + "projectId": "project-mobile", + "projectName": "mobile-chromium", + "results": [ + { + "workerIndex": 3, + "status": "passed", + "duration": 1102, + "error": null, + "stdout": [], + "stderr": [], + "retry": 0, + "startTime": "2026-08-03T09:30:05.000Z", + "attachments": [], + "steps": [] + } + ], + "status": "expected" + } + ], + "id": "spec-totals" + }, + { + "title": "runs checkout on a phone viewport", + "ok": false, + "tags": ["@mobile"], + "tests": [ + { + "timeout": 30000, + "annotations": [], + "expectedStatus": "passed", + "projectId": "project-mobile", + "projectName": "mobile-chromium", + "results": [], + "status": "skipped" + } + ], + "id": "spec-checkout-mobile" + } + ], + "suites": [] + } + ] + } + ] +} diff --git a/playwright/tests/test_pwrun.py b/playwright/tests/test_pwrun.py new file mode 100644 index 0000000..14d6572 --- /dev/null +++ b/playwright/tests/test_pwrun.py @@ -0,0 +1,165 @@ +#!/usr/bin/env python3 +"""Deterministic tests for the playwright/scripts/pwrun harness. + +Runs the script as a subprocess so the tests exercise the real CLI surface +(--help, --help --json, doctor, inventory, report, smoke). No node or browser +is required: report analysis and inventory run on stdlib alone, and smoke +degrades gracefully when the Node toolchain is missing. +""" +import json +import os +import shutil +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +SCRIPT = ROOT / "scripts" / "pwrun" +FIXTURE = ROOT / "tests" / "fixtures" / "sample-report.json" + + +def run_script(*args: str, cwd: str | None = None) -> subprocess.CompletedProcess: + return subprocess.run( + [sys.executable, str(SCRIPT), *args], + capture_output=True, + text=True, + cwd=cwd, + timeout=30, + ) + + +class HelpTests(unittest.TestCase): + def test_help_exits_zero_and_advertises_json(self): + proc = run_script("--help") + self.assertEqual(proc.returncode, 0) + self.assertIn("--json", proc.stdout) + for command in ("doctor", "inventory", "report", "smoke"): + self.assertIn(command, proc.stdout) + + def test_help_json_emits_parseable_json(self): + proc = run_script("--help", "--json") + self.assertEqual(proc.returncode, 0) + payload = json.loads(proc.stdout) + self.assertEqual(payload["name"], "pwrun") + self.assertIn("--json", [flag["name"] for flag in payload["flags"]]) + + def test_subcommand_help_exits_zero(self): + for command in ("doctor", "inventory", "report", "smoke"): + proc = run_script(command, "--help") + self.assertEqual(proc.returncode, 0, command) + self.assertIn("--json", proc.stdout) + + +class ReportTests(unittest.TestCase): + def test_report_summarizes_fixture(self): + proc = run_script("report", "--report", str(FIXTURE), "--json") + self.assertEqual(proc.returncode, 1, proc.stderr) # unexpected failures present + payload = json.loads(proc.stdout) + self.assertFalse(payload["ok"]) + self.assertEqual(payload["stats"]["expected"], 5) + self.assertEqual(payload["stats"]["unexpected"], 1) + self.assertEqual(payload["stats"]["skipped"], 1) + self.assertEqual(len(payload["failures"]), 1) + failure = payload["failures"][0] + self.assertEqual(failure["title"], "completes the purchase with a saved card") + self.assertIn("Place order", failure["error"]) + self.assertIn("5 expected, 1 unexpected", payload["summary"]) + + def test_report_flag_survives_subcommand_position(self): + # --report before the subcommand must survive argparse namespace merging. + proc = run_script("--report", str(FIXTURE), "report", "--json") + self.assertEqual(proc.returncode, 1, proc.stderr) + payload = json.loads(proc.stdout) + self.assertFalse(payload["ok"]) + self.assertEqual(payload["stats"]["unexpected"], 1) + + def test_report_requires_file(self): + proc = run_script("report", "--json") + self.assertEqual(proc.returncode, 2) + payload = json.loads(proc.stdout) + self.assertIn("--report", payload["error"]) + + def test_report_rejects_non_json(self): + with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as handle: + handle.write("this is not json") + bad_path = handle.name + try: + proc = run_script("report", "--report", bad_path, "--json") + finally: + os.unlink(bad_path) + self.assertEqual(proc.returncode, 1) + payload = json.loads(proc.stdout) # error path still emits parseable JSON + self.assertFalse(payload["ok"]) + + +class InventoryTests(unittest.TestCase): + def test_inventory_describes_suite(self): + with tempfile.TemporaryDirectory() as tmp: + (Path(tmp) / "playwright.config.ts").write_text( + "export default { projects: [ { name: 'chromium' }, { name: 'firefox' } ] };\n", + encoding="utf-8", + ) + (Path(tmp) / "e2e").mkdir() + (Path(tmp) / "e2e" / "checkout.spec.ts").write_text("import { test } from '@playwright/test';\n", encoding="utf-8") + proc = run_script("inventory", "--json", cwd=tmp) + self.assertEqual(proc.returncode, 0, proc.stderr) + payload = json.loads(proc.stdout) + self.assertTrue(payload["ok"]) + self.assertEqual(payload["config"], "playwright.config.ts") + self.assertIn("chromium", payload["projects"]) + self.assertEqual(payload["spec_count"], 1) + self.assertTrue(payload["specs"][0].endswith("e2e/checkout.spec.ts")) + + def test_inventory_no_config(self): + with tempfile.TemporaryDirectory() as tmp: + proc = run_script("inventory", "--json", cwd=tmp) + self.assertEqual(proc.returncode, 0) + payload = json.loads(proc.stdout) + self.assertIsNone(payload["config"]) + self.assertEqual(payload["spec_count"], 0) + + +class DoctorTests(unittest.TestCase): + def test_doctor_emits_json_without_toolchain(self): + proc = run_script("doctor", "--json") + self.assertEqual(proc.returncode, 0, proc.stderr) + payload = json.loads(proc.stdout) + self.assertIn("node_found", payload) + self.assertIn("config", payload) + self.assertIn("browsers_available", payload) + + def test_doctor_config_detection(self): + with tempfile.TemporaryDirectory() as tmp: + (Path(tmp) / "playwright.config.js").write_text("module.exports = {};\n", encoding="utf-8") + proc = run_script("doctor", "--json", cwd=tmp) + self.assertEqual(proc.returncode, 0) + payload = json.loads(proc.stdout) + self.assertEqual(payload["config"], "playwright.config.js") + + +class SmokeTests(unittest.TestCase): + @unittest.skipUnless(shutil.which("node") is None, "node present; missing-toolchain path not exercised") + def test_smoke_without_node_reports_missing_dependency(self): + proc = run_script("smoke", "--json") + self.assertEqual(proc.returncode, 127) + payload = json.loads(proc.stdout) + self.assertFalse(payload["ok"]) + self.assertIn("node", payload["error"]) + # The command parses without any extra flags; defaults are applied. + self.assertIn("url", payload) + + @unittest.skipIf(shutil.which("node") is None, "node absent; delegate path not exercised") + def test_smoke_with_node_emits_json_envelope(self): + proc = run_script("smoke", "--json") + # With node present but no guaranteed playwright install, the delegate + # exits 0 (pass), 1 (playwright/npx error surfaced as JSON), or 124. + self.assertIn(proc.returncode, (0, 1, 124)) + payload = json.loads(proc.stdout) + self.assertIn("ok", payload) + self.assertIn("command", payload) + + +if __name__ == "__main__": + unittest.main() diff --git a/qa-methodology/SKILL.md b/qa-methodology/SKILL.md index 8feb5c8..81cf1a3 100644 --- a/qa-methodology/SKILL.md +++ b/qa-methodology/SKILL.md @@ -33,6 +33,7 @@ Senior-to-principal QA and SDET methodology: test strategy, automation, regressi |---------|--------------| | Test strategy — what to test, at what level, with what priority | Root-cause debugging — route to [systematic-debugging](../systematic-debugging/SKILL.md) | | Test automation — framework selection, parallelism, flaky management | Security implementation and threat modeling — route to [secure-software-engineering](../secure-software-engineering/SKILL.md) | +| E2E automation strategy and coverage decisions | Operating a browser test tool (Playwright) — authoring/running specs, selectors, network mocking, scraping — route to [playwright](../playwright/SKILL.md) | | Regression suites — selection, impact analysis, suite evolution | Spec pipeline mechanics and gate verdicts — route to [spec-driven-development](../spec-driven-development/SKILL.md) | | Quality gates — blocking vs advisory, metrics, DORA | Eval framework governance and statistics — route to [agent-evals-and-observability](../agent-evals-and-observability/SKILL.md) | | Risk-based testing — P×I scoring, prioritization, registers | Verification verdicts against explicit criteria — route to [verification-methodology](../verification-methodology/SKILL.md) | @@ -123,6 +124,7 @@ Route to the named sibling skill instead: - [release-engineering](../release-engineering/SKILL.md) — composing test evidence into release-candidate readiness, promotion, go/no-go, production rollout, and rollback decisions; QA owns test strategy and gate semantics - [systematic-debugging](../systematic-debugging/SKILL.md) — root-cause analysis of production incidents, bug reproduction, fault localization - [secure-software-engineering](../secure-software-engineering/SKILL.md) — security implementation, threat modeling, secure defaults, dependency evaluation +- [playwright](../playwright/SKILL.md) — operating the Playwright tool itself: authoring and running E2E specs, selector robustness, network mocking, headless scraping, and headed debugging ## Stop and Exit Conditions