magnus919_agent-skills/mobile-development/evals/evals.json

{
  "schema_version": 1,
  "skill_name": "mobile-development",
  "evals": [
    {
      "id": "ios-build-signing-review",
      "prompt": "Our iOS team has a release build that only works on the lead developer's Mac. Reviewing their setup: the .xcodeproj references a distribution certificate and provisioning profile stored in a shared drive, builds are run from Xcode's UI, and the ExportOptions.plist is regenerated by hand each release. Walk me through reviewing and fixing the build and signing setup so any machine or CI runner can produce a signed .ipa.",
      "expected_output": "A build-and-signing review that separates building from signing. The review recommends a reproducible release build: a committed scheme and ExportOptions.plist (secrets-free), the distribution certificate and provisioning profile injected via CI secrets or a secure secrets store rather than a shared drive, and the archive/export flow scripted with xcodebuild -archive and xcodebuild -exportArchive so any machine with the same Xcode version produces the same signed artifact. It flags that regenerating ExportOptions.plist by hand is a review risk (a wrong export method or team produces a rejected upload) and that the certificate/profile must be versioned and renewed on a calendar. It verifies the .ipa actually signs with the distribution identity before upload and that build numbers increment per release.",
      "assertions": [
        "The review separates build configuration from signing assets and keeps secrets out of the repository",
        "The review scripts the archive/export flow with xcodebuild so builds reproduce on any machine or CI runner",
        "The review flags hand-regenerated ExportOptions.plist and shared-drive certificates as risks with concrete fixes",
        "The review covers certificate/profile expiry management and monotonic build numbers",
        "The review verifies the signed .ipa with the distribution identity before store upload"
      ]
    },
    {
      "id": "android-release-signing-readiness",
      "prompt": "We are about to release an Android app to Google Play for the first time. Our release keystore exists but only one engineer has it, the version code has never been set explicitly, and we have been shipping debug APKs to testers. Prepare the release build and signing for the Play Store and tell us exactly what to check before the first upload.",
      "expected_output": "A release-readiness plan for Android: configure signing in build.gradle.kts via a keystore referenced from environment variables or a gitignored keystore.properties (never committed passwords), set versionName and a monotonic versionCode explicitly, and build the Android App Bundle (bundleRelease) since Google Play requires AAB for new apps and signs per-device APKs via Play App Signing with the upload key. The plan backs up the keystore and document who holds it, verifies the signed artifact with apksigner, and lays out the Play Console flow: upload the AAB to internal/closed testing first, complete the data-safety declaration and store listing, then a staged production rollout with crash monitoring. It calls out that losing the keystore means the app identity cannot be updated.",
      "assertions": [
        "The plan configures release signing from secrets and commits no keystore or passwords",
        "The plan sets versionCode and versionName explicitly and builds the AAB required by Play",
        "The plan explains Play App Signing and the upload key versus the app signing key",
        "The plan verifies the artifact signature and tests via internal/closed tracks before production",
        "The plan covers keystore backup/custody and staged rollout with crash monitoring"
      ]
    },
    {
      "id": "mobile-testing-strategy",
      "prompt": "We just built a shopping app for iOS and Android with a shared React Native codebase. So far the only testing is developers tapping through the app on their own phones. Design a realistic mobile testing strategy covering the device and emulator matrix, the right test layers, and what runs in CI versus on a device farm.",
      "expected_output": "A layered mobile testing strategy: fast unit and component tests (Jest + React Native Testing Library) running in CI on every commit; a small set of critical E2E flows (sign-in, checkout, offline sync) with Detox on simulators and emulators in CI; and a device-farm run (Firebase Test Lab or BrowserStack) across the declared device matrix (min SDK, current SDK, mid-range device, both OSes) on every release candidate. The strategy explains what emulators/simulators hide (real networking, sensors, battery, low-memory behavior) and therefore mandates physical-device smoke tests of the signed release build, plus cold-install testing and debug-vs-release parity checks. It assigns ownership: developers own unit/component tests, a release gate requires the E2E suite and device-farm run to pass, and failures block the store submission.",
      "assertions": [
        "The strategy layers unit/component tests, E2E tests, and device-farm coverage with clear CI placement",
        "The strategy defines a concrete device matrix including min and current OS versions on both platforms",
        "The strategy explains the limits of simulators/emulators and requires physical-device release smoke tests",
        "The strategy covers cold-install and debug-vs-release parity checks",
        "The strategy assigns a release gate that blocks store submission until the suite and farm pass"
      ]
    },
    {
      "id": "store-submission-readiness",
      "prompt": "We are submitting our app to both the Apple App Store and Google Play next week. The app collects email addresses, tracks analytics events, and supports sign-in with Apple and Google. Give us a store-readiness review: what must be in place for each store, what will get us rejected, and what order to submit in.",
      "expected_output": "A store-readiness review that walks the submission checklist for both stores. For the App Store: TestFlight beta to internal and external testers first, privacy nutrition labels matching the collected data (email, analytics), Info.plist usage descriptions, complete listing with required screenshots, a demo account or demo mode since the app requires sign-in, and export compliance answers. For Google Play: the AAB uploaded to internal/closed testing, the data-safety form declaring the same data collection, privacy policy URL, and a staged production rollout. It flags the classic rejection causes: undeclared data collection, broken sign-in at review time, crashing on launch, and misleading metadata. It recommends submitting to a testing track on both stores for at least a few days before production review, and staggering the two releases so a regression can be caught on one platform before the other rolls out.",
      "assertions": [
        "The review covers TestFlight and internal/closed testing before production on both stores",
        "The review requires privacy disclosures (nutrition labels and data-safety) to match the app's actual data collection",
        "The review flags the common rejection causes: undeclared data, broken sign-in, launch crashes, misleading metadata",
        "The review covers listing assets, privacy policy, and demo access for sign-in-gated apps",
        "The review recommends staged rollout and staggered cross-platform release timing"
      ]
    },
    {
      "id": "framework-selection-and-scaffolding",
      "prompt": "We are building a new field-service app for technicians: it needs offline access to job lists, barcode scanning, GPS location capture, and push notifications, and our team is experienced in TypeScript but not Swift or Kotlin. We are deciding between native iOS + Android, Flutter, and React Native. Recommend a framework and walk through scaffolding the project correctly.",
      "expected_output": "A framework recommendation grounded in the constraints: the team's TypeScript experience and the need for shared offline, barcode, GPS, and push logic across both platforms point to React Native (Expo for new apps), with native fallback reserved only for features that need deep platform integration beyond existing modules. The response evaluates the alternatives honestly: native iOS/Android give the deepest platform control but double the implementation cost for a TypeScript team, and Flutter is a strong cross-platform option but adds Dart as a new language. It then scaffolds correctly: create-expo-app, TypeScript from the start, React Navigation or Expo Router for navigation, app.json config for bundle IDs, and a plan to verify each required capability (offline SQLite store, barcode library, expo-location, expo-notifications) early with a spike before committing to the full build.",
      "assertions": [
        "The recommendation is grounded in team skills and the required device capabilities, not hype",
        "The response compares native, Flutter, and React Native with honest tradeoffs",
        "The response scaffolds with the canonical tool for the chosen framework with TypeScript",
        "The response covers navigation, config, and bundle IDs at scaffold time",
        "The response verifies the hard device capabilities (offline, barcode, GPS, push) with an early spike"
      ]
    },
    {
      "id": "offline-sync-design",
      "prompt": "Our field-service app must work in basements and rural areas with no connectivity for hours. Technicians create and complete jobs offline, and those changes must reach the server without data loss or duplication, while the same job may be edited by a dispatcher at the same time. Design the offline storage and sync architecture.",
      "expected_output": "An offline-first architecture: local storage (SQLite-based, e.g., WatermelonDB/Realm or Room/drift depending on stack) is the source of truth while disconnected; every write is persisted locally and queued as a mutation with a client-generated ID so replay is idempotent. Sync runs when connectivity returns (with retry and exponential backoff, and a network-constraint-triggered job), uploading queued mutations in order and downloading changes since a cursor. Conflict resolution is explicit: per-field merge with timestamps for independent edits, and a visible conflict-resolution flow when the same field changed on both sides — never silent last-write-wins over newer remote data. The design covers partial sync (only the technician's assigned jobs), a clear sync-state indicator in the UI, and a strategy for handling the same job edited by a dispatcher concurrently, including server-side revision tokens.",
      "assertions": [
        "The design makes local storage the source of truth while offline and persists writes immediately",
        "The design queues mutations with idempotent replay and retry with backoff on reconnect",
        "The design defines explicit conflict resolution for concurrent edits, including the dispatcher case",
        "The design covers partial sync and a user-visible sync state indicator",
        "The design includes server-side revision tokens or equivalent to detect stale writes"
      ]
    }
  ]
}