diff --git a/README.md b/README.md index cebd7dd..7cb86cd 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,7 @@ AI agent skills for the [Stably Playwright SDK](https://docs.stably.ai/getting-s | [stably-sdk-rules](./skills/stably-sdk-rules) | AI rules for writing tests with Stably SDK | `npx skills add stablyai/agent-skills --skill stably-sdk-rules` | | [stably-cli](./skills/stably-cli) | Expert assistant for Stably CLI commands (including remote envs via `--env`) | `npx skills add stablyai/agent-skills --skill stably-cli` | | [stably-verify](./skills/stably-verify) | Run existing tests to verify code changes work — ideal for AI agent iteration loops | `npx skills add stablyai/agent-skills --skill stably-verify` | +| [google-auth-account-setup](./skills/google-auth-account-setup) | Step-by-step Google account setup for automated auth testing with Stably SDK | `npx skills add stablyai/agent-skills --skill google-auth-account-setup` | ## Installation @@ -37,6 +38,9 @@ npx skills add stablyai/agent-skills --skill stably-cli # Verify skill - use when validating code changes with existing tests npx skills add stablyai/agent-skills --skill stably-verify + +# Google auth setup - use when setting up Google accounts for auth testing +npx skills add stablyai/agent-skills --skill google-auth-account-setup ``` Or install from URL: diff --git a/skills/google-auth-account-setup/SKILL.md b/skills/google-auth-account-setup/SKILL.md new file mode 100644 index 0000000..428b702 --- /dev/null +++ b/skills/google-auth-account-setup/SKILL.md @@ -0,0 +1,138 @@ +--- +name: google-auth-account-setup +description: Step-by-step guide for setting up a Google account for automated authentication testing with Stably SDK. Covers Google 2FA/OTP configuration, environment variables, context.authWithGoogle() usage, and troubleshooting common Google auth issues (blocked sign-in, OTP failures, session problems). +license: MIT +metadata: + author: stably + version: '1.0.0' +--- + +# Google Auth Account Setup + +Complete guide for configuring a Google account for automated authentication testing with the Stably Playwright SDK. + +## Prerequisites + +- A **dedicated Google Workspace test account** — never use a personal account +- You will need: email, password, and the OTP secret (obtained in the setup steps below) + +## Step-by-Step Google Account Setup + +### 1. Sign in to the test Google account + +Go to [accounts.google.com](https://accounts.google.com) and sign in with the test account credentials. + +### 2. Navigate to Security settings + +From the left sidebar, click **Security**. + +### 3. Find the Authenticator setting + +Scroll to **"How you sign in to Google"** and click **"Authenticator"**. + +### 4. Add or change the authenticator app + +Click **"Add Authenticator"** (or **"Change Authenticator App"** if one is already set up). + +> **Warning:** Clicking "Change Authenticator App" invalidates any previous authenticator setup. + +### 5. Reveal the OTP secret + +When the QR code appears, click **"Can't scan QR code"** to reveal the OTP secret — a 32-character base32 string (letters A–Z, digits 2–7). + +### 6. Copy the OTP secret + +Copy the secret string. This becomes your `otpSecret` parameter (and the `GOOGLE_TEST_OTP_SECRET` environment variable). + +### 7. Complete Google's setup with a verification code + +Paste the OTP secret into the **Stably OTP helper** at [https://app.stably.ai/utils/google-otp-secret](https://app.stably.ai/utils/google-otp-secret) and copy the generated 6-digit code. + +### 8. Verify the code in Google + +Return to Google's verification screen, paste the 6-digit code, and click **"Verify"**. + +### 9. Ensure only Authenticator is enabled for 2FA + +Back on the Google Account security page, confirm that: +- 2-Step Verification is **ON** +- Under "Second Steps", only **Authenticator** is enabled +- Disable phone prompts, backup codes, and any other second-step methods +- Sign out of Google on mobile devices if needed (phone prompts can interfere) + +### 10. Store credentials as environment variables + +```bash +GOOGLE_TEST_EMAIL=qa@example.com +GOOGLE_TEST_PASSWORD=... +GOOGLE_TEST_OTP_SECRET=... # the 32-char base32 string from step 6 +``` + +Add these to your `.env` file (must be gitignored) or your CI secret manager. + +## Quick Code Reference + +### Recommended: context method + +```ts +await context.authWithGoogle({ + email: process.env.GOOGLE_TEST_EMAIL!, + password: process.env.GOOGLE_TEST_PASSWORD!, + otpSecret: process.env.GOOGLE_TEST_OTP_SECRET!, +}); +``` + +### Alternative: standalone import + +```ts +import { authWithGoogle } from "@stablyai/playwright-test"; + +await authWithGoogle({ + context, + email: process.env.GOOGLE_TEST_EMAIL!, + password: process.env.GOOGLE_TEST_PASSWORD!, + otpSecret: process.env.GOOGLE_TEST_OTP_SECRET!, +}); +``` + +### Signature + +```ts +authWithGoogle(options: { + context: BrowserContext; + email: string; + password: string; + otpSecret: string; + forceRefresh?: boolean; // bypass cached session +}): Promise +``` + +### Required environment variables + +| Variable | Description | +|----------|-------------| +| `GOOGLE_TEST_EMAIL` | Test account email address | +| `GOOGLE_TEST_PASSWORD` | Test account password | +| `GOOGLE_TEST_OTP_SECRET` | 32-char base32 OTP secret from step 6 | + +## Troubleshooting + +| Problem | Solution | +|---------|----------| +| **Google blocks sign-in** ("Try again later", "unusual activity") | Use a dedicated test account, avoid rapid successive logins, try from a different IP | +| **OTP code expired** | Codes rotate every 30 seconds — use the [Stably OTP helper](https://app.stably.ai/utils/google-otp-secret) for a fresh code | +| **2FA prompt still appears despite automation** | Ensure only Authenticator is enabled under "Second Steps" (disable phone prompts, backup codes, etc.) | +| **Session cache stale** | Use `forceRefresh: true` in `authWithGoogle` options | +| **OTP secret looks wrong** | Must be a 32-character base32 string (letters A–Z, digits 2–7), no spaces | + +## Best Practices + +- **Dedicated test accounts only** — never use personal Google accounts +- **Store credentials securely** — `.env` files (gitignored) or CI secret managers +- **Use Stably Environments** — `stably --env staging test` for secure credential storage in CI +- **Reuse auth sessions** — create one auth smoke test, save `storageState`, reuse for other tests +- **Rotate credentials** — rotate test account passwords and OTP secrets on a schedule + +## Full Documentation + +- [Stably Google Auth Docs](https://docs.stably.ai/stably2/auth/auth-with-google) diff --git a/skills/stably-sdk-rules/SKILL.md b/skills/stably-sdk-rules/SKILL.md index 72e575e..9f6262f 100644 --- a/skills/stably-sdk-rules/SKILL.md +++ b/skills/stably-sdk-rules/SKILL.md @@ -1,18 +1,13 @@ --- name: stably-sdk-rules -description: | - AI rules for writing tests with Stably Playwright SDK. - Use this skill when writing or modifying Playwright tests with Stably AI - features. Covers when to use Playwright vs Stably methods, plus minimal - patterns for aiAssert, extract, getLocatorsByAI, agent.act, Inbox, and - Google auth. +description: AI rules and complete reference for writing tests with Stably Playwright SDK. Covers when to use Playwright vs Stably methods, plus detailed patterns for aiAssert, extract, getLocatorsByAI, agent.act, Inbox, Google auth, and model selection. license: MIT metadata: author: stably - version: '1.1.0' + version: '2.0.0' --- -# Stably SDK Rules +# Stably SDK Rules & Reference ## Quick Rules @@ -29,7 +24,7 @@ metadata: 11. Always add `.describe("...")` to locators for trace readability. 12. For email isolation, use unique `Inbox.build({ suffix })` per test and clean up. -## Setup (Single Block) +## Setup ```bash npm install -D @playwright/test @stablyai/playwright-test @stablyai/email @@ -62,91 +57,322 @@ setApiKey("YOUR_KEY"); - Include explicit target, intent, and constraints. - Pass cross-step data through variables, not vague references. -## Minimal Usage Patterns +**Note:** `toMatchScreenshotPrompt` is deprecated — use `aiAssert` instead. Do NOT use Playwright's `toHaveScreenshot()` — it is not supported. Use `aiAssert` for ALL visual assertions. -### `aiAssert` +## AI Assertions (aiAssert) + +Use `aiAssert` for intent-based visual verification of dynamic UIs: ```ts -await expect(page).aiAssert("Shows revenue trend chart and spotlight card"); -await expect(page.locator(".header").describe("Header")).aiAssert("Has nav, avatar, and bell icon"); +// Page-level assertion +await expect(page).aiAssert( + "Shows revenue trend chart and spotlight card", + { timeout: 30_000 } +); + +// Scoped to specific element (preferred for precision) +await expect(page.locator(".header")) + .aiAssert("Nav with avatar and bell icon"); ``` -Use `fullPage: true` only when assertion needs off-screen content. +**Signature:** `expect(page|locator).aiAssert(prompt: string, options?: { timeout?: number, fullPage?: boolean, model?: AIModel })` -### `extract` +**Best Practices:** +- Use for **dynamic** UIs where deterministic assertions are insufficient +- Keep prompts specific with labels and units +- Scope with locators when possible (more precise, less noisy) +- **Consider `fullPage: true` carefully**: Only use when assertion requires content beyond the visible viewport. Viewport captures are faster and cheaper. -```ts -const orderId = await page.extract("Extract the order ID from the first row"); -``` +## AI Extraction (extract) -With schema: +Extract data from visual content: ```ts +// Simple string extraction +const txt = await page.extract("List revenue, active users, and churn rate"); + +// Typed extraction with Zod schema import { z } from "zod"; -const Schema = z.object({ revenue: z.string(), users: z.number() }); -const metrics = await page.extract("Get revenue and active users", { schema: Schema }); +const Metrics = z.object({ + revenue: z.string(), + activeUsers: z.number(), + churnRate: z.number() +}); +const m = await page.extract( + "Return revenue (currency), active users, churn %", + { schema: Metrics } +); ``` -### `getLocatorsByAI` +Before using `schema` with Zod, check the test project's `package.json` and install `zod` as a dev dependency if missing. If it already exists as a regular dependency, leave it as-is. -Requires Playwright `>= 1.54.1`. +**Signatures:** +- `page.extract(prompt: string, options?: { model?: AIModel }): Promise` +- `page.extract(prompt, { schema: T, model?: AIModel }): Promise>` + +## AI Locator Finding (getLocatorsByAI) + +Find elements using natural language based on accessibility properties: ```ts -const { locator, count } = await page.getLocatorsByAI("the login button"); +// Find a single element +const { locator: loginBtn, count } = await page.getLocatorsByAI("the login button"); expect(count).toBe(1); -await locator.describe("Login button located by AI").click(); +await loginBtn.click(); + +// Find multiple elements +const { locator: productCards, count: cardCount } = await page.getLocatorsByAI( + "all product cards in the grid" +); +await expect(productCards).toHaveCount(cardCount); +``` + +**Signature:** `page.getLocatorsByAI(prompt: string, options?: { model?: AIModel }): Promise<{ locator: Locator, count: number, reason: string }>` + +**Properties:** +- Returns a Playwright `Locator` usable for interactions and assertions +- `count` indicates how many elements were found (0 if none) +- `reason` contains the AI's explanation of what it found +- Requires Playwright v1.54.1 or higher + +**Best Practices:** +- Describe elements by accessible properties (labels, roles, text) rather than visual attributes (colors, positioning) +- Best for finding elements when CSS selectors or test IDs are unreliable + +## AI Agent (agent.act) + +Use the `agent` fixture for complex, autonomous workflows: + +```ts +test("complex workflow", async ({ agent, page }) => { + await page.goto("/orders"); + await agent.act("Find the first pending order and mark it as shipped", { page }); +}); + +// Or create manually +const agent = context.newAgent(); +await agent.act("Your task here", { page, maxCycles: 10 }); ``` -### `agent.act` +**Signature:** `agent.act(prompt: string, options: { page: Page, maxCycles?: number, model?: string }): Promise<{ success: boolean }>` + +**Default maxCycles:** 30 +**Supported models:** `anthropic/claude-sonnet-4-6` (default), `google/gemini-2.5-computer-use-preview-10-2025` + +### Passing Variables to Prompts + +Use template literals to pass variables: + +```ts +const duration = 24 * 7 * 60; +await agent.act(`Enter the duration of ${duration} seconds`, { page }); + +const username = "john.doe@example.com"; +await agent.act(`Login with username ${username}`, { page }); +``` + +### Self-Contained Prompts (CRITICAL) + +All prompts to Stably SDK AI methods must be self-contained with all necessary information: + +1. **No implicit references to outside context:** + - Bad: `agent.act("Verify the field you just filled in the form is 4", { page })` + - Good: `agent.act("Verify the 'timeout' field in the form has value 4", { page })` + - Bad: `agent.act("Pick something that's not in the previous step", { page })` + - Good: `const selectedItem = "Option A"; await agent.act(\`Pick an option other than ${selectedItem}\`, { page })` + +2. **Pass information between AI methods using explicit variables:** + ```ts + const orderId = await page.extract("Get the order ID from the first row"); + await agent.act(`Cancel order with ID ${orderId}`, { page }); + ``` + +3. **Include detailed instructions and domain knowledge:** + - Bad: `agent.act("Fill in the form", { page })` + - Good: `agent.act("Fill in the form with test data. On page 4 you might run into a popup asking for premium features - just click 'Skip' or 'Cancel' to ignore it", { page })` + +### Offload Work to Playwright + +The less actions/cycles agent.act() needs, the better it performs. Offload work to Playwright code: + +1. **Repetition:** Use loops in code, not in prompts + - Bad: "Click the button 5 times" + - Good: "Click the button" (in a loop that runs 5 times) + +2. **Calculations:** Calculate in code, pass result to prompt + - Bad: "enter the duration of 24*7*60 seconds" + - Good: `const sum = 24*7*60; agent.act(\`enter the duration of ${sum} seconds\`, { page })` + +3. **Conditionals:** Use code for if/else when possible + +### agent.act() Best Practices + +1. **Split complex prompts into smaller tasks:** + ```ts + // Bad - too many steps + await agent.act('Close popups, click menu, expand panel, find sliders', { page, maxCycles: 15 }); + + // Good - single responsibility + await agent.act('Close the tutorial popup if visible', { page, maxCycles: 5 }); + await agent.act('Click the color menu and expand Basic Color panel', { page, maxCycles: 8 }); + ``` + +2. **Be specific in prompts** - Include visual hints: + ```ts + // Bad + await agent.act('Adjust the slider', { page }); + + // Good + await agent.act('Move the "Brightness" slider (the one with the sun icon) to approximately 75%', { page }); + ``` + +3. **Use appropriate maxCycles:** + - Simple single action: 3-5 cycles + - Multi-step interaction: 8-10 cycles + - Complex workflow: 15-20 cycles + - Never rely on the default 30 cycles + +## Model Selection + +`aiAssert`, `extract`, and `getLocatorsByAI` support an optional `model` parameter: ```ts -await agent.act("Find the first pending order and mark it as shipped", { page }); +await expect(page).aiAssert("Shows the dashboard", { model: "google/gemini-3-flash-preview" }); +const data = await page.extract("Get the price", { model: "openai/o4-mini" }); +const { locator } = await page.getLocatorsByAI("the submit button", { model: "google/gemini-3-pro-preview" }); ``` -Good pattern: compute values in code, then pass concrete values into the prompt. +**Available models:** +- `"openai/o4-mini"` - OpenAI's efficient reasoning model +- `"google/gemini-3-pro-preview"` - Google's most capable model +- `"google/gemini-3-flash-preview"` - Google's fast, efficient model (good for simple tasks) + +**Tips:** +- Use `gemini-3-flash-preview` for fast, simple operations +- Use `gemini-3-pro-preview` or `o4-mini` for complex reasoning -### `Inbox` (Email Isolation) +## Email Inbox Testing (@stablyai/email) + +Use disposable email inboxes for testing email-dependent flows: ```ts +import { Inbox } from "@stablyai/email"; + +// Create a test-scoped inbox const inbox = await Inbox.build({ suffix: `test-${Date.now()}` }); -await page.getByLabel("Email").describe("Email input").fill(inbox.address); +// inbox.address → "org+test-1706621234567@mail.stably.ai" + +// Wait for an email +const email = await inbox.waitForEmail({ + from: "noreply@example.com", + subject: "verification", + timeoutMs: 60_000, +}); -const email = await inbox.waitForEmail({ subject: "verification", timeoutMs: 60_000 }); +// AI-powered extraction const { data: otp } = await inbox.extractFromEmail({ id: email.id, prompt: "Extract the 6-digit OTP code", }); -await inbox.deleteAllEmails(); +// Structured extraction with Zod +import { z } from "zod"; +const { data } = await inbox.extractFromEmail({ + id: email.id, + prompt: "Extract verification URL and expiration", + schema: z.object({ url: z.string().url(), expiresIn: z.string() }), +}); ``` -## Auth Flows (Google) +### Playwright Fixture Pattern -Use the helper instead of custom popup scripting: +For test isolation and automatic cleanup: ```ts -import { authWithGoogle } from "@stablyai/playwright-test/auth"; +import { test as base } from "@stablyai/playwright-test"; +import { Inbox } from "@stablyai/email"; -await authWithGoogle({ - context, - email: process.env.GOOGLE_AUTH_EMAIL!, - password: process.env.GOOGLE_AUTH_PASSWORD!, - otpSecret: process.env.GOOGLE_AUTH_OTP_SECRET!, +const test = base.extend<{ inbox: Inbox }>({ + inbox: async ({}, use, testInfo) => { + const inbox = await Inbox.build({ suffix: `test-${testInfo.testId}` }); + await use(inbox); + await inbox.deleteAllEmails(); + }, }); ``` -Required env vars: -- `GOOGLE_AUTH_EMAIL` -- `GOOGLE_AUTH_PASSWORD` -- `GOOGLE_AUTH_OTP_SECRET` +### Key Options + +**waitForEmail:** `from`, `subject`, `subjectMatch` (`'contains'` | `'exact'`), `timeoutMs` (default 120000), `pollIntervalMs` (default 3000) + +**extractFromEmail:** `id` (required), `prompt` (required), `schema` (optional Zod). Returns `{ data, reason }` + +**Best practices:** +- Always use unique suffixes for parallel test isolation +- Use the fixture pattern for automatic cleanup +- Prefer `waitForEmail` over polling with `listEmails` + +## Google Auth (authWithGoogle) + +Use the built-in helper instead of custom popup scripting: + +```ts +// Recommended: context method (no need to pass context) +await context.authWithGoogle({ + email: process.env.GOOGLE_TEST_EMAIL!, + password: process.env.GOOGLE_TEST_PASSWORD!, + otpSecret: process.env.GOOGLE_TEST_OTP_SECRET!, +}); + +// Alternative: standalone import +import { authWithGoogle } from "@stablyai/playwright-test"; +await authWithGoogle({ context, email, password, otpSecret }); +``` + +**Signature:** `authWithGoogle(options: { context: BrowserContext, email: string, password: string, otpSecret: string, forceRefresh?: boolean, apiKey?: string }): Promise` + +**Required env vars:** `GOOGLE_TEST_EMAIL`, `GOOGLE_TEST_PASSWORD`, `GOOGLE_TEST_OTP_SECRET` Use a dedicated test Google account only. -## Troubleshooting (Short) +For step-by-step Google account setup (2FA, OTP secret, environment variables), invoke the `google-auth-account-setup` skill. + +## When to Use SDK vs Playwright + +**Use Playwright** when: +- Simple, concrete checks (element visible, text matches, URL correct) +- Faster and more reliable for deterministic scenarios + +**Use Stably SDK** when: +1. Test accuracy and stability are paramount +2. Interactions are hard to express in Playwright or too brittle +3. Canvas-related operations or drag/click requiring coordinates → use `agent.act()` +4. Visual-heavy assertions → use `aiAssert` +5. Email verification flows → use `@stablyai/email` + +## Minimal Template + +```ts +import { test, expect } from "@stablyai/playwright-test"; + +test("AI-enhanced dashboard", async ({ page, agent }) => { + await page.goto("/dashboard"); + + // Use agent for complex workflows + await agent.act("Navigate to settings and enable notifications", { page }); + + // Use AI assertions for dynamic content + await expect(page).aiAssert( + "Dashboard shows revenue chart (>= 6 months) and account spotlight card" + ); +}); +``` + +## Troubleshooting -- `aiAssert` is slow/flaky: scope to a locator, tighten prompt, avoid unnecessary `fullPage: true`. -- `agent.act` fails: split into smaller tasks, pass explicit constraints, raise `maxCycles` only when needed. -- Email timeout: verify subject/from filter and use unique inbox suffixes. +- **Slow assertions** → Scope visuals with locators; reduce viewport +- **Agent stops early** → Increase `maxCycles` or break task into smaller steps +- **Email timeout** → Verify subject/from filter and use unique inbox suffixes ## Full References @@ -154,4 +380,3 @@ Use a dedicated test Google account only. - SDK setup skill: `skills/stably-sdk-setup/SKILL.md` - Package docs: https://www.npmjs.com/package/@stablyai/playwright-test - Email package docs: https://www.npmjs.com/package/@stablyai/email -- Local overview: `skills/stably-sdk-rules/README.md`