Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 46 additions & 5 deletions android-emulator-skill/skills/android-emulator-skill/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,16 @@ All scripts support `--help` for detailed options and `--json` for machine-reada

#### App Management (1 script)
4. **app_launcher.py** - App lifecycle management
- Launch apps by package name
- Launch apps by package name. `--launch` **waits for the activity to be
displayed** (`am start -W`) and fails if the system reports anything but
`Status: ok`, so the next command maps the app's screen rather than
whatever was still in front. The message reports the activity that came up
and how long it took.
- Terminate apps
- Install/uninstall APKs
- Deep link navigation
- Deep link navigation. `--open-url` waits the same way `--launch` does and
fails when the intent resolves to nothing, rather than reporting a URL it
merely handed to the system.
- List installed packages
- Check app state
- Options: `--launch`, `--terminate`, `--install`, `--uninstall`, `--open-url`, `--list`, `--state`, `--json`
Expand Down Expand Up @@ -128,17 +134,49 @@ All scripts support `--help` for detailed options and `--json` for machine-reada

#### Navigation & Interaction (4 scripts)
12. **screen_mapper.py** - Analyze current screen and list interactive elements
- Count elements by type
- Listing is the default, not a mode: **there is no `--list`.** The bare
command prints the summary; `--verbose` expands it to the per-element
breakdown, `--hints` adds navigation suggestions.
- Token-efficient summaries
- The default output **names every interactive control**, one line per kind
(`Button:`, `Control:`, `CheckBox:`, `EditText:` …), capped per kind. Those
names are exactly what `navigator.py --find-text` accepts, so step 2 of
Quick Start feeds step 3 directly. It used to print counts only, and on a
Compose screen the names existed solely under `--verbose`/`--json`.
- A control with no text of its own is named by the caption recovered from
its subtree or its row; one with a resource id is named by the **bare** id
(`com.android.settings:id/search_action_bar` prints as
`search_action_bar`), which is what `navigator` prints and what
`--find-id` takes. Compose test tags are already bare, so both toolkits
name things the same way.
- `--hints` prints the next command to run, already filled in.
- Options: `--serial`/`-s`, `--verbose`/`-v`, `--hints`, `--json`

13. **navigator.py** - Find and interact with elements semantically
- Find by text, type, resource ID
- Tap, enter text, get bounds
- Fuzzy matching support
- `--find-text` matches **any name the screen report printed**: an element's
own text or content-desc, the caption recovered for an unlabelled control,
or a resource id (bare or fully qualified, matched whole). A match on a
caption resolves to the control that owns it -- the control it sits
inside, or the one beside it in the same row -- so the tap lands on the
checkbox rather than 143px to its right. The owner must be **tappable**
(clickable / long-clickable / checkable): a scrolling container is
interactive, but it is not what a caption inside it names, and resolving
to it would tap the middle of the screen. A name that matches only a
passive label with no such control is refused, not tapped.
- `--tap` and `--enter-text` **require a target**: one of `--find-text`,
`--find-exact`, `--find-type`, `--find-id`, or explicit `--tap-at x,y`.
Without one it is a usage error (exit 2) and nothing is sent to the device.
- `--max-scrolls` is capped at 50 (so is `ANDROID_EMU_MAX_SCROLLS`), and a
`--scroll-to-find` search is also bounded in wall-clock time
(`ANDROID_EMU_SCROLL_SEARCH_DEADLINE`, default 120s -- see `--help`),
which bounds the screen dumps too. Each scroll prints a progress line to
stderr as it happens.
- Under `--json`, a successful `--tap`/`--enter-text` reports
**`tapped_at: [x, y]`** -- the coordinates that reached the device, so a
caller can check where the tap went without parsing prose. Every failure,
including a usage error, prints `{"error": ...}` and exits non-zero.
- **`--scroll-to-find`** searches below the fold. Without it a lookup sees only the visible
screen, and `Not found` is indistinguishable from "the item is two rows down". The default
path now says which it was: `(searched 1 screen; this screen scrolls -- retry with
Expand Down Expand Up @@ -178,7 +216,10 @@ All scripts support `--help` for detailed options and `--json` for machine-reada

#### Testing & Analysis (4 scripts) ✓ COMPLETE
16. **accessibility_audit.py** ⭐ NEW - WCAG compliance checking
- Missing content descriptions
- Missing content descriptions: any **operable** control (by its uiautomator
properties, not its class name) with no describing text in itself or its
subtree is critical. The old class-name gate could not fire on a Compose
screen at all, where controls are plain `android.view.View`.
- Touch target size verification
- EditText hint checking
- Image accessibility
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,14 @@

import argparse
import json
import re
import sys
from datetime import datetime
from pathlib import Path

from common.adb_exec import AdbError
from common.device_utils import get_device_density, get_ui_hierarchy
from common.env_config import env_int
from common.hierarchy import is_interactive, parse_bounds

# Tunable thresholds (overridable via env, ANDROID_EMU_ prefix).
A11Y_MAX_NESTING = env_int("ANDROID_EMU_A11Y_MAX_NESTING", 5)
Expand Down Expand Up @@ -67,15 +67,6 @@ def _fix_for(issue_type: str) -> str:
_SEVERITY_ORDER = {"critical": 0, "warning": 1, "info": 2}


def _parse_bounds(bounds_str: str) -> dict:
"""Parse a uiautomator bounds string '[l,t][r,b]' into a dict of ints."""
match = re.match(r"\[(-?\d+),(-?\d+)\]\[(-?\d+),(-?\d+)\]", bounds_str or "")
if not match:
return {}
left, top, right, bottom = (int(g) for g in match.groups())
return {"left": left, "top": top, "right": right, "bottom": bottom}


def _attr_bool(value: str) -> bool:
"""Coerce a uiautomator string attribute ('true'/'false') to a bool."""
return str(value).lower() == "true"
Expand Down Expand Up @@ -149,6 +140,28 @@ def _descendants(node: dict):
yield child
yield from AccessibilityAuditor._descendants(child)

@staticmethod
def _own_label(node: dict) -> str:
"""The text this node carries in its own right."""
attributes = node.get("attributes", {})
return (attributes.get("text") or "").strip() or (
attributes.get("content-desc") or ""
).strip()

@classmethod
def _has_label(cls, node: dict) -> bool:
"""Whether anything names this control -- its own text, or its subtree's.

A screen reader announces a node from its own ``text``/``contentDescription``
or from those of the nodes it contains. A caption that merely sits
*beside* the control is not in either place, which is why a sibling does
not count here even though `screen_mapper` uses one to name the control
for a sighted agent: that caption is exactly the accessibility defect.
"""
return bool(cls._own_label(node)) or any(
cls._own_label(child) for child in cls._descendants(node)
)

def audit_tree(self, hierarchy: dict) -> list:
"""Run every check over an already-fetched hierarchy.

Expand Down Expand Up @@ -216,35 +229,53 @@ def _audit_node(self, node: dict, depth: int = 0):
"""
attrs = node.get("attributes", {})
class_name = attrs.get("class", "")
bounds = _parse_bounds(attrs.get("bounds", ""))
clickable = _attr_bool(attrs.get("clickable", "false"))
enabled = _attr_bool(attrs.get("enabled", "true"))
# The rectangle, in the dict shape the report's `element` payload has
# always carried. Shaped at the one place that needs it rather than in a
# local `_parse_bounds`: this file used to own one of the skill's three
# bounds grammars, and a wrapper of that name is where a fourth would
# grow back (C5/C7).
box = parse_bounds(attrs.get("bounds"))
bounds = {"left": box[0], "top": box[1], "right": box[2], "bottom": box[3]} if box else {}
# The one eligibility rule, asked once and used by every check below
# that is about a control. `clickable and enabled` was a second rule
# living beside it: it counted a node whose rectangle is collapsed or
# unreadable, and missed every Compose control driven by `checkable`,
# `long-clickable` or `scrollable` rather than by `clickable` (C7).
interactive = is_interactive(node)
text = attrs.get("text", "")
content_desc = attrs.get("content-desc", "")
resource_id = attrs.get("resource-id", "")

# Check 1: Interactive elements need content description
if clickable and enabled and not content_desc and not text:
# Buttons, ImageButtons, etc. need descriptions
if any(
widget in class_name.lower() for widget in ["button", "imagebutton", "imageview"]
):
self.issues.append(
{
"type": "missing_content_description",
"severity": "critical",
"message": f"Interactive {class_name} missing content description",
"fix": _fix_for("missing_content_description"),
"element": {
"class": class_name,
"resource_id": resource_id,
"bounds": bounds,
},
}
)
# Check 1: a control a screen reader cannot announce.
#
# Eligibility is `hierarchy.is_interactive` and the label test is "is
# there any describing text in this node or below it" -- not a class
# name. The class-name gate ("button", "imagebutton", "imageview") could
# not fire on a Compose screen at all: Compose renders its controls as
# `android.view.View`, so the check that Quick Start step 5 exists to
# run reported zero criticals on every Compose app ever audited (L3).
# It also missed a clickable `LinearLayout` row, which is how most
# View-based lists are built.
if interactive and not self._has_label(node):
self.issues.append(
{
"type": "missing_content_description",
"severity": "critical",
"message": (
f"Interactive {class_name or 'element'} has no label: nothing in its "
f"own text, content-desc or subtree names it"
),
"fix": _fix_for("missing_content_description"),
"element": {
"class": class_name,
"resource_id": resource_id,
"bounds": bounds,
},
}
)

# Check 2: Touch target size. Bounds are pixels; the minimum is dp.
if clickable and enabled and bounds:
if interactive and bounds:
width = bounds.get("right", 0) - bounds.get("left", 0)
height = bounds.get("bottom", 0) - bounds.get("top", 0)
minimum_px = self.min_touch_target_px()
Expand Down Expand Up @@ -290,28 +321,22 @@ def _audit_node(self, node: dict, depth: int = 0):
# attribute -- verified across every recorded dump -- so the condition
# collapsed to "this field is empty" and flagged every correctly-hinted
# empty field. A field's label is discoverable, just not there: Compose
# puts a TextField's label in its subtree, and View layouts often place
# it in an adjacent node. Only flag a field with no describing text
# anywhere beneath it.
if "edittext" in class_name.lower():
described_by_child = any(
(child.get("attributes", {}).get("text") or "").strip()
or (child.get("attributes", {}).get("content-desc") or "").strip()
for child in self._descendants(node)
# puts a TextField's label in its subtree. Only flag a field with no
# describing text in it or beneath it -- the same label test check 1
# applies, so the two cannot drift into disagreeing about "labelled".
if "edittext" in class_name.lower() and not self._has_label(node):
self.issues.append(
{
"type": "edittext_missing_hint",
"severity": "warning",
"message": "EditText missing hint text",
"fix": _fix_for("edittext_missing_hint"),
"element": {
"class": class_name,
"resource_id": resource_id,
},
}
)
if not described_by_child and not text and not content_desc:
self.issues.append(
{
"type": "edittext_missing_hint",
"severity": "warning",
"message": "EditText missing hint text",
"fix": _fix_for("edittext_missing_hint"),
"element": {
"class": class_name,
"resource_id": resource_id,
},
}
)

# Check 5: Text readability
if text and len(text) > 100:
Expand All @@ -331,7 +356,7 @@ def _audit_node(self, node: dict, depth: int = 0):
)

# Check 6: Interactive elements should have a resource-id for testing
if clickable and enabled and not resource_id:
if interactive and not resource_id:
self.issues.append(
{
"type": "missing_resource_id",
Expand Down
Loading
Loading