Route-planning SMS verifiers are un-passable: `check_sms_via_adb` returns `bool`, but the verifier uses it as the SMS text

[threegold116]

Summary

Four agent-mcp route-planning tasks can never be scored 1.0 by any agent, because their
verifiers treat the return value of check_sms_via_adb(...) as the SMS body string, but that
function is declared and implemented to return a bool. When the agent actually sends the SMS,
the verifier crashes with AttributeError: 'bool' object has no attribute 'split' and the task is
scored 0.0 (eval error). When the agent does not send the SMS, the same verifier silently
returns "SMS not found" — so the design flaw is masked as a normal failure.

Net effect: these tasks are dead tasks — a perfect agent still gets 0.0.

Affected files

All four share the identical pattern (sms_content = check_sms_via_adb(..., content="") immediately
followed by sms_content.split("\n")):

Task	File:line
PlanTaxiRouteSmsTask	src/mobile_world/tasks/definitions/messages/plan_taxi_route_sms.py:65,69
PlanDrivingRouteSmsTask	src/mobile_world/tasks/definitions/messages/plan_driving_route_sms.py:63,67
PlanCyclingRouteSmsTask	src/mobile_world/tasks/definitions/messages/plan_cycling_route_sms.py:65,69
PlanCommuteRouteSmsTask	src/mobile_world/tasks/definitions/messages/plan_commute_route_sms.py:60,64

Root cause

check_sms_via_adb (src/mobile_world/runtime/app_helpers/system.py:121) is defined as:

def check_sms_via_adb(
    controller: AndroidController, phone_number: str, content: str | list[str]
) -> bool:
    """... Returns: bool: True if matching SMS is found, False otherwise """

The verifier (e.g. plan_taxi_route_sms.py:60-72) does:

sms_content = check_sms_via_adb(controller, phone_number=self.RECIPIENT_PHONE, content="")
if not sms_content:                                          # L66
    return 0.0, f"SMS not found sent to {self.RECIPIENT_PHONE}"
lines = [line.strip() for line in sms_content.split("\n") if line.strip()]   # L69  <-- crashes
first_line = lines[0]
# ... subsequent checks parse the SMS body line-by-line ...

Two problems compound:

1. Wrong type. check_sms_via_adb returns bool, never the SMS text. The verifier needs the
   raw multi-line SMS body to validate the 4 required lines (locations+coords / visiting order /
   per-segment distances / total distance), but it never has access to it.
2. content="" matches everything. Inside check_sms_via_adb, content="" becomes [""], and
   all("" in body ...) is always True. So any sent SMS to that number makes it return True.

Result:

• Agent sent an SMS → returns True → not True is False → falls through to
  True.split("\n") → AttributeError: 'bool' object has no attribute 'split' → task = 0.0.
• Agent did not send an SMS → returns False → early return 0.0, "SMS not found".

Either way the task scores 0.0, and the content-validation logic (L74+) is unreachable.

Minimal reproduction

1. Run any of the four tasks (e.g. PlanTaxiRouteSmsTask) with --enable_mcp.
2. Have the agent send a correctly-formatted SMS to the recipient (e.g. 13900139000).
3. Observe the eval log:

task_score: 0.0, reason: eval error: AttributeError: 'bool' object has no attribute 'split'

(Observed with GUI-Owl-1.5-32B on PlanTaxiRouteSmsTask.)

Suggested fix

Fetch the actual SMS body instead of a boolean.

Note: the existing get_sms_list_via_adb (system.py:186) is not a drop-in here — despite its
-> list[dict] annotation it actually returns list[str] (raw rows, result.output.split("\n"))
and queries content://sms/inbox, whereas these are sent messages (content://sms/sent).
The correct primitive is the sent-query + extract_sms_body logic already used inside
check_sms_via_adb.

Add a small helper (reusing the already-validated sent query + extract_sms_body, system.py:106):

def get_sent_sms_bodies_via_adb(controller: AndroidController, phone_number: str) -> list[str]:
    """Return the bodies (str) of SENT SMS to phone_number. Empty list if none."""
    query_cmd = f"adb -s {controller.device} shell content query --uri content://sms/sent"
    result = execute_adb(query_cmd, output=False, root_required=True)
    if not result.success or not result.output:
        return []
    bodies = []
    for line in result.output.strip().split("\nRow"):
        if not line.strip():
            continue
        if f"address={phone_number}" in line or phone_number in line:
            body = extract_sms_body(line)
            if body:
                bodies.append(body)
    return bodies

Then change the first lines of each verifier (taxi example, replacing L65-69):

        # Check 1: Verify SMS was sent
        bodies = get_sent_sms_bodies_via_adb(controller, self.RECIPIENT_PHONE)
        if not bodies:
            return 0.0, f"SMS not found sent to {self.RECIPIENT_PHONE}"
        sms_content = bodies[-1]            # a real str body
        lines = [ln.strip() for ln in sms_content.split("\n") if ln.strip()]
        # the existing per-line content checks (L74+) are now reachable

Applying the same change to all four files makes these tasks scorable again.