Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 27 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Gemini Android Computer Use Demo

This repository contains a reference implementation for controlling an Android emulator using the **Gemini 3.5 Flash** Computer Use API (`mobile` environment) via the Google GenAI SDK.
This repository contains a reference implementation for controlling an Android emulator using the **Gemini 3.6 Flash** Computer Use API (`mobile` environment) via the Google GenAI SDK.

## Overview

The agent operates in a continuous loop:
1. It captures a screenshot of the virtual device using ADB.
2. It sends the screenshot along with the user's task to Gemini 3.5 Flash.
3. The model returns structured tool commands (such as `click`, `type`, `long_press`, `drag_and_drop`, `press_key`, `go_back`, gwait`, `list_apps`, `open_app`, `take_screenshot`).
2. It sends the screenshot along with the user's task to Gemini 3.6 Flash.
3. The model returns structured tool commands (such as `click`, `type`, `long_press`, `drag_and_drop`, `press_key`, `go_back`, `wait`, `list_apps`, `open_app`, `take_screenshot`).
4. The client executing script maps the normalized coordinates (0-999) to the actual physical resolution of the screen and executes the action via ADB.
5. The loop repeats until the task is complete.

Expand Down Expand Up @@ -58,12 +58,34 @@ Then run the agent:
python agent.py "Find the latest blog post from philipp schmid and summarize it."
```

Alternatively, you can run the agent with a custom task:
### Command-Line Options

`agent.py` supports several CLI arguments:

| Option | Short | Default | Description |
| --- | --- | --- | --- |
| `--model` | `-m` | `gemini-3.6-flash` | Gemini model ID to use |
| `--thinking-level` | `-t` | `medium` | Thinking level configuration (`minimal`, `low`, `medium`, `high`) |
| `task` | N/A | *(default prompt)* | Task description for the agent |

Examples:

```bash
python agent.py "Open Settings and enable dark mode"
# Specify a different model ID
python agent.py --model gemini-3.5-flash-lite "Open Settings and enable dark mode"

# Adjust thinking level
python agent.py --thinking-level high "Open Clock and set an alarm for 7 AM"
```

### Supported Models

The following Gemini models are supported for computer use:

* `gemini-3.6-flash` (default)
* `gemini-3.5-flash-lite`
* `gemini-3.5-flash`

## Licensing & Disclaimer

Copyright 2026 Google LLC
Expand Down
64 changes: 54 additions & 10 deletions agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

import argparse
import base64
import json
import os
Expand Down Expand Up @@ -176,6 +177,7 @@ def screenshot(self) -> bytes:
return result.stdout



SYSTEM_PROMPT = """You are operating an Android phone.
* Use the provided tools to complete the task.
* Scroll down to inspect the full screen before assuming an element is missing.
Expand All @@ -185,12 +187,20 @@ def screenshot(self) -> bytes:
"""


def run_agent(task: str, device_id: str = None, max_turns: int = 100):
def run_agent(
task: str,
device_id: str = None,
model: str = "gemini-3.6-flash",
thinking_level: str = "medium",
max_turns: int = 100,
):
start_emulator()
client = genai.Client()
bridge = ADBBridge(device_id)

print(f"\nTask: {task}")
print(f"Model: {model}")
print(f"Thinking Level: {thinking_level}")
print("-" * 40)

screenshot_bytes = bridge.screenshot()
Expand All @@ -202,21 +212,22 @@ def run_agent(task: str, device_id: str = None, max_turns: int = 100):
"mime_type": "image/png",
},
]

previous_interaction_id = None
turn = 0

while turn < max_turns:
turn += 1

interaction = client.interactions.create(
model="gemini-3.5-flash",
model=model,
system_instruction=SYSTEM_PROMPT,
input=user_input,
tools=[{"type": "computer_use", "environment": "mobile"}],
generation_config={"thinking_config": {"thinking_level": thinking_level}},
previous_interaction_id=previous_interaction_id,
)

has_function_calls = any(
step.type == "function_call"
for step in interaction.steps
Expand Down Expand Up @@ -281,7 +292,40 @@ def run_agent(task: str, device_id: str = None, max_turns: int = 100):


if __name__ == "__main__":
task_desc = "Find the latest blog post from philipp schmid and summarize it."
if len(sys.argv) > 1:
task_desc = " ".join(sys.argv[1:])
run_agent(task_desc)
parser = argparse.ArgumentParser(description="Run Android Computer Use AI Agent")
parser.add_argument(
"--model",
"-m",
default="gemini-3.6-flash",
help="Gemini model ID to use (default: gemini-3.6-flash)",
)
parser.add_argument(
"--thinking-level",
"-t",
default="medium",
help="Thinking level for generation config (default: medium)",
)
parser.add_argument(
"--device-id",
"-d",
default=None,
help="ADB device ID (optional)",
)
parser.add_argument(
"task",
nargs="*",
help="Task description for the agent",
)

args = parser.parse_args()
task_desc = (
" ".join(args.task)
if args.task
else "Find the latest blog post from philipp schmid and summarize it."
)
run_agent(
task=task_desc,
device_id=args.device_id,
model=args.model,
thinking_level=args.thinking_level,
)