Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
5227daf
feat: add Chat SDK as submodule
fg-nava Aug 27, 2025
37f0a21
feat: decision log guidelines and start
KayTV Aug 27, 2025
68d2db9
template
KayTV Aug 27, 2025
b3529b5
edits
KayTV Aug 27, 2025
58474fe
feat: integrate Mastra web automation agent in chat client
fg-nava Aug 29, 2025
669f39d
feat: add browser WebSocket streaming service with CDP integration
fg-nava Aug 30, 2025
9a4a106
fix: update lockfile
fg-nava Aug 30, 2025
874e36a
feat: update client submodule with browser display panel
fg-nava Aug 30, 2025
830708e
docs: add horizontal sequence diagram and docs on browser streaming arch
fg-nava Aug 30, 2025
353566e
feat: add CDP control for user override
fg-nava Sep 2, 2025
040e549
feat: add playwright browser installation to build script
fg-nava Sep 2, 2025
6fd81c0
feat: add playwright setup script for browser installation
fg-nava Sep 2, 2025
455181e
fix: mcp isolated browser issue
fg-nava Sep 2, 2025
a908ee2
nit: switch to info from debug
fg-nava Sep 2, 2025
13103ca
feat: update client submodule to latest commit
fg-nava Sep 3, 2025
fca29ba
docs: add git submodule guide to README.md
fg-nava Sep 3, 2025
6908c5b
Merge pull request #18 from navapbc/feat/add-browser-websocket
fg-nava Sep 3, 2025
f0fb0b4
fix: add debugging to mastra client
fg-nava Sep 3, 2025
7aa742f
fix: make middleware more permissive in dev mode
fg-nava Sep 3, 2025
c33954c
fix: try chaning external host connection
fg-nava Sep 4, 2025
ab6a541
fix: revert arg change
fg-nava Sep 4, 2025
0156774
fix: upgrade to mastra@alpha to fix playground baseUrl bug
fg-nava Sep 4, 2025
94010a4
fix: revert middleware changes
fg-nava Sep 4, 2025
72b8619
fix: remove debug only arg in index.ts
fg-nava Sep 4, 2025
d914693
fix: better websocket event handling
fg-nava Sep 4, 2025
9c6e02f
feat: update for Mastra
KayTV Sep 11, 2025
6a8c42e
fix: resolve security vulnerabilities
fg-nava Sep 15, 2025
9e7d16d
Merge pull request #19 from navapbc/fix/debug-playwright-install
fg-nava Sep 16, 2025
014552d
fix: upgraded mastra version to resolve memory thread id issue
fg-nava Sep 16, 2025
1dce4a6
Merge pull request #21 from navapbc/fix/memory-version-mismatch
fg-nava Sep 16, 2025
b3fe9cb
feat: decision log guidelines and start
KayTV Aug 27, 2025
cc89837
template
KayTV Aug 27, 2025
9d0f5d7
edits
KayTV Aug 27, 2025
6edf620
Merge branch 'kaytv/decision-log' of https://github.com/navapbc/labs-…
KayTV Sep 16, 2025
c983801
docs
KayTV Sep 16, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[submodule "client"]
path = client
url = https://github.com/navapbc/ai-chatbot.git
branch = labs-asp
111 changes: 103 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,26 +18,36 @@ You'll need these installed on your computer:

## Step-by-Step Setup

### 1. Get the Code and Navigate to the Correct Directory
### 1. Get the Code with Submodules

This project uses Git submodules to manage the client frontend. You'll need to clone with submodules or set them up after cloning.

#### Option A: Clone with Submodules (Recommended)
```bash
# Clone the repository with submodules
git clone --recurse-submodules https://github.com/navapbc/labs-asp.git
cd labs-asp
```

#### Option B: If You Already Cloned Without Submodules
```bash
# Clone the repository
git clone https://github.com/navapbc/labs-asp-experiments.git
cd labs-asp-experiments/mastra-test-app
# If you already cloned the repo, initialize submodules
git submodule update --init --recursive
```

> **Important**: Make sure you're in the `mastra-test-app` directory for all the following commands. This is where all the app files are located.
> **Important**: This project uses the `client/` directory as a Git submodule that tracks the `labs-asp` branch of the AI chatbot repository.

### 2. Opening Terminal in Visual Studio Code

If you're using Visual Studio Code:

1. **Open the project folder**: Go to `File > Open Folder` and select the `mastra-test-app` directory
1. **Open the project folder**: Go to `File > Open Folder` and select the `labs-asp` directory
2. **Open the terminal**:
- Use the keyboard shortcut: `Ctrl+(backtick)` on Mac
- Or go to `Terminal > New Terminal` in the menu
- Or use `View > Terminal`

The terminal should automatically open in the correct `mastra-test-app` directory. You can verify this by running `pwd` (on Mac) to see your current directory path.
The terminal should automatically open in the correct `labs-asp` directory. You can verify this by running `pwd` (on Mac) to see your current directory path.

### 3. Install Dependencies
```bash
Expand Down Expand Up @@ -107,7 +117,7 @@ If the app displays errors or becomes unresponsive:

2. **Start fresh**:
- Open a new terminal (see "Opening Terminal in Visual Studio Code" above)
- Make sure you're in the `mastra-test-app` directory: `cd labs-asp-experiments/mastra-test-app`
- Make sure you're in the `labs-asp` directory: `cd labs-asp`
- Restart the app: `pnpm dev`

3. **If problems persist**:
Expand Down Expand Up @@ -135,6 +145,91 @@ pnpm dev
pnpm db:studio
```

## Git Submodule Management

This project uses Git submodules to manage the client frontend. The `client/` directory is a submodule that tracks the `labs-asp` branch of the AI chatbot repository.

### Initial Setup (One-Time Configuration)

Set up Git to automatically handle submodules and create a convenient alias:

```bash
# Configure Git to automatically handle submodules in most operations
git config --global submodule.recurse true

# Create an alias for pulling with submodules
git config --global alias.spull "pull --recurse-submodules"
```

### Daily Workflow Commands

```bash
# Pull latest changes from both main repo and submodules
git spull

# Alternative: Pull with submodules (if you don't have the alias)
git pull --recurse-submodules

# Update submodule to latest commit from its remote branch
git submodule update --remote client

# Check submodule status
git submodule status

# Initialize submodules if they're missing
git submodule update --init --recursive
```

### Working with Submodules

#### After a PR is Merged
When PRs are merged into the main repository, always pull with submodules:
```bash
git spull # Pulls both main repo and submodule changes automatically
```

#### If You Need to Update the Submodule Reference
Sometimes you'll need to update the main repository to point to a newer commit in the submodule:
```bash
# Update submodule to latest remote commit
git submodule update --remote client

# Add and commit the submodule reference update
git add client
git commit -m "feat: update client submodule to latest commit"
git push
```

#### Checking Submodule Configuration
You can view the submodule configuration in `.gitmodules`:
```bash
cat .gitmodules
```

This shows how the submodule is configured to track the `labs-asp` branch.

### Troubleshooting Submodules

#### Submodule Directory is Empty
```bash
git submodule update --init --recursive
```

#### Submodule is Out of Date
```bash
git submodule update --remote client
```

#### Reset Submodule to Match Main Repository
```bash
git submodule update --recursive
```

#### View What Branch the Submodule is Tracking
```bash
git config -f .gitmodules --get submodule.client.branch
```

### Admin-Only Commands
> **Note**: These commands are for admins only and will modify shared data:
```bash
Expand Down
1 change: 1 addition & 0 deletions client
Submodule client added at 343f90
143 changes: 143 additions & 0 deletions docs/BROWSER_STREAMING_ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
# Browser Streaming Architecture

## Overview

The browser streaming system provides real-time visualization of web automation tasks by streaming browser screenshots from the Mastra agent to the chat client via WebSocket.

## Architecture Flow

```
Chat Client Agent Server Browser Automation
| | |
|-- 1. User sends message ------>| |
| (with automation request) | |
| |--- 2. Start automation --------->|
| | (Playwright MCP tools) |
| | |
|-- 3. Tool detection ---------->| |
| triggers browser panel |<-- 4. Chrome CDP start ----------|
| | (auto-detect process) |
| | |
|-- 5. GET /api/browser-stream ->| |
| ?sessionId=chat_id | |
| | |
|<- 6. WebSocket URL ------------| |
| ws://localhost:8933 | |
| | |
|-- 7. WebSocket connect ------->| |
| to streaming service | |
| | |
|-- 8. start-streaming --------->| |
| {type: "start-streaming"} |--- 9. Page.startScreencast ----->|
| | (CDP command) |
| | |
|<- 10. streaming-started -------|<-- 11. screencast ready ---------|
| | |
|<- 12. frame data --------------|<-- continuous frames ------------|
| {type: "frame", | (base64 JPEG) |
| data: "base64_jpeg"} | |
| | |
|-- 13. Canvas render ---------->| |
| (HTML5 canvas display) | |
| | |
|<- 14. frame data --------------|<-- continuous frames ------------|
| (real-time streaming) | (1 FPS) |
| | |
|-- 15. stop-streaming --------->| |
| (user closes panel) |--- 16. Page.stopScreencast ----->|
| | |
|<- 17. streaming-stopped -------|<-- 18. cleanup complete ---------|
| | |
|-- 19. WebSocket close -------->| |
```

## Components

### Server Side (Agent Repo)

**Browser Streaming Service** (`src/mastra/browser-streaming.ts`)
- WebSocket server on port 8933
- Connects to Playwright's Chrome via DevTools Protocol (CDP)
- Captures screenshots using `Page.startScreencast()`
- Streams base64 JPEG frames to connected clients
- Auto-detects Chrome process and CDP port dynamically

**MCP Integration** (`src/mastra/mcp.ts`)
- Initializes browser streaming service alongside Playwright MCP
- Handles service lifecycle and cleanup

**Web Automation Agent** (`src/mastra/agents/web-automation-agent.ts`)
- Uses Playwright tools for browser automation
- Configured with 50-step limit via `stepCountIs(50)`
- Integrates with memory and database tools

### Client Side (Client Repo)

**Browser Panel** (`client/components/browser-panel.tsx`)
- React component with HTML5 canvas for displaying browser frames
- WebSocket client that connects to streaming service
- Auto-connects when panel becomes visible
- Connection management with error handling

**API Route** (`client/app/api/browser-stream/route.ts`)
- Next.js API endpoint providing WebSocket connection info
- Returns streaming service URL and session details
- Configurable via `BROWSER_STREAMING_PORT` and `BROWSER_STREAMING_HOST`

**Chat Integration** (`client/components/chat.tsx`)
- Detects browser tool usage in message parts
- Auto-shows browser panel when Playwright tools are used
- Split-screen layout: chat on left, browser view on right

**Message Display** (`client/components/message.tsx`)
- Tool name mapping for user-friendly descriptions
- Collapsible sections showing tool inputs/outputs
- Handles both `playwright_browser` and `mcp_playwright_browser` tools

## Data Flow

1. **Tool Detection**: Chat interface monitors messages for browser tool calls
2. **Panel Activation**: Browser panel automatically appears when tools detected
3. **WebSocket Setup**: Panel fetches connection info from API route
4. **CDP Connection**: Streaming service connects to Playwright's Chrome
5. **Frame Capture**: Chrome CDP streams screenshots via `startScreencast()`
6. **WebSocket Streaming**: Frames sent as base64 JPEG to connected clients
7. **Canvas Rendering**: Browser panel displays frames on HTML5 canvas

## Configuration

### Environment Variables

**Agent Repo:**
- `BROWSER_STREAMING_PORT` - WebSocket server port (default: 8933)

**Client Repo:**
- `BROWSER_STREAMING_HOST` - Streaming service host (default: localhost)
- `BROWSER_STREAMING_PORT` - Streaming service port (default: 8933)

### Session Management

- Each chat session gets unique browser streaming session ID
- Multiple concurrent sessions supported
- Automatic cleanup on client disconnect
- Connection retry logic with error handling

## Development Setup

```bash
# Clone with submodules
git clone --recurse-submodules https://github.com/navapbc/labs-asp.git
cd labs-asp
git checkout feat/add-browser-websocket

# Agent repo - start streaming service
pnpm install
pnpm dev

# Client repo - start chat interface
cd client
pnpm install
pnpm dev
```

The browser panel will automatically appear during web automation tasks and stream live browser content in real-time.
72 changes: 72 additions & 0 deletions docs/decisions/template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# [short title of solved problem and solution]

- Status: [proposed | rejected | accepted | deprecated | … | superseded by [ADR-YYYY-MM-DD](2025-01-21-example.md)] <!-- optional -->
- Deciders: [list everyone involved in the decision] <!-- optional -->
- Date: [YYYY-MM-DD when the decision was last updated] <!-- optional -->

Technical Story: [description | ticket/issue URL] <!-- optional -->

## Context and Problem Statement

[Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in the form of a question.]

## Decision Drivers <!-- optional -->

- [driver 1, e.g., a force, facing concern, …]
- [driver 2, e.g., a force, facing concern, …]
- … <!-- numbers of drivers can vary -->

## Considered Options

- [option 1]
- [option 2]
- [option 3]
- … <!-- numbers of options can vary -->

## Decision Outcome

Chosen option: "[option 1]", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | … | comes out best (see below)].

### Positive Consequences <!-- optional -->

- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, …]
- …

### Negative Consequences <!-- optional -->

- [e.g., compromising quality attribute, follow-up decisions required, …]
- …

## Pros and Cons of the Options <!-- optional -->

### [option 1]

[example | description | pointer to more information | …] <!-- optional -->

- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- … <!-- numbers of pros and cons can vary -->

### [option 2]

[example | description | pointer to more information | …] <!-- optional -->

- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- … <!-- numbers of pros and cons can vary -->

### [option 3]

[example | description | pointer to more information | …] <!-- optional -->

- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- … <!-- numbers of pros and cons can vary -->

## Links <!-- optional -->

- [Link type] [Link to ADR] <!-- example: Refined by [ADR-2025-01-21](2025-01-21-example.md) -->
- … <!-- numbers of links can vary -->
Loading