Skip to content

feat: add VITE_DISABLE_ONBOARDING to suppress the onboarding wizard#1928

Open
hieptl wants to merge 6 commits into
mainfrom
hieptl/oss-5736
Open

feat: add VITE_DISABLE_ONBOARDING to suppress the onboarding wizard#1928
hieptl wants to merge 6 commits into
mainfrom
hieptl/oss-5736

Conversation

@hieptl

@hieptl hieptl commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

HUMAN:

End-to-end evidence (beyond unit tests):

  1. Runtime injection smoke (production build + real static server). After npm run build:
  • node scripts/static-server.mjs --port 3001 --dir build --disable-onboardingcurl -s localhost:3001/ | grep __AGENT_CANVAS_DISABLE_ONBOARDING__ matches window.__AGENT_CANVAS_DISABLE_ONBOARDING__=true, and the startup log prints First-run onboarding: disabled.
  • AGENT_CANVAS_DISABLE_ONBOARDING=true node scripts/static-server.mjs --port 3001 --dir build → same injection + log (this is the path the SaaS Helm chart's env: passthrough uses).
  • Without either → no injection, page serves normally.
  1. Live browser differential (mock-API build, npm run build:mock, served by the real static server on two ports = two fresh origins/localStorage):
  • No flag, fresh profile → first-run onboarding wizard appears ("Add a backend"; with a healthy backend seeded via --session-api-key, the wizard opens on "Choose your agent" with OpenHands pre-selected).
  • --disable-onboarding, fresh profile with a healthy backend → lands directly on the "Let's Start Building!" home page with the chat composer; no wizard. In-page check confirmed window.__AGENT_CANVAS_DISABLE_ONBOARDING__ === true and localStorage["openhands-onboarded"] === null (suppression comes from the flag, not a fake completion marker).
  1. Suites: npm run lint clean; npm test → 3,959 passed / 512 files (includes 10 new tests below); npm run build and npm run build:lib pass.
  • A human has tested these changes.

AGENT:


Why

New SaaS users on app.all-hands.dev/canvas are forced through the onboarding wizard even though OpenHands Cloud already pre-provisions everything it collects (agent = OpenHands, free Cloud LLM). They should drop straight into the "Let's Start Building!" new-chat page. The wizard's two gates (root.tsx first-run screen; home-page OnboardingHost modal — the one SaaS users see) key solely off the per-browser openhands-onboarded localStorage flag, with no deployment-level override. A purely build-time flag cannot reach SaaS because it consumes the shared prebuilt image, so the flag follows the same dual-half convention as VITE_LOCK_TO_CLOUD / VITE_AUTH_REQUIRED.

Summary

  • Add isOnboardingDisabled() (src/api/agent-server-config.ts): true when build-time VITE_DISABLE_ONBOARDING === "true" or the runtime-injected window.__AGENT_CANVAS_DISABLE_ONBOARDING__ === true; wire it into useOnboardingCompletion() so both wizard gates close while locked-mode Cloud login and the ?previewOnboardingStep harness are unaffected.
  • Add --disable-onboarding to scripts/static-server.mjs (injects the window global into index.html, mirrors --lock-to-cloud), with an AGENT_CANVAS_DISABLE_ONBOARDING=true env fallback read at process entry so the SaaS Helm chart can enable it via its generic env: passthrough without a chart release. Document the flag in .env.sample, AGENTS.md, and docs/DEVELOPMENT.md.
  • Tests: 10 new cases extending existing files — getter contract (agent-server-config.test.ts), gate suppression for a fresh Cloud user (onboarding-host.test.tsx) and a fresh install (root.test.tsx), and static-server flag parsing + injection (static-server.test.ts).

Issue Number

N/A.

How to Test

  1. npm install, then VITE_DISABLE_ONBOARDING= npm test (see Notes on the .env caveat) and npm run lint.
  2. Default behavior unchanged: clear localStorage, npm run dev → the onboarding wizard appears.
  3. Build-time half: add VITE_DISABLE_ONBOARDING="true" to .env, restart npm run dev, clear localStorage → you land on the home screen, no wizard; ?previewOnboardingStep=0 still opens the preview. Remove the line afterwards (leaving it set makes wizard-visibility unit tests fail locally — same known hazard as VITE_LOCK_TO_CLOUD).
  4. Runtime half: npm run build, then node scripts/static-server.mjs --port 3001 --dir build --disable-onboarding (or AGENT_CANVAS_DISABLE_ONBOARDING=true node scripts/static-server.mjs --port 3001 --dir build) → curl -s localhost:3001/ contains window.__AGENT_CANVAS_DISABLE_ONBOARDING__=true; a fresh browser profile gets no wizard. Without the flag, the same build shows the wizard.

Video/Screenshots

oss-5736.mov

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

🐳 Docker images for this PR

GHCR package: https://github.com/OpenHands/agent-canvas/pkgs/container/agent-canvas

Component Value
Image ghcr.io/openhands/agent-canvas
Architectures amd64, arm64
Agent Server ghcr.io/openhands/agent-server:1.37.0-python
Automation openhands-automation==1.3.1
Commit ff8443d3e5d76439e23478c7a7d904a5539c3f0f

Pull (multi-arch manifest)

# Multi-arch manifest — Docker automatically pulls the correct architecture
docker pull ghcr.io/openhands/agent-canvas:sha-ff8443d

Run

docker run -it --rm \
  -p 8000:8000 \
  ghcr.io/openhands/agent-canvas:sha-ff8443d

All tags pushed for this build

ghcr.io/openhands/agent-canvas:sha-ff8443d-amd64
ghcr.io/openhands/agent-canvas:hieptl-oss-5736-amd64
ghcr.io/openhands/agent-canvas:pr-1928-amd64
ghcr.io/openhands/agent-canvas:sha-ff8443d-arm64
ghcr.io/openhands/agent-canvas:hieptl-oss-5736-arm64
ghcr.io/openhands/agent-canvas:pr-1928-arm64
ghcr.io/openhands/agent-canvas:sha-ff8443d
ghcr.io/openhands/agent-canvas:hieptl-oss-5736
ghcr.io/openhands/agent-canvas:pr-1928

About Multi-Architecture Support

  • Each tag (e.g., sha-ff8443d) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., sha-ff8443d-amd64) are also available if needed

@hieptl hieptl self-assigned this Jul 24, 2026
@vercel

vercel Bot commented Jul 24, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
agent-canvas Ready Ready Preview, Comment Jul 25, 2026 6:42am

Request Review

@github-actions github-actions Bot added the type: feat A new feature label Jul 24, 2026
Comment thread src/components/features/onboarding/use-onboarding-completion.ts Outdated
Comment thread scripts/static-server.mjs Outdated
Comment thread scripts/static-server.mjs
Comment thread __tests__/components/onboarding/onboarding-host.test.tsx Outdated
Comment thread AGENTS.md
Comment thread src/api/agent-server-config.ts Outdated
Comment thread __tests__/api/agent-server-config.test.ts Outdated

@VascoSch92 VascoSch92 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Two other things that are not blocking that you can address if you have the time:

  1. onboarding-host.tsx:10-23 — apply the docstring patch I gave above. Two lines: name both gates, and scope the "fresh browser sees onboarding once" invariant with "absent deployment-level suppression."
  2. static-server.mjs:572 — either accept 1/True/yes or console.warn when AGENT_CANVAS_DISABLE_ONBOARDING is set to something other than "true". A Helm typo currently no-ops silently.

@neubig neubig left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we consider #1942 as an alternative before merging this?

#1928 solves the SaaS problem with a new build-time flag, runtime environment variable, CLI option, and injected window global. #1942 instead uses the runtime signals Canvas already has: it suppresses the post-auth modal only when the active backend matches the deployment's locked Cloud host and settings report a usable LLM (a non-empty model plus API-key or subscription authentication).

That keeps the fresh-browser behavior from #1528 for Local and arbitrary Cloud backends, preserves Cloud login/recovery and preview mode, and avoids adding another deployment configuration surface. It also does not write a fake onboarding-completion marker.

Requesting changes so we can compare the scoped capability-based approach before committing to the new flag/API surface.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Mock-LLM E2E Tests

60/60 passed

Commit: ff8443d3 · Workflow run · Test artifacts

Details
Status Test Duration
automations/mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 1: setup LLM profile and register automation trajectory 6.4s
automations/mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 2: create automation and dispatch run via the UI 28.4s
automations/mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 3: verify automation and run on the automations page 6.2s
automations/mock-llm-preset-automation.spec.ts › preset automation → slash command conversation › automation card sends the correct slash command to a conversation 15.3s
automations/mock-llm-preset-automation.spec.ts › preset automation → slash command conversation › direct slash command from home page triggers skill activation 12.7s
backends/mock-llm-auth-modes.spec.ts › auth mode: fresh install with runtime-injected key › reaches the onboarding modal without pre-seeded localStorage 1.4s
backends/mock-llm-auth-modes.spec.ts › auth mode: non-public key rotation › recovers when localStorage has a stale session API key 5.3s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › shows first-run onboarding before the auth screen when no key is configured 1.4s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › rejects an incorrect key with an inline error 1.6s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › allows access after pasting the correct key 1.8s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › skips auth screen for returning user with valid stored key 793ms
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › re-prompts when the server rotates its key (stale localStorage) 1.5s
backends/mock-llm-cross-connect.spec.ts › cross-connect: frontend-only → backend-only › frontend-only connects to a separate backend-only instance 15.8s
backends/mock-llm-cross-connect.spec.ts › cross-connect: frontend-only → multiple backends › connects to two separate backends and switches between them 20.3s
backends/mock-llm-partial-stack.spec.ts › partial stack: --frontend-only › serves the frontend but returns 503 for backend routes 7.3s
backends/mock-llm-partial-stack.spec.ts › partial stack: --backend-only › serves backend APIs but returns 503 for the frontend root 13.3s
backends/mock-llm-partial-stack.spec.ts › partial stack: port conflict › fails with a clear error when the ingress port is occupied 86ms
backends/mock-llm-partial-stack.spec.ts › partial stack: port conflict › starts successfully on a free port after a conflict 6.0s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 1: create an LLM profile pointing at the mock LLM server 6.3s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 2: activate the mock-llm profile and verify settings API 10.8s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 3: run a conversation with the mock LLM 6.5s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 4: resume conversation from sidebar after navigating away 5.7s
conversations/mock-llm-image-upload.spec.ts › mock-LLM image upload › attaching an image embeds it as base64 in the LLM completion call 12.8s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 1: ensure mock LLM profile is configured 6.5s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 2: start conversation and attach workspace metadata 11.6s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 3: git control bar shows workspace pill and git actions 25.3s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 4: files tab can enable diff view for attached workspace 5.9s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 5: browser tab shows empty state 6.3s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 6: files tab defaults to file-tree view without attached workspace 7.4s
home/mock-llm-folder-workspace.spec.ts › mock-LLM folder browser → workspace → conversation › step 1: browse to a folder, add it as a workspace, and launch a conversation with the correct working_dir 7.7s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 1: GitHub card is visible on the MCP marketplace page 5.5s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 2: clicking GitHub add control opens the install modal with correct fields 5.6s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 3: full install flow — fill PAT, submit, verify installed 12.1s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 4: installed GitHub server can be deleted 5.8s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › install: invalid Slack credentials are blocked with a credential-check error 5.7s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › install: a valid token missing only a scope still installs (missing_scope is not a credential failure) 5.7s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › install: an older agent server that omits tool_result still installs (compat) 5.8s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › edit: Test Connection verifies the stored credentials and surfaces a credential failure 5.7s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › edit: Test Connection reports success for valid stored credentials 5.6s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › custom (non-catalog) server: Test Connection attaches no verification probe 5.7s
onboarding/mock-llm-onboarding-happy-path.spec.ts › onboarding happy path › completes the full onboarding flow and launches a conversation 4.4s
onboarding/mock-llm-onboarding-regressions.spec.ts › onboarding recent regressions › keeps the modal open on backdrop click and Escape 1.4s
onboarding/mock-llm-onboarding-regressions.spec.ts › onboarding recent regressions › defaults the LLM setup step to OpenAI GPT-5.5 1.5s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › scopes standalone styles to the agent-server-ui shell 1.3s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › renders critic results on agent messages and finish actions 1.4s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › loads older events when scrolling up 1.7s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › selected workspace persists after navigating away and returning 2.9s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › cleared sessionStorage yields empty workspace selection 1.3s
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 1: configure ACP agent via Settings → Agent UI 13.1s
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 2: reload and verify ACP settings are persisted in UI 5.6s
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 3: start ACP conversation and verify agent reply 6.2s
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 4: resume ACP conversation from sidebar after navigating away 5.7s
settings/mock-llm-model-switch.spec.ts › mock-LLM /model slash command › step 1: configure LLM, create switch-target profile, register trajectory 12.4s
settings/mock-llm-model-switch.spec.ts › mock-LLM /model slash command › step 2: start conversation, switch profile via /model, verify switch 6.8s
settings/mock-llm-profile-management.spec.ts › active profile deletion + reconciliation › active profile is deletable and reconciliation activates another profile 8.0s
settings/mock-llm-profile-management.spec.ts › same-model profile identity › chat header shows the correct profile when two profiles share the same model 14.4s
settings/mock-llm-profile-management.spec.ts › OpenHands provider hidden base_url preservation › re-saving an OpenHands profile from Basic view preserves hidden base_url 7.2s
skills/mock-llm-skills.spec.ts › skill loading: project, user, and deletion › project skill in workspace/.agents/skills/ triggers on matching keyword 12.8s
skills/mock-llm-skills.spec.ts › skill loading: project, user, and deletion › user skill in ~/.openhands/skills/ triggers on matching keyword 12.6s
skills/mock-llm-skills.spec.ts › skill loading: project, user, and deletion › deleting a user skill removes it from subsequent conversations 12.7s

Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses)

@github-actions

Copy link
Copy Markdown
Contributor

✅ Mock-LLM Docker E2E Test Results

60/60 passed

Commit: ff8443d3 · Workflow run · Test artifacts

Details
Status Test Duration
automations/mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 1: setup LLM profile and register automation trajectory 6.5s
automations/mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 2: create automation and dispatch run via the UI 32.5s
automations/mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 3: verify automation and run on the automations page 6.1s
automations/mock-llm-preset-automation.spec.ts › preset automation → slash command conversation › automation card sends the correct slash command to a conversation 15.4s
automations/mock-llm-preset-automation.spec.ts › preset automation → slash command conversation › direct slash command from home page triggers skill activation 12.6s
backends/mock-llm-auth-modes.spec.ts › auth mode: fresh install with runtime-injected key › reaches the onboarding modal without pre-seeded localStorage 1.4s
backends/mock-llm-auth-modes.spec.ts › auth mode: non-public key rotation › recovers when localStorage has a stale session API key 5.3s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › shows first-run onboarding before the auth screen when no key is configured 1.4s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › rejects an incorrect key with an inline error 1.5s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › allows access after pasting the correct key 1.9s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › skips auth screen for returning user with valid stored key 1.2s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › re-prompts when the server rotates its key (stale localStorage) 1.5s
backends/mock-llm-cross-connect.spec.ts › cross-connect: frontend-only → backend-only › frontend-only connects to a separate backend-only instance 22.8s
backends/mock-llm-cross-connect.spec.ts › cross-connect: frontend-only → multiple backends › connects to two separate backends and switches between them 20.6s
backends/mock-llm-partial-stack.spec.ts › partial stack: --frontend-only › serves the frontend but returns 503 for backend routes 7.4s
backends/mock-llm-partial-stack.spec.ts › partial stack: --backend-only › serves backend APIs but returns 503 for the frontend root 14.3s
backends/mock-llm-partial-stack.spec.ts › partial stack: port conflict › fails with a clear error when the ingress port is occupied 104ms
backends/mock-llm-partial-stack.spec.ts › partial stack: port conflict › starts successfully on a free port after a conflict 6.0s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 1: create an LLM profile pointing at the mock LLM server 6.3s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 2: activate the mock-llm profile and verify settings API 6.2s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 3: run a conversation with the mock LLM 6.3s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 4: resume conversation from sidebar after navigating away 5.7s
conversations/mock-llm-image-upload.spec.ts › mock-LLM image upload › attaching an image embeds it as base64 in the LLM completion call 12.6s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 1: ensure mock LLM profile is configured 6.6s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 2: start conversation and attach workspace metadata 11.3s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 3: git control bar shows workspace pill and git actions 25.3s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 4: files tab can enable diff view for attached workspace 5.9s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 5: browser tab shows empty state 6.3s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 6: files tab defaults to file-tree view without attached workspace 7.2s
home/mock-llm-folder-workspace.spec.ts › mock-LLM folder browser → workspace → conversation › step 1: browse to a folder, add it as a workspace, and launch a conversation with the correct working_dir 7.2s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 1: GitHub card is visible on the MCP marketplace page 5.5s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 2: clicking GitHub add control opens the install modal with correct fields 5.6s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 3: full install flow — fill PAT, submit, verify installed 12.2s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 4: installed GitHub server can be deleted 5.9s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › install: invalid Slack credentials are blocked with a credential-check error 5.7s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › install: a valid token missing only a scope still installs (missing_scope is not a credential failure) 5.9s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › install: an older agent server that omits tool_result still installs (compat) 5.9s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › edit: Test Connection verifies the stored credentials and surfaces a credential failure 5.8s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › edit: Test Connection reports success for valid stored credentials 5.8s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › custom (non-catalog) server: Test Connection attaches no verification probe 5.8s
onboarding/mock-llm-onboarding-happy-path.spec.ts › onboarding happy path › completes the full onboarding flow and launches a conversation 4.4s
onboarding/mock-llm-onboarding-regressions.spec.ts › onboarding recent regressions › keeps the modal open on backdrop click and Escape 1.6s
onboarding/mock-llm-onboarding-regressions.spec.ts › onboarding recent regressions › defaults the LLM setup step to OpenAI GPT-5.5 1.7s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › scopes standalone styles to the agent-server-ui shell 1.4s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › renders critic results on agent messages and finish actions 1.6s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › loads older events when scrolling up 1.8s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › selected workspace persists after navigating away and returning 2.0s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › cleared sessionStorage yields empty workspace selection 982ms
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 1: configure ACP agent via Settings → Agent UI 13.3s
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 2: reload and verify ACP settings are persisted in UI 5.7s
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 3: start ACP conversation and verify agent reply 6.7s
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 4: resume ACP conversation from sidebar after navigating away 5.7s
settings/mock-llm-model-switch.spec.ts › mock-LLM /model slash command › step 1: configure LLM, create switch-target profile, register trajectory 12.6s
settings/mock-llm-model-switch.spec.ts › mock-LLM /model slash command › step 2: start conversation, switch profile via /model, verify switch 6.7s
settings/mock-llm-profile-management.spec.ts › active profile deletion + reconciliation › active profile is deletable and reconciliation activates another profile 8.2s
settings/mock-llm-profile-management.spec.ts › same-model profile identity › chat header shows the correct profile when two profiles share the same model 14.4s
settings/mock-llm-profile-management.spec.ts › OpenHands provider hidden base_url preservation › re-saving an OpenHands profile from Basic view preserves hidden base_url 7.8s
skills/mock-llm-skills.spec.ts › skill loading: project, user, and deletion › project skill in workspace/.agents/skills/ triggers on matching keyword 12.7s
skills/mock-llm-skills.spec.ts › skill loading: project, user, and deletion › user skill in ~/.openhands/skills/ triggers on matching keyword 12.6s
skills/mock-llm-skills.spec.ts › skill loading: project, user, and deletion › deleting a user skill removes it from subsequent conversations 12.6s

Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses)

@hieptl

hieptl commented Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

Hello @neubig,

I took a close look at #1942, and I think your concern about avoiding another deployment-specific configuration option is a reasonable one.

One correction, though—and this is on me. #1928 has never written to localStorage, and since c1188db8 (pushed approximately eight hours before your review), it also no longer synthesizes completion. useOnboardingCompletion() now preserves its original, localStorage-only contract, and each onboarding gate independently ORs isOnboardingDisabled(), following the concern Vasco raised. The PR description still reflects the earlier hook-level implementation, so I'll update it to match the current behavior.

The main question that determines the right approach for me is whether #1942's assumption holds in practice. It only takes effect when the Cloud settings report llm_api_key_set: true (or subscription authentication). If newly created Cloud accounts already report that on the initial Canvas render, then I agree #1942 is the cleaner solution, and I'd be perfectly happy to close #1928 in its favor. If that provisioning is still pending under OSS-5729, however, then #1942 would effectively be a no-op for the very users this issue is intended to help.

There are also two behavioral implications that may be worth calling out explicitly:

  1. Because the completion marker is intentionally never written, the onboarding gate is no longer monotonic. If the underlying signal later becomes false—for example, after an API key rotation, a subscription lapse, or a settings 404 (which useSettings maps to DEFAULT_SETTINGS with llm_api_key_set: false)—an existing user would see the first-run onboarding flow again. A persisted completion flag would not exhibit that behavior.
  2. The change affects every --lock-to-cloud deployment, including self-hosted deployments that use Cloud-managed API keys. It also reverses the behavior introduced intentionally in [codex] Always show first-run onboarding #1528, where onboarding stopped auto-completing based on backend LLM readiness. If that's the intended direction, it would be helpful to mention it explicitly in the PR description.

If #1942 moves forward, I have one small suggestion: consider reusing useLlmConfigured() instead of introducing hasUsableCloudLlm. It derives the same Cloud readiness signal while already handling ACP agents, the hidden-settings feature flag, and loading/indeterminate states.

There is also a possible middle ground if the goal is to eliminate the configuration surface without relying on mutable backend state. We could instead gate on isActiveLockedCloudBackend && getLockedCloudAuthMode() === "cookie". That condition corresponds precisely to "Canvas is being served by the Cloud deployment it is locked to" (currently only the hosted SaaS deployment), where the onboarding flow is inherently redundant, while leaving self-hosted locked deployments unchanged. It's a small change—roughly six lines—and I'd be happy to implement it here if you think that's the better direction.

In any case, I'm happy with whichever of these three approaches you prefer. If you'd rather proceed with #1942 instead, I'm completely comfortable closing this PR.

Thank you very much! 🙏

@hieptl
hieptl requested a review from neubig July 25, 2026 07:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: feat A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants