Problem
When working through code, debugging one's own project, learning, or practicing problems, it helps to get a concise approach or hint grounded in what is on screen and in the user's own codebase.
Proposed solution
An advanced mode that captures a user-selected screen region (ScreenCaptureKit), reads it with OCR or a multimodal model, sends the problem or code to the agent, and streams back a concise approach or hint. Provide an adjustable hint level from a Socratic nudge to a full worked approach. Add practice-set integration that tracks progress and explains thoroughly after an attempt. Add codebase awareness for real work. Intended for the user's own development, debugging, and self-directed practice, not for completing an evaluated coding assessment while concealing assistance.
Privacy considerations
Screen reading is highly sensitive; it can capture whatever is on screen. Restrict capture to a user-selected region, make active capture obvious, require explicit opt-in, and send only the captured region to the agent. Combining this with Invisible Mode is not a way to defeat oversight in evaluated settings.
Problem
When working through code, debugging one's own project, learning, or practicing problems, it helps to get a concise approach or hint grounded in what is on screen and in the user's own codebase.
Proposed solution
An advanced mode that captures a user-selected screen region (ScreenCaptureKit), reads it with OCR or a multimodal model, sends the problem or code to the agent, and streams back a concise approach or hint. Provide an adjustable hint level from a Socratic nudge to a full worked approach. Add practice-set integration that tracks progress and explains thoroughly after an attempt. Add codebase awareness for real work. Intended for the user's own development, debugging, and self-directed practice, not for completing an evaluated coding assessment while concealing assistance.
Privacy considerations
Screen reading is highly sensitive; it can capture whatever is on screen. Restrict capture to a user-selected region, make active capture obvious, require explicit opt-in, and send only the captured region to the agent. Combining this with Invisible Mode is not a way to defeat oversight in evaluated settings.