[FR] Sending Events to Assistants

[AdamSobieski]

Introduction

What if software applications (e.g., in Web browsers) could send events or notifications to interoperating AI assistants?

In these regards, perhaps a new prompt role could be of use. Adding to those existing prompt roles of system, user, assistant, and function, one could call this new prompt role either: event, user-event, application, or note.

Software events or notifications could be serialized into JSON while potentially including multimodal content, e.g., images.

const json_selection_changed_event = AI_event_from_UI_event(e);

const response = await session.prompt([
  // ...
  { role: 'event', content: json_selection_changed_event },
  { role: 'user', content: 'Tell me about the selected product.' }
]);

Use Cases

Selection-focused Questions and Dialogue

Text Selections

End-users could select text and refer to these selections conversationally.

Image Selections

End-users could select point coordinates, create bounding rectangles, draw other selection shapes, or select defined regions from image maps atop images and refer to these selections conversationally.

Item Selections

End-users could select items (e.g., products from a displayed set) and refer to these selections conversationally, e.g.: "this item", "that item", or "the selected item".

[anssiko]

Do you think polyfilling text and image selections with the Selection API and Canvas API slicing respectively on top of the current Prompt API would be a possibility? That would help identify what gaps there might be.

[AdamSobieski]

Thank you. I can take a closer look at these topics to brainstorm, sketch solutions, and identify gaps. Also, recently, there are a growing number of AI assistants (computer-using agents) which can operate software applications, e.g., UFO². As I understand it, some use screenshots to "see" software applications, including users' selections in them.

[AdamSobieski]

With respect to providing AI assistants with access to focus- and selection-related data and events, the Selection API would be a part of solutions.

For a concrete example of item selection, let us consider an HTML custom element providing a carousel of items, e.g., products, for end-users to browse and to make selections from. What, to AI assistants, are the items selected from the carousel components? How would a description of these items and their corresponding data get to the AI assistant, into its context window, and in what form? Would product-related information come from the page or application or, instead, would the AI assistant retrieve these data from other data sources?

Adding some complexity, Web developers might want to be able to provide multiple sophisticated UI components on a page, each such component having a capability for end-users to make selections.

Towards a solution, instead of modeling a Web browser as being encapsulated by a single agent, what if a Web browser corresponded with, mapped with, a hierarchical multi-agent system – a tree of agents? There could, then, be a dynamic set of agents per page or application.

The example carousel component could, then, be modeled as an agent, being capable of describing itself, its tools, its resources, its prompts, and so forth. AI assistants would be able to discover and interact with the carousel component agent.

As considered, the Web browser's main agent would be responsible for dynamically listing and brokering access to its available subagents for interoperating AI assistants.

Brainstorming, inspired by MCP and A2A, perhaps a new API could involve a CustomAgentRegistry interface available from the global window object, resembling:

let agent = window.agents.defineAgent(element, name, title, description);

agent.defineTool(callback, name, title, description, inputSchema, outputSchema, annotations);
agent.undefineTool(name);

agent.defineResource(callbackOrBlobOrString, uri, name, title, description, mimeType, size);
agent.undefineResource(name);

agent.definePrompt(content, name, title, description, arguments);
agent.undefinePrompt(name);

let childAgent = agent.defineAgent(childElement, name, title, description);
agent.undefineAgent(childAgent);

window.agents.undefineAgent(agent);

let agent = window.agents.getAgentByElement(document.getElementById('carousel'));

A remaining detail would be how events should be subscribed to, raised by, sent to, received by, handled by, and unsubscribed from by agents, e.g., between a Web browser's main agent's subagent for a carousel component and an interoperating AI assistant. Any thoughts on these – presently abstract – ideas?