-
Notifications
You must be signed in to change notification settings - Fork 55
Description
In order to fulfill the goal "Provide web developers a uniform JavaScript API for accessing browser-provided language models", there web developers should be able not only to access the real language models, but also to test their code in a predictable way. Currently, testing interactions with the Prompt API requires making real calls to a language model. This presents several challenges for robust and reliable testing:
- Non-Deterministic Behavior: For robust testing, the perfectly predictable responses are required. Some features (like
downloadprogress
) is not feasible to test reliably without emulation. - Cost: Real API calls cost money, making extensive testing expensive, especially for iterative development and CI/CD pipelines.
- Rate Limits: API providers often impose rate limits, which can slow down or halt test execution. This makes it hard to run a large test suite quickly.
This feature request proposes adding a mechanism to mock the language model and its operations. This would allow developers to write tests that don't rely on external services, leading to faster, cheaper, more reliable, and more isolated tests.
This can be achieved by specifying simulation via WebDriver BiDi protocol extension module, analogous of Web Bluetooth automated testing.