Assistant: add setting for user-defined model listing

[sharon-wang]

Summary

• addresses Configure static list of available models (names, metadata, token limits, etc) #9971
• add positron.assistant.configuredModels setting to allow users to statically configure model settings per provider, which overrides dynamic model resolution
• update and add tests around model resolution

Approach

• each provider's resolveModels() method now calls retrieveModelsFromConfig() to check for statically defined models via positron.assistant.configuredModels or hardcoded models, then retrieveModelsFromApi() to dynamically retrieve the model listing if applicable for the provider
• move some duplicated code to shared util files modelDefinitions.ts and modelResolutionHelpers.ts
• continues some refactoring to incrementally work towards Assistant: reorg/refactor provider handling #10532

Release Notes

New Features

• Assistant: add initial support for user-defined model listing (Configure static list of available models (names, metadata, token limits, etc) #9971)

QA Notes

• Added new and updated existing Positron Assistant extension integration tests (run locally with npm run test-extension -- -l positron-assistant)
• The setting should work for all supported providers. Note that the currently selected model sometimes may not change when the setting is updated -- this is a known issue Assistant: The model provider and model selections can get into a bad state #9443.

[github-actions]

E2E Tests 🚀
This PR will run tests tagged with: @:critical

^{readme  valid tags}

[sharon-wang]

I've removed this as we no longer use the hardcoded model listing, as we've switched to using the Bedrock SDK to dynamically retrieve the models.

[melissa-barca]

Nice work! Thanks for including thorough test coverage

[melissa-barca]

Overall this is looking great. Thinking about this as a Workbench feature, it's possible an admin will misconfigure the setting and then need to play a back and forth game of telephone with the user and possibly our support/dev teams to debug the misconfiguration. We can try to make this process easier by adding in some validation that runs when the assistant extension activates. I think we have everything we need to verify the provider and surface any errors (probably as a notification and to the output pane?) The current experience isn't ideal if the configured model doesn't actually exist, but I'm not sure if there's a great way to improve this as we already log an error that includes the model we attempted to use.

[sharon-wang]

What do you think about adding a command like Check configured models, which iterates through the configured models setting and tries a connection check against each one? Note that this will require the user to already be authenticated to the providers, so I think this would work best for autoconfigured credentials like (via #10537), but individual users could log into a provider in the modal, then run this command.

I'm leaning towards a command as checking all the models on each extension activation would cause a delay, with how the code is currently structured (each provider is connected to in sequence). I can open an issue for this separately if that sounds like a decent approach to try!

positron.assistant.maxConnectionAttempts sets that we try up to 3 models by default, but could be increased if the user is having trouble with configured models, and as you mentioned we log each one that failed. This would be a hacky workaround in the meantime to check the models 😆

[melissa-barca]

I like that idea! I think it will be helpful for Admins and support if there are model issues. How do you feel about adding provider verification as part of this PR?

[sharon-wang]

I like that idea! I think it will be helpful for Admins and support if there are model issues. How do you feel about adding provider verification as part of this PR?

Added provider verification and a couple basic tests in 1dbf111!

settings.json

  "positron.assistant.configuredModels": {
    "bad provider":[]
  }

Here's what the warning looks like looks like -- runs during extension activation:

There's also a message in the output log:

[warning] Configured models contain unsupported providers: 'bad provider'. Please review your configuration for 'positron.assistant.configuredModels'

[sharon-wang]

What do you think about adding a command like Check configured models, which iterates through the configured models setting and tries a connection check against each one? Note that this will require the user to already be authenticated to the providers, so I think this would work best for autoconfigured credentials like (via #10537), but individual users could log into a provider in the modal, then run this command.

I'm leaning towards a command as checking all the models on each extension activation would cause a delay, with how the code is currently structured (each provider is connected to in sequence). I can open an issue for this separately if that sounds like a decent approach to try!

logged #10702 to capture this idea!