imageGenerationModel support in provider?

[Und3rf10w]

What are the thoughts on adding a imageGenerationModel as a property to a Provider?

I came up with these types as an example, but this is tied directly to Dalle.

type ImageGenerationModelV1CallSettings = {
  size: string
  n: number
  quality?: 'natural' | 'vivid'
  /**
  Response format. The output can either be url or b64json. Default is url.

     */
  responseFormat?: {
    type: 'url'
  } | {
    type: 'b64json'
  }
  /**
  Abort signal for cancelling the operation.
     */
  abortSignal?: AbortSignal
  /**
Additional HTTP headers to be sent with the request.
Only applicable for HTTP-based providers.
   */
  headers?: Record<string, string | undefined>
}

type ImageModelV1CallOptions = ImageGenerationModelV1CallSettings & {
  inputFormat: 'prompt'
  prompt: LanguageModelV1Prompt
}

type ImageGenerationModelV1FinishReason = 'completed' | 'content-filter' | 'error' | 'other' | 'unknown'

type DallePromiseLike = PromiseLike<{
  created: Date
  expires?: Date
  id?: string
  result?: {
    data: Array<{
      url: string
    }>
  }
  status?: string
  data?: Array<{
    url: string
    revised_prompt: string
  }>
  error?: {
    code: string
    message: string
  }
  lastActionDateTime: Date
}>

export interface ImageGenerationModelV1 {
    /**
  The language model must specify which language model interface
  version it implements. This will allow us to evolve the language
  model interface and retain backwards compatibility. The different
  implementation versions can be handled as a discriminated union
  on our side.
     */
  readonly specificationVersion: 'v1';
  /**
Name of the provider for logging purposes.
   */
  readonly provider: string;
  /**
Provider-specific model ID for logging purposes.
   */
  readonly modelId: string;
  /**
  Default object generation mode that should be used with this model when
  no mode is specified. Should be the mode with the best results for this
  model. `undefined` can be returned if object generation is not supported.
  
  This is needed to generate the best objects possible w/o requiring the
  user to explicitly specify the object generation mode.
     */
  readonly defaultObjectGenerationMode: 'json' | 'tool' | undefined
  doGenerate: (options: ImageModelV1CallOptions) => DallePromiseLike & {
    /**
     * Finish reason.
     */
    finishReason: ImageGenerationModelV1FinishReason
    /**
  Raw prompt and setting information for observability provider integration.
  */
    rawCall: {
    /**
  Raw prompt after expansion and conversion to the format that the
  provider uses to send the information to their API.
      */
      rawPrompt: unknown
      /**
  Raw settings that are used for the API call. Includes provider-specific
  settings.
      */
      rawSettings: Record<string, unknown>
    }
    /**
  Optional raw response information for debugging purposes.
  */
    rawResponse?: {
    /**
  Response headers.
      */
      headers?: Record<string, string>
    }
  }
}

Maybe the call settings and DallePromiseLike would be tied to like an OpenAIImageGenerationModel instead that implements from ImageModelV1?

I don't actually have any idea on how to properly handle doStream cases though.

Ideally, since most of the providers taht are available in the ProviderRegistry also expose image generation models (e.g. Dalle for OpenAI, Stable diffusion for vertex/bedrock), we would also be able to add those and manage those using the registry so we only have to track models in one place and they can all be self contained in the registry.

[lgrammel]

Move to #2524