✨ feat: Smooth WRR for Load Balancing

[zzzhouuu]

The current implementation routes all traffic to the first-configured provider when a model has multiple providers; without changing existing logic, introduce the Smooth WRR algorithm to achieve traffic balance among model providers.

Support adding multiple API keys for a single provider and distribute traffic according to their weights.

config example:

providers:
  groq1:
    type: openai
    base_url: https://api.groq.com/openai/v1
    api_key: sk-groq-api-key
  groq2:
    type: openai
    base_url: https://api.groq.com/openai/v1
    keys:
      - api_key: sk-groq-api-key-1
         weight: 1
      - api_key: sk-groq-api-key-2
         weight: 1
 models:
   kimi-k2-instruct-0905:
    providers:
      - provider: groq1
        model: moonshotai/kimi-k2-instruct-0905
        weight: 1
      - provider: groq2
        model: moonshotai/kimi-k2-instruct-0905
        weight: 1

[yvbbrjdr]

Thanks for your contributions! This is indeed very useful feature. I'll review it and merge soon.

[yvbbrjdr]

I'd actually like to make the api_key field in the provider accept both a string and a list of objects (key/weight pair), that way the config is cleaner.

[zzzhouuu]

Mainly for compatibility and to allow users who don’t need load balancing to configure simply, the API key is now used to store the key with the highest weight for the current provider, making it easier for the underlying code to call.