Scenario Internal AI Gateway

Goal: Your organization runs an AI gateway, such as LiteLLM, in front of its AI models. AI Studio sends every request through the gateway, which handles keys, budgets, and routing, and you learn which features of AI Studio are used.

You need: the gateway, with a key per user or per department.

How AI Studio sees a gateway

AI Studio has a provider type LITE_LLM for LiteLLM. Two things about it are easy to miss:

• It counts as a type of its own, not as self-hosted. Under the scheme LOCAL_TRUST_ONLY, it gets the level Very Low. If your gateway runs inside your organization, use a CUSTOM scheme and give LITE_LLM the level you trust it with.
• For data security, it isn't self-hosted either. ERI data sources with the security policy SELF_HOSTED stay closed to it until you list its entries in DataSourceSecuritySettings.TrustedProviderIds.

The level belongs to the type, so all LiteLLM entries share it, whichever models behind the gateway they use. When your gateway also routes to cloud models, and only some of its models may see confidential content, the trust list helps per entry, but the confidence level doesn't. Then consider a separate entry, or a self-hosted provider, for your internal models.

The configuration

CONFIG = {}

CONFIG["LLM_PROVIDERS"] = {}
CONFIG["LLM_PROVIDERS"][#CONFIG["LLM_PROVIDERS"]+1] = {
    ["Id"] = "2b4d1c6a-0f2e-4d9c-9a3b-5c6d7e8f9a0b",
    ["InstanceName"] = "AI gateway: internal model",
    ["UsedLLMProvider"] = "LITE_LLM",
    ["Host"] = "NONE",
    ["Hostname"] = "https://ai-gateway.intra.example.org",
    ["PromptForUserAPIKey"] = true,
    ["AdditionalJsonApiParameters"] = "",
    ["Model"] = {
        ["Id"] = "internal-model",
        ["DisplayName"] = "Internal model",
    },
}

CONFIG["SETTINGS"] = {}

-- The gateway runs inside the organization:
CONFIG["SETTINGS"]["DataConfidence.ConfidenceScheme"] = "CUSTOM"
CONFIG["SETTINGS"]["DataConfidence.CustomConfidenceScheme"] = {
    ["SELF_HOSTED"] = "HIGH",
    ["LITE_LLM"] = "HIGH",
}

-- Its entry may see confidential data sources:
CONFIG["SETTINGS"]["DataSourceSecuritySettings.TrustedProviderIds"] = {
    "2b4d1c6a-0f2e-4d9c-9a3b-5c6d7e8f9a0b",
}

-- Requests name the feature of AI Studio that sent them:
CONFIG["SETTINGS"]["DataApp.ShareFeatureUsageWithSelfHostedServerOperators"] = true

With PromptForUserAPIKey, each user adds their personal key of the gateway, such as a virtual key of LiteLLM, see Users Bring Their Own Keys. Alternatively, roll out one encrypted key per department, see Organization-Provided API Keys.

Usage statistics

With DataApp.ShareFeatureUsageWithSelfHostedServerOperators, AI Studio names the feature in the User-Agent of each request to self-hosted and LiteLLM providers, such as Component/CHAT or Component/TRANSLATION_ASSISTANT. LiteLLM shows it as a tag of the request.

Clarify data protection first. With personal keys, your gateway can tell users apart, and the usage data relates to persons. In some countries, such as Germany, evaluating it may need the consent of your works council. Evaluate per feature, not per person. Details are on Labeling and Usage Statistics.

Check it

1. Stage the configuration on a test device, see Testing and Exporting Configurations.
2. Send a message in the chat and run an assistant. The logs of the gateway show the requests with their features.
3. Select a confidential ERI data source in a chat with the gateway: AI Studio lets the gateway use it.

Where to go next

• Configuration Scenarios
• Confidence and Data Security