Skip to content

Commit be71290

Browse files
authored
[Feat] Add OpenAI compatible bedrock imported models. - qwen etc (#17097)
* test_bedrock_openai_imported_model * AmazonBedrockOpenAIConfig * add openai route for bedrock * docs fix * fix code qa check
1 parent db2c8e3 commit be71290

File tree

10 files changed

+699
-227
lines changed

10 files changed

+699
-227
lines changed

docs/my-website/docs/providers/bedrock.md

Lines changed: 1 addition & 201 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ ALL Bedrock models (Anthropic, Meta, Deepseek, Mistral, Amazon, etc.) are Suppor
77
| Property | Details |
88
|-------|-------|
99
| Description | Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs). |
10-
| Provider Route on LiteLLM | `bedrock/`, [`bedrock/converse/`](#set-converse--invoke-route), [`bedrock/invoke/`](#set-invoke-route), [`bedrock/converse_like/`](#calling-via-internal-proxy), [`bedrock/llama/`](#deepseek-not-r1), [`bedrock/deepseek_r1/`](#deepseek-r1), [`bedrock/qwen3/`](#qwen3-imported-models) |
10+
| Provider Route on LiteLLM | `bedrock/`, [`bedrock/converse/`](#set-converse--invoke-route), [`bedrock/invoke/`](#set-invoke-route), [`bedrock/converse_like/`](#calling-via-internal-proxy), [`bedrock/llama/`](#deepseek-not-r1), [`bedrock/deepseek_r1/`](#deepseek-r1), [`bedrock/qwen3/`](#qwen3-imported-models), [`bedrock/openai/`](./bedrock_imported.md#openai-compatible-imported-models-qwen-25-vl-etc) |
1111
| Provider Doc | [Amazon Bedrock ↗](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) |
1212
| Supported OpenAI Endpoints | `/chat/completions`, `/completions`, `/embeddings`, `/images/generations` |
1313
| Rerank Endpoint | `/rerank` |
@@ -1598,206 +1598,6 @@ curl -X POST 'http://0.0.0.0:4000/chat/completions' \
15981598
</Tabs>
15991599

16001600

1601-
## Bedrock Imported Models (Deepseek, Deepseek R1)
1602-
1603-
### Deepseek R1
1604-
1605-
This is a separate route, as the chat template is different.
1606-
1607-
| Property | Details |
1608-
|----------|---------|
1609-
| Provider Route | `bedrock/deepseek_r1/{model_arn}` |
1610-
| Provider Documentation | [Bedrock Imported Models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-import-model.html), [Deepseek Bedrock Imported Model](https://aws.amazon.com/blogs/machine-learning/deploy-deepseek-r1-distilled-llama-models-with-amazon-bedrock-custom-model-import/) |
1611-
1612-
<Tabs>
1613-
<TabItem value="sdk" label="SDK">
1614-
1615-
```python
1616-
from litellm import completion
1617-
import os
1618-
1619-
response = completion(
1620-
model="bedrock/deepseek_r1/arn:aws:bedrock:us-east-1:086734376398:imported-model/r4c4kewx2s0n", # bedrock/deepseek_r1/{your-model-arn}
1621-
messages=[{"role": "user", "content": "Tell me a joke"}],
1622-
)
1623-
```
1624-
1625-
</TabItem>
1626-
1627-
<TabItem value="proxy" label="Proxy">
1628-
1629-
1630-
**1. Add to config**
1631-
1632-
```yaml
1633-
model_list:
1634-
- model_name: DeepSeek-R1-Distill-Llama-70B
1635-
litellm_params:
1636-
model: bedrock/deepseek_r1/arn:aws:bedrock:us-east-1:086734376398:imported-model/r4c4kewx2s0n
1637-
1638-
```
1639-
1640-
**2. Start proxy**
1641-
1642-
```bash
1643-
litellm --config /path/to/config.yaml
1644-
1645-
# RUNNING at http://0.0.0.0:4000
1646-
```
1647-
1648-
**3. Test it!**
1649-
1650-
```bash
1651-
curl --location 'http://0.0.0.0:4000/chat/completions' \
1652-
--header 'Authorization: Bearer sk-1234' \
1653-
--header 'Content-Type: application/json' \
1654-
--data '{
1655-
"model": "DeepSeek-R1-Distill-Llama-70B", # 👈 the 'model_name' in config
1656-
"messages": [
1657-
{
1658-
"role": "user",
1659-
"content": "what llm are you"
1660-
}
1661-
],
1662-
}'
1663-
```
1664-
1665-
</TabItem>
1666-
</Tabs>
1667-
1668-
1669-
### Deepseek (not R1)
1670-
1671-
| Property | Details |
1672-
|----------|---------|
1673-
| Provider Route | `bedrock/llama/{model_arn}` |
1674-
| Provider Documentation | [Bedrock Imported Models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-import-model.html), [Deepseek Bedrock Imported Model](https://aws.amazon.com/blogs/machine-learning/deploy-deepseek-r1-distilled-llama-models-with-amazon-bedrock-custom-model-import/) |
1675-
1676-
1677-
1678-
Use this route to call Bedrock Imported Models that follow the `llama` Invoke Request / Response spec
1679-
1680-
1681-
<Tabs>
1682-
<TabItem value="sdk" label="SDK">
1683-
1684-
```python
1685-
from litellm import completion
1686-
import os
1687-
1688-
response = completion(
1689-
model="bedrock/llama/arn:aws:bedrock:us-east-1:086734376398:imported-model/r4c4kewx2s0n", # bedrock/llama/{your-model-arn}
1690-
messages=[{"role": "user", "content": "Tell me a joke"}],
1691-
)
1692-
```
1693-
1694-
</TabItem>
1695-
1696-
<TabItem value="proxy" label="Proxy">
1697-
1698-
1699-
**1. Add to config**
1700-
1701-
```yaml
1702-
model_list:
1703-
- model_name: DeepSeek-R1-Distill-Llama-70B
1704-
litellm_params:
1705-
model: bedrock/llama/arn:aws:bedrock:us-east-1:086734376398:imported-model/r4c4kewx2s0n
1706-
1707-
```
1708-
1709-
**2. Start proxy**
1710-
1711-
```bash
1712-
litellm --config /path/to/config.yaml
1713-
1714-
# RUNNING at http://0.0.0.0:4000
1715-
```
1716-
1717-
**3. Test it!**
1718-
1719-
```bash
1720-
curl --location 'http://0.0.0.0:4000/chat/completions' \
1721-
--header 'Authorization: Bearer sk-1234' \
1722-
--header 'Content-Type: application/json' \
1723-
--data '{
1724-
"model": "DeepSeek-R1-Distill-Llama-70B", # 👈 the 'model_name' in config
1725-
"messages": [
1726-
{
1727-
"role": "user",
1728-
"content": "what llm are you"
1729-
}
1730-
],
1731-
}'
1732-
```
1733-
1734-
</TabItem>
1735-
</Tabs>
1736-
1737-
### Qwen3 Imported Models
1738-
1739-
| Property | Details |
1740-
|----------|---------|
1741-
| Provider Route | `bedrock/qwen3/{model_arn}` |
1742-
| Provider Documentation | [Bedrock Imported Models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-import-model.html), [Qwen3 Models](https://aws.amazon.com/about-aws/whats-new/2025/09/qwen3-models-fully-managed-amazon-bedrock/) |
1743-
1744-
<Tabs>
1745-
<TabItem value="sdk" label="SDK">
1746-
1747-
```python
1748-
from litellm import completion
1749-
import os
1750-
1751-
response = completion(
1752-
model="bedrock/qwen3/arn:aws:bedrock:us-east-1:086734376398:imported-model/your-qwen3-model", # bedrock/qwen3/{your-model-arn}
1753-
messages=[{"role": "user", "content": "Tell me a joke"}],
1754-
max_tokens=100,
1755-
temperature=0.7
1756-
)
1757-
```
1758-
1759-
</TabItem>
1760-
1761-
<TabItem value="proxy" label="Proxy">
1762-
1763-
**1. Add to config**
1764-
1765-
```yaml
1766-
model_list:
1767-
- model_name: Qwen3-32B
1768-
litellm_params:
1769-
model: bedrock/qwen3/arn:aws:bedrock:us-east-1:086734376398:imported-model/your-qwen3-model
1770-
1771-
```
1772-
1773-
**2. Start proxy**
1774-
1775-
```bash
1776-
litellm --config /path/to/config.yaml
1777-
1778-
# RUNNING at http://0.0.0.0:4000
1779-
```
1780-
1781-
**3. Test it!**
1782-
1783-
```bash
1784-
curl --location 'http://0.0.0.0:4000/chat/completions' \
1785-
--header 'Authorization: Bearer sk-1234' \
1786-
--header 'Content-Type: application/json' \
1787-
--data '{
1788-
"model": "Qwen3-32B", # 👈 the 'model_name' in config
1789-
"messages": [
1790-
{
1791-
"role": "user",
1792-
"content": "what llm are you"
1793-
}
1794-
],
1795-
}'
1796-
```
1797-
1798-
</TabItem>
1799-
</Tabs>
1800-
18011601
### OpenAI GPT OSS
18021602

18031603
| Property | Details |

0 commit comments

Comments
 (0)