diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 9ccfa3bc5..b7213e8f3 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,70 +1,180 @@
-## 📌 PR 内容 / PR Description
-
--
+# 📌 PR 内容 / PR Description
-## 🔍 相关 Issue / Related Issue
-
-
-## ✅ 变更类型 / Type of Change
-
-- [ ] 修复 Bug / Bug fix (non-breaking change that fixes an issue)
-- [ ] 新功能 / New feature (non-breaking change that adds functionality)
-- [ ] 重构 / Refactor (no functionality change, code structure optimized)
-- [ ] 重大变更 / Breaking change (fix or feature that would cause existing functionality to change)
-- [ ] 文档更新 / Documentation update (changes to docs only)
-- [ ] 性能优化 / Performance optimization
-
-## 🧪 如何测试 / How Has This Been Tested?
-
-1.
-2.
-3.
-
-## 📷 截图 / Demo (Optional)
-
+---
+
+# 🎯 问题背景 / Problem Context
+
+
-
-## ⚡ 更新后的用法示例 / Usage After Update
-
+# 🔍 相关 Issue / Related Issue
+
+*
+
+---
+
+# ✅ 变更类型 / Type of Change
+
+* [ ] Bug fix
+* [ ] New feature
+* [ ] Refactor
+* [ ] Breaking change
+* [ ] Documentation
+* [ ] Performance optimization
+
+---
+
+# 🧪 如何测试 / How Has This Been Tested?
+
+1.
+2.
+3.
+
+---
+
+# 📷 Demo (Optional)
+
+*
+
+---
+
+# ⚡ 更新后的用法示例 / Usage After Update
+
```python
-# 示例 / Example
```
-## 🔄 重构前 / 重构后对比 (仅当 Type 为 Refactor) / Refactor Before & After (only for Refactor)
-
+---
+
+# 🔄 重构对比 / Refactor Comparison (如适用 / if applicable)
+
+## Before
+
+```python
+```
+
+## After
+
+```python
+```
+
+# ⚠️ 注意事项 / Additional Notes
+
+*
+
+---
+
+
+# 🧠 AI 参与情况 / AI Involvement
+
+
+- [ ] 未使用 AI / No AI used
+- [ ] 使用 AI 辅助（局部代码）/ AI-assisted (partial code)
+- [ ] 使用 AI 主导（大部分代码）/ AI-led (majority of code)
+
+**使用工具 / Tools Used：**
+- [ ] Cursor
+- [ ] CodeX
+- [ ] ClaudeCode
+- [ ] OpenCode
+- [ ] 其他 / Other：
+
+# 🤖 AI 生成过程 / AI Generation Process
+
+## 初始 Prompt / Initial Prompt
+
+```text
+```
+
+## AI 初始输出质量 / Initial Output Quality
+
+* [ ] 一次正确 / Correct on first try
+* [ ] 部分正确 / Partially correct
+* [ ] 基本不可用 / Mostly unusable
+
+## **问题点 / Issues：**
+
+## 🔧 人工干预过程 / Human Intervention
+
+
+
+### 第一次修正 / First Correction
+
+**修改后的 Prompt / 操作 / Updated Prompt or Action：**
+
+```text
+```
+
+### **原因 / Reason：**
+
+### 第二次修正（如有）/ Second Correction (if any)
+
+**修改后的 Prompt / 操作 / Updated Prompt or Action：**
+
+```text
+```
+
+### **原因 / Reason：**
-### 重构前 / Before:
+### 最终策略总结 / Final Strategy Summary
+
-### 重构后 / After:
+*
+
+---
+
+# 📊 AI vs 人工贡献占比 / AI vs Human Contribution
+
+
+
+* AI 生成代码占比 / AI-generated code：`%`
+* 人工修改占比 / Human modifications：`%`
+
+**主要人工修改点 / Key Human Modifications：**
+
+* [ ] 逻辑修正 / Logic fixes
+* [ ] 边界处理 / Edge case handling
+* [ ] 性能优化 / Performance optimization
+* [ ] 代码风格 / Code style
+* [ ] 架构调整 / Architecture adjustments
+
+---
+
+# ⚠️ AI 问题记录 / AI Failure Cases
+
+
+
+### ❌ 失败 Prompt 示例 / Failed Prompt Example
+
+```text
+```
+
+### ❌ 问题表现 / Issue Description
+
+*
+
+### ✅ 改进后 Prompt / Improved Prompt
+
+```text
+```
+# 🧩 可复用经验 / Reusable Patterns
-## ⚠️ 注意事项 / Additional Notes
-
+* Prompt 模板 / Prompt template：
+* 常见坑 / Common pitfalls：
+* 推荐做法 / Recommended practices：
diff --git a/docs/en/Best Practice/prompt.md b/docs/en/Best Practice/prompt.md
index 45185a1f2..86d326b55 100644
--- a/docs/en/Best Practice/prompt.md
+++ b/docs/en/Best Practice/prompt.md
@@ -31,14 +31,15 @@ import lazyllm
prompter.generate_prompt(dict(context='背景', input='输入'))
# {'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\n你是一个由LazyLLM开发的知识问答助手，你的任务是根据提供的上下文信息来回答用户的问题。上下文信息是背景，用户的问题是输入，现在请你做出回答。\n\n'}, {'role': 'user', 'content': ''}]}
-prompter.generate_prompt(dict(context='背景', input='输入'), return_dict=True)
+prompter.generate_prompt(dict(context='背景', input='输入'), format='openai')
```
In the example above, the ``generate_prompt`` function accepts a ``dict`` as input, filling in the slots in the ``instruction`` template with the provided values.
!!! Note
- - In the code above, there's a parameter ``return_dict`` worth noting. When ``return_dict`` is set to True, it returns a dictionary in the OpenAI format for online models.
+ - In the code above, there's a parameter ``format`` worth noting. When ``format='openai'``, it returns a dictionary in the OpenAI Chat Completions format for online models.
+ - The legacy ``return_dict=True`` remains supported (equivalent to ``format='openai'``, with a deprecation warning); prefer ``format`` in new code.
- Typically, you just need to assign the Prompter to a ``TrainableModule`` or ``OnlineChatModule`` without worrying about the ``generate_prompt`` function.
### Design Concept of LazyLLM Prompter
@@ -221,7 +222,7 @@ The tool will be read after step 4 in [Prompt Generation Process Analysis](#anal
>>> import lazyllm
>>> tools=[dict(type='function', function=dict(name='example'))]
>>> prompter = lazyllm.AlpacaPrompter('你是一个工具调用的Agent，我会给你提供一些工具，请根据用户输入，帮我选择最合适的工具并使用', extra_keys='input', tools=tools)
- >>> prompter.generate_prompt('帮我查询一下今天的天气', return_dict=True)
+ >>> prompter.generate_prompt('帮我查询一下今天的天气', format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\n\\n ### Instruction:\\n你是一个工具调用的Agent，我会给你提供一些工具，请根据用户输入，帮我选择最合适的工具并使用\\n\\nHere are some extra messages you can referred to:\\n\\n### input:\\n帮我查询一下今天的天气\\n\\n'}, {'role': 'user', 'content': ''}],
'tools': [{'type': 'function', 'function': {'name': 'example'}}]}
@@ -234,9 +235,9 @@ If we want the model to have multi-turn conversation capabilities, we need to co
>>> prompter = lazyllm.ChatPrompter('你是一个对话机器人，现在你要和用户进行友好的对话')
>>> prompter.generate_prompt('我们聊会儿天吧', history=[['你好', '你好，我是一个对话机器人，有什么能为您服务的']])
'<|start_system|>You are an AI-Agent developed by LazyLLM.你是一个对话机器人，现在你要和用户进行友好的对话\\n\\n<|end_system|>\\n\\n<|Human|>:你好<|Assistant|>:你好，我是一个对话机器人，有什么能为您服务的\\n<|Human|>:\\n我们聊会儿天吧\\n<|Assistant|>:\\n'
->>> prompter.generate_prompt('我们聊会儿天吧', history=[['你好', '你好，我是一个对话机器人，有什么能为您服务的']], return_dict=True)
+>>> prompter.generate_prompt('我们聊会儿天吧', history=[['你好', '你好，我是一个对话机器人，有什么能为您服务的']], format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\n你是一个对话机器人，现在你要和用户进行友好的对话\\n\\n'}, {'role': 'user', 'content': '你好'}, {'role': 'assistant', 'content': '你好，我是一个对话机器人，有什么能为您服务的'}, {'role': 'user', 'content': '我们聊会儿天吧'}]}
->>> prompter.generate_prompt('我们聊会儿天吧', history=[dict(role='user', content='你好'), dict(role='assistant', content='你好，我是一个对话机器人，有什么能为您服务的')], return_dict=True)
+>>> prompter.generate_prompt('我们聊会儿天吧', history=[dict(role='user', content='你好'), dict(role='assistant', content='你好，我是一个对话机器人，有什么能为您服务的')], format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\n你是一个对话机器人，现在你要和用户进行友好的对话\\n\\n'}, {'role': 'user', 'content': '你好'}, {'role': 'assistant', 'content': '你好，我是一个对话机器人，有什么能为您服务的'}, {'role': 'user', 'content': '我们聊会儿天吧'}]}
```
@@ -245,12 +246,12 @@ The conversation history will be read in step 4 of [Prompt Generation Process An
!!! Note
- Only ``ChatPrompter`` supports passing in conversation history.
- - When the input is in the format ``[[a, b], ...]``, both ``return_dict`` set to ``True`` or ``False`` are supported, whereas when the input is in the format ``[dict, dict]``, only ``return_dict`` set to ``True`` is supported.
+ - When the input is in the format ``[[a, b], ...]``, both ``format='openai'`` (or default ``None`` for string concatenation) are supported, whereas when the input is in the format ``[dict, dict]``, only API message formats such as ``format='openai'`` are supported.
#### Used with OnlineChatModule
-When the ``Prompter`` is used with the ``OnlineChatModule``, ``OnlineChatModule.__call__`` will call ``Prompter.generate_prompt`` and ``pass __input``,
-``history``, and ``tools`` to ``generate_prompt``. At this time, the ``return_dict`` of ``generate_prompt`` will be set to ``True``. Below is an example:
+When the ``Prompter`` is used with the ``OnlineChatModule``, ``OnlineChatModule.__call__`` will call ``Prompter.generate_prompt`` and pass ``__input``,
+``history``, and ``tools`` to ``generate_prompt``. The module sets ``format`` to ``'openai'`` or ``'anthropic'`` as appropriate. Below is an example:
```python
import lazyllm
@@ -262,8 +263,8 @@ module(dict(context='背景', input='输入'))
#### Used with TrainableModule
-When the ``Prompter`` is used with the ``TrainableModule``, ``TrainableModule.__call__`` will call ``Prompter.generate_prompt`` and ``pass __input``,
-``history``, and ``tools`` to ``generate_prompt``. At this time, the ``return_dict`` of ``generate_prompt`` will be set to ``True``. Below is an example:
+When the ``Prompter`` is used with the ``TrainableModule``, ``TrainableModule.__call__`` will call ``Prompter.generate_prompt`` and pass ``__input``,
+``history``, and ``tools`` to ``generate_prompt``. ``format`` stays at the default (``None``), producing the concatenated string for local inference. Below is an example:
```python
import lazyllm
diff --git a/docs/en/Cookbook/decentralized.md b/docs/en/Cookbook/decentralized.md
index c6668613b..8d6ad34f1 100644
--- a/docs/en/Cookbook/decentralized.md
+++ b/docs/en/Cookbook/decentralized.md
@@ -523,7 +523,7 @@ def generate_character_description(character_name):
that emphasizes their personalities.
Speak directly to {character_name}.
Do not add anything else.''',
- return_dict=True)
+ format='openai')
character_description = lazyllm.OnlineChatModule(model='Qwen3-32B', static_params=temp)(contents)
return character_description
diff --git a/docs/lazyllm-skill/assets/basic/prompter.md b/docs/lazyllm-skill/assets/basic/prompter.md
index 523a17c3b..a9eab2c80 100644
--- a/docs/lazyllm-skill/assets/basic/prompter.md
+++ b/docs/lazyllm-skill/assets/basic/prompter.md
@@ -78,17 +78,17 @@ from lazyllm import AlpacaPrompter
p = AlpacaPrompter('hello world {instruction}')
p.generate_prompt('this is my input')
'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\nhello world this is my input\n\n\n### Response:\n'
-p.generate_prompt('this is my input', return_dict=True)
+p.generate_prompt('this is my input', format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\nhello world this is my input\n\n'}, {'role': 'user', 'content': ''}]}
p = AlpacaPrompter('hello world {instruction}, {input}', extra_keys=['knowledge'])
p.generate_prompt(dict(instruction='hello world', input='my input', knowledge='lazyllm'))
'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\nhello world hello world, my input\n\nHere are some extra messages you can referred to:\n\n### knowledge:\nlazyllm\n\n\n### Response:\n'
-p.generate_prompt(dict(instruction='hello world', input='my input', knowledge='lazyllm'), return_dict=True)
+p.generate_prompt(dict(instruction='hello world', input='my input', knowledge='lazyllm'), format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\nhello world hello world, my input\n\nHere are some extra messages you can referred to:\n\n### knowledge:\nlazyllm\n\n'}, {'role': 'user', 'content': ''}]}
p = AlpacaPrompter(dict(system="hello world", user="this is user instruction {input}"))
p.generate_prompt(dict(input="my input"))
'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\nhello word\n\n\n\nthis is user instruction my input### Response:\n'
-p.generate_prompt(dict(input="my input"), return_dict=True)
+p.generate_prompt(dict(input="my input"), format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\nhello world'}, {'role': 'user', 'content': 'this is user instruction my input'}]}
```
@@ -127,7 +127,7 @@ p.generate_prompt({
p = ChatPrompter(dict(system="hello world", user="this is user instruction {input}"))
p.generate_prompt({'input': "my input", 'query': "this is user query"})
'You are an AI-Agent developed by LazyLLM.hello world\nthis is user instruction my input this is user query\n'
-p.generate_prompt({'input': "my input", 'query': "this is user query"}, return_dict=True)
+p.generate_prompt({'input': "my input", 'query': "this is user query"}, format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nhello world'}, {'role': 'user', 'content': 'this is user instruction my input this is user query'}]}
```
diff --git a/docs/zh/Best Practice/prompt.md b/docs/zh/Best Practice/prompt.md
index fdef3224c..281b88ec4 100644
--- a/docs/zh/Best Practice/prompt.md
+++ b/docs/zh/Best Practice/prompt.md
@@ -31,14 +31,15 @@ import lazyllm
prompter.generate_prompt(dict(context='背景', input='输入'))
# {'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\n你是一个由LazyLLM开发的知识问答助手，你的任务是根据提供的上下文信息来回答用户的问题。上下文信息是背景，用户的问题是输入，现在请你做出回答。\n\n'}, {'role': 'user', 'content': ''}]}
-prompter.generate_prompt(dict(context='背景', input='输入'), return_dict=True)
+prompter.generate_prompt(dict(context='背景', input='输入'), format='openai')
```
在上面的例子中， ``generate_prompt`` 的输入是一个 ``dict`` ，他会把值依次填入 ``instruction`` 提供的槽位中。
!!! Note "注意"
- - 上面代码中出现了一个值得您关注的参数 ``return_dict`` , 当 ``return_dict`` 为 True 时，会返回 OpenAI 格式的 dict 用于线上模型。
+ - 上面代码中出现了一个值得您关注的参数 ``format`` ，当 ``format='openai'`` 时，会返回 OpenAI Chat Completions 风格的 dict 用于线上模型。
+ - 旧版代码中的 ``return_dict=True`` 仍受支持（等价于 ``format='openai'``，并会提示弃用）；新代码请优先使用 ``format``。
- 一般情况下，您只需要将Prompter设置给 ``TrainableModule`` 或 ``OnlineChatModule`` 即可，而无需关心 ``generate_prompt`` 这一函数。
## LazyLLM Prompter 的设计思路
@@ -220,7 +221,7 @@ prompter.generate_prompt(dict(context='背景', input='输入'), return_dict=Tru
>>> import lazyllm
>>> tools=[dict(type='function', function=dict(name='example'))]
>>> prompter = lazyllm.AlpacaPrompter('你是一个工具调用的Agent，我会给你提供一些工具，请根据用户输入，帮我选择最合适的工具并使用', extra_keys='input', tools=tools)
- >>> prompter.generate_prompt('帮我查询一下今天的天气', return_dict=True)
+ >>> prompter.generate_prompt('帮我查询一下今天的天气', format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\n\\n ### Instruction:\\n你是一个工具调用的Agent，我会给你提供一些工具，请根据用户输入，帮我选择最合适的工具并使用\\n\\nHere are some extra messages you can referred to:\\n\\n### input:\\n帮我查询一下今天的天气\\n\\n'}, {'role': 'user', 'content': ''}],
'tools': [{'type': 'function', 'function': {'name': 'example'}}]}
@@ -234,9 +235,9 @@ prompter.generate_prompt(dict(context='背景', input='输入'), return_dict=Tru
>>> prompter = lazyllm.ChatPrompter('你是一个对话机器人，现在你要和用户进行友好的对话')
>>> prompter.generate_prompt('我们聊会儿天吧', history=[['你好', '你好，我是一个对话机器人，有什么能为您服务的']])
'<|start_system|>You are an AI-Agent developed by LazyLLM.你是一个对话机器人，现在你要和用户进行友好的对话\\n\\n<|end_system|>\\n\\n<|Human|>:你好<|Assistant|>:你好，我是一个对话机器人，有什么能为您服务的\\n<|Human|>:\\n我们聊会儿天吧\\n<|Assistant|>:\\n'
->>> prompter.generate_prompt('我们聊会儿天吧', history=[['你好', '你好，我是一个对话机器人，有什么能为您服务的']], return_dict=True)
+>>> prompter.generate_prompt('我们聊会儿天吧', history=[['你好', '你好，我是一个对话机器人，有什么能为您服务的']], format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\n你是一个对话机器人，现在你要和用户进行友好的对话\\n\\n'}, {'role': 'user', 'content': '你好'}, {'role': 'assistant', 'content': '你好，我是一个对话机器人，有什么能为您服务的'}, {'role': 'user', 'content': '我们聊会儿天吧'}]}
->>> prompter.generate_prompt('我们聊会儿天吧', history=[dict(role='user', content='你好'), dict(role='assistant', content='你好，我是一个对话机器人，有什么能为您服务的')], return_dict=True)
+>>> prompter.generate_prompt('我们聊会儿天吧', history=[dict(role='user', content='你好'), dict(role='assistant', content='你好，我是一个对话机器人，有什么能为您服务的')], format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\n你是一个对话机器人，现在你要和用户进行友好的对话\\n\\n'}, {'role': 'user', 'content': '你好'}, {'role': 'assistant', 'content': '你好，我是一个对话机器人，有什么能为您服务的'}, {'role': 'user', 'content': '我们聊会儿天吧'}]}
```
@@ -245,12 +246,12 @@ prompter.generate_prompt(dict(context='背景', input='输入'), return_dict=Tru
!!! Note "注意"
- 只有 ``ChatPrompter`` 支持传入历史对话
- - 当输入是 ``[[a, b], ...]`` 格式时，同时支持 ``return_dict`` 为 ``True`` 或 ``False`` ， 而当输入为 ``[dict, dict]`` 格式时，仅支持 ``return_dict`` 为 ``True``
+ - 当输入是 ``[[a, b], ...]`` 格式时，同时支持 ``format`` 为 ``'openai'``（或默认 ``None`` 走字符串拼接），而当输入为 ``[dict, dict]`` 格式时，仅支持 ``format='openai'`` 等 API 消息格式
### 和 OnlineChatModule 一起使用
-当 ``Prompter`` 和 ``OnlineChatModule`` 一起使用时， ``OnlineChatModule.__call__`` 会调用 ``Prompter.generate_prompt`` ，并且将 ``__input``,
-``history`` 和 ``tools`` 传给 ``generate_prompt`` ，此时 ``generate_prompt`` 的 ``return_dict`` 会被设置为 ``True``。下面给出一个例子：
+当 ``Prompter`` 和 ``OnlineChatModule`` 一起使用时， ``OnlineChatModule.__call__`` 会调用 ``Prompter.generate_prompt`` ，并且将 ``__input``、
+``history`` 和 ``tools`` 传给 ``generate_prompt`` ，此时 ``generate_prompt`` 的 ``format`` 会由模块设为 ``'openai'`` 或 ``'anthropic'`` 等。下面给出一个例子：
```python
import lazyllm
@@ -262,8 +263,8 @@ module(dict(context='背景', input='输入'))
### 和 TrainableModule 一起使用
-当 ``Prompter`` 和 ``TrainableModule`` 一起使用时， ``TrainableModule.__call__`` 会调用 ``Prompter.generate_prompt`` ，并且将 ``__input``,
-``history`` 和 ``tools`` 传给 ``generate_prompt`` ，此时 ``generate_prompt`` 的 ``return_dict`` 会被设置为 ``True``。下面给出一个例子：
+当 ``Prompter`` 和 ``TrainableModule`` 一起使用时， ``TrainableModule.__call__`` 会调用 ``Prompter.generate_prompt`` ，并且将 ``__input``、
+``history`` 和 ``tools`` 传给 ``generate_prompt`` ，此时 ``format`` 为默认（``None``），得到用于本地推理的拼接字符串。下面给出一个例子：
```python
import lazyllm
diff --git a/docs/zh/Cookbook/decentralized.md b/docs/zh/Cookbook/decentralized.md
index 001683bbd..b7211c63f 100644
--- a/docs/zh/Cookbook/decentralized.md
+++ b/docs/zh/Cookbook/decentralized.md
@@ -513,7 +513,7 @@ def generate_character_description(character_name):
that emphasizes their personalities.
Speak directly to {character_name}.
Do not add anything else.''',
- return_dict=True)
+ format='openai')
character_description = lazyllm.OnlineChatModule(model='Qwen3-32B', static_params=temp)(
contents
)
diff --git a/lazyllm/components/prompter/builtinPrompt.py b/lazyllm/components/prompter/builtinPrompt.py
index 2d705ed34..37294e51a 100644
--- a/lazyllm/components/prompter/builtinPrompt.py
+++ b/lazyllm/components/prompter/builtinPrompt.py
@@ -65,16 +65,16 @@ def _set_model_configs(self, system: str = None, sos: Union[None, str] = None, s
'tool_end_token', 'tool_args_token']:
if local[name] is not None: setattr(self, f'_{name}', local[name])
- def _get_tools(self, tools, *, return_dict):
- return tools if return_dict else '### Function-call Tools. \n\n' +\
+ def _get_tools(self, tools, *, for_chat_api: bool):
+ return tools if for_chat_api else '### Function-call Tools. \n\n' +\
f'{json.dumps(tools, ensure_ascii=False)}\n\n' if tools else ''
def _get_tools_name(self, tools):
return json.dumps([t['function']['name'] for t in tools], ensure_ascii=False) if tools else ''
- def _get_histories(self, history, *, return_dict): # noqa: C901
+ def _get_histories(self, history, *, for_chat_api: bool): # noqa: C901
if not self._history and not history: return ''
- if return_dict:
+ if for_chat_api:
content = []
for item in self._history + (history or []):
if isinstance(item, list):
@@ -119,9 +119,9 @@ def _get_histories(self, history, *, return_dict): # noqa: C901
else:
raise NotImplementedError('Cannot transform json history to {type(history[0])} now')
- def _get_instruction_and_input(self, input, *, return_dict=False, tools=None):
+ def _get_instruction_and_input(self, input, *, for_chat_api: bool = False, tools=None):
instruction = self._instruction_template
- fc_prompt = '' if return_dict or not tools else FC_PROMPT
+ fc_prompt = '' if for_chat_api or not tools else FC_PROMPT
if fc_prompt and FC_PROMPT_PLACEHOLDER not in instruction:
instruction = f'{instruction}\n\n{fc_prompt}'
instruction = instruction.replace(FC_PROMPT_PLACEHOLDER, fc_prompt)
@@ -183,6 +183,24 @@ def _generate_prompt_dict_impl(self, instruction, input, user, history, tools, l
'content': self._system + '\n' + instruction if instruction else self._system})
return dict(messages=history, tools=tools) if tools else dict(messages=history)
+ # Used for OnlineChatModule with Anthropic-format API
+ def _generate_prompt_anthropic_impl(self, instruction, input, user, history, tools, label):
+ result = self._generate_prompt_dict_impl(instruction, input, user, history, tools, label)
+ messages = result.get('messages', [])
+ system_text = None
+ non_system = []
+ for msg in messages:
+ if msg.get('role') == 'system':
+ system_text = msg['content']
+ else:
+ non_system.append(msg)
+ out = dict(messages=non_system)
+ if system_text is not None:
+ out['system'] = system_text
+ if tools:
+ out['tools'] = result['tools']
+ return out
+
def pre_hook(self, func: Optional[Callable] = None):
self._pre_hook = func
return self
@@ -203,17 +221,27 @@ def generate_prompt(self, input: Union[str, List, Dict[str, str], None] = None,
history: List[Union[List[str], Dict[str, Any]]] = None,
tools: Union[List[Dict[str, Any]], None] = None,
label: Union[str, None] = None,
- *, show: bool = False, return_dict: bool = False) -> Union[str, Dict]:
+ *, show: bool = False, return_dict: bool = False,
+ format: Optional[str] = None) -> Union[str, Dict]:
+ if return_dict and format is None:
+ LOG.log_once('return_dict is deprecated, use format="openai" instead.', level='warning')
+ format = 'openai'
input = copy.deepcopy(input)
if self._pre_hook:
input, history, tools, label = self._pre_hook(input, history, tools, label)
tools = tools or self._tools
- instruction, input = self._get_instruction_and_input(input, return_dict=return_dict, tools=tools)
- history = self._get_histories(history, return_dict=return_dict)
- tools = self._get_tools(tools, return_dict=return_dict)
+ for_chat_api = bool(format)
+ instruction, input = self._get_instruction_and_input(input, for_chat_api=for_chat_api, tools=tools)
+ history = self._get_histories(history, for_chat_api=for_chat_api)
+ tools = self._get_tools(tools, for_chat_api=for_chat_api)
self._check_values(instruction, input, history, tools)
instruction, user_instruction = self._split_instruction(instruction)
- func = self._generate_prompt_dict_impl if return_dict else self._generate_prompt_impl
+ if format == 'anthropic':
+ func = self._generate_prompt_anthropic_impl
+ elif format == 'openai':
+ func = self._generate_prompt_dict_impl
+ else:
+ func = self._generate_prompt_impl
result = func(instruction, input, user_instruction, history, tools, label)
if self._show or show: LOG.info(result)
return result
@@ -225,8 +253,12 @@ def get_response(self, output: str, input: Union[str, None] = None) -> str:
class EmptyPrompter(LazyLLMPrompterBase):
- def generate_prompt(self, input, history=None, tools=None, label=None, show=False, return_dict=False):
- if return_dict:
+ def generate_prompt(self, input, history=None, tools=None, label=None, *, show=False,
+ return_dict: bool = False, format=None):
+ if return_dict and format is None:
+ LOG.log_once('return_dict is deprecated, use format="openai" instead.', level='warning')
+ format = 'openai'
+ if format:
return {'messages': [{'role': 'user', 'content': input}]}
if self._show or show: LOG.info(input)
return input
diff --git a/lazyllm/docs/components.py b/lazyllm/docs/components.py
index fa27fa667..c7d25ab62 100644
--- a/lazyllm/docs/components.py
+++ b/lazyllm/docs/components.py
@@ -3022,7 +3022,7 @@ def extract_result(output, inputs):
>>> p = MyPrompter('ins {instruction}')
>>> p.generate_prompt('hello')
'You are an AI-Agent developed by LazyLLM.\\\\nins hello\\\\n\\\\n\\\\n\\\\n, ## Response::'
->>> p.generate_prompt('hello world', return_dict=True)
+>>> p.generate_prompt('hello world', format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\\\nins hello world\\\\n\\\\n'}, {'role': 'user', 'content': ''}]}
''')
@@ -3055,7 +3055,8 @@ def extract_result(output, inputs):
tools (Option[List[Dict]]: 可以使用的工具合集，大模型用作FunctionCall时使用，默认为None
label (Option[str]): 标签，训练或微调时使用，默认为None
show (bool): 标志是否打印生成的Prompt，默认为False
- return_dict (bool): 标志是否返回dict，一般情况下使用 ``OnlineChatModule`` 时会设置为True。如果返回dict，则仅填充 ``instruction``。默认为False
+ return_dict (bool): 已弃用，请改用 ``format="openai"``。当 ``format`` 为 ``None`` 且该参数为 True 时，行为与 ``format="openai"`` 相同，并会记录一次弃用告警。默认为 False。
+ format (Option[str]): 输出结构。``None`` 表示返回用于本地/微调的拼接字符串；``"openai"`` 表示返回 OpenAI Chat Completions 风格的 ``messages``（及可选 ``tools``）；``"anthropic"`` 表示 Anthropic Messages API 所需的 ``system``/``messages`` 结构。使用 ``OnlineChatModule`` 时由模块传入相应格式。若同时传入 ``return_dict`` 与本参数，以本参数为准。默认为 ``None``。
''')
add_english_doc('prompter.PrompterBase.generate_prompt', '''\
@@ -3068,7 +3069,8 @@ def extract_result(output, inputs):
tools (Option[List[Dict]]): A collection of tools that can be used, used when the large model performs FunctionCall, defaults to None.
label (Option[str]): Label, used during fine-tuning or training, defaults to None.
show (bool): Flag indicating whether to print the generated Prompt, defaults to False.
- return_dict (bool): Flag indicating whether to return a dict, generally set to True when using ``OnlineChatModule``. If returning a dict, only the ``instruction`` will be filled. Defaults to False.
+ return_dict (bool): Deprecated; prefer ``format="openai"``. When ``format`` is ``None`` and this is True, behaves like ``format="openai"`` and emits a one-time deprecation warning. Defaults to False.
+ format (Option[str]): Output structure. ``None`` returns a concatenated string for local/finetuning use; ``"openai"`` returns OpenAI-style ``messages`` (and optional ``tools``); ``"anthropic"`` returns Anthropic-style ``system``/``messages``. ``OnlineChatModule`` passes the appropriate value. If both ``return_dict`` and ``format`` are provided, ``format`` takes precedence. Defaults to ``None``.
''')
add_chinese_doc('prompter.PrompterBase.get_response', '''\
@@ -3124,6 +3126,8 @@ def extract_result(output, inputs):
tools (Option[List[Dict]]): 工具参数，可忽略，默认None。
label (Option[str]): 标签，可忽略，默认None。
show (bool): 是否打印返回内容，默认为False。
+ return_dict (bool): 已弃用，请改用 ``format="openai"``。当 ``format`` 为 ``None`` 且该参数为 True 时，与 ``format="openai"`` 行为一致并记录一次弃用告警。
+ format (Option[str]): 若为非空（例如 ``"openai"``），则返回 ``{"messages": [{"role": "user", "content": input}]}``；否则直接返回 ``input``。若同时传入 ``return_dict`` 与本参数，以本参数为准。
''')
add_english_doc('prompter.EmptyPrompter.generate_prompt', '''\
@@ -3137,6 +3141,8 @@ def extract_result(output, inputs):
tools (Option[List[Dict]]): Tool definitions, ignored. Defaults to None.
label (Option[str]): Label, ignored. Defaults to None.
show (bool): Whether to print the returned prompt. Defaults to False.
+ return_dict (bool): Deprecated; prefer ``format="openai"``. When ``format`` is ``None`` and this is True, behaves like ``format="openai"`` with a one-time deprecation warning.
+ format (Option[str]): If set (e.g. ``"openai"``), returns ``{"messages": [{"role": "user", "content": input}]}``; otherwise returns ``input`` unchanged. If both ``return_dict`` and ``format`` are provided, ``format`` takes precedence.
''')
add_english_doc('prompter.builtinPrompt.LazyLLMPrompterBase', '''\
@@ -3209,19 +3215,19 @@ def extract_result(output, inputs):
>>> p = AlpacaPrompter('hello world {instruction}')
>>> p.generate_prompt('this is my input')
'You are an AI-Agent developed by LazyLLM.\\\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\\\n\\\\n ### Instruction:\\\\nhello world this is my input\\\\n\\\\n\\\\n### Response:\\\\n'
->>> p.generate_prompt('this is my input', return_dict=True)
+>>> p.generate_prompt('this is my input', format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\\\n\\\\n ### Instruction:\\\\nhello world this is my input\\\\n\\\\n'}, {'role': 'user', 'content': ''}]}
>>>
>>> p = AlpacaPrompter('hello world {instruction}, {input}', extra_keys=['knowledge'])
>>> p.generate_prompt(dict(instruction='hello world', input='my input', knowledge='lazyllm'))
'You are an AI-Agent developed by LazyLLM.\\\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\\\n\\\\n ### Instruction:\\\\nhello world hello world, my input\\\\n\\\\nHere are some extra messages you can referred to:\\\\n\\\\n### knowledge:\\\\nlazyllm\\\\n\\\\n\\\\n### Response:\\\\n'
->>> p.generate_prompt(dict(instruction='hello world', input='my input', knowledge='lazyllm'), return_dict=True)
+>>> p.generate_prompt(dict(instruction='hello world', input='my input', knowledge='lazyllm'), format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\\\n\\\\n ### Instruction:\\\\nhello world hello world, my input\\\\n\\\\nHere are some extra messages you can referred to:\\\\n\\\\n### knowledge:\\\\nlazyllm\\\\n\\\\n'}, {'role': 'user', 'content': ''}]}
>>>
>>> p = AlpacaPrompter(dict(system="hello world", user="this is user instruction {input}"))
>>> p.generate_prompt(dict(input="my input"))
'You are an AI-Agent developed by LazyLLM.\\\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\\\n\\\\n ### Instruction:\\\\nhello word\\\\n\\\\n\\\\n\\\\nthis is user instruction my input### Response:\\\\n'
->>> p.generate_prompt(dict(input="my input"), return_dict=True)
+>>> p.generate_prompt(dict(input="my input"), format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\\\n\\\\n ### Instruction:\\\\nhello world'}, {'role': 'user', 'content': 'this is user instruction my input'}]}
''')
@@ -3259,7 +3265,7 @@ def extract_result(output, inputs):
>>> p.generate_prompt('this is my input')
'You are an AI-Agent developed by LazyLLM.hello world\\\\nthis is my input\\\\n'
->>> p.generate_prompt('this is my input', return_dict=True)
+>>> p.generate_prompt('this is my input', format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\\\nhello world'}, {'role': 'user', 'content': 'this is my input'}]}
- Using extra_keys
@@ -3284,7 +3290,7 @@ def extract_result(output, inputs):
>>> p.generate_prompt({'input': "my input", 'query': "this is user query"})
'You are an AI-Agent developed by LazyLLM.hello world\\\\nthis is user instruction my input this is user query\\\\n'
->>> p.generate_prompt({'input': "my input", 'query': "this is user query"}, return_dict=True)
+>>> p.generate_prompt({'input': "my input", 'query': "this is user query"}, format='openai')
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\\\nhello world'}, {'role': 'user', 'content': 'this is user instruction my input this is user query'}]}
''')
diff --git a/lazyllm/docs/module.py b/lazyllm/docs/module.py
index a49099277..6dd654d80 100644
--- a/lazyllm/docs/module.py
+++ b/lazyllm/docs/module.py
@@ -1633,6 +1633,64 @@
>>> print(response)
''')
+add_chinese_doc('llms.onlinemodule.supplier.claude.ClaudeChat', '''\
+Claude 在线聊天模块，继承自 OnlineChatModuleBase。
+封装了对 Anthropic Claude Messages API（/v1/messages）的调用，用于进行多轮问答交互。
+默认使用模型 `claude-opus-4-5`，支持流式输出、工具调用和调用链追踪。
+
+与 OpenAI 兼容格式不同，该模块使用 Anthropic 原生 API 格式：请求头使用 `x-api-key` 和
+`anthropic-version`，`system` 消息作为顶层字段传递，工具定义使用 `input_schema` 字段。
+传入的 OpenAI 格式工具定义会自动转换为 Anthropic 格式。
+
+Args:
+ model (str): 使用的模型名称，默认为 `claude-opus-4-5`。
+ base_url (str): API 基础 URL，默认为 `https://api.anthropic.com/v1/`。
+ api_key (Optional[str]): Anthropic API Key，若未提供，则从 `lazyllm.config['claude_api_key']` 读取。
+ stream (bool): 是否启用流式输出，默认为 True。
+ return_trace (bool): 是否返回调用链追踪信息，默认为 False。
+ **kwargs: 其他传递给基类 OnlineChatModuleBase 的参数。
+''')
+
+add_english_doc('llms.onlinemodule.supplier.claude.ClaudeChat', '''\
+Claude online chat module, inheriting from OnlineChatModuleBase.
+Wraps the Anthropic Claude Messages API (/v1/messages) for multi-turn Q&A interactions.
+Defaults to model `claude-opus-4-5`, supporting streaming, tool calls, and optional trace return.
+
+Unlike OpenAI-compatible format, this module uses the native Anthropic API format: the request
+header uses `x-api-key` and `anthropic-version`, the `system` message is passed as a top-level
+field, and tools use the `input_schema` field. OpenAI-format tool definitions are automatically
+converted to Anthropic format.
+
+Args:
+ model (str): The model name to use. Defaults to `claude-opus-4-5`.
+ base_url (str): Base URL of the API. Defaults to `https://api.anthropic.com/v1/`.
+ api_key (Optional[str]): Anthropic API key. If not provided, it is read from `lazyllm.config['claude_api_key']`.
+ stream (bool): Whether to enable streaming output. Defaults to True.
+ return_trace (bool): Whether to return trace information. Defaults to False.
+ **kwargs: Additional arguments passed to the base class OnlineChatModuleBase.
+''')
+
+add_example('llms.onlinemodule.supplier.claude.ClaudeChat', '''\
+>>> import lazyllm
+>>> # Set environment variable: export LAZYLLM_CLAUDE_API_KEY=your_api_key
+>>> chat = lazyllm.OnlineChatModule(source='claude', model='claude-opus-4-5')
+>>> response = chat('Hello, who are you?')
+>>> print(response)
+
+>>> # Tool call example (requires FunctionCallFormatter to preserve tool_calls in output)
+>>> from lazyllm.components.formatter import FunctionCallFormatter
+>>> tools = [{'type': 'function', 'function': {
+... 'name': 'get_weather',
+... 'description': 'Get the current weather for a city',
+... 'parameters': {'type': 'object', 'properties': {'city': {'type': 'string'}}, 'required': ['city']}
+... }}]
+>>> chat = lazyllm.OnlineChatModule(source='claude', model='claude-opus-4-5', stream=False)
+>>> chat._formatter = FunctionCallFormatter()
+>>> result = chat('Please call get_weather for Beijing', tools=tools)
+>>> print(result)
+''')
+
+
add_chinese_doc('llms.onlinemodule.supplier.openai.OpenAIEmbed', '''\
OpenAI 在线嵌入模块。
该类封装了对 OpenAI 嵌入 API 的调用，默认使用模型 `text-embedding-ada-002`，用于将文本编码为向量表示。
diff --git a/lazyllm/module/llms/onlinemodule/base/onlineChatModuleBase.py b/lazyllm/module/llms/onlinemodule/base/onlineChatModuleBase.py
index 26254242b..ba645268d 100644
--- a/lazyllm/module/llms/onlinemodule/base/onlineChatModuleBase.py
+++ b/lazyllm/module/llms/onlinemodule/base/onlineChatModuleBase.py
@@ -21,6 +21,7 @@ class LazyLLMOnlineChatModuleBase(LazyLLMOnlineBase, LLMBase):
VLM_MODEL_PREFIX = []
NO_PROXY = True
__lazyllm_registry_key__ = LLMType.CHAT
+ _message_format = 'openai'
def __init__(self, api_key: Union[str, List[str]], base_url: str, model_name: str,
stream: Union[bool, Dict[str, str]], return_trace: bool = False, skip_auth: bool = False,
@@ -70,6 +71,9 @@ def _get_models_list(self):
def _convert_msg_format(self, msg: Dict[str, Any]):
return msg
+ def _prepare_request_data(self, data: Dict[str, Any]) -> Dict[str, Any]:
+ return data
+
def _str_to_json(self, msg: str, stream_output: bool):
if isinstance(msg, bytes):
pattern = re.compile(r'^data:\s*')
@@ -131,7 +135,7 @@ def forward(self, __input: Union[Dict, str] = None, *, llm_chat_history: List[Li
runtime_url = self._get_chat_url(url) if url else self._chat_url
runtime_model = model or self._model_name
- params = {'input': __input, 'history': llm_chat_history, 'return_dict': True}
+ params = {'input': __input, 'history': llm_chat_history, 'format': self._message_format}
if tools: params['tools'] = tools
data = self._prompt.generate_prompt(**params)
data.update(self._static_params, **dict(model=runtime_model, stream=bool(stream_output)))
@@ -146,6 +150,7 @@ def forward(self, __input: Union[Dict, str] = None, *, llm_chat_history: List[Li
if msg.get('role') == 'user' and isinstance(msg.get('content'), str):
msg['content'] = self._format_vl_chat_query(msg['content'])
+ data = self._prepare_request_data(data)
proxies = {'http': None, 'https': None} if self.NO_PROXY else None
with requests.post(runtime_url, json=data, headers=self._header, stream=stream_output,
proxies=proxies) as r:
diff --git a/lazyllm/module/llms/onlinemodule/supplier/claude.py b/lazyllm/module/llms/onlinemodule/supplier/claude.py
new file mode 100644
index 000000000..fd8902ba0
--- /dev/null
+++ b/lazyllm/module/llms/onlinemodule/supplier/claude.py
@@ -0,0 +1,135 @@
+import json
+import re
+import requests
+from typing import Any, Dict, List, Optional, Union
+from urllib.parse import urljoin
+from ..base import OnlineChatModuleBase
+
+
+class ClaudeChat(OnlineChatModuleBase):
+ # Anthropic native Messages API (/v1/messages).
+ # Differs from OpenAI: x-api-key header, system as top-level field,
+ # max_tokens required, and SSE event types for streaming.
+
+ _ANTHROPIC_VERSION = '2023-06-01'
+ _DEFAULT_MAX_TOKENS = 4096
+ _message_format = 'anthropic'
+
+ def __init__(self, base_url: Optional[str] = None, model: Optional[str] = None,
+ api_key: str = None, stream: bool = True, return_trace: bool = False, **kwargs):
+ base_url = base_url or 'https://api.anthropic.com/v1/'
+ model = model or 'claude-3-5-sonnet-20241022'
+ super().__init__(api_key=api_key or self._default_api_key(),
+ base_url=base_url, model_name=model, stream=stream,
+ return_trace=return_trace, **kwargs)
+
+ def _get_system_prompt(self):
+ return 'You are Claude, an AI assistant made by Anthropic. You are helpful, harmless, and honest.'
+
+ def _get_chat_url(self, url):
+ if url.rstrip('/').endswith('v1/messages'):
+ return url
+ base = url.rstrip('/')
+ if base.endswith('/v1'):
+ return base + '/messages'
+ return urljoin(url if url.endswith('/') else url + '/', 'v1/messages')
+
+ @staticmethod
+ def _get_header(api_key: str) -> dict:
+ header = {'Content-Type': 'application/json',
+ 'anthropic-version': ClaudeChat._ANTHROPIC_VERSION}
+ if api_key:
+ header['x-api-key'] = api_key
+ return header
+
+ @staticmethod
+ def _convert_tools(tools: List[Dict]) -> List[Dict]:
+ # Convert OpenAI-format tools to Anthropic format.
+ # OpenAI: [{"type": "function", "function": {"name": ..., "description": ..., "parameters": {...}}}]
+ # Anthropic: [{"name": ..., "description": ..., "input_schema": {...}}]
+ result = []
+ for tool in tools:
+ fn = tool.get('function', tool)
+ result.append({
+ 'name': fn['name'],
+ 'description': fn.get('description', ''),
+ 'input_schema': fn.get('parameters', fn.get('input_schema', {'type': 'object', 'properties': {}})),
+ })
+ return result
+
+ def _prepare_request_data(self, data: Dict[str, Any]) -> Dict[str, Any]:
+ data = dict(data)
+ data.setdefault('max_tokens', self._DEFAULT_MAX_TOKENS)
+ if data.get('tools'):
+ data['tools'] = self._convert_tools(data['tools'])
+ return data
+
+ def _convert_msg_format(self, msg: Dict[str, Any]):
+ msg_type = msg.get('type', '')
+ if msg_type == 'message': # non-stream response
+ text = ''.join(b.get('text', '') for b in msg.get('content', []) if b.get('type') == 'text')
+ tool_calls = [
+ {'id': b['id'], 'type': 'function',
+ 'function': {'name': b['name'], 'arguments': json.dumps(b.get('input', {}), ensure_ascii=False)}}
+ for b in msg.get('content', []) if b.get('type') == 'tool_use'
+ ]
+ message: Dict[str, Any] = {'role': 'assistant', 'content': text}
+ if tool_calls:
+ message['tool_calls'] = tool_calls
+ usage = msg.get('usage', {})
+ return {'choices': [{'message': message}],
+ 'usage': {'prompt_tokens': usage.get('input_tokens', -1),
+ 'completion_tokens': usage.get('output_tokens', -1)}}
+ if msg_type == 'content_block_delta':
+ delta_obj = msg.get('delta', {})
+ if delta_obj.get('type') == 'text_delta':
+ return {'choices': [{'delta': {'role': 'assistant', 'content': delta_obj.get('text', '')}}]}
+ if delta_obj.get('type') == 'input_json_delta':
+ # Partial tool input — carry as tool_calls delta with index
+ partial = delta_obj.get('partial_json', '')
+ return {'choices': [{'index': msg.get('index', 0),
+ 'delta': {'tool_calls': [{'function': {'arguments': partial}}]}}]}
+ if msg_type == 'content_block_start':
+ block = msg.get('content_block', {})
+ if block.get('type') == 'tool_use':
+ # Emit the tool call header (id + name) as the first delta
+ return {'choices': [{'index': msg.get('index', 0),
+ 'delta': {'role': 'assistant', 'content': '',
+ 'tool_calls': [{'index': msg.get('index', 0),
+ 'id': block['id'], 'type': 'function',
+ 'function': {'name': block['name'], 'arguments': ''}}]}}]}
+ if msg_type == 'message_start':
+ usage = msg.get('message', {}).get('usage', {})
+ return {'choices': [{'delta': {'role': 'assistant', 'content': ''}}],
+ 'usage': {'prompt_tokens': usage.get('input_tokens', -1), 'completion_tokens': -1}}
+ if msg_type == 'message_delta':
+ usage = msg.get('usage', {})
+ return {'choices': [{'delta': {'role': 'assistant', 'content': ''}}],
+ 'usage': {'prompt_tokens': -1, 'completion_tokens': usage.get('output_tokens', -1)}}
+ return '' # ping / content_block_stop / message_stop → filtered out
+
+ def _str_to_json(self, msg: Union[str, bytes], stream_output: bool):
+ if isinstance(msg, bytes):
+ msg = re.sub(r'^data:\s*', '', msg.decode('utf-8'))
+ try:
+ message = self._convert_msg_format(json.loads(msg))
+ if not stream_output:
+ return message
+ color = stream_output.get('color') if isinstance(stream_output, dict) else None
+ for item in (message.get('choices', []) if isinstance(message, dict) else []):
+ delta = item.get('delta', {})
+ if (content := delta.get('content', '')) and not delta.get('tool_calls'):
+ self._stream_output(content, color)
+ return message
+ except Exception:
+ return ''
+
+ def _validate_api_key(self):
+ # Anthropic has no /v1/models endpoint; send a minimal request to verify the key.
+ try:
+ data = {'model': self._model_name, 'max_tokens': 1,
+ 'messages': [{'role': 'user', 'content': 'hi'}]}
+ response = requests.post(self._chat_url, json=data, headers=self._header, timeout=10)
+ return response.status_code == 200
+ except Exception:
+ return False
diff --git a/tests/basic_tests/Components/test_component.py b/tests/basic_tests/Components/test_component.py
index c6a5262fd..5bd5b1812 100644
--- a/tests/basic_tests/Components/test_component.py
+++ b/tests/basic_tests/Components/test_component.py
@@ -5,37 +5,36 @@
class TestPrompter(object):
def test_prompter(self):
p = lazyllm.Prompter(prompt='hello world2 <{input}>')
- assert not p._is_empty(), "Prompter should not be empty"
+ assert not p._is_empty(), 'Prompter should not be empty'
def test_generate_prompt(self):
p = lazyllm.Prompter(prompt='hello world2 <{input}>')
result = p.generate_prompt('123')
- assert result == 'hello world2 <123>', f"Expected 'hello world2 <123>', but got '{result}'"
+ assert result == 'hello world2 <123>', 'unexpected: ' + repr(result)
def test_generate_prompt_dict_input(self):
p = lazyllm.Prompter(prompt='hello world2 <{input}>')
result_dict_input = p.generate_prompt({'input': '123'})
- assert result_dict_input == 'hello world2 <123>', \
- f"Expected 'hello world2 <123>', but got '{result_dict_input}'"
+ assert result_dict_input == 'hello world2 <123>', 'unexpected: ' + repr(result_dict_input)
def test_from_template(self):
p = lazyllm.Prompter.from_template('alpaca')
expected_prompt = (
- "Below is an instruction that describes a task, paired with an input that provides further context. "
- "Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### "
- "Input:\n{input}\n\n### Response:\n"
+ 'Below is an instruction that describes a task, paired with an input that provides further context. '
+ 'Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### '
+ 'Input:\n{input}\n\n### Response:\n'
)
- assert p._prompt == expected_prompt, f"Expected prompt to be '{expected_prompt}', but got '{p._prompt}'"
+ assert p._prompt == expected_prompt, 'unexpected prompt: ' + repr(p._prompt)
def test_generate_prompt_with_template(self):
p = lazyllm.Prompter.from_template('alpaca')
result = p.generate_prompt(dict(instruction='ins', input='inp'))
expected_result = (
- "Below is an instruction that describes a task, paired with an input that provides further context. "
- "Write a response that appropriately completes the request.\n\n### Instruction:\nins\n\n### "
- "Input:\ninp\n\n### Response:\n"
+ 'Below is an instruction that describes a task, paired with an input that provides further context. '
+ 'Write a response that appropriately completes the request.\n\n### Instruction:\nins\n\n### '
+ 'Input:\ninp\n\n### Response:\n'
)
- assert result == expected_result, f"Expected '{expected_result}', but got '{result}'"
+ assert result == expected_result, 'unexpected: ' + repr(result)
class TestAlpacaPrompter(object):
@@ -43,15 +42,16 @@ def test_basic_prompter(self):
p = lazyllm.AlpacaPrompter('请完成加法运算, 输入为{instruction}')
r = p.generate_prompt('a+b')
assert r == 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n### Instruction:\n请完成加法运算, 输入为a+b\n\n\n\n### Response:\n' # noqa E501
- r = p.generate_prompt('a+b', return_dict=True)
+ r = p.generate_prompt('a+b', format='openai')
assert r == {'messages': [
{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n### Instruction:\n请完成加法运算, 输入为a+b\n\n'}, # noqa E501
{'role': 'user', 'content': ''}]}
+ assert p.generate_prompt('a+b', return_dict=True) == r
p = lazyllm.AlpacaPrompter('请完成加法运算', extra_keys='input')
r = p.generate_prompt('a+b')
assert r == 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n### Instruction:\n请完成加法运算\n\nHere are some extra messages you can referred to:\n\n### input:\na+b\n\n\n\n### Response:\n' # noqa E501
- r = p.generate_prompt('a+b', return_dict=True)
+ r = p.generate_prompt('a+b', format='openai')
assert r == {'messages': [
{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n### Instruction:\n请完成加法运算\n\nHere are some extra messages you can referred to:\n\n### input:\na+b\n\n'}, # noqa E501
{'role': 'user', 'content': ''}]}
@@ -59,7 +59,7 @@ def test_basic_prompter(self):
p = lazyllm.AlpacaPrompter(dict(system='请完成加法运算', user='输入为{instruction}'))
r = p.generate_prompt('a+b')
assert r == 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n### Instruction:\n请完成加法运算\n\n输入为a+b### Response:\n' # noqa E501
- r = p.generate_prompt('a+b', return_dict=True)
+ r = p.generate_prompt('a+b', format='openai')
assert r == {'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n### Instruction:\n请完成加法运算'}, {'role': 'user', 'content': '输入为a+b'}]} # noqa E501
@@ -68,7 +68,7 @@ def test_basic_prompter(self):
p = lazyllm.ChatPrompter('请完成加法运算, 输入为{instruction}')
r = p.generate_prompt('a+b')
assert r == 'You are an AI-Agent developed by LazyLLM.请完成加法运算, 输入为a+b\n\n\n\n\n\n\n\n'
- r = p.generate_prompt('a+b', return_dict=True)
+ r = p.generate_prompt('a+b', format='openai')
assert r == {'messages': [
{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\n请完成加法运算, 输入为a+b\n\n'},
{'role': 'user', 'content': ''}]}
@@ -76,7 +76,7 @@ def test_basic_prompter(self):
p = lazyllm.ChatPrompter('请完成加法运算', extra_keys='input')
r = p.generate_prompt('a+b')
assert r == 'You are an AI-Agent developed by LazyLLM.请完成加法运算\nHere are some extra messages you can referred to:\n\n### input:\na+b\n\n\n\n\n\n\n\n\n' # noqa E501
- r = p.generate_prompt('a+b', return_dict=True)
+ r = p.generate_prompt('a+b', format='openai')
assert r == {'messages': [
{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\n请完成加法运算\nHere are some extra messages you can referred to:\n\n### input:\na+b\n\n\n'}, # noqa E501
{'role': 'user', 'content': ''}]}
@@ -84,7 +84,7 @@ def test_basic_prompter(self):
p = lazyllm.ChatPrompter(dict(system='请完成加法运算', user='输入为{instruction}'))
r = p.generate_prompt('a+b')
assert r == 'You are an AI-Agent developed by LazyLLM.请完成加法运算\n\n\n\n输入为a+b\n\n'
- r = p.generate_prompt('a+b', return_dict=True)
+ r = p.generate_prompt('a+b', format='openai')
assert r == {'messages': [
{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\n请完成加法运算'},
{'role': 'user', 'content': '输入为a+b'}]}
@@ -94,7 +94,7 @@ def test_history(self):
history=[['输入为a+b', 'a+b'], ['输入为c+d', 'c+d']])
r = p.generate_prompt('e+f')
assert r == 'You are an AI-Agent developed by LazyLLM.请完成加法运算\n\n输入为a+ba+b输入为c+dc+d\n\n输入为e+f\n\n'
- r = p.generate_prompt('e+f', return_dict=True)
+ r = p.generate_prompt('e+f', format='openai')
assert r == {'messages': [
{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\n请完成加法运算'},
{'role': 'user', 'content': '输入为a+b'},
@@ -107,7 +107,7 @@ def test_history(self):
r = p.generate_prompt('e+f', history=[['输入为a+b', 'a+b'], ['输入为c+d', 'c+d']])
assert r == 'You are an AI-Agent developed by LazyLLM.请完成加法运算\n\n输入为a+ba+b输入为c+dc+d\n\n输入为e+f\n\n'
- r = p.generate_prompt('e+f', history=[['输入为a+b', 'a+b'], ['输入为c+d', 'c+d']], return_dict=True)
+ r = p.generate_prompt('e+f', history=[['输入为a+b', 'a+b'], ['输入为c+d', 'c+d']], format='openai')
assert r == {'messages': [
{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\n请完成加法运算'},
{'role': 'user', 'content': '输入为a+b'},
@@ -117,12 +117,12 @@ def test_history(self):
{'role': 'user', 'content': '输入为e+f'}]}
p = lazyllm.ChatPrompter(dict(system='请完成加法运算', user='输入为{instruction}'), history=[['输入为a+b', 'a+b']])
- r = p.generate_prompt('e+f', history=[{"role": "user", "content": '输入为c+d'},
- {"role": "assistant", "content": 'c+d'}])
+ r = p.generate_prompt('e+f', history=[{'role': 'user', 'content': '输入为c+d'},
+ {'role': 'assistant', 'content': 'c+d'}])
assert r == 'You are an AI-Agent developed by LazyLLM.请完成加法运算\n\n输入为a+ba+b输入为c+dc+d\n\n输入为e+f\n\n'
- r = p.generate_prompt('e+f', history=[{"role": "user", "content": '输入为c+d'},
- {"role": "assistant", "content": 'c+d'}], return_dict=True)
+ r = p.generate_prompt('e+f', history=[{'role': 'user', 'content': '输入为c+d'},
+ {'role': 'assistant', 'content': 'c+d'}], format='openai')
assert r == {'messages': [
{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\n请完成加法运算'},
{'role': 'user', 'content': '输入为a+b'},
@@ -134,11 +134,11 @@ def test_history(self):
def test_empty_prompt_with_history(self):
p = lazyllm.ChatPrompter('', history=[['输入为a+b', 'a+b']])
r11 = p.generate_prompt('c+d')
- r12 = p.generate_prompt('c+d', return_dict=True)
+ r12 = p.generate_prompt('c+d', format='openai')
p = lazyllm.ChatPrompter(None, history=[['输入为a+b', 'a+b']])
r21 = p.generate_prompt('c+d')
- r22 = p.generate_prompt('c+d', return_dict=True)
+ r22 = p.generate_prompt('c+d', format='openai')
assert r11 == r21 == 'You are an AI-Agent developed by LazyLLM.\n\n输入为a+ba+b\n\nc+d\n\n'
assert r12 == r22 == {'messages': [
@@ -152,18 +152,18 @@ def test_configs(self):
p._set_model_configs(sos='', eos='')
r = p.generate_prompt('a+b')
assert r == 'You are an AI-Agent developed by LazyLLM.请完成加法运算\n\n\n\n输入为a+b\n\n'
- r = p.generate_prompt('a+b', return_dict=True)
+ r = p.generate_prompt('a+b', format='openai')
assert r == {'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\n请完成加法运算'}, {'role': 'user', 'content': '输入为a+b'}]} # noqa E501
def test_config_with_history(self):
p = lazyllm.ChatPrompter(dict(system='请完成加法运算', user='输入为{instruction}'), history=[['输入为a+b', 'a+b']])
p._set_model_configs(sos='', eos='', soh='', eoh='', soa='', eoa='')
- r = p.generate_prompt('e+f', history=[{"role": "user", "content": '输入为c+d'},
- {"role": "assistant", "content": 'c+d'}])
+ r = p.generate_prompt('e+f', history=[{'role': 'user', 'content': '输入为c+d'},
+ {'role': 'assistant', 'content': 'c+d'}])
assert r == 'You are an AI-Agent developed by LazyLLM.请完成加法运算\n\n输入为a+ba+b输入为c+dc+d\n\n输入为e+f\n\n' # noqa E501
- r = p.generate_prompt('e+f', history=[{"role": "user", "content": '输入为c+d'},
- {"role": "assistant", "content": 'c+d'}], return_dict=True)
+ r = p.generate_prompt('e+f', history=[{'role': 'user', 'content': '输入为c+d'},
+ {'role': 'assistant', 'content': 'c+d'}], format='openai')
assert r == {'messages': [
{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\n请完成加法运算'},
{'role': 'user', 'content': '输入为a+b'},
@@ -175,16 +175,16 @@ def test_config_with_history(self):
class TestLLMType(object):
def test_llm_type(self):
- assert LLMType("llm") == LLMType.LLM
- assert LLMType("LLM") == LLMType.LLM
+ assert LLMType('llm') == LLMType.LLM
+ assert LLMType('LLM') == LLMType.LLM
assert LLMType(LLMType.LLM) == LLMType.LLM
- assert LLMType.LLM == "llm"
- assert "llm" == LLMType.LLM
- assert LLMType.LLM == "LLM"
- assert "LLM" == LLMType.LLM
+ assert LLMType.LLM == 'llm'
+ assert 'llm' == LLMType.LLM
+ assert LLMType.LLM == 'LLM'
+ assert 'LLM' == LLMType.LLM
assert LLMType.LLM in ('LLM',)
assert LLMType.LLM in (LLMType.LLM,)
assert LLMType.LLM in ('llm',)
assert 'llm' in (LLMType.LLM,)
- assert LLMType("CROSS_modal_embed") == LLMType.CROSS_MODAL_EMBED
- assert LLMType("CROSS_MODAL_EMBED") == LLMType.CROSS_MODAL_EMBED
+ assert LLMType('CROSS_modal_embed') == LLMType.CROSS_MODAL_EMBED
+ assert LLMType('CROSS_MODAL_EMBED') == LLMType.CROSS_MODAL_EMBED
diff --git a/tests/charge_tests/Models/test_chat.py b/tests/charge_tests/Models/test_chat.py
index f5fc02373..9c7db3c7c 100644
--- a/tests/charge_tests/Models/test_chat.py
+++ b/tests/charge_tests/Models/test_chat.py
@@ -70,3 +70,8 @@ def test_aiping_chat(self):
@pytest.mark.xfail
def test_ppio_chat(self):
self.common_chat(source='ppio')
+
+ @pytest.mark.ignore_cache_on_change(get_path('claude'))
+ @pytest.mark.xfail
+ def test_claude_chat(self):
+ self.common_chat(source='claude')