[Usage] How to use ollama with local server

[l2002924700]

Checked other resources

• I added a very descriptive title to this issue.
• I am sure the issue hasn't been already addressed by searching through https://github.com/agiresearch/AIOS/issues.
• The usage issue is not resolved by updating to the latest stable version in the main branch.

Your current environment

ubuntu 20.0.4
AIOS: v0.2.0.beta

How would you like to use aios

hi,
I setup the environment follow the readme, and I plan to use the backend ollama.
I run the ollama as the AIOS-LSFS/scripts/run_agent.sh,

• download the llama2:7b gguf;
• execute "ollama serve", the output is as follws:

2025/02/18 13:38:24 routes.go:1186: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-02-18T13:38:24.699+08:00 level=INFO source=images.go:432 msg="total blobs: 2"
time=2025-02-18T13:38:24.699+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-02-18T13:38:24.700+08:00 level=INFO source=routes.go:1237 msg="Listening on 127.0.0.1:11434 (version 0.5.11)"
time=2025-02-18T13:38:24.701+08:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-02-18T13:38:24.848+08:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.535.154.05"
time=2025-02-18T13:38:25.004+08:00 level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered"
time=2025-02-18T13:38:25.005+08:00 level=INFO source=types.go:130 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="31.0 GiB" available="28.8 GiB"

• then, execute run-agent --llm_name llama2:7b --llm_backend ollama --agent_name_or_path Cerebrum/example/agents/demo_agent --task "Tell me what is core idea of AIOS" --aios_kernel_url http://localhost:8000, the outputs are as follows:

🔧 Starting AIOS Demo with configuration:
LLM: llama2:7b (ollama)
Root Directory: root
Memory Limit: 524288000 bytes
Max Workers: 32
Timeout: 300 seconds
Agent: Cerebrum/example/agents/demo_agent
Task: Tell me what is core idea of AIOS
✅ Client initialized successfully
Status: {'llm': 'active', 'storage': 'active', 'memory': 'active', 'tool': 'active', 'scheduler': 'active', 'factory': 'active'}
🚀 Executing agent: demo_agent
📋 Task: Tell me what is core idea of AIOS
⚠️ Agent execution timed out

kernel outputs are as follows:

Saved agent to cache: /home/.cache/cerebrum/93dd529c-9885-4d82-b0ac-a4558e8b32c9/local.agent
Debug - BaseAgent init:
config_: {'name': 'demo_agent', 'description': ['You are an academic research assistant. ', 'Help users find relevant research papers, summarize key findings, and generate potential research questions.'], 'tools': ['demo_author/arxiv'], 'meta': {'author': 'demo_author', 'version': '0.0.1', 'license': 'CC0'}, 'build': {'entry': 'agent.py', 'module': 'DemoAgent'}, 'author': 'demo_author', 'version': '0.0.1', 'license': 'CC0', 'entry': 'agent.py', 'module': 'DemoAgent'}
self.config: {'name': 'demo_agent', 'description': ['You are an academic research assistant. ', 'Help users find relevant research papers, summarize key findings, and generate potential research questions.'], 'tools': ['demo_author/arxiv'], 'meta': {'author': 'demo_author', 'version': '0.0.1', 'license': 'CC0'}, 'build': {'entry': 'agent.py', 'module': 'DemoAgent'}, 'author': 'demo_author', 'version': '0.0.1', 'license': 'CC0', 'entry': 'agent.py', 'module': 'DemoAgent'}
tool name demo_author/arxiv
reload arxiv
local True
'arxiv'
INFO: 127.0.0.1:39934 - "GET /agents/854008/status HTTP/1.1" 200 OK
'arxiv'
INFO: 127.0.0.1:39938 - "GET /agents/854008/status HTTP/1.1" 200 OK

How can i fix the issues? Thank you

[dongyuanjushi]

Could you try a newer version of LLMs supported by ollama, for example, llama3:8b to see whether it still has the problem?

[l2002924700]

Could you try a newer version of LLMs supported by ollama, for example, llama3:8b to see whether it still has the problem?

ok. I will have a try as your suggestion

[l2002924700]

Could you try a newer version of LLMs supported by ollama, for example, llama3:8b to see whether it still has the problem?

When I use llama3:8b, the results is same

[dongyuanjushi]

I see, that could be the connection issue to ollama models in the kernel, let me take a look.

[l2002924700]

I see, that could be the connection issue to ollama models in the kernel, let me take a look.

ok, Thank you for your help

[dongyuanjushi]

The GPU environment at my side looks good. The CPU environment may not support LLMs exceeding 7B as it could cause timeout due to long-context. Try a smaller model, e.g., Qwen-2.5 series (1.5B and 3B) to see how it works.

[l2002924700]

The GPU environment at my side looks good. The CPU environment may not support LLMs exceeding 7B as it could cause timeout due to long-context. Try a smaller model, e.g., Qwen-2.5 series (1.5B and 3B) to see how it works.

Thank you, let me have a try.

[l2002924700]

The GPU environment at my side looks good. The CPU environment may not support LLMs exceeding 7B as it could cause timeout due to long-context. Try a smaller model, e.g., Qwen-2.5 series (1.5B and 3B) to see how it works.

Sorry, I encounter another issue: when I execute the command as ur suggestion:

un-agent --llm_name qwen2.5:3b --llm_backend ollama --agent_name_or_path AIOS/Cerebrum/cerebrum/example/agents/language_tutor --task "How should I use the word help?" --aios_kernel_url "http://localhost:8000" --timeout 3000000

the results have errors as follws:
esult': 'An unexpected error occurred: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n<html>\n<!-- FileName: index.html\n Language: [en]\n-->\n<!--Head-->\n<!--/Head-->\n<!--Body-->\n<!--Contents-->\n<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">\n<html>\n<head>\n<title>Non business website </title>\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8">\n<style type="text/css">\n.auto-style1 {\n\tfont-family: "Clear Gothic";\n\ttext-align: left;\n}\n.auto-style3 {\n\tfont-family: "Clear Gothic";\n\tfont-size: medium;\n}\n.auto-style6 {\n\tfont-family: "Clear Gothic";\n\tfont-size: medium;\n}\n.auto-style7 {\n\tfont-family: "Clear Gothic";\n\tcolor: #FF0000;\n\tfont-size: xx-large;\n\ttext-align: center;\n}\n</style>\n</head>\n\n<body scroll ="no">\n<center>\n<table width="710" border="1" cellpadding="2" cellspacing="0" bordercolor="#A9BCF5">\n<tr>\n<td>\n<table style="width: 700">\n\t<tr>\n\t\t<td class="auto-style7"><strong>Non-business Site Denied</strong></td>\n\t</tr>\n</table>\n<p>\n<table width="700" height="100px" border="0" cellpadding="0" cellspacing="0">\n <tr>\n <td height="50%" style="width: 100px">\n <div align="center">\n <table width="700" border="0" cellspacing="0" cellpadding="0">\n <tr>\n <td>\n <div class="auto-style1"><span class="auto-style6"> <strong>This website restricts connections according to Information Security Policy of SRCX</strong></span><strong><br class="auto-style6"></strong>\n\t\t</strong><span class="auto-style6"> <strong>
how can I fix the issue?
thank you in advanced

[l2002924700]

For the issue above, I add breakpoint, I find that when the porgram execute the" tools = llm_syscall.query.tools" in the function address_syscall which locate aios/llm_core/cores/local/ollama.py file, it return null. Why it return null and how can i fix the issue?
thank you in advanced.

[dongyuanjushi]

let me try to reproduce this issue and see how it happens

[dongyuanjushi]

Check out to v0.2.1 to see whether it can work. If that works, could you submit a PR to change the pointer from 0.2.0.beta to the latest version? Thx

[l2002924700]

hello, cpuld u please tell me how to fix the issue(below code in AIOS/aios/llm_core/cores/local/ollama.py

tools = llm_syscall.query.tools
print(tools)
null

Check out to v0.2.1 to see whether it can work. If that works, could you submit a PR to change the pointer from 0.2.0.beta to the latest version? Thx

[dongyuanjushi]

hello, cpuld u please tell me how to fix the issue(below code in AIOS/aios/llm_core/cores/local/ollama.py

tools = llm_syscall.query.tools
print(tools)
null

Check out to v0.2.1 to see whether it can work. If that works, could you submit a PR to change the pointer from 0.2.0.beta to the latest version? Thx

The language tutor agent does not import extra tools, so the query.tool param is null. Does this block the overall execution of this agent?