Model cache leads to very long startup time

[ckuethe]

Platform

Linux/Ubuntu

Lemonade Version

10.8.1

GPU / APU Model

No response

Component

Other / Not sure

Bug Description

After upgrading to lemonade 10.8.1 the server takes a very long time (10's of minutes) while the cache is built

Steps to Reproduce

1. upgrade to lemonade 10.8.1
2. start lemond
3. observe that the model cache takes a very long time to build

Expected vs Actual Behavior

Expected: lemond should start quickly, the UI should be responsive, ...
Actual: cache build takes a confusingly long time, which also causes lemonade backends and lemonade list to hang

Log Output

Additional Context

No response

[jeremyfowers]

Can you share the debug log and ideally help identify what part of the code is taking a long time for you?

[ckuethe]

yeah, I'll look into it later today. I need to add some additional debug prints to show progress so I can narrow it down.

[ckuethe]

It's currently hanging on auto flm_status = SystemInfoCache::get_flm_status(); which pulls FLM status from /v1/system-info.

And curl localhost:13305/v1/system-info also hangs. Need to figure out where that's going wrong.

[ckuethe]

Looks like a deadlock in system_info.cpp: std::lock_guard<std::mutex> lock(s_system_info_mutex);. After adding some debug prints:

2026-06-25 14:30:51.366 [Debug] (ModelManager) Checking FLM models
2026-06-25 14:30:51.366 [Debug] (SystemInfoCache) Fetching system info with cache
2026-06-25 14:30:51.366 [Debug] (SystemInfoCache::get_system_info_with_cache) Acquired lock for system info cache
2026-06-25 14:30:51.366 [Debug] (SystemInfoCache::get_system_info_with_cache) Fetching hardware info
2026-06-25 14:30:51.366 [Debug] (SystemInfo::create_system_info) Creating SystemInfo instance for current OS
2026-06-25 14:30:51.461 [Debug] (SystemInfoCache::get_system_info_with_cache) Hardware computation completed
2026-06-25 14:30:51.461 [Debug] (SystemInfoCache::get_system_info_with_cache) Fetching recipe info
2026-06-25 14:30:51.461 [Debug] (SystemInfo::create_system_info) Creating SystemInfo instance for current OS
2026-06-25 14:30:51.461 [Debug] (SystemInfoCache::get_system_info_with_cache) Computing recipes based on detected devices
2026-06-25 14:30:51.461 [Info] (SystemInfo::build_recipes_info) Building recipes info...
2026-06-25 14:30:51.461 [Info] (SystemInfo::build_recipes_info) Checking CPU...
2026-06-25 14:30:51.461 [Info] (SystemInfo::build_recipes_info) Checking AMD GPUs...
2026-06-25 14:30:51.461 [Info] (SystemInfo::build_recipes_info) Checking NVIDIA GPUs...
2026-06-25 14:30:51.461 [Info] (SystemInfo::build_recipes_info) Checking AMD NPUs...
2026-06-25 14:30:51.461 [Info] (SystemInfo::build_recipes_info) Checking Metal devices...
2026-06-25 14:30:51.462 [Debug] (SystemInfo::get_rocm_arch) Detecting ROCm architecture...
2026-06-25 14:30:51.462 [Debug] (SystemInfoCache) Fetching system info with cache
<hang>

[ckuethe]

Git bisect narrowed it down to 37c1a56 #2295

I don't understand yet where the deadlock is coming from but reverting that change avoids the deadlock but also prevents rocm detection from working.