Fix diagram analysis for local LLMs by using base64 encoding instead of URLs

[androemeda]

Summary

Fixes a bug in diagram analysis that caused failures when using local LLM deployments. Local LLMs cannot process remote image URLs and require base64-encoded image data instead. This change makes diagram analysis work correctly with both local and cloud-based LLM configurations.

Problem

The analyse_diagram function was passing the diagram as a remote URL directly to the model:

{"type": "image_url", "image_url": {"url": diagram_url}}

Local LLMs (Ollama, LocalAI, etc.) do not support fetching remote URLs during inference and expect image data to be provided directly in base64 format. This caused the following error:

image URLs are not currently supported, please use base64 encoded data instead

Fix

Replaced the remote URL approach with base64 encoding — the image bytes are fetched in Python and encoded before being passed to the model:

{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{diagram_base64}"}}

This is fully backward compatible — OpenAI and Azure OpenAI continue to work correctly with the base64 format.

Changes

• backend/app/services/llms_service.py — replaced get_diagram_url() with get_diagram_image() to fetch raw image bytes, added base64 encoding before passing image to the model.

Testing

Endpoint: GET http://localhost:8000/analyse/{biomodel_id}/diagram

Test case: http://localhost:8000/analyse/306747843/diagram

1. Configure a local LLM in your environment (PROVIDER=local)
2. Make a request to the diagram analysis endpoint with a valid biomodel ID
3. Before: Returns "image URLs are not currently supported, please use base64 encoded data instead"
4. After: Returns a successful diagram analysis response