Skip to content

[Performance]: Cache the OpenRouter model catalog with a TTL #56

Description

@shauryagangrade

Problem

list_free_models hits https://openrouter.ai/api/v1/models on every call.
/models and startup both fetch the full catalog each time — a wasted
network round-trip and a slow menu on a flaky connection.

Proposed approach

  • Add a short-lived in-process cache (e.g. 10-60 min TTL) so repeated calls
    within a session reuse the last response.
  • Keep a 0-TTL escape hatch for anything that needs fresh data.
  • Respect the existing best-effort contract (empty list + error on failure).

Where to look

  • gcode/models.py:18-55 (list_free_models).

Acceptance criteria

  • Two consecutive list_free_models() calls inside the TTL make one HTTP request.
  • Cache invalidation on expiry is tested.

Difficulty

Low — great first issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    good first issueGood for newcomersperformanceLatency, throughput, and efficiency improvements

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions