Skip to content

Stream tokens in nanotune chat instead of waiting for the full reply #68

Description

@will-lamerton

Problem

nanotune chat waits for the entire response before rendering anything. chatCompletion() in src/lib/llama-cpp.ts does a plain fetch and awaits the full JSON body.

On a long reply the UI sits on a "Generating..." spinner for many seconds with no indication of progress. Small local models are exactly the case where you want to see tokens land — it's how you tell the model is working versus stuck.

llama-server already supports SSE streaming on /v1/chat/completions with "stream": true.

Proposal

  • Add a streaming variant alongside chatCompletion (keep the non-streaming one — benchmark doesn't need streaming and relies on the final timings block)
  • Render tokens into the chat transcript as they arrive
  • Keep the existing per-turn stats line; SSE responses still carry timings in the final chunk

Acceptance criteria

  • Tokens appear in nanotune chat as they're generated
  • TTFT / tok/s / token count stats still shown after each turn
  • Interrupting mid-generation cleanly cancels the in-flight request
  • benchmark behaviour is unchanged

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions