Skip to content
This repository was archived by the owner on Oct 6, 2025. It is now read-only.

Conversation

@doringeman
Copy link
Collaborator

Makes use of docker/compose@4bbc6c6 and docker/model-runner#101.

E.g.,

$ # terminal 1

$ DEBUG=1 MODEL_RUNNER_PORT=8080 make run 2>&1 | grep "with user agent: "
$ # terminal 2

$ cat test-compose-models.yaml
services:
  ai_runner:
    image: alpine
    models:
      - ai_model
models:
  ai_model:
    model: ai/llama3.2
    context_size: 1024
    runtime_flags:
      - "--no-prefill-assistant"

$ MODEL_RUNNER_HOST=http://localhost:8080 docker compose -f test-compose-models.yaml up
# in terminal 1 you'll see:
# time="2025-07-08T15:18:31+03:00" level=debug msg="Tracked index.docker.io/ai/llama3.2:latest latest
# with user agent: docker-model-runner docker-model-cli/dev compose/v2.23.3-812-g9e17a091b" component=metrics

$ MODEL_RUNNER_HOST=http://localhost:8080 docker model run ai/smollm2 hi
# in terminal 1 you'll see:
# time="2025-07-08T15:18:38+03:00" level=debug msg="Tracked index.docker.io/ai/smollm2:latest latest
# with user agent: docker-model-runner docker-model-cli/dev" component=metrics

$ curl http://localhost:8080/engines/v1/chat/completions -X POST -H "Content-Type: application/json" -d '{
    "model": "dorin/smollm2",
    "messages": [
      {"role": "user", "content": "Capital of Romania?"}
    ]
  }'
# in terminal 1 you'll see:
# time="2025-07-08T15:18:50+03:00" level=debug msg="Tracked index.docker.io/ai/smollm2:latest latest
# with user agent: docker-model-runner curl/8.7.1" component=metrics

@doringeman doringeman requested a review from a team July 8, 2025 12:32
@doringeman doringeman merged commit c181f44 into docker:main Jul 9, 2025
3 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants