Skip to content
This repository was archived by the owner on Oct 6, 2025. It is now read-only.

Conversation

@ekcasey
Copy link
Contributor

@ekcasey ekcasey commented Aug 22, 2025

Bumps model-distribution to include docker/model-distribution#123. This allows packaging of sharded models.

Using DMR build from docker/model-runner#133:

## Package
> docker model package --gguf /Users/emilycasey/Library/Caches/llama.cpp/Felladrin_gguf-sharded-Q4_K_S-DeepSeek-R1-Distill-Qwen-1.5B_model.shard-00001-of-00006.gguf shard/test
Adding GGUF file from "/Users/emilycasey/Library/Caches/llama.cpp/Felladrin_gguf-sharded-Q4_K_S-DeepSeek-R1-Distill-Qwen-1.5B_model.shard-00001-of-00006.gguf"
Loading model to Model Runner...
Transferred: 97.52 MB
Model loaded successfully

## Run
> docker model run shard/test
Interactive chat mode started. Type '/bye' to exit.
> hi
Hello! How can I assist you today?
> /bye
Chat session ended.

## Inspect bundle
> tree ~/.docker/models/bundles
/Users/emilycasey/.docker/models/bundles
└── sha256
    ├── 608d88953bbbb3d41964dd205882a0081227a50378050ee20a84994fe9ed005a
    │   ├── config.json
    │   ├── model-00001-of-00006.gguf
    │   ├── model-00002-of-00006.gguf
    │   ├── model-00003-of-00006.gguf
    │   ├── model-00004-of-00006.gguf
    │   ├── model-00005-of-00006.gguf
    │   └── model-00006-of-00006.gguf

@ekcasey ekcasey requested a review from xenoscopic August 22, 2025 04:37
Copy link
Contributor

@xenoscopic xenoscopic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be helpful to avoid confusion if package's Short description included something like "Sharded GGUF files can be package by specifying the path to the first GGUF shard" (or something similar, I'm not sure the exact way to say it).

@ekcasey
Copy link
Contributor Author

ekcasey commented Aug 22, 2025

It might be helpful to avoid confusion if package's Short description included something like "Sharded GGUF files can be package by specifying the path to the first GGUF shard" (or something similar, I'm not sure the exact way to say it)

I couldn't find a wording that was short enough to look okay in docker model --help output so I added a Long help text with more details to be shown only on docker model package --help.

@ekcasey ekcasey marked this pull request as ready for review August 22, 2025 17:48
@ekcasey ekcasey merged commit f6ad1e8 into main Aug 22, 2025
6 checks passed
@ekcasey ekcasey deleted the shards branch August 22, 2025 17:51
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants