diff --git a/docs/reference/docker_model_pull.yaml b/docs/reference/docker_model_pull.yaml index f10e4368..f0843b02 100644 --- a/docs/reference/docker_model_pull.yaml +++ b/docs/reference/docker_model_pull.yaml @@ -16,6 +16,11 @@ examples: |- You can pull GGUF models directly from [Hugging Face](https://huggingface.co/models?library=gguf). + **Note about quantization:** If no tag is specified, the command tries to pull the `Q4_K_M` version of the model. + If `Q4_K_M` doesn't exist, the command pulls the first GGUF found in the **Files** view of the model on HuggingFace. + To specify the quantization, provide it as a tag, for example: + `docker model pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_S` + ```console docker model pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF ``` diff --git a/docs/reference/model_pull.md b/docs/reference/model_pull.md index bae41fc3..246cc59d 100644 --- a/docs/reference/model_pull.md +++ b/docs/reference/model_pull.md @@ -22,6 +22,11 @@ docker model pull ai/smollm2 You can pull GGUF models directly from [Hugging Face](https://huggingface.co/models?library=gguf). +**Note about quantization:** If no tag is specified, the command tries to pull the `Q4_K_M` version of the model. +If `Q4_K_M` doesn't exist, the command pulls the first GGUF found in the **Files** view of the model on HuggingFace. +To specify the quantization, provide it as a tag, for example: +`docker model pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_S` + ```console docker model pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF ```