Skip to content

Add GGUF VRAM Calculator and two quantization guides #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
McFatcop1 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add GGUF VRAM Calculator and two quantization guides #47

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ New to local AI? Start here.
Interactive tools for planning and optimizing your local AI setup.

- [Local AI Planning Tool](https://insiderllm.com/tools/vram-calculator/) - Interactive VRAM calculator with hardware, model, and task entry points
- [GGUF VRAM Calculator](https://learnaitodayonline.com/vram-calculator/) - Which quant fits your card, from measured GGUF file sizes and per-layer KV cache for hybrid-attention models

## Hardware Guides

Expand All @@ -62,6 +63,8 @@ Figuring out what hardware you need (or what to do with what you have).
- [Mixtral VRAM Requirements](https://insiderllm.com/guides/mixtral-vram-requirements/) - Every quantization level for 8x7B and 8x22B
- [KV Cache and VRAM](https://insiderllm.com/guides/kv-cache-optimization-guide/) - Why context length eats your VRAM and how to fix it
- [num_ctx VRAM Overflow](https://insiderllm.com/guides/num-ctx-vram-overflow-slow-inference/) - The silent performance killer nobody warns about
- [Which Quant to Download](https://learnaitodayonline.com/which-quant-to-download/) - Matching GGUF quantisation to the VRAM you actually have
- [KV Cache Quantization](https://learnaitodayonline.com/kv-cache-quantization/) - Trading cache precision for context length
- [GPU Benchmarks for LLM Inference](https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference) - Community benchmark database

### Buying Guides
Expand Down