LLM VRAM memory requirements estimator
Accurately estimate the GPU VRAM and system memory required for local deployment of large language models such as Llama, DeepSeek, Qwen, Gemma, and Mistral. Supports quantization format, Context Length, GPU compatibility judgment, and reverse query of "which models can my GPU run?"
Model size
quantization format bits per param
Context Length & GPU Configuration
Quick selection of common models
Click to automatically bring in the corresponding model size
⭐ GPU reverse query Exclusive features
Choose your GPU to see which models can be executed
VRAM demand estimate results
-- GB
-- GB
-- GB
-- GB
Common GPU compatibility comparisons
| GPU | VRAM | Runnable | Suggestions |
|---|
Performance and deployment recommendations
Impact of Context Length on VRAM
Change trends of KV Cache and total VRAM under different Context Length
A complete guide to LLM VRAM and on-premises deployment
A closer look at VRAM, quantization, and GPU selection
What is VRAM?
VRAM (Video Random Access Memory) is a dedicated memory on the graphics card (GPU), similar to the system RAM in a computer. VRAM is used to store data required for GPU real-time computing, including textures, frame buffers (Frame Buffer), shader programs, and the most important in recent years -Model weights and intermediate calculations for large language models (LLM)。
When deploying LLM locally, the size of VRAM directly determines the size of the model that can be loaded and the speed of inference. If the VRAM required by the model exceeds the capacity of the GPU, the model will not run fully on that GPU, or will be forced to use slowdown schemes such as CPU Offloading.
Why does LLM need a lot of VRAM?
The VRAM consumption of large language models mainly comes from three aspects:
- Model weights (Weights): All parameters of the neural network. A 70B model requires approximately 70 × 2 = 140 GB of memory in FP16 format.
- KV Cache: When generating each Token, the Transformer model needs to store the Key and Value values of all previous Tokens. As the context length (Context Length) increases, the KV Cache grows linearly, and may exceed the size of the weight itself in long context scenarios.
- Calculating intermediate values (Activations): The intermediate layer output during forward propagation, the size depends on the batch size and model architecture.
For example, running a 70B model with FP16 processing 32K context requires ~140 GB (weights) + 10 GB (KV Cache) + small overhead ≈ 150+ GB VRAM, which is far beyond the capacity of a single RTX 4090 (24 GB), so quantization or a multi-GPU configuration is required.
What is Quantization?
Quantization is a model compression technique that reduces model size by reducing the number of bits per parameter. Raw models are usually stored using FP16 (16-bit floating point, 2 bytes per parameter) or BF16. Quantization represents these parameters in fewer bits, such as INT8 (8 bits, 1 byte), Q4_K_M (~4 bits, 0.5 bytes) or even Q2_K (~2 bits, 0.25 bytes).
The volume of the quantized model is significantly reduced, but it will bring a certain degree of quality loss. Q4_K_M in GGUF format (popularized by llama.cpp) is currently recognized as the best balance point in the industry, reducing memory requirements by about 75% while retaining most of the model quality.
Q4, Q5, Q8 differences and selection suggestions
There are trade-offs between model size, quality and speed for different quantization levels:
| quantization format | number of bits | parameter size per B | quality preservation | Suggested scenarios |
|---|---|---|---|---|
| FP16 / BF16 | 16-bit | ~2 GB | 100% | Professional research, highest quality requirements |
| Q8_0 | 8-bit | ~1 GB | ~99% | High quality, medium VRAM scene |
| Q6_K | 6-bit | ~0.75 GB | ~98% | Good quality and size balance |
| Q5_K_M | 5-bit | ~0.625 GB | ~96% | Recommended balance point |
| Q4_K_M | 4-bit | ~0.5 GB | ~93% | ⭐ Best value for money (industry standard) |
| Q3_K_M | 3-bit | ~0.375 GB | ~85% | Extremely low VRAM scenario |
| Q2_K | 2-bit | ~0.25 GB | ~75% | Only used when VRAM is extremely low |
Suggestions: For most users, Q4_K_M is the gold standard - reducing model size by approximately 75% while retaining approximately 93% of quality. If VRAM is sufficient, Q5_K_M or Q8_0 can provide higher quality. Q3_K_M is the last line of defense in extremely low VRAM scenarios.
How does Context Length affect memory?
The size of KV Cache is related to Context Lengthlinear relationship. When the context length doubles, the KV Cache also doubles. Under short contexts (2K~8K), KV Cache accounts for a relatively small amount; but under long contexts (32K~128K+), KV Cache may become the main source of VRAM consumption.
For example, an 8B model has a weight of about 4 GB under Q4_K_M and a KV Cache of about 0.5 GB under an 8K context; but under a 128K context, the KV Cache swells to about 8 GB - exceeding the weights themselves. This is exactly why long context models require more VRAM or use optimization techniques (such as sliding window attention, context caching).
What is GPU Offload?
GPU Offload (also known as Layer Offloading) is a hybrid inference strategy that allocates some neural network layers to the CPU system memory for operation when all the weights of the model cannot be placed in the GPU VRAM. The GPU computes the layers it can accommodate, while the CPU handles the remaining layers, with intermediate results passed between the two via the PCIe bus.
- Advantages: Ability to run models that would otherwise be unloadable on VRAM-limited GPUs.
- Disadvantages: Significantly slower (perhaps 5~20x slower than pure GPU) because PCIe bandwidth is much lower than GPU memory bandwidth.
When using Ollama or llama.cpp, you can pass --num-gpu-layers The parameter controls the number of Offload layers. It is usually recommended to first put the model completely into VRAM (without Offload), and then gradually increase the CPU layer if VRAM is insufficient.
LLM VRAM Frequently Asked Questions FAQ
Q1: What models can my RTX 4090 (24GB) run?
The RTX 4090's 24GB VRAM can smoothly run models below 8B (FP16), or use Q4_K_M quantization to run 32B~40B models. For example: Llama 3 8B (FP16, ~16 GB), Qwen3 32B (Q4_K_M, ~16.5 GB), Mistral 7B (FP16, ~14 GB). If you use Q4_K_M, you can even try the 70B model with CPU Offload.
Q2: Is there a big quality difference between FP16 and Q4_K_M?
Actual measurements show that Q4_K_M retains about 93% of FP16 quality, and the difference is almost unnoticeable in most daily tasks (chat, translation, summarization, code generation). But in professional reasoning, mathematical calculations, or scenarios that require a high degree of accuracy, FP16 or Q8_0 still have advantages. Choosing Q4_K_M can reduce VRAM requirements by 75%, making it the best choice for most users.
Q3: What are the alternatives when VRAM is insufficient?
There are four options: (1) Reduce quantization level——Switching from FP16 to Q4_K_M reduces memory requirements by 75%; (2) Shorten Context Length——Reduce KV Cache occupation; (3) Enable GPU Offload--Move some layers to CPU memory (speed will decrease); (4) Use multiple GPUs- Spread the model across multiple display cards. In practice, it is common to try option 1 first and then gradually evaluate other options.
Q4: Do Ollama and LM Studio have the same VRAM requirements?
The basic requirements are the same, as both use llama.cpp or a similar engine under the hood. However, the actual VRAM occupancy may be slightly different: (1) Ollama will occupy VRAM approximately equal to the model size when loading the model. It supports --num-gpu-layers Control Offload; (2) LM Studio provides more detailed GPU Offload slider adjustment; (3) ollama will release VRAM after a period of inactivity, while LM Studio remains occupied. Overall, the core VRAM requirements of both are consistent.
Q5: Why can't my 24GB VRAM GPU load a 16GB model?
This is because when the model is loaded, it not only requires space to store weights, but also requires overhead such as KV Cache, intermediate calculations (Activations), and CUDA core occupancy. These overheads are approximately 10~20%. Additionally, the operating system and other applications use small amounts of VRAM. It is recommended to reserve at least 2~4 GB of buffer space, that is, a 24 GB GPU can actually use about 20~22 GB.
Q6: How much system RAM is required?
It depends on the inference mode. Pure CPU corollary: System RAM needs to be at least 1.2x the model size (a 70B Q4_K_M ~35 GB model → 48~64 GB RAM recommended). Full GPU corollary: RAM requirements are low, 16~32 GB is sufficient. GPU Offload mode: Needs something in between. At least 32 GB of system RAM is recommended for most native AI applications, and 64 GB is better, especially for users who frequently perform long context or multi-model switching.