How Much GPU Memory Do You Actually Need to Run an LLM?

I've watched more people hit "CUDA out of memory" errors while trying to fine-tune their first model than any other single roadblock in getting started with local LLMs. This is the explanation I wish had existed before I hit that wall myself.

"How much GPU do I need?" is one of the most common questions from anyone starting to work with language models locally, and the honest answer is: it depends enormously on what you're actually trying to do. Running a model and training one have wildly different memory footprints, and understanding exactly why closes the gap between guessing and planning.

The Baseline: Model Weights

Every parameter in a model needs to be stored in memory, and the storage format directly determines how much space that takes. A 7-billion-parameter model stored in FP16 (16-bit floating point, the standard inference precision) needs roughly 14GB just for the weights themselves, 2 bytes per parameter times 7 billion parameters. This baseline number is the starting point for any memory estimate, but it's rarely the whole story, especially once you move beyond simple inference.

Why Inference Alone Doesn't Need Much More Than the Weights

Running a model purely for inference, generating text from a prompt without any training happening, needs the model weights plus a relatively modest amount of additional memory for activations, the intermediate values computed as data flows through the network, plus the KV cache, which stores attention key/value pairs to avoid redundant computation for each new token generated. For most conversational use cases, this overhead adds maybe 10-20% on top of the base weight size, which is exactly why a 7B model that needs 14GB for weights alone can often run comfortably on a single 16GB or 24GB consumer GPU.

Why Fine-Tuning Is a Completely Different Story

The moment you move from running a model to training or fine-tuning one, memory requirements jump dramatically, and this catches a lot of people off guard. Full fine-tuning needs to store not just the weights, but also gradients (roughly the same size as the weights themselves) and optimizer states, and this is the part that surprises people most, the Adam optimizer, the standard choice for training neural networks, needs to track momentum and variance for every single parameter, adding roughly 4x the model weight size just for optimizer bookkeeping. Add it all up and full fine-tuning of a 7B model can require 5-6x more memory than simply running it, easily exceeding what a single consumer GPU can provide.

How LoRA Made Fine-Tuning Accessible Again

LoRA (Low-Rank Adaptation) was a genuine turning point for accessible fine-tuning. Instead of updating every parameter in the model, LoRA freezes the entire original model and only trains small additional "adapter" matrices inserted into each layer, a tiny fraction of the total parameter count. Since gradients and optimizer states only need to be tracked for these small adapters, not the full model, the memory overhead for training drops enormously, this is precisely why fine-tuning a 7B or even 13B model became genuinely feasible on a single consumer GPU with LoRA, something that would otherwise demand enterprise-grade hardware with full fine-tuning.

The Precision Trade-off

Beyond LoRA, reducing numeric precision is the other major lever for fitting larger models into limited memory. Moving from FP16 to INT8 halves memory again, and INT4 quantization halves it once more, at some cost to model accuracy that varies depending on the specific quantization method and how sensitive the task is to precision loss. Techniques like QLoRA combine both approaches, quantizing the frozen base model to 4-bit precision while still applying LoRA adapters on top, stacking memory savings from both directions simultaneously, which is how fine-tuning models in the tens of billions of parameters became possible on hardware that would have been laughably insufficient for full fine-tuning at that scale.

Planning Before You Buy or Rent Hardware

Getting a rough memory estimate before committing to specific GPU hardware, whether buying a consumer card or renting cloud GPU time, saves real money and frustration. Our GPU VRAM Estimator breaks down exactly where memory goes, weights, gradients, optimizer states, activations, based on your model size, precision, and whether you're doing inference, LoRA fine-tuning, or full fine-tuning, so you can compare against real GPU specs before running into an out-of-memory error mid-training.

A Rule of Thumb Worth Remembering

As a rough planning heuristic: multiply a model's parameter count in billions by 2 to estimate inference memory in GB (FP16), by roughly 12-16 for full fine-tuning, and by roughly 3-4 for LoRA fine-tuning. These multipliers are approximations, not exact figures, but they're accurate enough to immediately rule out obviously insufficient hardware before you've invested time setting up a training run that was never going to fit in the first place, a five-minute estimate versus hours of wasted setup and debugging.