How Much VRAM Do You Need for AI Model Training? LoRA, QLoRA and Full Fine-Tuning

Five components of training VRAM

Estimating training memory from checkpoint size alone is wrong. VRAM can contain model weights, trainable gradients, optimizer states, forward activations and framework/kernel workspace at the same time. Full fine-tuning carries most of these for all parameters; LoRA and QLoRA reduce the trainable subset.

The same 8B model can use radically different memory at batch 1/context 2048 versus batch 4/context 8192. Activation memory grows with batch and sequence length, so parameter count alone is not a capacity decision.

Why full fine-tuning is expensive

In full fine-tuning every parameter receives gradients. Optimizers such as AdamW keep extra state for trainable parameters, and even with mixed precision, master weights and optimizer state remain significant. Training footprint can therefore be several times the FP16 weight size.

Even 7B/8B full fine-tuning often does not fit on a single consumer GPU; model sharding, ZeRO/FSDP or multiple GPUs are commonly required. This is why LoRA/QLoRA are practical single-GPU approaches.

What LoRA and QLoRA change in memory

LoRA freezes the base model and trains low-rank adapters, so gradients and optimizer states are kept mainly for adapter parameters. QLoRA goes further by loading the base model in 4-bit form.

4-bit does not mean total training VRAM becomes one quarter. Activations are typically computed in BF16/FP16, while intermediate tensors, attention workspaces and batch/context effects remain.

What can a 24 GB GPU handle?

On 24 GB GPUs, 7B/8B QLoRA is a comfortable starting class; 14B is often feasible with careful settings. 32B can be possible under tight configurations, but activations and long context quickly become limiting.

Before deciding, estimate model loading with `accelerate estimate-memory`, then validate peak training memory on a real mini-batch using PyTorch metrics and `nvidia-smi`. The estimator itself documents that it only estimates loading memory.

accelerate estimate-memory MODEL_ID --library_name transformers

nvidia-smi

python - <<'PY'
import torch
print(torch.cuda.get_device_name())
print(round(torch.cuda.get_device_properties(0).total_memory/1024**3,2),'GB')
PY

Why context length explodes VRAM

Sequence length creates activations across every layer. Going from 2K to 8K does not change weight size but can dramatically increase activation footprint. If lowering batch is insufficient, context length is a primary variable to inspect.

How gradient accumulation and checkpointing work together

Gradient accumulation keeps the physical micro-batch small while simulating a larger effective batch. Gradient checkpointing recomputes activations during backward. Together they can reduce VRAM substantially at the cost of step time.

Effective batch is roughly per-device batch × accumulation × GPU count. Learning-rate and scheduler decisions should consider effective batch, not only micro-batch.

Order of fixes when VRAM is insufficient

Start with batch 1 and the required context. Then evaluate BF16/FP16, gradient checkpointing, accumulation, 8-bit optimizers and QLoRA. If it still does not fit, reduce model size or move to FSDP/DeepSpeed/offload.

Practical GPU buying classes

12-16 GB: small/medium QLoRA and diffusion LoRA. 24 GB: strong single-GPU class for 7B-14B QLoRA. 32-48 GB: more room for 14B-32B and longer contexts. 80 GB: professional class for larger adapters and distributed work. These are starting points, not guarantees.