Turkey (Turkish)
Germany (German)
Worldwide (English)
This error means the loss tensor used for backward is not connected to an autograd graph; it is not the same category as CUDA OOM or dataset-format errors.
Halkali merkez neighborhood fatih st ozgur apt no 46 , Kucukcekmece , Istanbul , 34303 , TR
Diagnose SFTTrainer/PEFT grad_fn errors using frozen parameters, adapter trainability, no_grad/detach, k-bit preparation, checkpointing and custom loss.
This error means the loss tensor used for backward is not connected to an autograd graph; it is not the same category as CUDA OOM or dataset-format errors.
For quantized PEFT training, Hugging Face recommends `prepare_model_for_kbit_training()`.
Frozen base parameters are normal in LoRA; what matters is that adapter parameters are trainable and the loss has a gradient path to them.
PyTorch backward requires the loss to be connected to trainable operations. If loss is produced under no_grad, detached, or connected to no trainable parameters, it has no grad_fn.
Frozen base weights are expected in PEFT. But if the count of requires_grad parameters is zero, adapters may not have been injected, inference mode may be enabled, or the Trainer may be using the wrong model instance.
python - <<'PY'
trainable=sum(p.numel() for p in model.parameters() if p.requires_grad)
total=sum(p.numel() for p in model.parameters())
print(trainable, total, trainable/total if total else 0)
for n,p in model.named_parameters():
if p.requires_grad: print(n)
PYWhen training adapters on 4/8-bit models, PEFT recommends `prepare_model_for_kbit_training()`. It prepares details such as normalization and gradient-related behavior for k-bit training.
If a custom loop, loss or wrapper runs forward under `torch.no_grad()`, the entire graph disappears. Calling `.item()` for logging is fine; detaching the tensor that will be used for backward is not.
Some PEFT plus gradient-checkpointing combinations require input-gradient handling. Check how your current Transformers/PEFT versions use `enable_input_require_grads()` or k-bit preparation instead of copying old workarounds blindly.
Converting loss to NumPy or a Python float and then creating a new tensor destroys autograd history. Custom loss computations must remain as PyTorch tensor operations connected to model outputs.
If you wrap a base model with adapters but pass the original base model to Trainer, LoRA parameters are absent from the training graph. Verify the Trainer actually holds the PEFT model.
Run one batch outside Trainer and inspect `loss.requires_grad`, `loss.grad_fn`, trainable parameter count and adapter gradients after backward. This separates model-graph issues from Trainer/Accelerate issues.
python - <<'PY'
# outputs=model(**batch)
# print(outputs.loss.requires_grad, outputs.loss.grad_fn)
# outputs.loss.backward()
# for n,p in model.named_parameters():
# if p.requires_grad and p.grad is not None:
# print(n, p.grad.norm().item()); break
PYMaking every base parameter trainable destroys the purpose of LoRA, can explode VRAM usage and hides the real graph issue. The correct goal is gradients on intended adapter parameters.
| Check | Bad result | Meaning |
|---|---|---|
| Trainable param count | 0 | No trainable adapter |
| loss.requires_grad | False | Detached graph |
| loss.grad_fn | None | Loss not connected |
| Adapter .grad | None | Gradient not reaching adapter |
Do not blindly upgrade packages in a working training environment. Record GPU, driver, CUDA/PyTorch runtime, Transformers, Accelerate, PEFT, TRL, bitsandbytes/Diffusers, model revision and dataset fingerprint for every run. Reproduce minimally before changing production training.
No. The base is expected to be frozen; adapters should be trainable.
It may hide the error but can accidentally turn LoRA into full fine-tuning and blow VRAM.
Evaluate model size, precision, context, batch, LoRA/QLoRA or full fine-tuning and multi-GPU needs together instead of choosing by GPU name alone.