Turkey (Turkish)
Germany (German)
Worldwide (English)
If chat-control tokens are wrong, the same conversation can become a completely different token sequence. Training and inference formatting must match.
Halkali merkez neighborhood fatih st ozgur apt no 46 , Kucukcekmece , Istanbul , 34303 , TR
Fix fine-tuned models that never stop, emit role tokens or return empty answers by checking EOS/PAD, chat templates, generation prompts, masking and tokenizer.
If chat-control tokens are wrong, the same conversation can become a completely different token sequence. Training and inference formatting must match.
`add_generation_prompt=True` adds control tokens that start a new assistant response; `continue_final_message` is for prefilling and the two are mutually exclusive.
PAD and EOS can share an ID in some setups, but incorrect loss masking or generation configuration can accidentally ignore EOS supervision.
If generation only stops at `max_new_tokens`, inspect EOS supervision and generation config. Literal role markers suggest template/tokenizer mismatch. Immediate EOS with empty answers can indicate masked assistant content.
Decode the exact training sequence and inference sequence side by side. Verify BOS/EOS, role headers and turn terminators are where the model expects them.
python - <<'PY'
# print(tokenizer.special_tokens_map)
# print(tokenizer.eos_token, tokenizer.eos_token_id)
# print(tokenizer.pad_token, tokenizer.pad_token_id)
# print(tokenizer.chat_template)
PYIf the EOS or turn-end token at the end of the assistant answer is not supervised, the model may not learn to stop. Inspect a collated batch to ensure the terminator was not changed to -100 by masking.
For tokenizers without a dedicated pad token, PAD=EOS is common. But if the collator masks every EOS ID as padding, real sequence-ending EOS supervision can disappear. Verify masking by padding positions/attention masks rather than blindly by token ID.
Llama, Qwen, Gemma, Mistral and other instruct models transform the same messages into different control-token sequences. Copying another model's ChatML template can break training when special tokens differ.
If you add role/EOS tokens to the tokenizer without resizing model embeddings, you can get index errors or untrained-token behavior. After resizing, verify how embedding/lm_head weights are saved with PEFT.
`eos_token_id`, multiple EOS IDs, pad token, min/max new tokens and stopping criteria change generation behavior. If the model stops only because of hard limits, adding stopping criteria hides the symptom without fixing training.
If the base model stops correctly under the same template and deterministic decoding but the adapter does not, the issue is likely in adapter training/masking. If the base model also fails, fix tokenizer/template/generation first.
Save the tokenizer special-token map; inspect five training examples as IDs and decoded text; verify assistant start/end tokens are supervised; compare base/adapter with identical decoding; ensure tokenizer files ship with the checkpoint.
| Symptom | Likely cause | Check |
|---|---|---|
| Never stops | No EOS supervision | Assistant final labels |
| Prints role tokens | Wrong template | Decode training sequence |
| Empty answer | Masking/truncation | Supervised-token count |
| Base good, adapter bad | Fine-tune formatting | Base vs adapter A/B |
Do not blindly upgrade packages in a working training environment. Record GPU, driver, CUDA/PyTorch runtime, Transformers, Accelerate, PEFT, TRL, bitsandbytes/Diffusers, model revision and dataset fingerprint for every run. Reproduce minimally before changing production training.
Not always. The key is ensuring real EOS tokens are not mistakenly masked as padding.
It may not have learned EOS, may use the wrong EOS ID, or the generation pipeline may not recognize the correct terminator.
Evaluate model size, precision, context, batch, LoRA/QLoRA or full fine-tuning and multi-GPU needs together instead of choosing by GPU name alone.