Training
This guide covers training LLaDA models with Safe dLLM, including SFT, CPT, and edit-based safety fine-tuning.
Training Types
Safe dLLM supports three main training paradigms:
Type |
Script |
Purpose |
Typical Duration (4×A100) |
|---|---|---|---|
SFT |
|
Supervised fine-tuning on instruction data |
2-4 hours (100k samples) |
CPT |
|
Continued pre-training on long texts |
12-24 hours |
Safety SFT |
|
Safety-aligned fine-tuning |
2-4 hours |
Data Preparation
WildGuardMix Four-Category Split
First, split the WildGuardMix dataset into four safety categories:
python data/split_wildguardmix.py
Output:
data/wildguardmix/
├── wildguardmix_safe_prompts_safe_response.jsonl # Group A
├── wildguardmix_safe_prompts_harmful_response.jsonl # Group B
├── wildguardmix_harmful_prompt_safe_response.jsonl # Group C
└── wildguardmix_harmful_prompt_harmful_response.jsonl # Group D
Categories:
Group A: Safe prompts → safe responses (training)
Group B: Safe prompts → harmful responses (refusal training)
Group C: Harmful prompts → safe responses (refusal training)
Group D: Harmful prompts → harmful responses (safety testing)
Build Arrow Dataset
Convert JSONL data to HuggingFace Arrow format:
python data/build_mixed_arrow_dataset.py \
--config configs/sft_llada.yaml
The script:
Loads each dataset from
config.dataset.arrow_dirsSamples according to
sampling_ratiosConcatenates all datasets
Shuffles globally with
training.seedSaves to
experiment.output_dir/mixed_arrow/Writes metadata
mixing_meta.json
Custom Dataset
Prepare JSONL:
{"prompt": "Question?", "response": "Answer.", "system": ""}
{"prompt": "Another?", "response": "Answer.", "system": "You are helpful."}
Convert to Arrow:
from datasets import Dataset
import json
data = [json.loads(l) for l in open("data.jsonl")]
dataset = Dataset.from_list(data)
dataset.save_to_directory("output_arrow/")
Supervised Fine-Tuning (SFT)
Configuration
Edit configs/sft_llada.yaml:
model:
pretrained_model: "/path/to/llada_base" # Required!
dataset:
arrow_dirs:
- "/path/to/mixed_sft_dataset"
training:
gradient_accumulation_steps: 8
batch_size_lm: 1
mixed_precision: "bf16"
num_train_epochs: 2
save_steps: 500
optimizer:
learning_rate: 1e-5 # Typical: 1e-5 to 5e-5
Single-GPU Debug
CUDA_VISIBLE_DEVICES=0 python train/sft_llada.py \
config=configs/sft_llada.yaml \
training.num_train_epochs=0.1 \
training.gradient_accumulation_steps=1
Checkpoints:
✅ No CUDA OOM
✅ Loss decreases
✅ Checkpoint saved
Multi-GPU with Slurm
2 GPUs (quick iteration):
sbatch scripts/submit_1_node_2_gpus_dpspd2_sft_llada.slurm
4 GPUs (standard, recommended):
sbatch scripts/submit_1_node_4_gpus_dpspd2_sft_llada.slurm
Effective batch size = batch_size_lm × gradient_accumulation_steps × num_gpus
Monitoring Training
# Check job queue
squeue -u $USER
# View job details
sacct -j $JOBID --format=JobID,State,Elapsed,MaxRSS
# Tail training log
tail -f /scratch/aszalay1/tianze/safe_dllm/logs/train.log
# W&B offline
wandb offline sync wandb/offline-run-*
Training Log Example:
[2025-04-14 10:00] epoch 0/2: 15%|██▌ | 375/2500 [01:23<04:56, 4.50s/it]
[2025-04-14 10:00] step 500: loss=1.234, lr=1.00e-05
[2025-04-14 10:05] Saving checkpoint at step 500...
[2025-04-14 10:05] checkpoint saved to /path/to/output/checkpoint-500
Resume Training
Training auto-resumes if same wandb.run_name is used:
wandb:
resume: 'auto'
Re-submit the same Slurm script to continue.
Continued Pre-Training (CPT)
CPT differs from SFT:
Aspect |
SFT |
CPT |
|---|---|---|
Data format |
Instruction-response pairs |
Long text sequences |
Loss masking |
Response region only |
Full sequence (prompt weighted down) |
Attention |
Diffusion mask bias |
Document boundary causal mask |
Typical epochs |
2-3 |
1 |
Run CPT:
sbatch scripts/submit_1_node_8_gpus_dpspd2_cpt_latent_llada.slurm
Requires 8 GPUs and long-context datasets (4k+ tokens).
Edit-Based Safety Fine-Tuning
Principle
Edit-based fine-tuning activates the edit_head after standard unmasking:
Standard diffusion unmasking proceeds normally
When token confidence ≥
edit_threshold, edit_head refines itAdditional
post_unmask_stepsiterations improve quality
Configuration
training:
unmask_confidence_threshold: 0.9 # Trigger edit threshold
rollout:
edit_threshold: 0.9 # Confidence threshold
post_unmask_steps: 5 # Extra edit iterations
Scripts
Standard edit SFT:
sbatch scripts/submit_1_node_4_gpus_dpspd2_safety_continue_training_edit_llada.slurm
Harmful prompt only (recommended):
sbatch scripts/submit_1_node_4_gpus_dpspd2_safety_continue_training_only_harmfulprompt_edit_llada.slurm
Multi-turn with high temperature:
sbatch scripts/submit_1_node_4_gpus_dpspd2_safety_continue_training_multiturn_hightemp_only_harmfulprompt_edit_llada.slurm
Checkpoint Management
Output structure:
/scratch/.../safe_dllm/
├── checkpoints/
│ └── sft_llada_mixed_100k/
│ ├── checkpoint-500/
│ │ ├── pytorch_model.bin
│ │ ├── config.json
│ │ └── trainer_state.json
│ ├── checkpoint-1000/
│ └── best/ ← Best model (lowest loss)
└── wandb/
└── offline-run-abc123/
Select checkpoint:
# List all checkpoints
ls /path/to/output/checkpoint-*/
# Use specific checkpoint for evaluation
python run_safety_benchmark.py \
--model_path /path/to/checkpoint-1000 \
--dataset harmbench
Troubleshooting
Low GPU Utilization
Increase batch_size_lm or reduce gradient_accumulation_steps. Check data loader bottleneck.
Loss Not Decreasing
Verify data loading:
python quick_test_dataset.pyCheck
diffusion_maskalignment (should be 0=prompt, 1=response, -1=padding)Lower learning rate:
1e-5 → 5e-6
Training Unstable
Gradient explosion:
max_grad_norm: 1.0(already default)Learning rate too high: reduce to
5e-6or1e-6Data quality: verify Arrow dataset
Fast Iteration Strategy
Single GPU, 1 epoch, 10k samples → verify pipeline
4 GPUs, full data, 0.5 epoch → check loss
Full training: 2-3 epochs
Mixed Data Training
Configure sampling ratios in build_mixed_arrow_dataset.py:
dataset_specs = [
("/path/to/math_data", 0.4), # 40% math
("/path/to/code_data", 0.4), # 40% code
("/path/to/safety_data", 0.2), # 20% safety
]
Post-Training Validation
# Quick safety eval (100 samples)
python run_safety_benchmark.py \
--model_path /path/to/checkpoint \
--dataset wildguardmix_D \
--max_samples 100
# Full benchmark
python eval_bench.py config=configs/llada_eval.yaml dataset.eval_dataset=GSM8K
# Compare with baseline
# Record ASR/accuracy for base vs fine-tuned model
FAQ
Q: How much data for SFT?
A: At least 50k-100k high-quality instruction pairs. The mixed dataset in this project includes math, code, and safety data.
Q: How to choose epochs?
A: 2-3 epochs for 100k data. For larger datasets, 1-2 epochs suffice. Watch for overfitting (loss increases on eval).
Q: Can I train on CPU?
A: Not recommended. SFT/CPT require significant GPU compute. CPU training only for debugging data loading.
Q: What if CUDA out of memory?
A:
Reduce
gradient_accumulation_steps(but this reduces effective batch size)Use
mixed_precision: "bf16"Reduce
batch_size_lmto 1Use gradient checkpointing:
gradient_checkpointing_enable: true
Q: How to fine-tune DREAM model?
A: Use train/sft_dream.py with configs/sft_dream.yaml. DREAM is a different diffusion architecture.
See Also
guide/config - Configuration details
guide/evaluation - Safety evaluation
guide/attacks - Adversarial testing
guide/slurm - Cluster job submission