Training

This guide covers training LLaDA models with Safe dLLM, including SFT, CPT, and edit-based safety fine-tuning.

Training Types

Safe dLLM supports three main training paradigms:

Type

Script

Purpose

Typical Duration (4×A100)

SFT

train/sft_llada.py

Supervised fine-tuning on instruction data

2-4 hours (100k samples)

CPT

train/cpt_latent_llada_fullscale.py

Continued pre-training on long texts

12-24 hours

Safety SFT

train/sft_llada.py with edit config

Safety-aligned fine-tuning

2-4 hours

Data Preparation

WildGuardMix Four-Category Split

First, split the WildGuardMix dataset into four safety categories:

python data/split_wildguardmix.py

Output:

data/wildguardmix/
├── wildguardmix_safe_prompts_safe_response.jsonl      # Group A
├── wildguardmix_safe_prompts_harmful_response.jsonl   # Group B
├── wildguardmix_harmful_prompt_safe_response.jsonl    # Group C
└── wildguardmix_harmful_prompt_harmful_response.jsonl # Group D

Categories:

  • Group A: Safe prompts → safe responses (training)

  • Group B: Safe prompts → harmful responses (refusal training)

  • Group C: Harmful prompts → safe responses (refusal training)

  • Group D: Harmful prompts → harmful responses (safety testing)

Build Arrow Dataset

Convert JSONL data to HuggingFace Arrow format:

python data/build_mixed_arrow_dataset.py \
    --config configs/sft_llada.yaml

The script:

  1. Loads each dataset from config.dataset.arrow_dirs

  2. Samples according to sampling_ratios

  3. Concatenates all datasets

  4. Shuffles globally with training.seed

  5. Saves to experiment.output_dir/mixed_arrow/

  6. Writes metadata mixing_meta.json

Custom Dataset

Prepare JSONL:

{"prompt": "Question?", "response": "Answer.", "system": ""}
{"prompt": "Another?", "response": "Answer.", "system": "You are helpful."}

Convert to Arrow:

from datasets import Dataset
import json

data = [json.loads(l) for l in open("data.jsonl")]
dataset = Dataset.from_list(data)
dataset.save_to_directory("output_arrow/")

Supervised Fine-Tuning (SFT)

Configuration

Edit configs/sft_llada.yaml:

model:
  pretrained_model: "/path/to/llada_base"  # Required!

dataset:
  arrow_dirs:
    - "/path/to/mixed_sft_dataset"

training:
  gradient_accumulation_steps: 8
  batch_size_lm: 1
  mixed_precision: "bf16"
  num_train_epochs: 2
  save_steps: 500

optimizer:
  learning_rate: 1e-5  # Typical: 1e-5 to 5e-5

Single-GPU Debug

CUDA_VISIBLE_DEVICES=0 python train/sft_llada.py \
    config=configs/sft_llada.yaml \
    training.num_train_epochs=0.1 \
    training.gradient_accumulation_steps=1

Checkpoints:

  • ✅ No CUDA OOM

  • ✅ Loss decreases

  • ✅ Checkpoint saved

Multi-GPU with Slurm

2 GPUs (quick iteration):

sbatch scripts/submit_1_node_2_gpus_dpspd2_sft_llada.slurm

4 GPUs (standard, recommended):

sbatch scripts/submit_1_node_4_gpus_dpspd2_sft_llada.slurm

Effective batch size = batch_size_lm × gradient_accumulation_steps × num_gpus

Monitoring Training

# Check job queue
squeue -u $USER

# View job details
sacct -j $JOBID --format=JobID,State,Elapsed,MaxRSS

# Tail training log
tail -f /scratch/aszalay1/tianze/safe_dllm/logs/train.log

# W&B offline
wandb offline sync wandb/offline-run-*

Training Log Example:

[2025-04-14 10:00] epoch 0/2:  15%|██▌  | 375/2500 [01:23<04:56, 4.50s/it]
[2025-04-14 10:00] step 500: loss=1.234, lr=1.00e-05
[2025-04-14 10:05] Saving checkpoint at step 500...
[2025-04-14 10:05] checkpoint saved to /path/to/output/checkpoint-500

Resume Training

Training auto-resumes if same wandb.run_name is used:

wandb:
  resume: 'auto'

Re-submit the same Slurm script to continue.

Continued Pre-Training (CPT)

CPT differs from SFT:

Aspect

SFT

CPT

Data format

Instruction-response pairs

Long text sequences

Loss masking

Response region only

Full sequence (prompt weighted down)

Attention

Diffusion mask bias

Document boundary causal mask

Typical epochs

2-3

1

Run CPT:

sbatch scripts/submit_1_node_8_gpus_dpspd2_cpt_latent_llada.slurm

Requires 8 GPUs and long-context datasets (4k+ tokens).

Edit-Based Safety Fine-Tuning

Principle

Edit-based fine-tuning activates the edit_head after standard unmasking:

  1. Standard diffusion unmasking proceeds normally

  2. When token confidence ≥ edit_threshold, edit_head refines it

  3. Additional post_unmask_steps iterations improve quality

Configuration

training:
  unmask_confidence_threshold: 0.9  # Trigger edit threshold

rollout:
  edit_threshold: 0.9              # Confidence threshold
  post_unmask_steps: 5             # Extra edit iterations

Scripts

Standard edit SFT:

sbatch scripts/submit_1_node_4_gpus_dpspd2_safety_continue_training_edit_llada.slurm

Harmful prompt only (recommended):

sbatch scripts/submit_1_node_4_gpus_dpspd2_safety_continue_training_only_harmfulprompt_edit_llada.slurm

Multi-turn with high temperature:

sbatch scripts/submit_1_node_4_gpus_dpspd2_safety_continue_training_multiturn_hightemp_only_harmfulprompt_edit_llada.slurm

Checkpoint Management

Output structure:

/scratch/.../safe_dllm/
├── checkpoints/
│   └── sft_llada_mixed_100k/
│       ├── checkpoint-500/
│       │   ├── pytorch_model.bin
│       │   ├── config.json
│       │   └── trainer_state.json
│       ├── checkpoint-1000/
│       └── best/  ← Best model (lowest loss)
└── wandb/
    └── offline-run-abc123/

Select checkpoint:

# List all checkpoints
ls /path/to/output/checkpoint-*/

# Use specific checkpoint for evaluation
python run_safety_benchmark.py \
    --model_path /path/to/checkpoint-1000 \
    --dataset harmbench

Troubleshooting

Low GPU Utilization

Increase batch_size_lm or reduce gradient_accumulation_steps. Check data loader bottleneck.

Loss Not Decreasing

  1. Verify data loading: python quick_test_dataset.py

  2. Check diffusion_mask alignment (should be 0=prompt, 1=response, -1=padding)

  3. Lower learning rate: 1e-5 → 5e-6

Training Unstable

  • Gradient explosion: max_grad_norm: 1.0 (already default)

  • Learning rate too high: reduce to 5e-6 or 1e-6

  • Data quality: verify Arrow dataset

Fast Iteration Strategy

  1. Single GPU, 1 epoch, 10k samples → verify pipeline

  2. 4 GPUs, full data, 0.5 epoch → check loss

  3. Full training: 2-3 epochs

Mixed Data Training

Configure sampling ratios in build_mixed_arrow_dataset.py:

dataset_specs = [
    ("/path/to/math_data", 0.4),    # 40% math
    ("/path/to/code_data", 0.4),    # 40% code
    ("/path/to/safety_data", 0.2),  # 20% safety
]

Post-Training Validation

# Quick safety eval (100 samples)
python run_safety_benchmark.py \
    --model_path /path/to/checkpoint \
    --dataset wildguardmix_D \
    --max_samples 100

# Full benchmark
python eval_bench.py config=configs/llada_eval.yaml dataset.eval_dataset=GSM8K

# Compare with baseline
# Record ASR/accuracy for base vs fine-tuned model

FAQ

Q: How much data for SFT?

A: At least 50k-100k high-quality instruction pairs. The mixed dataset in this project includes math, code, and safety data.

Q: How to choose epochs?

A: 2-3 epochs for 100k data. For larger datasets, 1-2 epochs suffice. Watch for overfitting (loss increases on eval).

Q: Can I train on CPU?

A: Not recommended. SFT/CPT require significant GPU compute. CPU training only for debugging data loading.

Q: What if CUDA out of memory?

A:

  • Reduce gradient_accumulation_steps (but this reduces effective batch size)

  • Use mixed_precision: "bf16"

  • Reduce batch_size_lm to 1

  • Use gradient checkpointing: gradient_checkpointing_enable: true

Q: How to fine-tune DREAM model?

A: Use train/sft_dream.py with configs/sft_dream.yaml. DREAM is a different diffusion architecture.

See Also

  • guide/config - Configuration details

  • guide/evaluation - Safety evaluation

  • guide/attacks - Adversarial testing

  • guide/slurm - Cluster job submission