Slurm Jobs
Complete guide to submitting and managing Slurm jobs for Safe dLLM.
Slurm Scripts Overview
scripts/
├── submit_1_node_2_gpus_dpspd2_sft_llada.slurm # SFT (2 GPUs)
├── submit_1_node_4_gpus_dpspd2_sft_llada.slurm # SFT (4 GPUs)
├── submit_1_node_4_gpus_dpspd2_sft_edit_llada.slurm # Edit SFT (4 GPUs)
├── submit_1_node_8_gpus_dpspd2_cpt_latent_llada.slurm # CPT (8 GPUs)
├── submit_1_node_2_gpus_eval_bench.slurm # Benchmark eval (2 GPUs)
└── submit_1_node_2_gpus_eval_safety.slurm # Safety eval + attacks (2 GPUs)
Naming convention:
1_node_2_gpus- Single node, 2 GPUsdpspd2- DP+SPD2 data parallel (DistributedDataParallel + Sharded)eval_bench- Benchmark evaluation (MBPP/GSM8K/MATH/LiveBench)eval_safety- Safety evaluation (HarmBench/WildGuardMix + attacks)
SBATCH Directives
Common header across all scripts:
#!/bin/bash
#SBATCH --job-name=eval_safety # Job name (shown in squeue)
#SBATCH --partition=a100 # GPU partition (a100/h100/nvl)
#SBATCH --account=aszalay1_ssci # Billing account
#SBATCH --nodes=1 # Number of nodes
#SBATCH --gres=gpu:2 # Number of GPUs
#SBATCH --cpus-per-task=24 # CPU cores per task
#SBATCH --mem=512G # Memory (GB)
#SBATCH --time=24:00:00 # Max runtime (DD-HH:MM:SS)
#SBATCH --exclude=c001,h01 # Nodes to exclude
#SBATCH --export=ALL,OPENAI_API_KEY # Export environment variables
Environment Setup
module purge # Clean environment
module load gcc/12.3.0 # Load GCC
module load cuda/12.6.3 # Load CUDA
unset CUDA_HOME # Avoid conflicts
unset LD_LIBRARY_PATH
source .venv/bin/activate # Activate virtual env
# Cache directories (required!)
export TORCH_EXTENSIONS_DIR="/scratch/.../torch_extensions"
export TRITON_CACHE_DIR="/scratch/.../triton"
export HF_HOME="/scratch/.../huggingface"
export WANDB_MODE=offline # Offline mode (no internet)
export DEBUG_MODE="1"
export CUDA_VISIBLE_DEVICES="0,1"
Training Job Submission
SFT (2 GPUs) - Quick Debug
sbatch scripts/submit_1_node_2_gpus_dpspd2_sft_llada.slurm
Resources:
GPU: 2×A100 (40GB/80GB)
CPU: 24 cores
Memory: 512GB
Time: 24 hours
Use for:
Small datasets (<100k)
Fast iteration
Debugging data pipeline
SFT (4 GPUs) - Standard (Recommended)
sbatch scripts/submit_1_node_4_gpus_dpspd2_sft_llada.slurm
Resources:
GPU: 4×A100
Effective batch size:
1 × 8(accum) × 4(GPU) = 32100k data ≈ 2-3 hours
Use for:
Standard SFT training
Best throughput/memory balance
Edit-based SFT
# Standard edit fine-tuning
sbatch scripts/submit_1_node_4_gpus_dpspd2_sft_edit_llada.slurm
# Harmful prompt only (recommended)
sbatch scripts/submit_1_node_4_gpus_dpspd2_safety_continue_training_only_harmfulprompt_edit_llada.slurm
# Multi-turn high temp + harmful only
sbatch scripts/submit_1_node_4_gpus_dpspd2_safety_continue_training_multiturn_hightemp_only_harmfulprompt_edit_llada.slurm
CPT (8 GPUs) - Large Scale
sbatch scripts/submit_1_node_8_gpus_dpspd2_cpt_latent_llada.slurm
Resources:
GPU: 8×A100/H100
Long-context data (4k+ tokens)
Typical runtime: 12-24 hours
Evaluation Job Submission
Benchmark Evaluation (eval_bench)
Script: submit_1_node_2_gpus_eval_bench.slurm
Basic usage:
# Run all 4 datasets
sbatch scripts/submit_1_over_2_gpus_eval_bench.slurm
Default datasets:
MBPP (code)
LiveBench (code)
GSM8K (math)
MATH500 (math)
Parameter Overrides
Method 1: Positional arguments
sbatch scripts/submit_1_node_2_gpus_eval_bench.slurm \
configs/llada_eval.yaml \ # $1: config file
"debug_mbpp" \ # $2: run_name
"/path/to/model" \ # $3: model_path
0.95 \ # $4: edit_threshold
5 \ # $5: post_unmask_steps
"MBPP" # $6: datasets (comma-separated)
Method 2: Environment variables
sbatch --export=ALL, \
RUN_NAME=debug_run, \
MODEL_PATH=/path/to/model, \
EDIT_THRESHOLD=0.95, \
POST_UNMASK_STEPS=5, \
DATASETS=MBPP,GSM8K \
scripts/submit_1_node_2_gpus_eval_bench.slurm
Complete Examples
Example 1: Quick MBPP test with edit model
sbatch submit_1_node_2_gpus_eval_bench.slurm \
configs/llada_eval.yaml \
"mbpp_edit_test" \
"/scratch/models/llada_edit_095" \
0.95 5 MBPP
Example 2: Math datasets, no edit
sbatch submit_1_node_2_gpus_eval_bench.slurm \
configs/llada_eval.yaml \
"math_base" \
"/scratch/models/llada_base" \
-1 0 "GSM8K,MATH500"
Example 3: All datasets via –export
sbatch --export=ALL, \
RUN_NAME=all_benchmarks, \
MODEL_PATH=/path/to/model, \
EDIT_THRESHOLD=-1, \
POST_UNMASK_STEPS=0, \
DATASETS=MBPP,GSM8K,MATH500,LiveBench \
submit_1_node_2_gpus_eval_bench.slurm
Parameter Reference (eval_bench)
Position |
Variable |
Type |
Default |
Description |
|---|---|---|---|---|
$1 |
EVAL_CONFIG |
str |
|
YAML config path |
$2 |
RUN_NAME |
str |
YAML value |
Experiment name |
$3 |
MODEL_PATH |
str |
YAML value |
Model checkpoint path |
$4 |
EDIT_THRESHOLD |
float |
YAML value |
Edit threshold (-1 or 0.85-0.95) |
$5 |
POST_UNMASK_STEPS |
int |
YAML value |
Post-unmask steps |
$6 |
DATASETS_CSV |
str |
|
Comma-separated dataset list |
Dataset mapping (internal):
MBPP → code
LiveBench → code
GSM8K → math
MATH500 → math
Safety Evaluation (eval_safety)
Script: submit_1_node_2_gpus_eval_safety.slurm
Basic usage:
# Run all attack types (null, prefill, pair, renellm)
sbatch scripts/submit_1_node_2_gpus_eval_safety.slurm
Parameter overrides:
sbatch submit_1_node_2_gpus_eval_safety.slurm \
configs/llada_eval.yaml \ # $1
"safety_test" \ # $2: run_name
"/path/to/model" \ # $3: model_path
0.95 \ # $4: edit_threshold
5 \ # $5: post_unmask_steps
"pair,renellm" \ # $6: attack_types (comma-separated)
50 # $7: max_samples
Via environment:
sbatch --export=ALL, \
RUN_NAME=safety_eval, \
MODEL_PATH=/path/to/model, \
EDIT_THRESHOLD=0.9, \
POST_UNMASK_STEPS=5, \
ATTACK_TYPES=prefill, \
MAX_SAMPLES=100 \
submit_1_node_2_gpus_eval_safety.slurm
Complete Examples
Example 1: Quick prefill test
sbatch submit_1_node_2_gpus_eval_safety.slurm \
configs/llada_eval.yaml \
"prefill_quick" \
"/scratch/models/llada_base" \
-1 0 prefill 10
Example 2: PAIR attack on edit model
sbatch submit_1_node_2_gpus_eval_safety.slurm \
configs/llada_eval.yaml \
"pair_edit_095" \
"/scratch/models/llada_edit_095" \
0.95 5 pair 50
Example 3: Compare edit thresholds
for thr in 0.85 0.90 0.95; do
sbatch submit_1_node_2_gpus_eval_safety.slurm \
configs/llada_eval.yaml \
"thr_${thr}" \
"/path/to/edit_model" \
$thr 5 "all" 100
done
Parameter Reference (eval_safety)
Position |
Variable |
Type |
Default |
Description |
|---|---|---|---|---|
$1 |
EVAL_CONFIG |
str |
|
Config file |
$2 |
RUN_NAME |
str |
YAML value |
Experiment name |
$3 |
MODEL_PATH |
str |
YAML value |
Model path |
$4 |
EDIT_THRESHOLD |
float |
YAML value |
Edit threshold |
$5 |
POST_UNMASK_STEPS |
int |
YAML value |
Post-unmask steps |
$6 |
ATTACK_TYPES |
str |
|
Attack types (comma-separated or |
$7 |
MAX_SAMPLES |
int |
|
Max samples |
Supported attack types:
null- Zeroshot (no attack)prefill- Prefill attackpair- PAIR attackrenellm- ReNeLLM attack
Job Management
Submit Job
sbatch script.slurm
# Output: Submitted batch job 123456
Check Status
# All my jobs
squeue -u $USER
# Detailed
squeue -j 123456 --format="%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"
# Columns: JobID, Partition, Name, User, State, Time, Nodes, Reason
Cancel Job
# Single job
scancel 123456
# All my jobs
scancel -u $USER
# By name
scancel -n "eval_safety"
Job Details
# Job statistics
sacct -j 123456 --format=JobID,State,Elapsed,MaxRSS,AllocTRES
Monitor Resources
# Node status
sinfo -o "%15P %.5D %.10a %.10l %G" | grep a100
# GPU usage (on compute node)
nvidia-smi
# Via scontrol
scontrol show node $NODENAME
Logs & Output
Standard Output/Error
Default locations:
slurm-123456.out # stdout
slurm-123456.err # stderr
Custom locations (in script):
#SBATCH --output=/scratch/.../logs/%j_%x.out
#SBATCH --error=/scratch/.../logs/%j_%x.err
# %j = Job ID, %x = Job name, %A = Array job ID
Training Logs
/scratch/aszalay1/tianze/safe_dllm/
├── logs/
│ └── train.log
├── wandb/
│ └── offline-run-xxx/
└── checkpoints/
└── sft_llada_xxx/
Real-time Log Viewing
tail -f /path/to/slurm-123456.out
# Or with watch
watch -n 1 "tail /path/to/slurm-123456.out | tail -20"
Common Slurm Issues
Job Stuck in Pending
Causes:
Required resources unavailable (GPU type/quantity)
No permission for partition
Job limit reached
Troubleshoot:
# Check partition status
sinfo -p a100
# Check your limits
sacctmgr show assoc user=$USER
# Pending reason
squeue -j 123456 --format="%R"
Solutions:
Adjust
--partition(a100 → nvl)Reduce
--gres=gpucountWait for resources
Job Killed (COMPLETED with non-zero exit)
Causes:
Timeout (
--timetoo short)Memory insufficient (
--memtoo small)GPU OOM
Troubleshoot:
# Exit code
sacct -j 123456 --format=JobID,ExitCode
# Resource usage
sacct -j 123456 --format=JobID,MaxRSS,MaxVMSize,Elapsed
Environment Variables Not Passed
Problem: os.environ["OPENAI_API_KEY"] is None in Python
Solutions:
# Method 1: --export flag
#SBATCH --export=ALL,OPENAI_API_KEY
# Method 2: submit with --export
sbatch --export=ALL,OPENAI_API_KEY=$OPENAI_API_KEY script.slurm
# Method 3: export inside script (recommended)
export OPENAI_API_KEY="$OPENAI_API_KEY"
Module Load Failures
# List available modules
module avail
# Search for module
module spider cuda
# Load default version
module load cuda # instead of cuda/12.6.3
# List loaded modules
module list
Utility Scripts
Batch Submission with Different Parameters
#!/bin/bash
# Test different edit_threshold values
for thr in -1 0.85 0.90 0.95; do
sbatch scripts/submit_1_node_2_gpus_eval_safety.slurm \
configs/llada_eval.yaml \
"thr_${thr}" \
"/path/to/model" \
$thr 5 "prefill" 100
sleep 5 # Avoid submitting too fast
done
Job Status Monitor
#!/bin/bash
# Monitor script
while true; do
clear
echo "=== $(date) ==="
squeue -u $USER --format="%.18i %.9P %.8j %.8u %.2t %.10M %.6D"
sleep 30
done
Auto-Retry Failed Jobs
#!/bin/bash
# Check job status, resubmit if failed
JOBID=$1
STATUS=$(sacct -j $JOBID --format=State --noheader | head -1)
if [[ "$STATUS" == "FAILED" ]]; then
echo "Job $JOBID failed, resubmitting..."
# Resubmit with saved command
fi
Best Practices
Resource Allocation Guidelines
Task Type |
GPU |
CPU |
Memory |
Time |
Notes |
|---|---|---|---|---|---|
SFT debug |
2×A100 |
24 |
256G |
4h |
Small data |
SFT standard |
4×A100 |
24 |
512G |
24h |
100k data |
CPT |
8×A100 |
48 |
1T |
48h |
Large data |
Safety eval |
2×A100 |
24 |
256G |
8h |
HarmBench 400 samples |
Attack testing |
2×A100 |
24 |
512G |
24h |
PAIR/ReNeLLM slower |
Naming Conventions
run_name: "sft_llada_{data}_{model}_{date}"
Examples:
sft_llada_mixed100k_base_260414eval_harmbench_edit095_260415pair_attack_llada_sft_260416
Directory Organization
/scratch/aszalay1/tianze/safe_dllm/
├── checkpoints/
│ └── sft_llada_mixed_100k_260414/
│ └── checkpoint-500/
├── results/
│ ├── benchmark/
│ │ ├── harmbench_llada_base_260414.jsonl
│ │ └── harmbench_llada_sft_260415.jsonl
│ └── attack/
│ ├── pair_attack_260416.jsonl
│ └── renellm_cache/
├── logs/
│ ├── train_260414.log
│ └── eval_260415.log
└── wandb/
└── offline-run-xxx/
Debugging Slurm Jobs
Interactive Debugging
# Allocate interactive job (2 GPUs, 1 hour)
srun --gres=gpu:2 --mem=100G --time=1:00:00 --pty bash
# Run inside interactive shell
python train/sft_llada.py config=configs/sft_llada.yaml
Dry-run Test
# Test script syntax
sbatch --test-only script.slurm
# Check for errors after parsing
sbatch --export=ALL,MODEL_PATH=/test script.slurm 2>&1 | grep -i error
Environment Check
Add to top of job script:
echo "CUDA available: $(python -c 'import torch; print(torch.cuda.is_available())')"
echo "GPUs: $(nvidia-smi --list-gpus | wc -l)"
python -c "import torch; print(f'CUDA version: {torch.version.cuda}')"
See Also
guide/training - Training workflows
guide/evaluation - Evaluation setup
guide/attacks - Attack methods
guide/config - Configuration files