Slurm Jobs

Complete guide to submitting and managing Slurm jobs for Safe dLLM.

Slurm Scripts Overview

scripts/
├── submit_1_node_2_gpus_dpspd2_sft_llada.slurm           # SFT (2 GPUs)
├── submit_1_node_4_gpus_dpspd2_sft_llada.slurm           # SFT (4 GPUs)
├── submit_1_node_4_gpus_dpspd2_sft_edit_llada.slurm      # Edit SFT (4 GPUs)
├── submit_1_node_8_gpus_dpspd2_cpt_latent_llada.slurm    # CPT (8 GPUs)
├── submit_1_node_2_gpus_eval_bench.slurm                 # Benchmark eval (2 GPUs)
└── submit_1_node_2_gpus_eval_safety.slurm                # Safety eval + attacks (2 GPUs)

Naming convention:

  • 1_node_2_gpus - Single node, 2 GPUs

  • dpspd2 - DP+SPD2 data parallel (DistributedDataParallel + Sharded)

  • eval_bench - Benchmark evaluation (MBPP/GSM8K/MATH/LiveBench)

  • eval_safety - Safety evaluation (HarmBench/WildGuardMix + attacks)

SBATCH Directives

Common header across all scripts:

#!/bin/bash
#SBATCH --job-name=eval_safety         # Job name (shown in squeue)
#SBATCH --partition=a100              # GPU partition (a100/h100/nvl)
#SBATCH --account=aszalay1_ssci       # Billing account
#SBATCH --nodes=1                     # Number of nodes
#SBATCH --gres=gpu:2                  # Number of GPUs
#SBATCH --cpus-per-task=24            # CPU cores per task
#SBATCH --mem=512G                    # Memory (GB)
#SBATCH --time=24:00:00              # Max runtime (DD-HH:MM:SS)
#SBATCH --exclude=c001,h01           # Nodes to exclude
#SBATCH --export=ALL,OPENAI_API_KEY  # Export environment variables

Environment Setup

module purge                          # Clean environment
module load gcc/12.3.0               # Load GCC
module load cuda/12.6.3              # Load CUDA

unset CUDA_HOME                      # Avoid conflicts
unset LD_LIBRARY_PATH

source .venv/bin/activate            # Activate virtual env

# Cache directories (required!)
export TORCH_EXTENSIONS_DIR="/scratch/.../torch_extensions"
export TRITON_CACHE_DIR="/scratch/.../triton"
export HF_HOME="/scratch/.../huggingface"
export WANDB_MODE=offline            # Offline mode (no internet)
export DEBUG_MODE="1"
export CUDA_VISIBLE_DEVICES="0,1"

Training Job Submission

SFT (2 GPUs) - Quick Debug

sbatch scripts/submit_1_node_2_gpus_dpspd2_sft_llada.slurm

Resources:

  • GPU: 2×A100 (40GB/80GB)

  • CPU: 24 cores

  • Memory: 512GB

  • Time: 24 hours

Use for:

  • Small datasets (<100k)

  • Fast iteration

  • Debugging data pipeline

Edit-based SFT

# Standard edit fine-tuning
sbatch scripts/submit_1_node_4_gpus_dpspd2_sft_edit_llada.slurm

# Harmful prompt only (recommended)
sbatch scripts/submit_1_node_4_gpus_dpspd2_safety_continue_training_only_harmfulprompt_edit_llada.slurm

# Multi-turn high temp + harmful only
sbatch scripts/submit_1_node_4_gpus_dpspd2_safety_continue_training_multiturn_hightemp_only_harmfulprompt_edit_llada.slurm

CPT (8 GPUs) - Large Scale

sbatch scripts/submit_1_node_8_gpus_dpspd2_cpt_latent_llada.slurm

Resources:

  • GPU: 8×A100/H100

  • Long-context data (4k+ tokens)

  • Typical runtime: 12-24 hours

Evaluation Job Submission

Benchmark Evaluation (eval_bench)

Script: submit_1_node_2_gpus_eval_bench.slurm

Basic usage:

# Run all 4 datasets
sbatch scripts/submit_1_over_2_gpus_eval_bench.slurm

Default datasets:

  • MBPP (code)

  • LiveBench (code)

  • GSM8K (math)

  • MATH500 (math)

Parameter Overrides

Method 1: Positional arguments

sbatch scripts/submit_1_node_2_gpus_eval_bench.slurm \
    configs/llada_eval.yaml \      # $1: config file
    "debug_mbpp" \                 # $2: run_name
    "/path/to/model" \             # $3: model_path
    0.95 \                         # $4: edit_threshold
    5 \                            # $5: post_unmask_steps
    "MBPP"                         # $6: datasets (comma-separated)

Method 2: Environment variables

sbatch --export=ALL, \
    RUN_NAME=debug_run, \
    MODEL_PATH=/path/to/model, \
    EDIT_THRESHOLD=0.95, \
    POST_UNMASK_STEPS=5, \
    DATASETS=MBPP,GSM8K \
    scripts/submit_1_node_2_gpus_eval_bench.slurm

Complete Examples

Example 1: Quick MBPP test with edit model

sbatch submit_1_node_2_gpus_eval_bench.slurm \
    configs/llada_eval.yaml \
    "mbpp_edit_test" \
    "/scratch/models/llada_edit_095" \
    0.95 5 MBPP

Example 2: Math datasets, no edit

sbatch submit_1_node_2_gpus_eval_bench.slurm \
    configs/llada_eval.yaml \
    "math_base" \
    "/scratch/models/llada_base" \
    -1 0 "GSM8K,MATH500"

Example 3: All datasets via –export

sbatch --export=ALL, \
    RUN_NAME=all_benchmarks, \
    MODEL_PATH=/path/to/model, \
    EDIT_THRESHOLD=-1, \
    POST_UNMASK_STEPS=0, \
    DATASETS=MBPP,GSM8K,MATH500,LiveBench \
    submit_1_node_2_gpus_eval_bench.slurm

Parameter Reference (eval_bench)

Position

Variable

Type

Default

Description

$1

EVAL_CONFIG

str

configs/llada_eval.yaml

YAML config path

$2

RUN_NAME

str

YAML value

Experiment name

$3

MODEL_PATH

str

YAML value

Model checkpoint path

$4

EDIT_THRESHOLD

float

YAML value

Edit threshold (-1 or 0.85-0.95)

$5

POST_UNMASK_STEPS

int

YAML value

Post-unmask steps

$6

DATASETS_CSV

str

all

Comma-separated dataset list

Dataset mapping (internal):

MBPP → code
LiveBench → code
GSM8K → math
MATH500 → math

Safety Evaluation (eval_safety)

Script: submit_1_node_2_gpus_eval_safety.slurm

Basic usage:

# Run all attack types (null, prefill, pair, renellm)
sbatch scripts/submit_1_node_2_gpus_eval_safety.slurm

Parameter overrides:

sbatch submit_1_node_2_gpus_eval_safety.slurm \
    configs/llada_eval.yaml \      # $1
    "safety_test" \                # $2: run_name
    "/path/to/model" \             # $3: model_path
    0.95 \                         # $4: edit_threshold
    5 \                            # $5: post_unmask_steps
    "pair,renellm" \               # $6: attack_types (comma-separated)
    50                             # $7: max_samples

Via environment:

sbatch --export=ALL, \
    RUN_NAME=safety_eval, \
    MODEL_PATH=/path/to/model, \
    EDIT_THRESHOLD=0.9, \
    POST_UNMASK_STEPS=5, \
    ATTACK_TYPES=prefill, \
    MAX_SAMPLES=100 \
    submit_1_node_2_gpus_eval_safety.slurm

Complete Examples

Example 1: Quick prefill test

sbatch submit_1_node_2_gpus_eval_safety.slurm \
    configs/llada_eval.yaml \
    "prefill_quick" \
    "/scratch/models/llada_base" \
    -1 0 prefill 10

Example 2: PAIR attack on edit model

sbatch submit_1_node_2_gpus_eval_safety.slurm \
    configs/llada_eval.yaml \
    "pair_edit_095" \
    "/scratch/models/llada_edit_095" \
    0.95 5 pair 50

Example 3: Compare edit thresholds

for thr in 0.85 0.90 0.95; do
    sbatch submit_1_node_2_gpus_eval_safety.slurm \
        configs/llada_eval.yaml \
        "thr_${thr}" \
        "/path/to/edit_model" \
        $thr 5 "all" 100
done

Parameter Reference (eval_safety)

Position

Variable

Type

Default

Description

$1

EVAL_CONFIG

str

configs/llada_eval.yaml

Config file

$2

RUN_NAME

str

YAML value

Experiment name

$3

MODEL_PATH

str

YAML value

Model path

$4

EDIT_THRESHOLD

float

YAML value

Edit threshold

$5

POST_UNMASK_STEPS

int

YAML value

Post-unmask steps

$6

ATTACK_TYPES

str

all

Attack types (comma-separated or all)

$7

MAX_SAMPLES

int

null

Max samples

Supported attack types:

  • null - Zeroshot (no attack)

  • prefill - Prefill attack

  • pair - PAIR attack

  • renellm - ReNeLLM attack

Job Management

Submit Job

sbatch script.slurm
# Output: Submitted batch job 123456

Check Status

# All my jobs
squeue -u $USER

# Detailed
squeue -j 123456 --format="%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"

# Columns: JobID, Partition, Name, User, State, Time, Nodes, Reason

Cancel Job

# Single job
scancel 123456

# All my jobs
scancel -u $USER

# By name
scancel -n "eval_safety"

Job Details

# Job statistics
sacct -j 123456 --format=JobID,State,Elapsed,MaxRSS,AllocTRES

Monitor Resources

# Node status
sinfo -o "%15P %.5D %.10a %.10l %G" | grep a100

# GPU usage (on compute node)
nvidia-smi

# Via scontrol
scontrol show node $NODENAME

Logs & Output

Standard Output/Error

Default locations:

slurm-123456.out  # stdout
slurm-123456.err  # stderr

Custom locations (in script):

#SBATCH --output=/scratch/.../logs/%j_%x.out
#SBATCH --error=/scratch/.../logs/%j_%x.err

# %j = Job ID, %x = Job name, %A = Array job ID

Training Logs

/scratch/aszalay1/tianze/safe_dllm/
├── logs/
│   └── train.log
├── wandb/
│   └── offline-run-xxx/
└── checkpoints/
    └── sft_llada_xxx/

Real-time Log Viewing

tail -f /path/to/slurm-123456.out

# Or with watch
watch -n 1 "tail /path/to/slurm-123456.out | tail -20"

Common Slurm Issues

Job Stuck in Pending

Causes:

  • Required resources unavailable (GPU type/quantity)

  • No permission for partition

  • Job limit reached

Troubleshoot:

# Check partition status
sinfo -p a100

# Check your limits
sacctmgr show assoc user=$USER

# Pending reason
squeue -j 123456 --format="%R"

Solutions:

  • Adjust --partition (a100 → nvl)

  • Reduce --gres=gpu count

  • Wait for resources

Job Killed (COMPLETED with non-zero exit)

Causes:

  • Timeout (--time too short)

  • Memory insufficient (--mem too small)

  • GPU OOM

Troubleshoot:

# Exit code
sacct -j 123456 --format=JobID,ExitCode

# Resource usage
sacct -j 123456 --format=JobID,MaxRSS,MaxVMSize,Elapsed

Environment Variables Not Passed

Problem: os.environ["OPENAI_API_KEY"] is None in Python

Solutions:

# Method 1: --export flag
#SBATCH --export=ALL,OPENAI_API_KEY

# Method 2: submit with --export
sbatch --export=ALL,OPENAI_API_KEY=$OPENAI_API_KEY script.slurm

# Method 3: export inside script (recommended)
export OPENAI_API_KEY="$OPENAI_API_KEY"

Module Load Failures

# List available modules
module avail

# Search for module
module spider cuda

# Load default version
module load cuda  # instead of cuda/12.6.3

# List loaded modules
module list

Utility Scripts

Batch Submission with Different Parameters

#!/bin/bash
# Test different edit_threshold values

for thr in -1 0.85 0.90 0.95; do
    sbatch scripts/submit_1_node_2_gpus_eval_safety.slurm \
        configs/llada_eval.yaml \
        "thr_${thr}" \
        "/path/to/model" \
        $thr 5 "prefill" 100
    sleep 5  # Avoid submitting too fast
done

Job Status Monitor

#!/bin/bash
# Monitor script
while true; do
    clear
    echo "=== $(date) ==="
    squeue -u $USER --format="%.18i %.9P %.8j %.8u %.2t %.10M %.6D"
    sleep 30
done

Auto-Retry Failed Jobs

#!/bin/bash
# Check job status, resubmit if failed
JOBID=$1
STATUS=$(sacct -j $JOBID --format=State --noheader | head -1)

if [[ "$STATUS" == "FAILED" ]]; then
    echo "Job $JOBID failed, resubmitting..."
    # Resubmit with saved command
fi

Best Practices

Resource Allocation Guidelines

Task Type

GPU

CPU

Memory

Time

Notes

SFT debug

2×A100

24

256G

4h

Small data

SFT standard

4×A100

24

512G

24h

100k data

CPT

8×A100

48

1T

48h

Large data

Safety eval

2×A100

24

256G

8h

HarmBench 400 samples

Attack testing

2×A100

24

512G

24h

PAIR/ReNeLLM slower

Naming Conventions

run_name: "sft_llada_{data}_{model}_{date}"

Examples:

  • sft_llada_mixed100k_base_260414

  • eval_harmbench_edit095_260415

  • pair_attack_llada_sft_260416

Directory Organization

/scratch/aszalay1/tianze/safe_dllm/
├── checkpoints/
│   └── sft_llada_mixed_100k_260414/
│       └── checkpoint-500/
├── results/
│   ├── benchmark/
│   │   ├── harmbench_llada_base_260414.jsonl
│   │   └── harmbench_llada_sft_260415.jsonl
│   └── attack/
│       ├── pair_attack_260416.jsonl
│       └── renellm_cache/
├── logs/
│   ├── train_260414.log
│   └── eval_260415.log
└── wandb/
    └── offline-run-xxx/

Debugging Slurm Jobs

Interactive Debugging

# Allocate interactive job (2 GPUs, 1 hour)
srun --gres=gpu:2 --mem=100G --time=1:00:00 --pty bash

# Run inside interactive shell
python train/sft_llada.py config=configs/sft_llada.yaml

Dry-run Test

# Test script syntax
sbatch --test-only script.slurm

# Check for errors after parsing
sbatch --export=ALL,MODEL_PATH=/test script.slurm 2>&1 | grep -i error

Environment Check

Add to top of job script:

echo "CUDA available: $(python -c 'import torch; print(torch.cuda.is_available())')"
echo "GPUs: $(nvidia-smi --list-gpus | wc -l)"
python -c "import torch; print(f'CUDA version: {torch.version.cuda}')"

See Also

  • guide/training - Training workflows

  • guide/evaluation - Evaluation setup

  • guide/attacks - Attack methods

  • guide/config - Configuration files