Models

API reference for LLaDA model implementations.

class models.llada.modeling_llada.LayerNormBase(config: ModelConfig, *, size: int | None = None, elementwise_affine: bool | None = True, eps: float = 1e-05)[source]

Bases: Module

__init__(config: ModelConfig, *, size: int | None = None, elementwise_affine: bool | None = True, eps: float = 1e-05)[source]

Initialize internal Module state, shared by both nn.Module and ScriptModule.

abstractmethod forward(x: Tensor) → Tensor[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

classmethod build(config: ModelConfig, size: int | None = None, **kwargs) → LayerNormBase[source]
reset_parameters()[source]
class models.llada.modeling_llada.LayerNorm(config: ModelConfig, size: int | None = None, low_precision: bool = False, elementwise_affine: bool | None = None, eps: float = 1e-05)[source]

Bases: LayerNormBase

The default LayerNorm implementation which can optionally run in low precision.

__init__(config: ModelConfig, size: int | None = None, low_precision: bool = False, elementwise_affine: bool | None = None, eps: float = 1e-05)[source]

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x: Tensor) → Tensor[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class models.llada.modeling_llada.RMSLayerNorm(config: ModelConfig, size: int | None = None, elementwise_affine: bool | None = None, eps: float = 1e-05)[source]

Bases: LayerNormBase

RMS layer norm, a simplified LayerNorm implementation

__init__(config: ModelConfig, size: int | None = None, elementwise_affine: bool | None = None, eps: float = 1e-05)[source]

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x: Tensor) → Tensor[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class models.llada.modeling_llada.GemmaRMSLayerNorm(config: ModelConfig, size: int | None = None, elementwise_affine: bool | None = None, eps: float = 1e-05)[source]

Bases: LayerNormBase

Gemma RMS layer norm, a simplified LayerNorm implementation

__init__(config: ModelConfig, size: int | None = None, elementwise_affine: bool | None = None, eps: float = 1e-05)[source]

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x: Tensor) → Tensor[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class models.llada.modeling_llada.RotaryEmbedding(config: ModelConfig, cache: BufferCache)[source]

Bases: Module

[Rotary positional embeddings (RoPE)](https://arxiv.org/abs/2104.09864).

__init__(config: ModelConfig, cache: BufferCache)[source]

Initialize internal Module state, shared by both nn.Module and ScriptModule.

get_rotary_embedding(seq_len: int, device: device) → Tuple[Tensor, Tensor][source]
rotate_half(x: Tensor) → Tensor[source]
apply_rotary_pos_emb(pos_sin: Tensor, pos_cos: Tensor, t: Tensor) → Tensor[source]
forward(q: Tensor, k: Tensor) → Tuple[Tensor, Tensor][source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class models.llada.modeling_llada.Activation(config: ModelConfig)[source]

Bases: Module

__init__(config: ModelConfig)[source]

Initialize internal Module state, shared by both nn.Module and ScriptModule.

abstractmethod forward(x: Tensor) → Tensor[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

abstract property output_multiplier: float
classmethod build(config: ModelConfig) → Activation[source]
class models.llada.modeling_llada.GELU(approximate: str = 'none')[source]

Bases: GELU

property output_multiplier: float
class models.llada.modeling_llada.ReLU(inplace: bool = False)[source]

Bases: ReLU

property output_multiplier: float
class models.llada.modeling_llada.SwiGLU(config: ModelConfig)[source]

Bases: Activation

forward(x: Tensor) → Tensor[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

property output_multiplier: float
class models.llada.modeling_llada.LLaDABlock(layer_id: int, config: ModelConfig, cache: BufferCache)[source]

Bases: Module

A base class for transformer block implementations.

__init__(layer_id: int, config: ModelConfig, cache: BufferCache)[source]

Initialize internal Module state, shared by both nn.Module and ScriptModule.

reset_parameters()[source]
set_activation_checkpointing(strategy: ActivationCheckpointingStrategy | None)[source]
attention(q: Tensor, k: Tensor, v: Tensor, attention_bias: Tensor | None = None, layer_past: Tuple[Tensor, Tensor] | None = None, use_cache: bool = False) → Tuple[Tensor, Tuple[Tensor, Tensor] | None][source]
abstractmethod forward(x: Tensor, attention_bias: FloatTensor | None = None, layer_past: Tuple[Tensor, Tensor] | None = None, use_cache: bool = False) → Tuple[Tensor, Tuple[Tensor, Tensor] | None][source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

classmethod build(layer_id: int, config: ModelConfig, cache: BufferCache) → LLaDABlock[source]
class models.llada.modeling_llada.LLaDASequentialBlock(layer_id: int, config: ModelConfig, cache: BufferCache)[source]

Bases: LLaDABlock

This is a typical transformer block where the output is computed as MLP(LN(x + Attention(LN(x)))) (plus another skip connection).

__init__(layer_id: int, config: ModelConfig, cache: BufferCache)[source]

Initialize internal Module state, shared by both nn.Module and ScriptModule.

reset_parameters()[source]
forward(x: Tensor, attention_bias: Tensor | None = None, layer_past: Tuple[Tensor, Tensor] | None = None, use_cache: bool = False) → Tuple[Tensor, Tuple[Tensor, Tensor] | None][source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class models.llada.modeling_llada.LLaDAModel(config: ModelConfig, init_params: bool = True)[source]

Bases: Module

__init__(config: ModelConfig, init_params: bool = True)[source]

Initialize internal Module state, shared by both nn.Module and ScriptModule.

set_activation_checkpointing(strategy: ActivationCheckpointingStrategy | None)[source]
property device: device
reset_parameters()[source]
get_alibi_attention_bias(seq_len: int, device: device) → Tensor[source]
forward(input_ids: LongTensor, input_embeddings: FloatTensor | None = None, attention_mask: Tensor | None = None, attention_bias: Tensor | None = None, past_key_values: Sequence[Tuple[Tensor, Tensor]] | None = None, use_cache: bool = False, last_logits_only: bool = False, output_hidden_states: bool | None = None) → LLaDAOutput[source]
Parameters:
  • input_ids – A tensor of shape (batch_size, seq_len).

  • input_embeddings – A tensor of shape (batch_size, seq_len, d_model) with input embeddings. When provided, it is treated as the output of the input embedding layer.

  • attention_mask –

    A tensor of shape (batch_size, seq_len) that indicates which input IDs are masked. A 1 value in the mask means that the corresponding input ID should not be ignored. A 0 means that the corresponding input ID is masked.

    This has the same meaning as the attention_mask in HuggingFace’s transformers library.

  • attention_bias –

    A tensor of shape (batch_size, 1, seq_len, seq_len), (1, 1, seq_len, seq_len), or (seq_len, seq_len). This is used to introduce causal or other biases.

    If the tensor is a bool or byte tensor, a True or 1 at attention_bias[:, :, i, j] indicates that the i-th element in the sequence is allowed to attend to the j-th element in the sequence.

    If the tensor is a float tensor, it will just be added to the attention scores before the softmax.

    The default is causal, which corresponds to a lower-diagonal byte matrix of ones.

  • past_key_values – Pre-computed keys and values for each attention block. Can be used to speed up sequential decoding. The input_ids which have their past given to this model should not be passed as input_ids as they have already been computed.

  • use_cache – If True, return key and value tensors for each block.

  • last_logits_only – If True, only compute the logits for the last token of each sequence. This can speed up decoding when you only care about the next token.

class models.llada.modeling_llada.LLaDAOutput(logits, attn_key_values, hidden_states)[source]

Bases: NamedTuple

logits: FloatTensor

A tensor of shape (batch_size, seq_len, vocab_size) representing the log probabilities for the next token before normalization via (log) softmax.

attn_key_values: List[Tuple[Tensor, Tensor]] | None

Attention keys and values from each block.

hidden_states: Tuple[Tensor] | None

Hidden states from each block.

class models.llada.modeling_llada.LLaDAGenerateOutput(token_ids, scores)[source]

Bases: NamedTuple

token_ids: LongTensor

The generated token IDs, a tensor of shape (batch_size, beam_size, max_steps). These do not include the original input IDs.

scores: FloatTensor

The scores of the generated sequences, a tensor of shape (batch_size, beam_size).