Models
API reference for LLaDA model implementations.
- class models.llada.modeling_llada.LayerNormBase(config: ModelConfig, *, size: int | None = None, elementwise_affine: bool | None = True, eps: float = 1e-05)[source]
Bases:
Module- __init__(config: ModelConfig, *, size: int | None = None, elementwise_affine: bool | None = True, eps: float = 1e-05)[source]
Initialize internal Module state, shared by both nn.Module and ScriptModule.
- abstractmethod forward(x: Tensor) Tensor[source]
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- classmethod build(config: ModelConfig, size: int | None = None, **kwargs) LayerNormBase[source]
- class models.llada.modeling_llada.LayerNorm(config: ModelConfig, size: int | None = None, low_precision: bool = False, elementwise_affine: bool | None = None, eps: float = 1e-05)[source]
Bases:
LayerNormBaseThe default
LayerNormimplementation which can optionally run in low precision.- __init__(config: ModelConfig, size: int | None = None, low_precision: bool = False, elementwise_affine: bool | None = None, eps: float = 1e-05)[source]
Initialize internal Module state, shared by both nn.Module and ScriptModule.
- forward(x: Tensor) Tensor[source]
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class models.llada.modeling_llada.RMSLayerNorm(config: ModelConfig, size: int | None = None, elementwise_affine: bool | None = None, eps: float = 1e-05)[source]
Bases:
LayerNormBaseRMS layer norm, a simplified
LayerNormimplementation- __init__(config: ModelConfig, size: int | None = None, elementwise_affine: bool | None = None, eps: float = 1e-05)[source]
Initialize internal Module state, shared by both nn.Module and ScriptModule.
- forward(x: Tensor) Tensor[source]
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class models.llada.modeling_llada.GemmaRMSLayerNorm(config: ModelConfig, size: int | None = None, elementwise_affine: bool | None = None, eps: float = 1e-05)[source]
Bases:
LayerNormBaseGemma RMS layer norm, a simplified
LayerNormimplementation- __init__(config: ModelConfig, size: int | None = None, elementwise_affine: bool | None = None, eps: float = 1e-05)[source]
Initialize internal Module state, shared by both nn.Module and ScriptModule.
- forward(x: Tensor) Tensor[source]
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class models.llada.modeling_llada.RotaryEmbedding(config: ModelConfig, cache: BufferCache)[source]
Bases:
Module[Rotary positional embeddings (RoPE)](https://arxiv.org/abs/2104.09864).
- __init__(config: ModelConfig, cache: BufferCache)[source]
Initialize internal Module state, shared by both nn.Module and ScriptModule.
- forward(q: Tensor, k: Tensor) Tuple[Tensor, Tensor][source]
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class models.llada.modeling_llada.Activation(config: ModelConfig)[source]
Bases:
Module- __init__(config: ModelConfig)[source]
Initialize internal Module state, shared by both nn.Module and ScriptModule.
- abstractmethod forward(x: Tensor) Tensor[source]
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- classmethod build(config: ModelConfig) Activation[source]
- class models.llada.modeling_llada.SwiGLU(config: ModelConfig)[source]
Bases:
Activation- forward(x: Tensor) Tensor[source]
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class models.llada.modeling_llada.LLaDABlock(layer_id: int, config: ModelConfig, cache: BufferCache)[source]
Bases:
ModuleA base class for transformer block implementations.
- __init__(layer_id: int, config: ModelConfig, cache: BufferCache)[source]
Initialize internal Module state, shared by both nn.Module and ScriptModule.
- attention(q: Tensor, k: Tensor, v: Tensor, attention_bias: Tensor | None = None, layer_past: Tuple[Tensor, Tensor] | None = None, use_cache: bool = False) Tuple[Tensor, Tuple[Tensor, Tensor] | None][source]
- abstractmethod forward(x: Tensor, attention_bias: FloatTensor | None = None, layer_past: Tuple[Tensor, Tensor] | None = None, use_cache: bool = False) Tuple[Tensor, Tuple[Tensor, Tensor] | None][source]
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- classmethod build(layer_id: int, config: ModelConfig, cache: BufferCache) LLaDABlock[source]
- class models.llada.modeling_llada.LLaDASequentialBlock(layer_id: int, config: ModelConfig, cache: BufferCache)[source]
Bases:
LLaDABlockThis is a typical transformer block where the output is computed as
MLP(LN(x + Attention(LN(x))))(plus another skip connection).- __init__(layer_id: int, config: ModelConfig, cache: BufferCache)[source]
Initialize internal Module state, shared by both nn.Module and ScriptModule.
- forward(x: Tensor, attention_bias: Tensor | None = None, layer_past: Tuple[Tensor, Tensor] | None = None, use_cache: bool = False) Tuple[Tensor, Tuple[Tensor, Tensor] | None][source]
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class models.llada.modeling_llada.LLaDAModel(config: ModelConfig, init_params: bool = True)[source]
Bases:
Module- __init__(config: ModelConfig, init_params: bool = True)[source]
Initialize internal Module state, shared by both nn.Module and ScriptModule.
- forward(input_ids: LongTensor, input_embeddings: FloatTensor | None = None, attention_mask: Tensor | None = None, attention_bias: Tensor | None = None, past_key_values: Sequence[Tuple[Tensor, Tensor]] | None = None, use_cache: bool = False, last_logits_only: bool = False, output_hidden_states: bool | None = None) LLaDAOutput[source]
- Parameters:
input_ids – A tensor of shape (batch_size, seq_len).
input_embeddings – A tensor of shape (batch_size, seq_len, d_model) with input embeddings. When provided, it is treated as the output of the input embedding layer.
attention_mask –
A tensor of shape (batch_size, seq_len) that indicates which input IDs are masked. A 1 value in the mask means that the corresponding input ID should not be ignored. A 0 means that the corresponding input ID is masked.
This has the same meaning as the attention_mask in HuggingFace’s transformers library.
attention_bias –
A tensor of shape (batch_size, 1, seq_len, seq_len), (1, 1, seq_len, seq_len), or (seq_len, seq_len). This is used to introduce causal or other biases.
If the tensor is a bool or byte tensor, a True or 1 at attention_bias[:, :, i, j] indicates that the i-th element in the sequence is allowed to attend to the j-th element in the sequence.
If the tensor is a float tensor, it will just be added to the attention scores before the softmax.
The default is causal, which corresponds to a lower-diagonal byte matrix of ones.
past_key_values – Pre-computed keys and values for each attention block. Can be used to speed up sequential decoding. The input_ids which have their past given to this model should not be passed as input_ids as they have already been computed.
use_cache – If True, return key and value tensors for each block.
last_logits_only – If True, only compute the logits for the last token of each sequence. This can speed up decoding when you only care about the next token.
- class models.llada.modeling_llada.LLaDAOutput(logits, attn_key_values, hidden_states)[source]
Bases:
NamedTuple- logits: FloatTensor
A tensor of shape (batch_size, seq_len, vocab_size) representing the log probabilities for the next token before normalization via (log) softmax.
Hidden states from each block.
- class models.llada.modeling_llada.LLaDAGenerateOutput(token_ids, scores)[source]
Bases:
NamedTuple- token_ids: LongTensor
The generated token IDs, a tensor of shape (batch_size, beam_size, max_steps). These do not include the original input IDs.
- scores: FloatTensor
The scores of the generated sequences, a tensor of shape (batch_size, beam_size).