Autoregressive Factorization and MLE

Factorizing the joint

Training versus generation

Teacher forcing provides the true prefix during training. Generation must reuse model samples, so early errors can propagate. A causal mask prevents access to future tokens.

MLE as a loss

Logs convert products to sums. Negating the maximization objective produces NLL, and cross entropy selects the target class log probability.

Code checks

I shifted inputs and targets by one step, checked the vocabulary axis, and masked padding tokens out of the loss.

Sequential cost

Sampling remains sequential over long outputs, motivating comparisons with latent-variable and diffusion approaches.