Course Study Note
Day 2 — Autoregressive Models and MLE
I factorized joint probabilities into conditionals and connected maximum likelihood with negative log-likelihood.
- Date
- 2025-07-08
- Course
- 2025 Generative Models Intensive — From VAE to Stable Diffusion
Study checkpoint for this day
Study checkpoint for this day
Topics from the day
- Autoregressive Models
I reviewed sequential prediction and teacher forcing.
- Maximum Likelihood Estimation
I followed the conversion from a product of probabilities to a log-sum loss.
Factorizing a joint distribution
| (p(x_1, ldots, x_T)=\prod_t p(x_t | x_{<t})) turns a high-dimensional problem into next-value prediction. PixelRNN applies this view over two image axes. |
Teacher forcing
Training can feed the true prefix, while generation must reuse model samples. Early mistakes can therefore propagate. A causal Transformer uses the same dependency rule even though its training operations are parallelized.
From MLE to a loss
Taking logs turns products into sums. Negating the objective converts maximization into a negative log-likelihood loss; cross entropy selects the log probability assigned to the target class.
The next step was handling a latent variable whose posterior is not directly tractable.