https://fredericowieser.github.io/tags/attention/Recent content in Attention on Fred WieserHugoenfrederico.wieser@proton.me (Fred Wieser)frederico.wieser@proton.me (Fred Wieser)Tue, 23 Jun 2026 01:14:06 +0100https://fredericowieser.github.io/posts/modern_transformer_pretraining/Sat, 20 Jun 2026 00:00:00 +0000frederico.wieser@proton.me (Fred Wieser)https://fredericowieser.github.io/posts/modern_transformer_pretraining/<p>Open-weight models such as GLM-5.2 make the gap between closed and open models feel much smaller. The useful way to read that history is not as a list of model names, but as a sequence modelling story.</p> <p>A sequence model first chooses a representation, then a dependency graph, then a way to spend compute. Raw text is mapped into tokens $z_1,\ldots,z_T$, tokens become vectors $X\in\mathbb{R}^{T\times d}$, and the model repeatedly mixes information across positions and across channels.</p>