https://fredericowieser.github.io/tags/state-space-models/Recent content in State Space Models on Fred WieserHugoenfrederico.wieser@proton.me (Fred Wieser)frederico.wieser@proton.me (Fred Wieser)Mon, 24 Aug 2026 20:14:05 +0100https://fredericowieser.github.io/posts/modern_transformer_pretraining/Sat, 20 Jun 2026 00:00:00 +0000frederico.wieser@proton.me (Fred Wieser)https://fredericowieser.github.io/posts/modern_transformer_pretraining/<h2 id="markov-and-n-gram-models">Markov and n-gram models</h2> <p>$$ P(x_{0:T}) = P(x_0)\prod_{t=1}^{T}P(x_t \mid x_{t-1}) $$</p> <figure class="tikz-diagram"><figcaption>Token-level dependencies</figcaption><img src="https://fredericowieser.github.io/generated/tikz/markov.svg" alt="Token-level Markov chain graph" loading="lazy"> </figure> <p>$$ P(x_t \mid x_{&lt;t}) \approx P(x_t \mid x_{t-n+1:t-1}) $$</p> <figure class="tikz-diagram"><figcaption>Token-level dependencies</figcaption><img src="https://fredericowieser.github.io/generated/tikz/ngram.svg" alt="Token-level n-gram dependency graph" loading="lazy"> </figure> <h2 id="hidden-markov-models">Hidden Markov models</h2> <p>$$ P(z_{0:T},x_{0:T}) = P(z_0)\prod_{t=1}^{T}P(z_t \mid z_{t-1})P(x_t \mid z_t) $$</p> <p>$$ z_{t-1}\rightarrow z_t,\qquad z_t\rightarrow x_t $$</p> <h2 id="recurrent-and-gated-state-models">Recurrent and gated state models</h2> <p>$$ h_t = f_\theta(x_t,h_{t-1}) $$</p> <p>$$ c_t = f_t\odot c_{t-1} + i_t\odot\widetilde{c}_t $$</p> <p>$$ s_t = A s_{t-1} + Bx_t,\qquad y_t = Cs_t $$</p>