https://notes.leo-draw.xyz/posts/Recent content in Posts on NotesHugoenMon, 24 Aug 2026 00:00:00 +0000https://notes.leo-draw.xyz/posts/token-sampling/Mon, 24 Aug 2026 00:00:00 +0000https://notes.leo-draw.xyz/posts/token-sampling/<p>How a language model turns last-step <strong>logits</strong> into the next token. The model itself only produces a score per vocabulary id. Sampling decides which id is actually emitted.</p> <p>Input: <code>logits</code> with shape <code>[batch, vocab_size]</code>.<br> Output: next token ids with shape <code>[batch, 1]</code>.</p> <p>Typical knobs: <code>temperature</code>, <code>top_k</code>, <code>top_p</code>. Common defaults for greedy decoding: <code>temperature = 0</code>, <code>top_k = 0</code> (off), <code>top_p = 1.0</code> (off).</p> <h2 id="pipeline">Pipeline</h2> <p>Each step below is optional except the first branch and the final sample. Greedy (<code>temperature == 0</code>) skips the rest.</p>