How Large Language Models Work

A large language model, or LLM, is a neural network trained to do one thing: given some text, predict what comes next. Everything else (answering questions, writing code, summarizing a 90-page budget) grows out of that single skill done at enormous scale. That sounds too simple to work. It worked anyway, and the people who built it were as surprised as anyone.

Tokens: how a model sees text

Models don't read letters or words. They read tokens, chunks from a fixed vocabulary of a hundred thousand or more pieces. Common words are one token. Rare words get split into parts. Spaces usually stick to the front of the next word.

The Millbrook water board voted 4 to 1 to raise rates by 12 percent.

[The] [·Mill] [brook] [·water] [·board] [·voted] [·] [4] [·to] [·] [1]
[·to] [·raise] [·rates] [·by] [·] [12] [·percent] [.]

Tokens explain some odd chatbot behavior. Ask a model how many r's are in "strawberry" and older ones often got it wrong, because they never saw letters, only chunks like "str" and "awberry." Unusual names and long numbers get chopped into pieces, which is one reason models garble them. And AI companies bill by the token, so tokens are also money.

Next-token prediction

Feed the model some tokens. It outputs a probability for every token in its vocabulary: how likely each one is to come next. Pick one, stick it on the end, feed the whole thing back in, and repeat. A 300-word answer is about 400 trips through that loop.

Here's the part that trips people up. The model is picking plausible continuations. It has no separate fact-checking step. A true sentence and a false one that sounds just as typical can get similar odds. Lesson 6 is about what that means.

Predicting one token at a time doesn't mean no planning happens inside. In a 2025 study, Anthropic researchers traced the internal activity of their model writing rhyming couplets and found it picked the rhyme word before writing the line that led to it.

Attention, in plain words

Consider: "The board tabled the motion because it was incomplete." What does "it" mean? You know instantly it's the motion. Change one word, "because it had run out of time," and now "it" is the board. Meaning depends on other words, sometimes far away.

Attention, the core of the 2017 transformer, handles this. For each token, the model asks a question of every other token in the text: how relevant are you to me? It scores every pair, then blends in information from the tokens that scored highest. "It" ends up carrying a strong dose of "motion." A transformer does this in dozens of layers, each with many attention "heads" looking for different kinds of relationships at the same time: one tracking who did what, another matching quotation marks, another tracking tense. Because every token looks at every other token at once, the work runs in parallel on GPUs, which is what made training on trillions of tokens possible.

Training: three stages

Ask a base model "What is the capital of France?" and it might continue with "What is the capital of Germany? What is the capital of Spain?" because that's what a quiz page looks like. Fine-tuning teaches it to answer. OpenAI's 2022 InstructGPT paper laid out this recipe, and its launch post for ChatGPT says it was trained the same way: people wrote example conversations, then ranked model replies, and the rankings trained a reward model that tuned the chatbot.

RLHF has side effects. Human raters tend to like confident, agreeable answers, so models can learn to flatter and to sound sure when they shouldn't. Companies work against this, with mixed results.

The context window

The context window is how many tokens the model can hold at once: your messages, its replies, any documents you pasted, and hidden instructions from the company. It is working memory. When a conversation outgrows it, the oldest parts drop off or get summarized. As of October 2026, Anthropic lists a 1 million token window for its current Claude models, and OpenAI's open gpt-oss models support 128,000. A million tokens is roughly 750,000 words, about the length of eight or nine typical novels. Even so, models tend to use information at the start and end of a long context better than the middle. Don't assume a huge window means it read every page carefully.

Nothing you type changes the model's weights. Each chat starts fresh unless the app adds a memory feature, which works by quietly pasting notes back into the context.

Temperature

After the model scores the candidates, the app picks one. Temperature controls how adventurous that pick is. Here are the odds from the diagram reshaped at three temperatures:

The math: the model's raw scores (here 4.0, 3.0, 2.0 and 0.5) are divided by the temperature, then turned into percentages with a function called softmax. At 1.0, "rates" gets e to the 4.0 (about 54.6) out of a total of about 83.7, which is 65.2 percent. At 0.5 the scores double before that step and the leader runs away with it. At 2.0 they halve and the field evens out. Low temperature gives steady, repetitive text; high temperature gives variety and, past a point, nonsense. Temperature does not make a model more truthful. A model sure of a wrong fact will state it at any temperature.