How Large Language Models Work
None of its own. This is a stretch page, and nothing on it is examined. It explains natural language processing, which A4.1.1 names as an application of machine learning, and connects to A4.2.1, A4.2.3, A4.3.2, A4.3.6 and A4.3.8 (all HL) and to the ethics points A4.4.1 and A4.4.2. SL and HL students are both welcome.
A language model turns text into numbers and uses patterns learned during training to generate a response. Follow how it chooses each next token, why its replies can vary, and where its limits matter. For an overview, start with the video below. (Written July 2026 from sources dated 2019 to 2026. Claims that depend on the current state of the field carry a date.)
Table of Contents
- The Three-Minute Version
- Why Language Is Hard for a Computer
- Tokenisation: Text Becomes Tokens
- Embeddings: Tokens Become Meaning
- Attention: Context Changes Meaning
- From a Trained Model to a Usable Assistant
- Limits and Reliability
- What This Page Leaves Out
- Where This Connects
The Three-Minute Version
A passage with its final word missing, words turned into lists of numbers, and attention drawn as thicker and thinner lines between words. The segment ends at 46:01; everything after is a different topic.
The video gives the core idea in brief: a large language model is trained to predict the next token (roughly, the next word) from the text it has been given, and attention is how words far back in that text can influence the prediction. The sections below take it step by step: text becomes tokens, tokens become numbers that carry meaning, context changes meaning, and a trained predictor becomes a usable assistant.
Why Language Is Hard for a Computer
Stretch. Related to A4.1.1, which names natural language processing as an application of machine learning.
Language is ambiguous in two separate ways, and both matter.
A single word can mean different things. If a friend texts “there’s a bat in the sports hall”, nothing inside the word “bat” tells you whether to fetch a scorer’s book or open a window. Only the surrounding situation settles it.
A whole sentence can be ambiguous even when every word is clear. In “Maya photographed the cyclist with the drone”, Maya might have used the drone to take the photo, or the cyclist might have been carrying it.
Both examples show the same thing: the meaning of a word or sentence depends on its context, not only on the words themselves. Much of the rest of this page is about how a model captures that context.
Tokenisation: Text Becomes Tokens
Stretch. Builds on A4.2.1, data cleaning (HL): tokenisation is the text version of preparing data before training, and the choices made here decide what the model can learn.
A neural network cannot take letters. It takes numbers, and before text can become numbers, someone has to decide what counts as one unit. Those units are called tokens, and the splitting is called tokenisation.
There are three standard ways to split, each with a cost:
One sentence, three splits. Subword splitting (middle row) is the compromise real systems use: “brightest” becomes “bright” plus “est”, so what the model learns about “est” in one place transfers to words it has never met.
Punctuation usually gets its own token, because where a comma falls changes meaning. Different organisations split differently, and the details are still an active area, so from here on this page simplifies to one word per token, as most explanations do.
The choice of training text also decides what gets learned. The body of text a model learns from is called its corpus (the collection of text used for training). Count which words follow which across a single detective novel and you get back that book’s characters, places and plot. Count across a large, varied collection and you get back ordinary English. The counting method is the same in both cases; only the corpus changed. This is the text version of a point from data preparation (A4.2.1): what goes into training shapes what comes out.
Try it: split a sentence three ways. Take “Don’t the quickest self-driving cars need the newest sensors?” and split it by word, by character, and by meaningful part. Decide what to do with the apostrophe and the hyphen, then answer: which split would help a model cope with a word it has never seen?
Show a worked answer
A word-based split immediately needs a rule for “Don’t” (one word or two?) and “self-driving” (split at the hyphen or not?). A character-based split can spell anything, but it produces a much longer sequence of pieces that carry little meaning on their own. A possible subword split is “Do” + “n’t”, “quick” + “est”, “self” + “-“ + “driving”, “car” + “s”, “new” + “est”, “sensor” + “s”. The subword split copes best with new words: a model that has learned what “est” and “s” do can handle “smartest gadgets” by splitting them into known parts, even if it has not met either word whole. The exact splits depend on the tokeniser.
Extension: in “a test of a good test is a fair test” there are ten tokens (every word counted, repeats included) but only six distinct words. Corpus sizes are usually quoted in tokens, so it is worth knowing the difference.
Embeddings: Tokens Become Meaning
Stretch. Related to A4.2.3 (compact numerical representations) and A4.3.2 (nearest-neighbour methods), both HL.
A neural network works with numbers, so each token is given a number, its token ID. The ID identifies the token, but a single number carries no information about meaning: nothing in it says that “bat” can be an animal, a piece of sports equipment or a verb.
So each token also gets a vector: a list of numbers, often hundreds of them, called its embedding. The values are learned during training, and no position is assigned a fixed meaning such as “how royal is it”. An individual number means little on its own; what matters is how one token’s list compares with others. Tokens used in similar contexts end up with similar lists, so if you treat each list as coordinates, they sit near each other in the embedding space.
How can training produce these similarities without someone labelling every pair of related words?
One approach learns from words that occur in similar contexts. Consider:
She packed her __ before the trip.
“Suitcase”, “boots” and “charger” could each fill the gap. “Excellent” would not fit. During training, words that can fill the same gaps across a very large number of sentences are nudged towards similar embeddings. No dictionary or hand labelling is needed: the training text itself provides the signal.
Words used in similar contexts can have similar embeddings. Similarity between two embeddings is often measured with a dot product, which combines two vectors into a single number. Its value depends on both their directions and their magnitudes; for vectors of similar length, it is higher when they point in similar directions.
Because the space is learned rather than designed, directions in it end up carrying meaning. The classic research demonstration is that the vector for “king”, minus “man”, plus “woman”, lands near “queen”. The same arithmetic works on patterns nobody planned: “kitten” minus “cat” plus “dog” lands near “puppy”, because a young-animal direction exists in the space.
Bias in embeddings. The same property that makes “king” minus “man” plus “woman” land near “queen” means an embedding space takes in whatever associations appear in the training text, including stereotypes about gender, race and class. A model trained on web text can learn those stereotypes as directions in the space, in the same way it learns everything else. This is one mechanism behind the training-data bias discussed in Ethics of Machine Learning (A4.4.1, in the syllabus).
The same trick is not limited to words. The image generation page shows photos and captions being placed into one shared space, which is what lets a sentence steer a picture.
Watch: the gentle route into word meaning (two Crash Course videos)
An introduction to word embeddings and next-word prediction: meaning from context at 3:33, why raw co-occurrence counts are too bulky at 5:15, and learned word clusters at 10:49. From 2019, so it ends before attention, which the sections below cover.
A lab that builds a small language model from start to finish: tokenisation (including contractions) at 3:46, rare and unknown words from 5:10, the embedding table at 7:59, and sampling the next word instead of picking the single most likely one at 11:37. Also from 2019.
Attention: Context Changes Meaning
Stretch. Builds on A4.3.8, artificial neural networks (HL): the machinery in this section is built from the weighted sums and layers described there.
An embedding gives each token a fixed starting representation: “bat” gets the same list of numbers whether it means the animal or the sports equipment. As the opening section showed, a fixed meaning is not enough, because the surrounding words decide which meaning applies. The sections below explain how a model lets context change a token’s representation.
First attempt: read one word at a time
A recurrent neural network reads one token at a time and passes a hidden state, a vector of numbers, from one step to the next. During training, it learns which information to retain in that state.
The output layer assigns a score to each token in the vocabulary. These scores are converted into a probability distribution (a spread of likelihoods across every possible next token), so the model’s output is a set of probabilities rather than a single stored reply. Sampling from this distribution is why asking the same question twice can produce different answers.
A recurrent network compresses the preceding text into a fixed-size state. It can lose information as the sequence gets longer, making distant relationships harder to retain.
Recurrence works, then runs out: a fixed-size state cannot hold a long passage.
The observation that became attention
Consider how context helps you predict a missing word:
Try it: which words actually helped? Ask a friend to write a paragraph of five or six sentences, ending in a sentence whose final word is strongly pinned down by something early in the passage, then hand it to you with that last word deleted. Supply the missing word, and then highlight only the words that helped you decide. Count how many words did nothing.
What the exercise shows
You will probably find that only a few words helped, some of them a long way back from the gap, and that most words contributed little or nothing. Your highlighting is an attention pattern: it shows which parts of the context mattered for this one prediction. A model needs to do the same thing, giving different parts of the context different weights.
That observation is attention: when making a prediction, the model weights information from the other tokens in its available context by how relevant they are, however far back they sit. Relevant tokens get large weights and the rest get small ones.
An attention pattern for one prediction. “Norway” earns the thick line even though it sits far from the gap; “Canada” gets a thin one (it is the wrong country, but it is at least a country); filler words get nothing.
How attention is computed
Attention uses three vectors for each token:
- The token produces a query: effectively a question, “what should I be paying attention to?”
- Every other token produces a key: its offer, “here is what I am”.
- Comparing the query against each key (with the dot product again) gives attention scores, which are normalised into percentages that total 100.
- Each token also offers a value: the information it will hand over if attended to. The scores weight an average of those values, and the result updates the asking token’s embedding.
Query, key, value for one word. After this update, the same word “it” appearing twice in one passage can mean two different things, despite both copies starting from an identical embedding. The “bat” from the opening is resolved the same way: by whatever it attends to.
The transformer
A transformer is the architecture built on attention. Instead of reading one token at a time, it processes all tokens at once, which is what makes it fast on parallel hardware such as GPUs (the connection to A4.1.2, which is in the syllabus). Processing everything at once loses the word order, so a representation of each token’s position is added back in, called positional encoding. The whole attend-and-update operation is then repeated in many layers, with feed-forward blocks (ordinary layers of weighted sums) between the attention steps; much of the model’s factual knowledge appears to live in those blocks, though that account is a research hypothesis (stated as of 2022), not settled fact. “Large” in large language model means the learned numbers, the parameters or weights, run into the hundreds of billions in the largest models of the mid-2020s.
Multi-headed attention uses several attention mechanisms in parallel, allowing them to learn different relationships. Masking blocks attention to later tokens during next-token training, so the model cannot use the answer it is learning to predict.
Go deeper: the mathematics, properly (3Blue1Brown)
These four videos explain the neural-network and attention calculations in more detail. Their examples date from around 2022, so treat their parameter counts and named models as examples from that time; the mechanisms they explain are the ones this page describes.
An overview of language models, including human feedback at 3:56, hardware at 4:14 and emergent behaviour at 6:28.
- Transformers, the tech behind LLMs (27 min): the embedding matrix and semantic directions from 12:27, context size at 19:49, softmax and temperature from 20:23.
- Attention, step by step (26 min): queries and keys from 4:29, masking at 11:09, values at 13:10, multi-headed attention at 19:20. More advanced than the others: it works through the vector and matrix calculations.
- How might LLMs store facts (22 min): the feed-forward layers, with a concrete account of a single neuron (a weighted sum, a bias and an activation). The “might” in the title matters: it presents a hypothesis about how these layers store facts.
From a Trained Model to a Usable Assistant
Stretch. Related to A4.3.6, reinforcement learning (HL), whose reward idea reappears in training from human feedback, and to A4.3.2 (HL), because nearest-neighbour search reappears in retrieval.
A transformer on its own predicts one next token, which is not yet a chatbot. Three more pieces close the gap: generating a complete response one token at a time, further training that adapts responses to people’s preferences, and retrieval that supplies information from documents.
Generating a whole reply, one token at a time
A language model predicts a single next token. To get a paragraph, the system runs in a loop: predict a token, stick it onto the end of the input, run again on the longer input, and repeat.
The generation loop: each new token is added to the text and fed back in to predict the next one, so the reply is produced one token at a time. Systems can cache calculations for earlier tokens to avoid repeating that work at each step.
The loop also explains the prompt: the prompt supplies tokens that the model can attend to at every step as it generates its response. Common prompting techniques work by giving it more relevant tokens to attend to:
- Describe your situation to specify the task and its constraints.
- List the steps you want followed to specify the process.
- Show an example to specify the format and level of detail.
Try it: improve a prompt, with reasons. Take the vague request “make me a revision plan”. Improve it three ways: add your situation (subjects, exam date, weak topics), give the steps you want in order, and paste a short example of the format you want back. After each change, name exactly what the model gained.
Compare your answer
Each change gave the model more relevant tokens to attend to: your constraints, the steps to follow and an example of the format you want. That is the mechanism behind all three techniques. Check whether the resulting plan uses each piece of information.
Tuning towards what people prefer
There is no single right answer to “write me a friendly email”, so training cannot mark responses right or wrong the way arithmetic can. The fix is indirect. People are shown pairs of responses and asked which is better. Those judgements train a separate reward model that learns to predict how a person would rate any response. That reward model then coaches the language model: responses it scores highly get reinforced, responses it scores poorly get discouraged.
This is reinforcement learning from human feedback. The reinforcement learning inside it is the reward-seeking idea from A4.3.6, applied to a task where the world provides no score of its own, so the score comes from a reward model trained on human preferences. The aim is to make responses more likely to match those preferences.
The two-model arrangement is the idea. Keep the boxes separate in your head: one model answers, the other judges.
Reading documents it was never trained on
A model has not seen your private messages or documents. If the answer to “what time do I need to be at the station on Friday?” is in a booking confirmation, the model needs access to that text, for example because it is pasted into the conversation or found by a connected retrieval tool.
Retrieval augmented generation (RAG) does this as part of the system: find the relevant document and add it to the prompt above the question. Finding the right document is the harder half. In one common method, every document (or chunk of a long one) is turned into an embedding, as words were in the embeddings section above, and so is the question. The search then looks for the nearest stored vectors: the same nearest-neighbour idea you met as a classifier in A4.3.2, doing a different job. A store searched this way, by position in the embedding space rather than by keyword, is a vector database. Large collections often use approximate nearest-neighbour search to reduce search time, accepting that it may miss the single closest match. See the RAG research paper for an example of a system combining retrieval and generation.
Retrieval augmented generation: find relevant passages, add them to the question, and pass both to the language model.
Try it: retrieval by hand. Write three short “documents” on paper (a club notice, a fake booking email, a homework reminder) and one question whose answer sits in exactly one of them. Ask a friend to find and answer it, then ask them how they chose. They will describe matching on meaning, not on exact shared words, which is precisely what the embedding search automates.
Limits and Reliability
Stretch. Related to A4.4.1: accountability and the difficulty of explaining a model’s decisions.
A fluent response can still be wrong. The model predicts likely next tokens, which is a different activity from a writer checking facts and deciding what to say: predicting likely tokens does not by itself check whether each claim is supported by evidence.
Hallucination is the generation of false or unsupported information, such as an invented quotation or reference, often in a confident, fluent style. It can arise in two ways: something untrue in the training text is learned as if it were reliable, or the generation loop takes an unlikely path and keeps building on it, one plausible token at a time. The style can stay fluent while the content is wrong, so check factual claims against reliable sources. The CS50x lecture demonstrates a confident recitation of a poem that does not exist (from 46:02).
The model’s behaviour is not written by hand: it comes from adjusting billions of numerical weights during training. Tracing how those weights produced a particular answer is difficult, even for the people who build these systems (as of 2026, still an active research area). This is the explainability problem often described as a “black box”.
What these systems can do has changed quickly between the oldest source for this page (a 2019 video that explains language models with recurrent networks) and the newest (2026). Treat any claim about what a model can do, including on this page, as true for a particular year and model: check the date and the model being discussed.
What This Page Leaves Out
This page does not cover the following topics, which are useful to look up next:
- The named subword algorithms real systems use for tokenisation (the idea is here; the specific methods are not).
- Instruction tuning, a separate training stage between next-token pre-training and human feedback.
- Reasoning and “thinking” models, which generate hidden working before answering.
- Multimodal models (one model handling images, audio and text together), and agents that use tools and take actions.
- How long documents are handled beyond the basic cost problem, and the efficiency tricks (there are many) that make big models cheaper to run.
- How anyone measures whether a chat model is good, benchmarks and evaluations included.
- Prompt injection and jailbreaks: attacks that try to redirect a model or bypass its safeguards, including through instructions hidden in text it reads.
Where This Connects
- In the syllabus: What Machine Learning Is for natural language processing (A4.1.1) and parallel hardware (A4.1.2), which transformers depend on because they process all tokens at once.
- In the syllabus (HL): Training and Evaluating a Model for data preparation (A4.2.1), compact representations (A4.2.3) and neural networks (A4.3.8); Types of Learning for nearest neighbours (A4.3.2) and reinforcement learning (A4.3.6).
- In the syllabus: Ethics of Machine Learning (A4.4.1): absorbed bias, hallucination, accountability and the environmental cost of training all connect straight back to the mechanisms on this page.
- Stretch: How AI Image Generation Works reuses the embedding idea to put pictures and captions in one shared space; AI Beyond Machine Learning covers AI methods that work from rules and search rather than training.
Sources and dates for this page
Written July 2026, drawing on: CS50x 2026 “Artificial Intelligence” (2026); 3Blue1Brown’s deep learning series chapters 5 to 7 and “LLMs explained briefly” (examples from around 2022); Crash Course AI episodes 7 and 8 (2019); and CS50’s “Fundamentals of AI” lectures Communicating and Generating (2026), which are long live recordings and so are linked rather than embedded. Additional reading: Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. All worked examples and diagrams on this page are original.