How Large Language Models Work
IB Syllabus: none directly. This is a stretch page: nothing on it is examined. It sits behind A4.1.1, where natural language processing appears as a real-world application of machine learning, and it deepens A4.2.1, A4.2.3, A4.3.2, A4.3.6 and A4.3.8 (all HL) plus the ethics points A4.4.1 and A4.4.2. SL and HL both welcome.
Why read this. You probably use a chatbot most weeks. This page explains what is actually happening when it answers: how text becomes numbers, how those numbers pick up meaning, why the same question can get two different replies, and what the system is genuinely not doing. In a hurry? Watch the three-minute video just below, then jump to What the model is not doing. Everything in between is the full route. (Written July 2026 from sources spanning 2019 to 2026; claims that depend on the state of the art carry a date.)
Table of Contents
- The Three-Minute Version
- Why Language Is Hard for a Computer
- Movement One: Text Becomes Tokens
- Movement Two: Tokens Become Meaning
- Movement Three: Context Changes Meaning
- Movement Four: From a Trained Model to a Usable One
- What the Model Is Not Doing
- What This Page Leaves Out
- Where This Connects
The Three-Minute Version
A passage with its final word missing, words turned into lists of numbers, and attention drawn as thicker and thinner lines between words. The segment ends at 46:01; everything after is a different topic.
That is the entire idea, compressed: a large language model is a system trained to guess the next word, and “attention” is how far-away words get a say in the guess. The rest of this page unpacks it slowly, in four movements: text becomes tokens, tokens become meaning, context changes meaning, and a trained predictor becomes a usable assistant.
Why Language Is Hard for a Computer
Stretch. Builds on A4.1.1, which names natural language processing among the applications of machine learning. Worth a moment because students usually underestimate the problem being solved.
Language is ambiguous in two separate ways, and both matter.
A single word can mean different things. If a friend texts “there’s a bat in the sports hall”, nothing inside the word “bat” tells you whether to fetch a scorer’s book or open a window. Only the surrounding situation settles it.
A whole sentence can be ambiguous even when every word is clear. In “Maya photographed the cyclist with the drone”, it is genuinely unclear whether Maya used the drone to take the photo or the cyclist was carrying one. The words are simple; the structure has two readings.
The lesson from both: meaning is not stored inside words. It comes from context. Everything on this page is machinery for capturing that one fact.
Movement One: Text Becomes Tokens
Stretch. Builds on A4.2.1, data cleaning (HL): tokenisation is the text version of preparing data before training, and the choices made here decide what the model can learn.
A neural network cannot take letters. It takes numbers, and before text can become numbers, someone has to decide what counts as one unit. Those units are called tokens, and the splitting is called tokenisation.
There are three standard ways to split, each with a cost:
One sentence, three splits. Subword splitting (middle row) is the compromise real systems use: “brightest” becomes “bright” plus “est”, so what the model learns about “est” in one place transfers to words it has never met.
Punctuation usually gets its own token, because where a comma falls changes meaning. Different organisations split differently, and the details are still an active area, so from here on this page simplifies to one word per token, as most explanations do.
The second half of this movement matters just as much: the choice of training text decides what gets learned. The body of text a model learns from is called its corpus (just a word for the collection of text used for training). Count which words follow which across a single detective novel and you get back that book’s suspects, alleys and plot. Count across a huge, varied collection and you get back ordinary English. Nothing about the counting changed; only the corpus did. This is the text version of a lesson from the syllabus: what goes into training decides what comes out.
Try it: split a sentence three ways. Take “Don’t the quickest self-driving cars need the newest sensors?” and split it by word, by character, and by meaningful part. Decide what to do with the apostrophe and the hyphen, then answer: which split would help a model cope with a word it has never seen?
Show a worked answer
By word you get stuck immediately on “Don’t” (one word or two?) and “self-driving” (the hyphen splits it or not). By character everything is spellable but meaningless. By meaningful part: “Do” + “n’t”, “quick” + “est”, “self” + “-“ + “driving”, “car” + “s”, “new” + “est”, “sensor” + “s”. The subword split wins on new words: a model that has learned what “est” and “s” do can handle “smartest gadgets” without ever having met either word.
Extension: in “a test of a good test is a fair test” there are ten tokens (every word counted, repeats included) but only six distinct words. Corpus sizes are usually quoted in tokens, so it is worth knowing the difference.
Movement Two: Tokens Become Meaning
Stretch. Builds on A4.2.3, dimensionality reduction (HL): an embedding is a compact numerical representation of something large, which is exactly that point’s territory. Also touches A4.3.2, because “find the nearest neighbour” reappears here.
A network needs numbers, so each token gets some. The first idea is one number per word, and it fails on the spot: a single value cannot say that “bat” is an animal, a cricket accessory, and a verb.
So each word gets a vector: a list of numbers, often hundreds of them. This list is called the word’s embedding. Now the point students most often miss: the individual numbers mean nothing on their own. Nobody decides that position 17 stores “how royal is it”. The numbers only mean something in relation to other words’ numbers: words with similar meanings end up with similar lists, which is to say they end up near each other when you treat the lists as coordinates.
That raises the obvious question, and it deserves a straight answer: how could a machine ever know two words are similar in the first place? Nobody labels that.
The answer is gaps. Look at this sentence:
She packed her __ before the trip.
“Suitcase” fits. So do “boots” and “charger”. “Excellent” does not. Words that can fill the same gaps in millions of sentences get nudged towards similar embeddings during training. No dictionary, no labelling, just the company words keep.
Words that fill the same gaps drift close together in the embedding space. Closeness between two vectors is measured with the dot product (a single number that is high when two vectors point the same way), and that closeness is the entire content of the representation.
Because the space is learned rather than designed, directions in it end up carrying meaning. The classic research demonstration is that the vector for “king”, minus “man”, plus “woman”, lands near “queen”. The same arithmetic works on patterns nobody planned: “kitten” minus “cat” plus “dog” lands near “puppy”, because a young-animal direction exists in the space.
The ethics arrives with the geometry. The same property means an embedding space absorbs whatever associations sit in the training text, including social ones about gender, race and class. A model trained on the web learns the web’s stereotypes as directions, exactly as it learns everything else. This is the mechanism behind the training-data bias discussed in Ethics of Machine Learning (A4.4.1, in the syllabus).
The same trick is not limited to words. The image generation page shows photos and captions being placed into one shared space, which is what lets a sentence steer a picture.
Watch: the gentle route into word meaning (two Crash Course videos)
The friendliest introduction anywhere to word embeddings and next-word prediction: distributional meaning at 3:33, why raw co-occurrence counts are too bulky at 5:15, learned word clusters at 10:49. From 2019, so it stops before attention; nothing in it is wrong, it just ends early.
A language model actually built, end to end: tokenisation with contractions at 3:46, rare and unknown words from 5:10, the embedding table at 7:59, and sampling rather than always taking the top word at 11:37. Also from 2019.
Movement Three: Context Changes Meaning
Stretch. Builds on A4.3.8, artificial neural networks (HL): everything here is made of the weighted sums and layers that point describes. This is where they lead.
An embedding gives each word a fixed starting meaning. But the opening section showed that fixed meanings are not enough: “bat” needs the sports hall around it. This movement is about machinery that lets context rewrite meaning.
First attempt: read one word at a time
The earliest approach is a recurrent neural network: read the sentence one token at a time, and carry a bundle of numbers forward from each step to the next. That bundle is the hidden state. Two things are worth stressing. The hidden state is just numbers. And nobody specifies what goes in it: the network works out for itself, during training, what is worth carrying forward.
This is also the right moment to see the output layer, because it quietly explains something students notice daily. For language, the output layer has one unit per word in the vocabulary, each holding how likely that word is to come next. The model’s answer is a distribution (a spread of likelihoods across every possible word), not a single stored reply. That is why asking the same question twice can produce different answers: the system samples from the spread.
Recurrence has a breaking point. However clever the network, there is only so much of a paragraph, let alone a chapter, that can be squeezed into one fixed-size bundle of numbers before information is lost.
Recurrence works, then runs out: a fixed-size state cannot hold a long passage.
The observation that became attention
Before naming the fix, do the exercise that motivates it.
Try it: which words actually helped? Ask a friend to write a paragraph of five or six sentences, ending in a sentence whose final word is strongly pinned down by something early in the passage, then hand it to you with that last word deleted. Supply the missing word, and then highlight only the words that helped you decide. Count how many words did nothing.
What the exercise shows
Almost everyone finds the same thing: a handful of words helped, sometimes from a long way back, and most words contributed nothing at all. Your highlighting is an attention pattern. You have just drawn the thing this section names, before it was named, which is the honest order to meet it in.
That observation is attention: when predicting, let every word reach back and take from the words that matter, however far away they sit, and ignore the rest.
An attention pattern for one prediction. “Norway” earns the thick line even though it sits far from the gap; “Canada” gets a thin one (it is the wrong country, but it is at least a country); filler words get nothing.
How attention is computed
The mechanism has three parts with deliberately plain names. For every token:
- The token produces a query: effectively a question, “what should I be paying attention to?”
- Every other token produces a key: its offer, “here is what I am”.
- Comparing the query against each key (with the dot product again) gives attention scores, which are normalised into percentages that total 100.
- Each token also offers a value: the information it will hand over if attended to. The scores weight an average of those values, and the result updates the asking token’s embedding.
Query, key, value for one word. After this update, the same word “it” appearing twice in one passage can mean two different things, despite both copies starting from an identical embedding. The “bat” from the opening is resolved the same way: by whatever it attends to.
The transformer
A transformer is the architecture built on attention. Instead of reading one token at a time, it processes all tokens at once, which is what makes it fast on parallel hardware such as GPUs (the connection to A4.1.2, which is in the syllabus). Processing everything at once loses the word order, so a representation of each token’s position is added back in, called positional encoding. The whole attend-and-update operation is then repeated in many layers, with feed-forward blocks (ordinary layers of weighted sums) between the attention steps; much of the model’s factual knowledge appears to live in those blocks, though that account is a research hypothesis (stated as of 2022), not settled fact. “Large” in large language model means the learned numbers, the parameters or weights, run into the hundreds of billions in the largest models of the mid-2020s.
Three honest hedges, because each rests on a single source: positional encoding, multi-headed attention (many attention mechanisms running in parallel, each learning a different way context changes meaning) and masking (blocking later words from influencing earlier ones during training) are all real components, and this page gives them one line each rather than pretending to teach them.
Go deeper: the mathematics, properly (3Blue1Brown)
These four videos are the best deep treatment available. They date from 2022, so parameter counts and named models are a snapshot of that moment; the mechanisms are the durable part.
The best short overview, and the only brief one that also reaches human feedback (3:56), the hardware story (4:14) and emergent behaviour (6:28).
- Transformers, the tech behind LLMs (27 min): the embedding matrix and semantic directions from 12:27, context size at 19:49, softmax and temperature from 20:23.
- Attention, step by step (26 min): queries and keys from 4:29, masking at 11:09, values at 13:10, multi-headed attention at 19:20. Genuinely advanced; the presenter says it takes time, and he is right.
- How might LLMs store facts (22 min): the feed-forward half of the block, and the clearest concrete account anywhere of a neuron, a weighted sum, a bias and an activation. Note the “might” in the title: it presents a hypothesis.
Movement Four: From a Trained Model to a Usable One
Stretch. Anchored to A4.3.6, reinforcement learning (HL), whose reward idea reappears here doing a job nobody mentions in class, and to A4.3.2, because nearest-neighbour search returns as well. There is no student-friendly video for this movement (the lectures that cover it are long unedited recordings), so the diagrams below carry it.
Most explanations stop at the transformer. That leaves a machine that predicts one next token, which is not yet a chatbot. Three more pieces close the gap.
Generating a whole reply, one token at a time
A language model predicts a single next token. To get a paragraph, the system runs in a loop: predict a token, stick it onto the end of the input, run again on the longer input, and repeat.
The generation loop. Nothing composes a whole reply in advance; the answer exists one likely token at a time. (Since the same early tokens get reprocessed on every pass, real systems store and reuse that earlier work rather than redoing it.)
Seeing the loop corrects a very common quiet assumption, that the model plans its answer and then types it out. It also explains the prompt. A prompt is not a magic spell; it is simply more tokens for attention to work on. Every piece of standard prompting advice is the same mechanism:
- Describe your situation: more of the right tokens to attend to.
- List the steps you want followed: more of the right tokens to attend to.
- Show an example of the output you want: more of the right tokens to attend to.
Try it: improve a prompt, with reasons. Take the vague request “make me a revision plan”. Improve it three ways: add your situation (subjects, exam date, weak topics), give the steps you want in order, and paste a short example of the format you want back. After each change, name exactly what the model gained.
Compare your answer
The answer is the same all three times: the model gained more of the right tokens to attend to. That is the entire mechanism. This is worth remembering because it is the only version of prompting advice that comes with an explanation attached rather than folklore.
Tuning towards what people prefer
There is no single right answer to “write me a friendly email”, so training cannot mark responses right or wrong the way arithmetic can. The fix is indirect. People are shown pairs of responses and asked which is better. Those judgements train a separate reward model that learns to predict how a person would rate any response. That reward model then coaches the language model: responses it scores highly get reinforced, responses it scores poorly get discouraged.
This is reinforcement learning from human feedback, and the reinforcement learning inside it is exactly the reward-chasing idea from A4.3.6, applied where the world provides no score of its own, so one is manufactured from human preferences.
The two-model arrangement is the idea. Keep the boxes separate in your head: one model answers, the other judges.
Reading documents it was never trained on
A model has not seen your email. If the answer to “what time do I need to be at the station on Friday” sits in a booking confirmation, no amount of clever prediction can know it.
The fix is refreshingly unmagical: find the relevant document and paste it into the prompt above the question. That is the whole trick, and it is called retrieval augmented generation. Finding the document is the interesting half. Every document (or chunk of a long one) is turned into an embedding, exactly as words were in movement two. So is the question. The search is then “find the nearest stored point”, the same nearest-neighbour idea you met as a classifier in A4.3.2, doing a different job. A store searched this way, by position in the space rather than by keyword, is a vector database. Because comparing against every stored vector gets expensive, real systems settle for an approximate nearest neighbour: they give up the guarantee of the single closest match in exchange for a much cheaper search.
Retrieval augmented generation. The final panel is deliberately ordinary: the “augmentation” is a paste. Students are usually surprised by how unmysterious this is, and that surprise is the point.
Try it: retrieval by hand. Write three short “documents” on paper (a club notice, a fake booking email, a homework reminder) and one question whose answer sits in exactly one of them. Ask a friend to find and answer it, then ask them how they chose. They will describe matching on meaning, not on exact shared words, which is precisely what the embedding search automates.
What the Model Is Not Doing
Stretch, and the most important section on the page. Anchored to A4.4.1 (in the syllabus): accountability and the black-box problem start exactly here.
The model has no understanding of the world. It predicts likely next tokens, which is a genuinely different activity from a writer deciding what to say and to whom. Fluency is not knowledge.
That difference is exactly why hallucination happens: a smooth, confident, wrong answer. It arises two ways: something untrue in the training text gets treated as reliable, or the generation loop takes an unlikely path and keeps building on it, one plausible token at a time. The style stays perfect while the substance fails, because style is what the system is actually good at. There is a memorable demonstration in the CS50x lecture: a confident recitation of a poem that does not exist (watch from 46:02).
Nobody sets the behaviour directly, either. Everything above emerges from tuning billions of weights against data, which is why even the people who build these systems often cannot say why a particular answer came out (still true in 2026, and an active research area). When an exam question or an ethics discussion reaches for “the black box”, this is the concrete thing it means.
A dated honesty note, because this page will age: what these systems can reliably do has changed fast between the oldest source here (2019, when recurrent networks were the standard story) and the newest (2026). Treat any specific capability claim you read anywhere, including here, as a snapshot with a year attached.
What This Page Leaves Out
An honest list, so the gaps are limits rather than surprises. None of these is covered by the sources this page is built on:
- The named subword algorithms real systems use for tokenisation (the idea is here; the specific methods are not).
- Instruction tuning, a separate training stage between next-token pre-training and human feedback.
- Reasoning and “thinking” models, which generate hidden working before answering.
- Multimodal models (one model handling images, audio and text together), and agents that use tools and take actions.
- How long documents are handled beyond the basic cost problem, and the efficiency tricks (there are many) that make big models cheaper to run.
- How anyone measures whether a chat model is good, benchmarks and evaluations included.
- Prompt injection and jailbreaks: attacks that hide instructions in the text a model reads. You will hear plenty about these; a proper treatment needs its own sources.
Where This Connects
- In the syllabus: What Machine Learning Is (A4.1.1 names natural language processing among the applications; this page is behind that door). The transformer’s appetite for parallel hardware is the story of A4.1.2.
- In the syllabus (HL): Training and Evaluating a Model for data preparation (A4.2.1), compact representations (A4.2.3) and neural networks (A4.3.8); Types of Learning for nearest neighbours (A4.3.2) and reinforcement learning (A4.3.6).
- In the syllabus: Ethics of Machine Learning (A4.4.1): absorbed bias, hallucination, accountability and the environmental cost of training all connect straight back to the mechanisms on this page.
- Stretch: How AI Image Generation Works reuses the embedding-space idea to put pictures and captions in one space; AI Beyond Machine Learning shows the half of AI that works without any training at all.
Sources and dates for this page
Built July 2026 by rewording and reorganising ideas from: CS50x 2026 “Artificial Intelligence” (embedded above, 2026); 3Blue1Brown’s deep learning series chapters 5 to 7 and “LLMs explained briefly” (2022); Crash Course AI episodes 7 and 8 (2019); and two lectures from CS50’s “Fundamentals of AI” series (“Communicating” and “Generating”, 2026), which are long unedited live recordings, linked here for completeness rather than embedded: Lecture 4 and Lecture 5. All worked examples and diagrams on this page are original.