The Hidden Machinery Behind an LLM’s “Thinking”

imagine a basic model predicting house prices: 1,000 square feet multiplied by a learned weight of 300 yields a $300,000 output. Real neural networks scale this up, combining many inputs and weights across multiple layers. The output of one layer becomes the input for the next.

These computational units are neurons, and all learned values are parameters. This simple multiplication offers an intuition, but it vastly understates an LLM's complexity. A frontier LLM like Llama 3 405B has 126 layers; GPT-4 is estimated at 96 to 128 layers.

Stacking layers enables models to learn increasingly intricate patterns. While "around 100 layers" illustrates a common depth, specific architectures vary significantly. Each layer transforms input data, allowing the network to abstract features from raw information, much like recognizing edges before identifying an object.

Text begins its journey not as words, but as tokens—numerical IDs representing common word fragments or punctuation. “Bank” might be a single token ID. A specialized lookup table then maps each token ID to a numerical embedding: a vector, or list of numbers, that provides a usable starting representation for the model.

Consider the word "bank." Its meaning shifts dramatically between "I sat on the river bank" and "I put my money in the bank." A fixed dictionary entry cannot capture this nuance. For the model to resolve this ambiguity, it must consider the surrounding words.

This is where causal self-attention, the core mechanism of the foundational Transformer architecture, comes into play. Within each layer, every token at a given position can "attend" to, or draw information from, all preceding tokens in the sequence. It dynamically updates its numerical representation by mixing in the most relevant contextual cues.

For "bank" in "river bank," the attention mechanism might heavily weigh "river," while in "money in the bank," it would focus on "money." This contextual blending ensures that the token’s vector accurately reflects its specific usage. After about 100 such layers, the final token embedding encapsulates the complete context of the entire message, enabling the model to make highly informed predictions for the next token.

A transformer block, the core computational unit in LLMs, performs two distinct operations. First, an attention mechanism mixes information across token positions. This allows a model to weigh the relevance of every preceding word to the current word, resolving ambiguities like whether "bank" refers to a river’s edge or a financial institution by considering its neighbors.

After attention contextualizes the token, a feed-forward network (FFN) processes its vector independently. Think of the FFN as a pattern detector: it takes the adjusted numerical representation of "bank" and checks for embedded patterns, like "Is this about money?" If a pattern matches, the FFN amplifies what the model has learned about that specific concept. Researchers like Geva et al. have shown these layers act as associative memories, retrieving factual associations.

While this simplified view captures the essence, real transformer blocks also incorporate mechanisms like residual connections and normalization layers. These components, omitted from the video's analogy for clarity, stabilize training and improve performance. For deeper dives into transformer architecture, explore the seminal paper Attention Is All You Need. Modern LLMs like Llama 3 405B leverage up to 126 such layers to build intricate contextual representations.

After roughly 100 layers of attention and feed-forward networks, the model’s final contextual representation for the last token position arrives at the output layer. This vector then converts into logits: numerical scores for every token in the model’s vocabulary, which can range from 32,000 to over 128,000 unique tokens. A softmax function can then transform these logits into probabilities, indicating the likelihood of each token being the next word.

Decoding then selects the next token. While simply picking the highest-probability token is one strategy, advanced methods like sampling introduce an element of randomness, allowing for more diverse and creative outputs within the probability distribution. This single token then appends to the input, and the entire process repeats, generating the next token in sequence.

This iterative, one-token-at-a-time process, known as autoregressive generation, builds an entire response. Crucially, the model does not plan its whole answer in advance; it predicts each token based on the previous ones. While advanced reasoning models can spend more inference-time computation generating intermediate steps or "thoughts," they still produce their ultimate output sequentially, token by token.

How does an LLM choose its next word?

It calculates scores for tokens in its vocabulary based on the context, then selects a token according to those scores and its decoding settings.

What does attention do in a transformer?

Attention lets each token’s representation draw on relevant tokens in the context, helping the model distinguish meanings such as “river bank” and “bank account.”

Do LLMs think like people?

Not in the human sense. Standard LLMs generate text through learned computations and next-token prediction, though multi-step outputs can resemble reasoning.

What are parameters in an LLM?

Parameters are learned numerical values, including weights, that shape how the model transforms its input and produces predictions.