AI
Models

1 · What it is

A language model does not receive language in the form a human experiences it. Text must first become numerical structure. The useful chain is text → tokens → token IDs → vectors → contextual representations.

A token is not necessarily a word. It can be a word, part of a word, punctuation, a number or another recurring character sequence. Each token receives an integer ID, but that ID is only a label. The mathematically useful representation comes when the ID is mapped to a vector: an embedding.

2 · Why it matters

The model cannot directly manipulate reality. It manipulates representations of information. If something is absent from the input or represented poorly, the model has less to work with. This is the foundation for later distinctions between knowledge encoded in parameters, information supplied in context and information retrieved through tools.

3 · How it works

A token embedding can be represented as a vector such as x = [0.2, -0.7, 0.4, 0.1]. Real model vectors have far more dimensions. Neural networks transform these vectors using learned operations.

These transformations are composed repeatedly, often with nonlinear operations between them. Crucially, humans do not assign meanings such as “dimension 17 = animals.” Training discovers numerical structure that helps the model perform its objective.

Initial token representations are not enough. Consider “The dog chased the cat” versus “The cat chased the dog.” The same words occur, but the relationships differ. Modern models therefore construct contextual representations: a token's internal representation changes according to surrounding tokens. Attention, studied under Architecture, is central to this process.

4 · Concrete mental model

Imagine language projected into a vast learned mathematical landscape. Pieces of language become points and patterns of geometry encode useful relationships. During processing, those representations are repeatedly transformed and contextualized.

human experience
      ↓
     text
      ↓
    tokens
      ↓
numerical vectors
      ↓
contextual representations
      ↓
transformed internal states

Where the analogy breaks: there is no literal 3D map containing objects labelled “cat,” “France” or “democracy.” Representation is distributed across many dimensions, layers, activations and parameters, and it changes with context.

5 · Common misunderstandings

6 · How it manifests in interaction

Long prompts containing background, constraints, examples, exceptions and objectives do more than “explain the task.” They alter the information structure available to the model. Structuring a problem as objective → constraints → architecture → stages → failure modes gives the model a different computational object than an underspecified request.

7 · What it enables / prevents

Learned representations can support syntax, semantics, entities, relationships, styles, abstractions, code structures and patterns of reasoning without each being manually programmed. But capability depends on usable representation. A model may be capable of answering a question while lacking the relevant information at inference time. Capability is not access.

8 · Connections

The objective determines which representations become useful. Parameters store the learned transformations. Architecture determines how representations interact. Learning adjusts the parameters. Inference executes the transformations. Context changes the representations currently active. Reasoning depends on transformations over representations. Memory asks where information resides. Tools introduce new information and operations.

1 · What it is

A machine-learning system needs a criterion by which one output is better than another during training. For a language model, the foundational pretraining task is remarkably simple: given the tokens so far, predict the next token.

The                         → predict "capital"
The capital                 → predict "of"
The capital of              → predict "France"
The capital of France       → predict "is"
The capital of France is    → predict "Paris"

2 · Why it matters

The objective supplies the learning pressure. Without it, there is no mathematical direction saying one parameter configuration is better than another. This separates machine learning from traditional rule programming: humans design the architecture, data pipeline and optimization process; the training process discovers parameters that perform the objective.

data → model → prediction → compare with target → error → adjust parameters → repeat

3 · How it works

The model computes a probability distribution across candidate next tokens. Once a token is selected, it joins the sequence and the model runs again. This is autoregressive generation: previous output becomes part of the input conditioning later output.

Training then asks how changes to parameters would affect loss. The gradient supplies that direction; gradient descent moves parameters slightly in the direction that reduces error.

4 · Why prediction can produce broad capability

To predict language well, a system benefits from learning structure about what language describes. “Water freezes at ___ degrees Celsius” rewards physical knowledge. “Alice is older than Bob; Bob is older than Claire; therefore the oldest is ___” rewards relational structure. Code completion rewards programming structure. The objective is simple; a strong strategy for satisfying it is not.

5 · Pretraining is not the whole assistant

A raw model trained only to continue internet text need not behave like an assistant. Modern systems use additional post-training to shape instruction-following, preferences, constraints and useful response behavior.

large-scale pretraining
        ↓
broad predictive capability
        ↓
post-training
        ↓
assistant-like behavior
        ↓
inference-time system + context + tools
        ↓
product experience

6 · Concrete mental model

Picture a vast parameter landscape in which height represents prediction error. Training repeatedly estimates a local downhill direction and adjusts the model. The analogy is useful for optimization, but the real space may have billions of dimensions, training uses batches rather than one global view, and optimization need not find a unique or globally best solution.

Training is not inserting answers into storage. It is optimizing a gigantic numerical system against an objective.

7 · Common misunderstandings

8 · How it manifests in interaction

Your prompt changes the conditioning information, not the model's trained parameters. Correcting a framing inside a conversation can radically change later output without any gradient descent occurring. Because generation is autoregressive, an early mistaken framing can also condition later tokens and create error propagation.

9 · What it enables / prevents

A broad predictive objective allows one learned system to support translation, summarization, coding, classification, explanation, extraction, transformation and some forms of reasoning without a separate hand-coded rule set for every task.

But predictive optimization alone does not guarantee truth, current information, exact arithmetic, logical consistency or recognition of ignorance. A surrounding system can compensate with retrieval, calculators, verification and other tools. At that point, we are analyzing a system containing a model, not the model alone.

10 · Connections

Representation gives the objective something mathematical to operate over. Parameters are what optimization changes. Architecture constrains the computations available. Learning is the mechanism that turns error into parameter updates. Inference uses the trained system without ordinary training updates. Context conditions current behavior. Reasoning asks what computations can occur inside prediction. Memory and tools separate learned structure from external information and capability.