Tokenization & Embeddings

Character-Level Tokenisation

Split text into individual characters (a, b, c, space, punctuation…). Very small vocabulary (~100–300 tokens), handles any word without out-of-vocabulary issues, but sequences become very long. A 100-word paragraph becomes ~500 character tokens. Attention must span far more steps to capture word-level meaning, and the model must learn word boundaries from scratch. Used in some early models and character-level language models, but rarely in LLMs.

Word-Level Tokenisation

Split on whitespace and punctuation. Short sequences, human-interpretable tokens. But vocabulary size explodes: English alone has 170,000+ words, and with inflections, compounds, technical jargon, and code, you quickly need millions of entries. Worse, any word not in the training vocabulary becomes Out-Of-Vocabulary (OOV) — the model sees it as [UNK] and loses all information about it. Morphologically rich languages (Finnish, Turkish) or mixed-language text are especially hurt.

Subword Tokenisation (The Sweet Spot)

Modern LLMs split text into subword units — fragments that are commonly occurring substrings. Common words stay as single tokens ("the", "is"); rare or novel words split into meaningful pieces ("unhappiness" → "un" + "happ" + "iness"). Vocabulary is manageable (32k–130k tokens), sequences are reasonable length, and OOV is nearly eliminated because any word can be decomposed to individual bytes if necessary. BPE, WordPiece, and SentencePiece/Unigram are the three main approaches.

The Embedding Matrix

The embedding layer is a lookup table: a matrix of shape vocab_size × d_model. Each row is the learnable embedding vector for one token. Given a token ID, the model simply retrieves that row. The entire matrix is trained end-to-end via gradient descent — no hand-crafting required. The same matrix is often reused (transposed) as the final output projection layer to reduce parameters, a technique called weight tying.

E = nn.Embedding(vocab_size=50257, d_model=768)
# GPT-2 small: 50257 × 768 = 38.6M parameters
# just in the embedding layer

x_ids = [1234, 567, 890]   # token IDs
x_emb = E(x_ids)           # shape: [3, 768]

Embedding Dimensions by Model

The embedding dimension d_model scales with model size:

• GPT-2 Small: d_model = 768
• GPT-2 XL: d_model = 1600
• GPT-3 175B: d_model = 12288
• Llama 3 8B: d_model = 4096
• Llama 3 70B: d_model = 8192
• Llama 3.1 405B: d_model = 16384

Larger d_model means a richer representational space — more "axes" to encode meaning along — but proportionally more compute and memory.

The King–Queen Analogy

Word2Vec (2013) famously showed that static word embeddings encode semantic analogies as geometric relationships:

vec("king") - vec("man") + vec("woman")
  ≈ vec("queen")

vec("Paris") - vec("France") + vec("Germany")
  ≈ vec("Berlin")

This works because embeddings are trained to place similar words near each other. Directions in embedding space correspond to semantic dimensions (gender, geography, tense). LLM embeddings carry similar structure, but enriched by context.

Polysemy: The "Bank" Problem

The word "bank" has at least two major senses. In static embeddings, the single vector is a blurry average of all senses — useful for neither. With contextual embeddings from BERT or GPT, the vector for "bank" in "I deposited money at the bank" clusters with financial words (credit, account, loan), while "bank" in "we sat on the river bank" clusters with geographic words (river, shore, water). The surrounding context flows into the representation through attention.

How Context Enters the Embedding

In a Transformer, the initial embedding is a static lookup — identical to Word2Vec at layer 0. But each subsequent layer updates the representation by mixing in information from other positions via attention. By layer 12 of BERT-base, the "bank" token's vector has been enriched by attending to "river" or "money" depending on context. The final hidden state is the contextual embedding. Different layers capture different aspects: lower layers encode syntactic patterns, higher layers encode semantic/pragmatic content.

Cosine Similarity

The standard metric for comparing embedding vectors. Measures the angle between two vectors, ignoring magnitude. Ranges from −1 (opposite) to +1 (identical direction):

cos_sim(a, b) = (a · b) / (||a|| × ||b||)

import numpy as np
def cosine_similarity(a, b):
    return np.dot(a, b) / (
        np.linalg.norm(a) * np.linalg.norm(b)
    )

# Example:
# cosine_sim("dog", "cat")    ≈ 0.92  (close)
# cosine_sim("dog", "justice") ≈ 0.21  (far)

Sentence Embeddings

Token-level embeddings represent individual tokens. For document-level tasks (semantic search, clustering), we need a single vector for an entire sentence or paragraph. Two common approaches:

• Mean Pooling: average all token embeddings in the sequence
• CLS Token: BERT prepends a special [CLS] token; its final embedding is used as the sequence representation
• Sentence-BERT: fine-tuned to produce high-quality sentence vectors via siamese networks

Nearest-Neighbour Search

Given a query embedding, find the k vectors in a database most similar to it. Naive brute-force is O(n×d) — too slow at scale. Approximate Nearest Neighbour (ANN) algorithms like FAISS (Facebook), ScaNN (Google), and HNSW trade a small accuracy loss for massive speedups, enabling billion-scale search in milliseconds. These power the retrieval step in RAG (Retrieval-Augmented Generation) pipelines.