How ChatGPT Understands Your Questions: A Deep Dive into LLMs, Tokenization, and Transformers

TL;DR: ChatGPT doesn't "read" your question the way you do. It converts text into numbers, processes those numbers through billions of mathematical operations inside a Transformer architecture, and generates a statistically coherent response one token at a time. This article explains every step of that pipeline with real code, diagrams, and honest depth.

Introduction

Most people interact with ChatGPT as if it were a knowledgeable colleague on the other side of a chat window. You type a question. A thoughtful answer appears. The experience feels conversational, almost human.

But underneath that interface is a system that has no concept of "understanding" in the philosophical sense — no inner experience, no dictionary lookup, no internet search at query time. What it has instead is an extraordinarily sophisticated mathematical function trained on hundreds of billions of words, capable of predicting what tokens should follow a given sequence of tokens.

This article is for developers, engineering students, and technically curious people who want to move beyond the surface-level explanation. We'll cover:

What a Large Language Model actually is
The full pipeline from your keystroke to the model's response
Why computers can't work with raw text and how tokenization solves that
How the Transformer architecture processes language
Real, runnable code you can experiment with today

Assumed knowledge: basic programming familiarity (Python helps), comfort with the idea that machine learning models involve matrices and arithmetic. You don't need a PhD.

1. What Is a Large Language Model (LLM)?

LLM stands for Large Language Model. Each word in that acronym matters:

Large — these models have hundreds of billions of parameters (numerical weights). GPT-4 is estimated to have over a trillion. Scale is not incidental; it's the primary driver of capability.
Language — the input and output domain is natural language: text in, text out.
Model — it's a mathematical function, not a database, not a search engine, not a rule-based system.

The core problem LLMs solve is language understanding and generation at scale. Before LLMs, NLP (Natural Language Processing) required hand-crafted rules or task-specific models. You'd have one model for sentiment analysis, a separate one for translation, another for summarization. LLMs replaced this fragmented landscape with a single, general-purpose architecture that can do all of those things — and many more — because it learned language itself, not just specific tasks.

Popular LLMs Today

Model	Creator	Parameters (est.)	Notable Use
GPT-4	OpenAI	~1 trillion	ChatGPT, Copilot
Claude 3	Anthropic	Unknown	Claude.ai
Gemini 1.5 Pro	Google DeepMind	Unknown	Google products
LLaMA 3	Meta	8B – 70B	Open-source research
Mistral Large	Mistral AI	Unknown	Enterprise API
Falcon 180B	TII	180B	Open-source

Common Applications You Already Use

Code completion (GitHub Copilot, Cursor)
Search summarization (Google AI Overviews, Bing Chat)
Customer support automation
Document drafting and editing
Translation and localization
Medical and legal document analysis

Every one of these applications runs the same fundamental pipeline we're about to dissect.

2. What Happens When You Send a Message to ChatGPT?

Let's trace the full lifecycle of a single prompt: "What is the capital of France?"

Diagram showing the full pipeline from user prompt to ChatGPT response: User → Tokenizer → Embeddings → Transformer Blocks → Softmax Sampling → Response

Step 1: Your Text Is Tokenized

The moment you hit send, the first thing that happens is tokenization — your raw string is split into tokens (more on this in the next section). The model never sees the word "France". It sees token IDs like [2061, 318, 262, 3139, 286, 4881, 30].

Step 2: Tokens Are Embedded Into Vectors

Each token ID is looked up in an embedding table — a giant matrix where each row is a dense numerical vector representing that token. A typical embedding has 768 to 12,288 dimensions depending on model size. These vectors carry semantic meaning learned during training.

Step 3: The Transformer Processes the Sequence

The sequence of embedding vectors is fed through many layers of a Transformer (typically 96+ layers in GPT-4-scale models). Each layer refines the representation using an operation called self-attention, which lets every token look at every other token in the context window.

Step 4: A Probability Distribution Is Generated

After the final Transformer layer, a projection matrix maps the output back to vocabulary size — for GPT-4, that's ~100,000 tokens. A softmax function converts the raw scores into a probability distribution. The model is saying: "Given everything I've seen, here's the probability of each possible next token."

Step 5: Sampling Produces the Response

A token is sampled from that distribution (influenced by the temperature parameter — we'll cover this). The chosen token is appended to the context, and the process repeats until an end-of-sequence token is generated or the maximum length is reached.

Note: ChatGPT is not searching the internet in real time (unless it has a tool/plugin enabled). The knowledge encoded in its weights comes entirely from training data collected before its knowledge cutoff.

Why Responses Aren't Copied From the Internet

A common misconception is that ChatGPT retrieves and pastas text from web pages. It does not. The model has no access to the internet during inference (by default). What it does is reconstruct plausible text based on statistical patterns learned during training. The response to "What is the capital of France?" emerges from the model having seen the association between France and Paris millions of times in training text — not from a lookup.

This distinction explains why LLMs can hallucinate: they're generating statistically plausible continuations, not verifying facts against a ground truth.

3. Why Computers Don't Understand Human Language

At the hardware level, a computer is a machine that performs arithmetic on binary numbers. Everything — images, audio, text, video — must be represented as numbers before a CPU or GPU can process it.

For images, this is intuitive: a pixel is an RGB triplet (255, 128, 0). For audio, it's a time-series of amplitude samples. For text, it's less obvious.

You might think: "We already have ASCII and Unicode. 'A' is 65. Problem solved."

Not quite. Character-level encoding ('A'=65, 'B'=66, ...) gives every character a number, but it encodes no semantic information. The character 'k' and 'K' are 75 and 107 respectively — a difference of 32 — but they're semantically nearly identical. The words king and queen would have wildly different numerical representations despite being conceptually related.

To do anything useful with language, we need representations that capture meaning and context, not just arbitrary code-point assignments. That's the problem tokenization and embeddings solve together.

4. Tokenization

Tokenization is the process of splitting raw text into discrete units called tokens, and mapping those units to integer IDs that the model can process.

A token is not always a word. It might be:

A whole word: capital → token 3139
A word fragment: tokenization might split into token + ization
A single character or punctuation mark
A whitespace-prefixed word: " France" (note the leading space)

Why Not Just Split on Spaces?

Word-level tokenization has fatal problems:

Vocabulary explodes: English has ~170,000 words, plus names, numbers, code, slang, other languages...
Rare or novel words become [UNKNOWN] tokens, losing information entirely
Morphological variants (run, running, ran) are treated as completely separate tokens

The solution used by GPT-style models is Byte-Pair Encoding (BPE), a compression-inspired algorithm that builds a vocabulary of subword units from the training corpus.

BPE in Plain English

Start with a vocabulary of individual characters (plus a special end-of-word symbol)
Count all adjacent pairs of symbols in the corpus
Merge the most frequent pair into a new symbol
Repeat until the vocabulary reaches the target size (e.g., 50,257 for GPT-2)

The result is a vocabulary where common words get their own token, but rare words are decomposed into recognizable subword pieces — dramatically reducing unknown tokens while keeping vocabulary manageable.

Hands-On: Tokenizing Text With `tiktoken`

tiktoken is OpenAI's official, fast tokenizer library. Install it and try this:

pip install tiktoken

import tiktoken

# Load the tokenizer used by GPT-4 and GPT-3.5-turbo
enc = tiktoken.get_encoding("cl100k_base")

prompt = "What is the capital of France?"
token_ids = enc.encode(prompt)
token_strings = [enc.decode([tid]) for tid in token_ids]

print("Token IDs:   ", token_ids)
print("Token count: ", len(token_ids))
print("Tokens:      ", token_strings)

Expected output:

Token IDs:    [3923, 374, 279, 6864, 315, 9822, 30]
Token count:  7
Tokens:       ['What', ' is', ' the', ' capital', ' of', ' France', '?']

Now try something that reveals subword splitting:

technical_text = "Tokenization is supercalifragilistic and ChatGPT uses transformers."
ids = enc.encode(technical_text)
tokens = [enc.decode([i]) for i in ids]

for i, (tid, tok) in enumerate(zip(ids, tokens)):
    print(f"  [{i:2d}] ID={tid:6d}  repr={repr(tok)}")

Expected output (abbreviated):

  [ 0] ID= 30454  repr='Token'
  [ 1] ID=  2065  repr='ization'
  [ 2] ID=   374  repr=' is'
  [ 3] ID=  2307  repr=' super'
  [ 4] ID= 10053  repr='cali'
  [ 5] ID=  1658  repr='fra'
  [ 6] ID= 29736  repr='gil'
  [ 7] ID=  2483  repr='istic'
  ...

Notice how Tokenization splits into Token + ization, and the nonsense word supercalifragilistic gets fragmented into recognizable subword pieces. This is BPE at work.

Tip: Token count directly affects cost and context usage in the OpenAI API. A rule of thumb: ~1 token ≈ 0.75 English words, or ~4 characters.

The Context Window

Every LLM has a context window — the maximum number of tokens it can process in a single forward pass. GPT-3.5-turbo has a 16K token context. GPT-4 Turbo supports 128K tokens. Claude 3 supports up to 200K.

Everything inside the context window — your system prompt, conversation history, and current message — is processed simultaneously. Nothing outside the context window can influence the response. This has important implications:

Long conversations get truncated when they exceed the window
Retrieval-Augmented Generation (RAG) is a technique to work around context limits
Longer contexts are significantly more expensive to process (quadratic in the attention mechanism)

5. Transformers: The Architecture That Changed Everything

Prior to 2017, the dominant architectures for sequence modeling were RNNs (Recurrent Neural Networks) and LSTMs (Long Short-Term Memory networks). These processed text sequentially — one token at a time, left to right — making them slow to train and prone to "forgetting" context from early in the sequence.

In 2017, researchers at Google published Attention Is All You Need, introducing the Transformer architecture. It eliminated recurrence entirely, replacing it with a mechanism called self-attention that processes all tokens in parallel and allows every position to attend to every other position directly.

The impact was immediate and total. Within a few years, virtually every state-of-the-art NLP model was a Transformer.

Architecture diagram of a Transformer block showing self-attention, residual connections, layer normalization, and feed-forward network layers

Self-Attention: The Core Intuition

Imagine you're reading the sentence: "The bank by the river was steep."

To understand what "bank" means here, you need to relate it to "river" — a word several positions away. Self-attention is the mechanism that lets the model do exactly this: for each token, it computes a weighted sum of all other tokens' representations, where the weights reflect relevance.

For each token, three vectors are computed:

Q (Query) — "What am I looking for?"
K (Key) — "What information do I have?"
V (Value) — "What should I pass forward if selected?"

Attention scores are computed as the dot product of Q and K, scaled and softmaxed:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V

In practice, this is implemented with matrix operations across the entire sequence simultaneously, which is why GPUs (optimized for matrix math) are essential.

Multi-Head Attention

A single attention operation captures one type of relationship. Multi-head attention runs several attention operations in parallel ("heads"), each learning to focus on different linguistic patterns — one head might track syntactic dependencies, another semantic similarity, another coreference.

import torch
import torch.nn as nn
import math

class MultiHeadSelfAttention(nn.Module):
    """
    A simplified but correct Multi-Head Self-Attention implementation.
    d_model: embedding dimension (e.g., 768 for GPT-2 small)
    n_heads: number of attention heads (e.g., 12)
    """
    def __init__(self, d_model: int, n_heads: int):
        super().__init__()
        assert d_model % n_heads == 0, "d_model must be divisible by n_heads"
        
        self.d_model = d_model
        self.n_heads = n_heads
        self.d_head = d_model // n_heads  # dimension per head
        
        # Fused projection for Q, K, V
        self.qkv_proj = nn.Linear(d_model, 3 * d_model, bias=False)
        self.out_proj = nn.Linear(d_model, d_model, bias=False)
    
    def forward(self, x: torch.Tensor, mask: torch.Tensor = None):
        """
        x: (batch_size, seq_len, d_model)
        Returns: (batch_size, seq_len, d_model)
        """
        B, T, C = x.shape  # batch, sequence length, channels (d_model)
        
        # Compute Q, K, V for all heads at once
        qkv = self.qkv_proj(x)  # (B, T, 3 * d_model)
        q, k, v = qkv.split(self.d_model, dim=2)  # each: (B, T, d_model)
        
        # Reshape to (B, n_heads, T, d_head)
        def reshape_for_attention(tensor):
            return tensor.view(B, T, self.n_heads, self.d_head).transpose(1, 2)
        
        q, k, v = map(reshape_for_attention, (q, k, v))
        
        # Scaled dot-product attention
        scale = math.sqrt(self.d_head)
        scores = torch.matmul(q, k.transpose(-2, -1)) / scale  # (B, n_heads, T, T)
        
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float('-inf'))
        
        attn_weights = torch.softmax(scores, dim=-1)  # (B, n_heads, T, T)
        
        # Weighted sum of values
        attended = torch.matmul(attn_weights, v)  # (B, n_heads, T, d_head)
        
        # Concatenate heads and project
        attended = attended.transpose(1, 2).contiguous().view(B, T, C)  # (B, T, d_model)
        return self.out_proj(attended)


# Quick sanity check
batch_size, seq_len, d_model, n_heads = 2, 16, 768, 12
model = MultiHeadSelfAttention(d_model=d_model, n_heads=n_heads)
sample_input = torch.randn(batch_size, seq_len, d_model)
output = model(sample_input)
print(f"Input shape:  {sample_input.shape}")
print(f"Output shape: {output.shape}")
# Output: Input shape:  torch.Size([2, 16, 768])
#         Output shape: torch.Size([2, 16, 768])

The Full Transformer Block

A complete Transformer layer (or "block") wraps multi-head attention with:

Layer Normalization — stabilizes training
Residual connections — allows gradients to flow freely, enabling very deep networks
Feed-Forward Network (FFN) — two linear layers with a nonlinearity between them, applied independently to each token position

class TransformerBlock(nn.Module):
    """
    A single Transformer decoder block as used in GPT-style models.
    """
    def __init__(self, d_model: int, n_heads: int, ffn_multiplier: int = 4):
        super().__init__()
        self.ln1 = nn.LayerNorm(d_model)
        self.attn = MultiHeadSelfAttention(d_model, n_heads)
        self.ln2 = nn.LayerNorm(d_model)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, d_model * ffn_multiplier),
            nn.GELU(),  # GPT-2 and later use GELU instead of ReLU
            nn.Linear(d_model * ffn_multiplier, d_model),
        )
    
    def forward(self, x: torch.Tensor, mask: torch.Tensor = None):
        # Pre-norm formulation (used in GPT-2 and most modern LLMs)
        x = x + self.attn(self.ln1(x), mask)  # residual + attention
        x = x + self.ffn(self.ln2(x))          # residual + feed-forward
        return x


# A tiny 4-layer GPT-like model
class MiniGPT(nn.Module):
    def __init__(self, vocab_size: int, d_model: int, n_heads: int, 
                 n_layers: int, max_seq_len: int):
        super().__init__()
        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(max_seq_len, d_model)
        self.blocks = nn.ModuleList([
            TransformerBlock(d_model, n_heads) for _ in range(n_layers)
        ])
        self.ln_final = nn.LayerNorm(d_model)
        self.head = nn.Linear(d_model, vocab_size, bias=False)
    
    def forward(self, token_ids: torch.Tensor):
        B, T = token_ids.shape
        positions = torch.arange(T, device=token_ids.device).unsqueeze(0)
        
        # Token + positional embeddings
        x = self.token_embedding(token_ids) + self.position_embedding(positions)
        
        # Causal mask: token i can only attend to tokens 0..i
        causal_mask = torch.tril(torch.ones(T, T, device=token_ids.device))
        
        # Pass through all Transformer blocks
        for block in self.blocks:
            x = block(x, causal_mask)
        
        x = self.ln_final(x)
        logits = self.head(x)  # (B, T, vocab_size)
        return logits


mini_gpt = MiniGPT(
    vocab_size=50257, d_model=256, n_heads=8, n_layers=4, max_seq_len=512
)
total_params = sum(p.numel() for p in mini_gpt.parameters())
print(f"MiniGPT parameter count: {total_params:,}")
# MiniGPT parameter count: 41,456,129  (~41M params — tiny by modern standards)

Note: GPT-2 small (the publicly released model from 2019) has 117M parameters and 12 Transformer blocks. GPT-3 has 96 blocks and 175B parameters. The architecture is the same; scale is different.

6. Temperature and Sampling: Why ChatGPT Isn't Deterministic

After the final Transformer block, the model produces logits — raw, unnormalized scores for each token in the vocabulary. These get converted to probabilities via softmax. The temperature parameter T controls how "sharp" that distribution is:

P(token_i) = exp(logit_i / T) / Σ exp(logit_j / T)

Side-by-side comparison of low temperature vs high temperature probability distributions over next token choices, showing sharpness vs flatness

import numpy as np

def softmax_with_temperature(logits: np.ndarray, temperature: float) -> np.ndarray:
    """Apply temperature-scaled softmax to logits."""
    scaled = logits / temperature
    exp_scaled = np.exp(scaled - np.max(scaled))  # numerical stability
    return exp_scaled / exp_scaled.sum()

# Simulate logits for a 5-token vocabulary
# Imagine the top candidates are: ["Paris", "Lyon", "Marseille", "Berlin", "London"]
tokens = ["Paris", "Lyon", "Marseille", "Berlin", "London"]
logits = np.array([8.2, 4.1, 3.7, 1.2, 0.9])

for temp in [0.1, 0.7, 1.0, 1.8]:
    probs = softmax_with_temperature(logits, temp)
    print(f"\nTemperature = {temp}")
    for token, p in zip(tokens, probs):
        bar = "█" * int(p * 40)
        print(f"  {token:12s}  {p:.4f}  {bar}")

Expected output:

Temperature = 0.1
  Paris         1.0000  ████████████████████████████████████████
  Lyon          0.0000  
  Marseille     0.0000  
  Berlin        0.0000  
  London        0.0000  

Temperature = 0.7
  Paris         0.9731  ██████████████████████████████████████
  Lyon          0.0185  
  Marseille     0.0079  
  Berlin        0.0004  
  London        0.0002  

Temperature = 1.0
  Paris         0.8912  ███████████████████████████████████
  Lyon          0.0670  ██
  Marseille     0.0363  █
  Berlin        0.0033  
  London        0.0022  

Temperature = 1.8
  Paris         0.5741  ██████████████████████
  Lyon          0.1682  ██████
  Marseille     0.1466  █████
  Berlin        0.0614  ██
  London        0.0498  █

At low temperature (0.1), the model almost always picks the top token — deterministic and focused, but repetitive. At high temperature (1.8), it's more random — creative but prone to errors and hallucinations. The default temperature for ChatGPT in most contexts is around 0.7–1.0.

7. The Complete Pipeline: Putting It All Together

Here's the full flow from prompt to response as a Mermaid diagram:

flowchart TD
    A(["User types: What is the capital of France?"]) --> B[Tokenizer]
    B --> C["Token IDs: [3923, 374, 279, 6864, 315, 9822, 30]"]
    C --> D[Token + Positional Embeddings]
    D --> E[Transformer Block 1]
    E --> F[Transformer Block 2]
    F --> G["... N Transformer Blocks"]
    G --> H[Final Layer Norm]
    H --> I[Linear Projection → Vocab Logits]
    I --> J[Softmax + Temperature Scaling]
    J --> K[Sample Next Token]
    K --> L{End of sequence?}
    L -- No --> C
    L -- Yes --> M(["Response: The capital of France is Paris."])

    style A fill:#1e293b,color:#f8fafc
    style M fill:#1e293b,color:#f8fafc
    style K fill:#4f46e5,color:#fff

Notice the loop: the sampled token is appended to the context and the entire sequence is re-processed to generate the next token. This is called autoregressive generation. Generating 200 tokens means running 200 forward passes.

8. Trade-offs and Pitfalls

Concern	Reality	Mitigation
Hallucination	The model generates plausible-sounding text that may be factually wrong	RAG, grounding, citations, human review
Context window limits	Long documents get truncated; older context may be "forgotten"	Chunking, summarization, larger models
Tokenization artifacts	Non-English text uses more tokens per word; code formatting can inflate counts	Be aware of costs; use language-native models where possible
Temperature tuning	Too low → repetitive; too high → incoherent	Task-specific defaults (0.0 for factual, ~0.8 for creative)
Quadratic attention cost	Attention scales as O(n²) with sequence length	Flash Attention, sparse attention, linear attention research
Training data bias	The model reflects biases present in its training corpus	RLHF fine-tuning, content filters, ongoing research
Knowledge cutoff	The model doesn't know about events after its training cutoff	Tool use, RAG with real-time data

Warning: Never treat LLM output as ground truth for high-stakes domains (medicine, law, finance) without independent verification. Hallucinations are not bugs that will eventually be fixed — they're a fundamental property of probabilistic text generation.

9. Best Practices for Working With LLMs

When writing prompts:

Be explicit about the format you want. "Return a JSON object with keys summary and keywords" yields more predictable output than "summarize this".
Provide examples (few-shot prompting). LLMs are extremely good at pattern-matching; showing the pattern you want is more reliable than describing it.
Constrain generation with system prompts when using the API — they strongly influence behavior.

When building applications:

Set temperature=0 or close to it for deterministic tasks (extraction, classification, code).
Always validate and sanitize LLM output before using it programmatically. Use structured output modes (response_format: {type: "json_object"}) when available.
Measure token usage aggressively. A prompt that's 50 tokens longer costs more per request and fills context faster.
Cache responses for identical prompts in production to reduce latency and cost.

When evaluating models:

Don't evaluate only on benchmark tasks. Evaluate on YOUR specific use case with YOUR data.
Track output quality over time — model updates can change behavior in ways that break your application silently.

Conclusion

The journey from your typed question to ChatGPT's response is a remarkable chain of mathematical operations:

Tokenization converts your text into integer IDs using Byte-Pair Encoding
Embeddings map those IDs into high-dimensional vectors that carry semantic meaning
Transformer blocks — each containing multi-head self-attention and a feed-forward network — process the entire sequence in parallel, allowing every token to attend to every other token
Sampling with temperature selects the next token from the output probability distribution
This process loops autoregressively until the response is complete

None of this involves "understanding" in the human sense. There is no inner model of the world, no lookup table of facts, no reasoning engine checking for logical consistency. What there is is a function of extraordinary power and scale that has compressed patterns from hundreds of billions of words into a set of numerical weights — and learned, in the process, to generate text that is often indistinguishable from expert human writing.

The practical next step: run the tiktoken tokenizer on your own prompts. Observe where tokens split. Experiment with temperature. Build the MiniGPT class above and train it on a small text corpus (Shakespeare's works fit in a few MB). There is no better way to internalize this architecture than to watch it learn.

How ChatGPT Understands Your Questions: A Deep Dive into LLMs, Tokenization, and Transformers

How ChatGPT Understands Your Questions: A Deep Dive into LLMs, Tokenization, and Transformers

Introduction

1. What Is a Large Language Model (LLM)?

Popular LLMs Today

Common Applications You Already Use

2. What Happens When You Send a Message to ChatGPT?

Step 1: Your Text Is Tokenized

Step 2: Tokens Are Embedded Into Vectors

Step 3: The Transformer Processes the Sequence

Step 4: A Probability Distribution Is Generated

Step 5: Sampling Produces the Response

Why Responses Aren't Copied From the Internet

3. Why Computers Don't Understand Human Language

4. Tokenization

Why Not Just Split on Spaces?

BPE in Plain English

Hands-On: Tokenizing Text With `tiktoken`

The Context Window

5. Transformers: The Architecture That Changed Everything

Self-Attention: The Core Intuition

Multi-Head Attention

The Full Transformer Block

6. Temperature and Sampling: Why ChatGPT Isn't Deterministic

7. The Complete Pipeline: Putting It All Together

8. Trade-offs and Pitfalls

9. Best Practices for Working With LLMs

Conclusion

Further Reading

Comments

More from this blog

Building Scalable Systems: Caching, Rate Limiting, and Observability

Kafka Explained Like You're 5: Events, Partitions, and Consumer Groups

Modern Next.js: The Complete Guide to App Router, Server Components, and Full-Stack Architecture

React Fundamentals: Components, JSX, State, and Re-rendering Explained

Command Palette

How ChatGPT Understands Your Questions: A Deep Dive into LLMs, Tokenization, and Transformers

Introduction

1. What Is a Large Language Model (LLM)?

Popular LLMs Today

Common Applications You Already Use

2. What Happens When You Send a Message to ChatGPT?

Step 1: Your Text Is Tokenized

Step 2: Tokens Are Embedded Into Vectors

Step 3: The Transformer Processes the Sequence

Step 4: A Probability Distribution Is Generated

Step 5: Sampling Produces the Response

Why Responses Aren't Copied From the Internet

3. Why Computers Don't Understand Human Language

4. Tokenization

Why Not Just Split on Spaces?

BPE in Plain English

Hands-On: Tokenizing Text With tiktoken

The Context Window

5. Transformers: The Architecture That Changed Everything

Self-Attention: The Core Intuition

Multi-Head Attention

The Full Transformer Block

6. Temperature and Sampling: Why ChatGPT Isn't Deterministic

7. The Complete Pipeline: Putting It All Together

8. Trade-offs and Pitfalls

9. Best Practices for Working With LLMs

Conclusion

Further Reading

Comments

More from this blog

Hands-On: Tokenizing Text With `tiktoken`