The model does not “look up” answers in a database at inference time (unless you add retrieval or tools). It completes patterns it saw during training—grammar, APIs, refund policies, code idioms—weighted by the prompt you send. That is why the same model can be brilliant on familiar tasks and confidently wrong on niche facts: it is optimizing for plausible continuation, not guaranteed truth.

Foundation model is the industry term for the base weights (GPT-4 class, Claude, Llama). Application is your system prompt, RAG chunks, tool results, guardrails, and UI—everything that turns raw completion into a product feature.

Each generation step follows the same pattern:

1. Encode prompt — text → token IDs via the model’s tokenizer.
2. Forward pass — transformer layers output a vector of logits (scores) over the vocabulary.
3. Softmax — logits become a probability distribution over all possible next tokens.
4. Sample — pick one token (argmax at temperature 0, or stochastic with temperature / top-p / top-k).
5. Append & repeat — add token to context; stop at max_tokens, stop sequence, or EOS token.

Output length directly drives latency and cost: step 2–4 run once per generated token. A 500-token answer is roughly 500 forward passes (batched efficiently on GPU, but still proportional).

A decoder-only transformer (GPT-style) stacks repeated blocks. Each block roughly does:

• Self-attention — every token gathers information from prior tokens.
• Feed-forward MLP — per-token non-linear transform.
• Residual + layer norm — stabilizes deep stacks (dozens to 100+ layers on frontier models).

Deeper stacks tend to capture more abstract patterns (multi-step reasoning, code structure) at the cost of compute and memory. Model size (parameters) and context length are separate knobs—a 8B model can still have a 128K context window.

Tokenization: text → token IDs

Raw Unicode text is split into tokens using subword algorithms: BPE (GPT, Llama), WordPiece (older BERT lineage), SentencePiece (many multilingual models). Each token maps to an integer ID in a fixed vocabulary (often 32K–200K entries).

Token ≠ word. English averages ~4 characters per token, but:

• unbelievable → often 3 tokens (un, believ, able)
• One emoji → 1–4 tokens depending on tokenizer
• Code, JSON, and base64 → more tokens per character than prose
• CJK text → often one token per character or small groups

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
text = "Refund policy: partial months are not refundable on annual plans."
ids = enc.encode(text)
print("tokens:", len(ids))
print("ids:", ids[:12], "...")

// jtokkit — OpenAI-compatible token counting in JVM services
EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
Encoding enc = registry.getEncoding(EncodingType.CL100K_BASE);
String text = "Refund policy: partial months are not refundable on annual plans.";
List<Integer> ids = enc.encode(text);
System.out.println("tokens: " + ids.size());

Embeddings and attention

Embedding layer: each token ID → dense vector (e.g. 4096 dimensions). Similar tokens sit near each other in space; the model learns these vectors during training.

Attention: for each token, the model computes how much to “look at” every previous token and builds a weighted mix. Classic example: “The animal didn't cross the street because it was tired.” — attention links it to animal, not street. Without that mechanism, pronoun resolution and long-range dependencies fail.

Output head: final hidden state → logits over vocabulary → softmax → probabilities → sample next token.

Temperature

Temperature scales logits before softmax. Lower → sharper distribution (more deterministic); higher → flatter (more random).

Top-p (nucleus sampling)

Sort tokens by probability; take the smallest set whose cumulative mass ≥ p (e.g. 0.9); sample only within that set. Adapts to context—narrow when one token dominates, wide when many are plausible. Most production apps use temperature 0–0.7 + top_p 0.9–0.95.

Top-k

Sample only from the top k tokens (e.g. 40). Simpler but less adaptive than top-p. Some APIs expose both; if in doubt, prefer top-p.

# Extraction: temperature 0 + JSON schema (OpenAI strict mode in next guide)
extract = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0,
    response_format={"type": "json_object"},
    messages=[{"role": "user", "content": "Extract vendor name from: Acme Corp invoice #8821"}],
)

# Marketing draft: higher temperature, human review before send
draft = client.chat.completions.create(
    model="gpt-4o",
    temperature=0.9,
    top_p=0.95,
    messages=[{"role": "user", "content": "Write three subject lines for a product launch email."}],
)

// Spring AI — pin temperature per use case
ChatResponse extraction = chatClient.prompt()
    .options(ChatOptionsBuilder.builder().withTemperature(0.0).build())
    .user("Extract vendor name from: Acme Corp invoice #8821")
    .call();

ChatResponse draft = chatClient.prompt()
    .options(ChatOptionsBuilder.builder().withTemperature(0.9).withTopP(0.95).build())
    .user("Write three subject lines for a product launch email.")
    .call();

Representative limits (check provider docs—changes often)

Tokens are not characters

English prose ≈ 4 characters per token. Code, tables, and non-Latin text diverge widely. A 50-page PDF pasted as text can blow a budget you sized for “50 pages × 500 words × 4 chars.”

What happens at the limit

• Hard error — API returns 400 context length exceeded; you must truncate or summarize.
• Silent truncation — some clients drop oldest messages; critical instructions at the start may survive while recent user text is cut.
• Quality degradation — even within limit, lost in the middle: models recall start and end of long context better than the middle. Put critical RAG chunks and rules at the beginning or end—not buried mid-prompt.

Long context ≠ free

Billing is per input token. Attention compute inside the model scales roughly O(n²) with sequence length during prefill— providers price and rate-limit accordingly. Stuffing 500K tokens “because it fits” can cost dollars per request and add seconds of latency.

Token budget template (single request)

1. Pre-training

Train on massive unlabeled text (web, books, code) with next-token prediction. The model learns grammar, facts (statistically), reasoning patterns, and languages—embedded in weights, static until retrained. Cost: millions of GPU-hours; only foundation labs do this routinely.

2. Supervised fine-tuning (SFT)

Curated instruction → response pairs teach “assistant” behavior: follow questions, refuse unsafe requests, use markdown, etc. This is why base Llama behaves differently from Llama-Instruct—you want instruct/chat variants for products.

3. RLHF (reinforcement learning from human feedback)

Humans rank multiple model answers; a reward model learns preferences; policy tuning nudges outputs toward helpful/harmless. Explains alignment quirks—models refuse more than raw pre-training would, and style varies by vendor.

4. Constitutional AI (Anthropic)

AI-generated critiques replace some human ranking for scale—same goal as RLHF with different data pipeline. As an app builder, you still add your policies via system prompt and guardrails.

Reference pricing (order of magnitude — verify current rates)

Routing pattern: cheap first, escalate on uncertainty

Classify or answer with Haiku / 4o-mini; if confidence low (logprobs, self-check, or classifier), retry with Sonnet / GPT-4o. Cascade routing saves 60–80% at scale when most tickets are simple—see the cost guide in this track.

Minimal call (non-streaming)

Log usage.total_tokens (OpenAI) or input/output token fields on every response—cost attribution starts here.

from openai import OpenAI

client = OpenAI()  # OPENAI_API_KEY from environment
response = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0,
    max_tokens=256,
    messages=[
        {"role": "system", "content": "You are a concise technical assistant."},
        {"role": "user", "content": "Explain what a token is in one sentence."},
    ],
)
print(response.choices[0].message.content)
print("prompt:", response.usage.prompt_tokens, "completion:", response.usage.completion_tokens)

import anthropic

client = anthropic.Anthropic()  # ANTHROPIC_API_KEY
message = client.messages.create(
    model="claude-3-5-haiku-20241022",
    max_tokens=256,
    temperature=0,
    system="You are a concise technical assistant.",
    messages=[{"role": "user", "content": "Explain what a token is in one sentence."}],
)
print(message.content[0].text)
print("in:", message.usage.input_tokens, "out:", message.usage.output_tokens)

import boto3, json

bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
body = {
    "anthropic_version": "bedrock-2023-05-31",
    "max_tokens": 256,
    "temperature": 0,
    "system": "You are a concise technical assistant.",
    "messages": [{"role": "user", "content": "Explain what a token is in one sentence."}],
}
resp = bedrock.invoke_model(
    modelId="anthropic.claude-3-haiku-20240307-v1:0",
    body=json.dumps(body),
)
payload = json.loads(resp["body"].read())
print(payload["content"][0]["text"])

@RestController
@RequestMapping("/v1")
public class ChatController {
  private final ChatClient chatClient;

  public ChatController(ChatClient.Builder builder) {
    this.chatClient = builder
        .defaultSystem("You are a concise technical assistant.")
        .build();
  }

  @GetMapping("/chat")
  public String chat(@RequestParam String question) {
    return chatClient.prompt()
        .user(question)
        .call()
        .content();
  }
}

AnthropicChatModel model = AnthropicChatModel.builder()
    .apiKey(System.getenv("ANTHROPIC_API_KEY"))
    .modelName("claude-3-5-haiku-20241022")
    .temperature(0.0)
    .maxTokens(256)
    .build();

String answer = model.generate(
    "Explain what a token is in one sentence."
).content().text();

var chat = new BedrockAnthropicChatModel(
    BedrockAnthropicChatOptions.builder()
        .withModel("anthropic.claude-3-haiku-20240307-v1:0")
        .withTemperature(0.0f)
        .withMaxTokens(256)
        .build());

String text = chat.call(new Prompt("Explain what a token is in one sentence."))
    .getResult().getOutput().getContent();

Streaming (production default for user-facing chat)

Set stream=True (OpenAI) or streaming API on Anthropic/Bedrock. Forward tokens to the client over SSE so users see progress; measure TTFT separately from total duration. The next guide in this track covers provider-specific streaming and error handling mid-stream.

Architecture sketch

Client → your API → (optional RAG/tools) → LLM gateway → provider. Gateway owns API keys, model routing, rate limits, retries, caching, and per-team cost accounting— do not scatter provider keys across microservices.

Resilience

• 429 rate limits — exponential backoff with jitter; respect Retry-After.
• 503 / timeouts — retry idempotent reads; cap total deadline below client timeout.
• Model fallback — GPT-4o → 4o-mini → cached/static response for non-critical paths.
• Provider fallback — OpenAI down → Anthropic via unified client (LiteLLM, Portkey, or in-house).
• Circuit breaker — stop calling a failing provider after N errors; alert on-call.

Observability (log every call)

Production readiness checklist

• Structured logging per LLM call with token counts and cost estimate
• max_tokens and input length validated before the API call
• Temperature 0 (or pinned) on deterministic code paths
• Rate limit and timeout handling tested under load
• Streaming to client for chat UX; graceful partial response on stream errors
• Secrets in vault—not env files in the container image
• Golden eval suite in CI (Track 5)—block merge on regression
• Human review path for high-stakes outputs (medical, legal, money movement)