Model selection & cost optimization

Output is almost always more expensive than input because generation runs autoregressively—one forward pass per token. A chat feature that returns 800 tokens when 200 would suffice is burning budget on max_tokens and prompt design, not on model choice alone.

Prices change. OpenAI, Anthropic, and Google adjust list rates when new model generations ship; enterprise contracts (Bedrock, Azure OpenAI, Vertex) may differ from public API pricing. Before you commit to a margin model or customer pricing, pull current rates from provider docs and re-run your spreadsheet monthly.

Input vs output: why the split matters

Consider a support bot averaging 2,000 input tokens (system prompt + RAG chunks + user message) and 400 output tokens, at 50,000 requests per day on GPT-4o at ~$5 / $15 per million:

• Input cost/day: 2,000 × 50,000 × ($5 / 1,000,000) = $500
• Output cost/day: 400 × 50,000 × ($15 / 1,000,000) = $300
• Total: ~$800/day (~$24K/month) — before caching, routing, or failures/retries

The same traffic on GPT-4o-mini at ~$0.15 / $0.60 drops to roughly $18/day—a 40× swing driven entirely by model ID. That is why routing and caps belong in architecture, not as a post-launch optimization.

Most production traffic is “easy”: FAQ lookup, intent classification, short summarization, formatting JSON. Reserve frontier models for the long tail where small models fail evals. At 70–90% easy-case rates, cascades routinely cut spend 60–80% with negligible quality loss if escalation triggers are tuned on real data.

Cascade pattern: Haiku → Sonnet → Opus

1. Tier 0 (Haiku / 4o-mini / Flash) — answer or classify with temperature 0 and a strict output schema.
2. Confidence check — logprobs threshold, JSON schema validation, self-consistency vote, or a dedicated “needs escalation” classifier.
3. Tier 1 (Sonnet / GPT-4o) — retry with richer model if check fails; pass prior attempt as context (“your draft was rejected because…”).
4. Tier 2 (Opus / o1-class) — only for flagged high-value or high-complexity paths (legal review queue, executive summary).

Confidence-based escalation signals

Routing classifier with a small model

Train or prompt a tiny model to predict tier=cheap|standard|frontier from the user query and metadata (tenant plan, document length, detected language). Keep the classifier prompt under 500 tokens; run at temperature 0. Log predicted tier vs actual escalation rate—if you escalate >40% of “cheap” routes, the classifier needs more labels or features.

from dataclasses import dataclass
from openai import OpenAI

client = OpenAI()

TIERS = [
    {"name": "cheap", "model": "gpt-4o-mini", "price_in": 0.15, "price_out": 0.60},
    {"name": "standard", "model": "gpt-4o", "price_in": 5.0, "price_out": 15.0},
]

@dataclass
class RouteResult:
    text: str
    model: str
    input_tokens: int
    output_tokens: int
    escalated: bool

def call_model(model: str, messages: list) -> RouteResult:
    resp = client.chat.completions.create(
        model=model, temperature=0, max_tokens=512, messages=messages,
        response_format={"type": "json_object"},
    )
    u = resp.usage
    return RouteResult(
        text=resp.choices[0].message.content or "",
        model=model,
        input_tokens=u.prompt_tokens,
        output_tokens=u.completion_tokens,
        escalated=False,
    )

def needs_escalation(draft_json: str) -> bool:
    # Replace with Pydantic validation, logprobs, or classifier score
    return '"confidence"' not in draft_json or '"low"' in draft_json

def cascade_chat(messages: list) -> RouteResult:
    cheap = call_model(TIERS[0]["model"], messages)
    if not needs_escalation(cheap.text):
        return cheap
    standard = call_model(TIERS[1]["model"], messages + [
        {"role": "assistant", "content": cheap.text},
        {"role": "user", "content": "Previous answer failed validation. Try again."},
    ])
    standard.escalated = True
    return standard

public record RouteResult(String text, String model, int inputTokens,
                          int outputTokens, boolean escalated) {}

public class CascadeRouter {
  private final ChatClient cheapClient;
  private final ChatClient standardClient;

  public CascadeRouter(ChatClient.Builder builder) {
    this.cheapClient = builder.clone().defaultOptions(
        ChatOptionsBuilder.builder().withModel("gpt-4o-mini").withTemperature(0.0).build()
    ).build();
    this.standardClient = builder.clone().defaultOptions(
        ChatOptionsBuilder.builder().withModel("gpt-4o").withTemperature(0.0).build()
    ).build();
  }

  public RouteResult cascade(List<Message> messages) {
    var cheap = invoke(cheapClient, messages);
    if (!needsEscalation(cheap.text())) return cheap;
    var retryMessages = new ArrayList<>(messages);
    retryMessages.add(new AssistantMessage(cheap.text()));
    retryMessages.add(new UserMessage("Previous answer failed validation. Try again."));
    var standard = invoke(standardClient, retryMessages);
    return new RouteResult(standard.text(), "gpt-4o", standard.inputTokens(),
        standard.outputTokens(), true);
  }

  private boolean needsEscalation(String json) {
    return !json.contains("\"confidence\"") || json.contains("\"low\"");
  }
}

Anthropic prefix caching (~90% input discount on cache hits)

Mark cacheable blocks with cache_control: {type: "ephemeral"} on system content or tool definitions. The first request pays full input price to populate cache; subsequent requests with the same prefix read cached tokens at roughly 10% of normal input cost (verify current Anthropic pricing—rates evolve). TTL is typically minutes to hours; ideal for stable system prompts over high QPS.

OpenAI automatic prefix cache

OpenAI applies automatic caching when the prompt prefix matches a recent request (often 1,024+ token prefixes). You do not opt in explicitly—check usage.prompt_tokens_details.cached_tokens on responses. Structure prompts so static content comes first and variable user/RAG content comes last to maximize hit rate.

Semantic caching (GPTCache, Redis + embeddings)

Application-level cache flow:

1. Embed user query (small embedding model—cheap).
2. Vector search Redis / pgvector for neighbors above similarity threshold (e.g. cosine > 0.92).
3. On hit, return stored answer; on miss, call LLM and write (embedding, answer) with TTL.

Libraries like GPTCache wrap this pattern. Works best for repetitive FAQ, internal policy bots, and developer doc assistants—not for personalized or time-sensitive answers unless you partition cache keys by tenant + doc version.

When batch fits

• Nightly eval runs over golden datasets (Track 5 regression suites)
• Bulk document summarization, classification, or enrichment pipelines
• Backfill after schema or prompt version changes
• Embedding or caption generation for offline indexes
• Monthly report generation, invoice parsing, CRM hygiene

When real-time wins

• User-facing chat, copilot, or search-with-answer
• Agent tool loops that need sub-second tool selection
• Streaming UX (SSE to browser)
• Any workflow blocked on LLM output to continue the transaction

Hybrid architectures are common: real-time path for interactive features, batch path for analytics and eval at half price. Same prompt templates—different transport. Idempotency keys on batch line items prevent duplicate charges when you retry uploads.

When local wins

• Volume — millions of tokens/day where GPU amortization beats API list price (model size dependent).
• Privacy — data cannot leave VPC; regulated workloads (health, finance, gov).
• Fine-tuned weights — custom LoRA adapters on Llama; no hosted equivalent.
• Offline / edge — ships without reliable cloud connectivity.

Ops cost trade-off

A single L4 or A10G instance running 8B–70B quantized models might cost $500–2,000/month reserved—but you own utilization risk, model updates, security patches, and eval regressions when you bump weights. Staff time often exceeds GPU rent for teams without existing ML platform. Rule of thumb: prove ROI with hosted APIs and traffic metrics first; self-host when marginal token cost × volume exceeds fully loaded infra cost for 6+ months.

Template

monthly_cost = (avg_input × price_in + avg_output × price_out) / 1_000_000 × requests_per_day × 30

Extend with cache hit rate, escalation rate, and retry multiplier: effective_cost = base_cost × (1 - cache_savings) × (1 + escalation_rate × tier_multiplier) × retry_factor.

Worked example: RAG support feature

Same volume on GPT-4o at $5 / $15: per-request ≈ $0.0228, monthly ≈ $13,680—31× higher. A cascade with 85% mini / 15% 4o blends to roughly ~$2,400/month if escalations double token use on tier-2.

Per-feature dashboards

Tag every LLM call with feature, tenant_id, model, and computed cost_usd. Aggregate in your metrics stack (Datadog, Grafana, CloudWatch) or warehouse (BigQuery, Snowflake):

• Daily spend by feature — catches runaway cron jobs
• p50 / p95 tokens in and out — informs prompt compression work
• Escalation rate by route — tunes cascade thresholds
• Cost per successful task — ties dollars to business outcome, not raw calls

from dataclasses import dataclass

@dataclass(frozen=True)
class ModelPrice:
    name: str
    price_in_per_m: float
    price_out_per_m: float

PRICES = {
    "gpt-4o-mini": ModelPrice("gpt-4o-mini", 0.15, 0.60),
    "gpt-4o": ModelPrice("gpt-4o", 5.0, 15.0),
    "claude-haiku": ModelPrice("claude-haiku", 0.25, 1.25),
}

def request_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    p = PRICES[model]
    return (input_tokens * p.price_in_per_m + output_tokens * p.price_out_per_m) / 1_000_000

def monthly_cost(model: str, avg_in: int, avg_out: int, req_per_day: int, days: int = 30) -> float:
    per_req = request_cost(model, avg_in, avg_out)
    return per_req * req_per_day * days

# RAG support example
print(f"mini/month: ${monthly_cost('gpt-4o-mini', 3500, 350, 20_000):,.2f}")
print(f"4o/month:   ${monthly_cost('gpt-4o', 3500, 350, 20_000):,.2f}")

public record ModelPrice(String name, double priceInPerM, double priceOutPerM) {}

public final class LlmCostCalculator {
  private static final Map<String, ModelPrice> PRICES = Map.of(
      "gpt-4o-mini", new ModelPrice("gpt-4o-mini", 0.15, 0.60),
      "gpt-4o", new ModelPrice("gpt-4o", 5.0, 15.0)
  );

  public static double requestCost(String model, int inputTokens, int outputTokens) {
    var p = PRICES.get(model);
    return (inputTokens * p.priceInPerM() + outputTokens * p.priceOutPerM()) / 1_000_000.0;
  }

  public static double monthlyCost(String model, int avgIn, int avgOut, int reqPerDay, int days) {
    return requestCost(model, avgIn, avgOut) * reqPerDay * days;
  }
}

Decision dimensions

• Task complexity — extraction vs multi-hop reasoning vs open-ended writing
• Latency SLO — TTFT and total ms budget for UX tier (chat vs async email)
• Cost envelope — target $/successful task or $/tenant/month
• Privacy / residency — data leaves region? BAA required? air-gap?
• Context needs — max document size without truncation
• Tooling — JSON schema, function calling, vision, audio

Benchmark awareness (MMLU, HumanEval, etc.)

MMLU measures broad knowledge multiple-choice; HumanEval measures code synthesis pass@1. Leaderboard gains of 2–3 points rarely predict your invoice-parser accuracy. Use benchmarks to eliminate clearly weak candidates, then run golden-set evals (Track 5) on production prompts. Log regression when providers silently upgrade model snapshots.

FinOps checklist

• Central LLM gateway — one place for keys, routing, rate limits, and cost attribution
• Per-request cost log — model, tokens, feature, tenant, USD estimate
• Budget alerts — daily/weekly thresholds at org and feature level (PagerDuty / Slack)
• Per-tenant quotas — hard caps on tokens/day; graceful degradation message
• max_tokens enforced server-side — never trust client-supplied limits alone
• Prompt version registry — tie spend spikes to prompt deploys
• Monthly FinOps review — top 5 expensive features, escalation rates, cache hit rates
• Provider invoice reconciliation — billing CSV vs application logs within 5%

Per-tenant quotas pattern

Store rolling token usage in Redis keyed by tenant_id:yyyy-mm-dd. Before each call, increment estimated tokens; reject with 429 and upgrade CTA when over plan limit. Enterprise tenants get higher caps and optional burst allowance with finance approval.

Track 1 wrap-up

You now have the full LLM foundation stack: how models work, APIs and context (coming guides), embeddings, multimodal inputs, and this cost playbook. Multimodal features (PDF, image, audio) multiply token counts—size those paths with the same calculator and routing patterns before enabling them for all tenants.