System Design Fundamentals

Computers are fast at some things and catastrophically slow at others—and the gap spans six orders of magnitude from L1 cache to cross-region network round-trips. When you design a system, every component choice is really a bet on where data lives relative to the CPU that needs it.

The hierarchy in one sentence

If your hot path touches disk or network more than once per user request, you have a latency problem that no amount of "faster CPUs" will fix. The fix is always move data closer (cache), touch fewer things (denormalize, batch), or do less work (async, precompute).

Practical rules derived from the numbers

• Keep hot data in RAM. Redis/Memcached for session, feed fragments, rate-limit counters.
• Minimize serial RPC chains. Each same-DC hop costs ~0.5 ms; fan-out in parallel, not sequence.
• Batch disk writes. One 10 KB write is nearly as expensive as one 1 MB write on SSD (amortize seek/flush).
• Never assume cross-region is fast. 50 ms RTT means synchronous replication caps write throughput and inflates p99.
• Profile before optimizing CPU. If you're I/O-bound, faster algorithms won't help—change the data path.

Throughput (requests/sec, QPS, TPS) measures system capacity. Latency measures user-perceived speed for a single operation. A system can have high throughput with terrible latency (batch processing) or low latency with low throughput (single-threaded optimized path).

Little's Law: the bridge between them

L = λ × W — average concurrent requests (L) equals arrival rate (λ) times average time in system (W). If your API handles 10,000 RPS with 100 ms average latency, you have ~1,000 in-flight requests at any moment. Double traffic without scaling → latency doubles (queue buildup).

Example: checkout service
  Arrival rate (λ)  = 2,000 RPS
  Avg latency (W)   = 50 ms = 0.05 s
  Concurrent (L)    = 2,000 × 0.05 = 100 in-flight requests

  If λ doubles to 4,000 RPS without adding capacity:
  W increases → queue depth grows → p99 explodes

Percentiles: why averages lie

Average latency hides tail behavior. A service with 10 ms average but 2 s p99 feels broken to 1% of users—that's millions of users at scale. Always design and commit to percentile SLOs, not averages.

SLI, SLO, SLA — the observability contract

• SLI (Indicator): What you measure — e.g., "proportion of requests completing in < 200 ms."
• SLO (Objective): Target — e.g., "99.9% of requests < 200 ms over 30 days."
• SLA (Agreement): Business contract with consequences — e.g., "99.9% uptime or credits."

Error budget = 1 − SLO. At 99.9% availability, you have 43.8 minutes/month of allowed downtime. Spend error budget on risky launches; conserve it during incidents.

The four golden signals (Google SRE)

flowchart LR
  Client[Client]
  LB[Load Balancer\n~0.1 ms]
  API[API Server\n~5 ms]
  Cache[Redis Cache\n~1 ms]
  DB[(PostgreSQL\n~5 ms)]
  Client --> LB --> API
  API --> Cache
  Cache -->|miss| DB
  Cache -->|hit| API
  DB --> API
  API --> LB --> Client

Vertical vs horizontal scaling

The AKF Scale Cube

Martin Abbott and Michael Fisher's AKF Scale Cube defines three independent axes of scaling. Mature systems scale on all three simultaneously.

• X-axis — Horizontal duplication: Clone the entire service N times behind a load balancer. Stateless web/API tier. Easiest; no code changes.
• Y-axis — Functional decomposition: Split by feature/service—User Service, Order Service, Payment Service. Microservices. Reduces blast radius; adds network overhead.
• Z-axis — Data partitioning: Split by shard key—user_id % N, geographic region, tenant_id. Scales data and write throughput. Hardest; requires routing and rebalancing.

Stateless vs stateful services

The nines table

Each additional nine costs roughly 10× in engineering and infrastructure. Most consumer products target 99.9%–99.99%. Don't over-engineer five nines for a blog.

MTBF and MTTR

Availability = MTBF / (MTBF + MTTR)

• MTBF (Mean Time Between Failures): How long the system runs before breaking. Increase via redundancy, chaos testing, better hardware.
• MTTR (Mean Time To Recovery): How long to restore service after failure. Decrease via automation, runbooks, circuit breakers, graceful degradation.

Example: API cluster
  MTBF = 720 hours (30 days between incidents)
  MTTR = 0.5 hours (30 min to recover)
  Availability = 720 / (720 + 0.5) = 99.93%

  To reach 99.99% without improving MTBF:
  MTTR must drop to ~0.05 hours (3 minutes)

Redundancy patterns

Common failure modes

• Single Point of Failure (SPOF): One component whose failure takes down the system—eliminate with redundancy.
• Cascading failure: Overloaded node fails → retries flood siblings → they fail too. Fix: circuit breakers, bulkheads, rate limits.
• Split brain: Network partition causes two nodes to both think they're primary. Fix: quorum, fencing tokens, STONITH.
• Thundering herd: Cache expires → all requests hit DB simultaneously. Fix: jittered TTL, mutex on miss, pre-warming.
• Retry storm: Clients retry failed requests → amplifies load 3–10×. Fix: exponential backoff with jitter, idempotency keys.

The three properties

• Consistency (C): Every read receives the most recent write or an error. All nodes see the same data at the same time (linearizable).
• Availability (A): Every request receives a non-error response, without guarantee that it reflects the latest write.
• Partition tolerance (P): The system continues operating despite arbitrary message loss or delay between nodes (network splits).

In practice, partitions happen—switch failures, GC pauses masquerading as timeouts, cross-AZ link cuts. P is not optional in distributed systems. The real choice is CP vs AP during partition: reject requests (sacrifice availability) or serve potentially stale data (sacrifice consistency).

Full form: If Partition → choose A or C. Else (normal operation) → choose L or C.

Consistency is not binary. It's a spectrum from "every read sees the latest write globally" (linearizability) to "reads may be arbitrarily stale" (eventual). Pick the weakest model that satisfies your product requirements.

Read-your-writes in practice

After a user updates their avatar, they must see the new avatar immediately—even if other users see the old one for seconds. Implementation options:

• Sticky sessions: Route user to the same replica that handled the write (fragile on scale-down).
• Session token with version: Client sends last-write timestamp; server reads from replica at least that fresh.
• Write-through cache: Update cache synchronously on write; reads hit cache first.
• Primary reads for session: User's writes go to primary; their reads also hit primary (others hit replicas).

When to use each model

Powers of two (memorize these)

Other useful numbers

• 1 day = 86,400 seconds ≈ 100K seconds (round for mental math)
• 1 month ≈ 2.5M seconds
• 1 year ≈ 30M seconds
• Peak QPS ≈ 2–3× average (sometimes 10× for viral events)
• 1 server (well-tuned API) ≈ 1K–10K RPS (use 1K for conservative estimates)
• 1 SSD ≈ 100K IOPS; 1 HDD ≈ 100 IOPS
• 1 Gbps = 125 MB/s; 10 Gbps = 1.25 GB/s

Estimation framework (5 steps)

1. Clarify assumptions: DAU, actions per user per day, read/write ratio, payload size.
2. Calculate QPS: (DAU × ops/user) / 86400; multiply by peak factor.
3. Calculate storage: DAU × ops × bytes/record × retention days.
4. Calculate bandwidth: peak QPS × average response/request size.
5. Calculate servers: peak QPS / per-server capacity; add 30% headroom.

Worked example: Twitter-scale timeline reads

Assumptions:

• 300M DAU
• Each user reads timeline 10×/day (scroll sessions)
• Each user posts 0.5 tweets/day (150M tweets/day)
• Peak traffic = 3× average
• Tweet size ≈ 300 bytes; timeline page ≈ 20 tweets = 6 KB response
• Retention: 5 years of tweets

READ QPS
  = 300M DAU × 10 reads / 86,400 sec
  = 3B / 86,400 ≈ 35,000 avg
  Peak = 35K × 3 ≈ 105,000 read QPS

WRITE QPS
  = 300M × 0.5 / 86,400 ≈ 1,750 avg
  Peak ≈ 5,250 write QPS

STORAGE (5 years)
  = 150M tweets/day × 300 bytes × 365 × 5
  = 150M × 300 × 1,825 ≈ 82 × 10^15 bytes ≈ 82 PB raw
  With indexes (3×) ≈ 250 PB

BANDWIDTH (peak reads)
  = 105K RPS × 6 KB ≈ 630 MB/s ≈ 5 Gbps

APP SERVERS (stateless API @ 1K RPS each)
  = 105K / 1K ≈ 105 servers + 30% headroom ≈ 140 servers