Skip to main content
ThitaInfo BlogsThitaInfo Blogs

Command Palette

Search for a command to run...

Building Scalable Systems: Caching, Rate Limiting, and Observability

Updated
•18 min read•View as Markdown
Building Scalable Systems: Caching, Rate Limiting, and Observability
S
Saurabh Prajapati

Building Scalable Systems: Caching, Rate Limiting, and Observability

TL;DR: Scaling isn't just about adding more servers. It's about protecting your system with intelligent caching, guarding your APIs with rate limiting, and gaining enough visibility through observability to understand what's happening before things fall apart. This article walks through all three layers with real code and architecture diagrams.

Introduction

Imagine it's 9:00 AM on a Tuesday. Your startup just got featured on a major tech publication. In the next 90 seconds, your database CPU hits 100%, response times climb past 10 seconds, and your on-call phone starts ringing. You had 5,000 daily active users yesterday. You now have 200,000 simultaneous requests.

This isn't a hypothetical. It's the Slashdot effect, the Hacker News hug of death, the viral moment every startup fears and secretly dreams about — and most systems aren't built to survive it.

Scalability is the property of a system to handle growing amounts of work gracefully. But the word "gracefully" is doing a lot of heavy lifting. A system that collapses at 10x traffic isn't just slow — it's completely unavailable. A system that scales gracefully degrades predictably, protects its core resources, and gives operators enough visibility to respond.

This article is for backend engineers and architects who want to move beyond "it works on my machine" and build systems that survive real-world load. We assume familiarity with web servers, databases, and at least one backend language. We'll use TypeScript/Node.js for code examples, but the concepts apply universally.

Part 1: Why Applications Struggle to Scale

Before we solve the problem, we need to understand where the pain actually comes from.

Vertical scaling means making one machine bigger — more CPU, more RAM, faster disks. It works, but it has a hard ceiling and it creates a single point of failure. Horizontal scaling means adding more machines and distributing load across them. It's theoretically unbounded, but it introduces distributed systems complexity: you now have to think about shared state, consistency, and coordination.

The most common bottlenecks in web applications are:

  • Database queries — especially N+1 query patterns and missing indexes
  • Synchronous blocking I/O — waiting on network calls serially
  • Repeated computation — calculating the same result for every request
  • Stateful sessions — server-side sessions that can't be load-balanced away
  • Unprotected endpoints — no rate limiting, so a single abusive client can saturate resources

The first three bottlenecks are where caching helps. The fourth is solved by moving to JWTs or distributed session stores. The fifth is where rate limiting becomes essential.

Architecture diagram showing vertical vs horizontal scaling and common web application bottlenecks including database, computation, and I/O layers

Part 2: Caching — The First Line of Defense

Caching is the practice of storing the result of an expensive operation so that future requests for the same result can be served faster. The fundamental insight is that most real-world applications have highly skewed access patterns: a tiny fraction of your data accounts for the vast majority of reads.

Cache Hit vs. Cache Miss

When a request comes in:

  • Cache hit: The data is in the cache. Return it immediately. No database query needed.
  • Cache miss: The data isn't cached. Fetch it from the source, store it in the cache, then return it.
flowchart LR
    Client([Client]) -->|Request| App[Application Server]
    App -->|Check cache| Cache[(Cache)]
    Cache -->|Hit ✓| App
    Cache -->|Miss ✗| DB[(Database)]
    DB -->|Result| App
    App -->|Store in cache| Cache
    App -->|Response| Client

The goal is to maximize your cache hit ratio — the percentage of requests served from cache. A cache hit ratio of 95% means your database only sees 5% of the traffic it would otherwise receive.

Where to Cache

Layer Location Latency Use Case
Browser Cache Client device 0ms Static assets, API responses with Cache-Control headers
CDN Cache Edge nodes (geographically distributed) 5–50ms HTML pages, images, public API responses
Application Cache In-process memory (e.g., LRU cache) <1ms Frequently read config, small lookup tables
Distributed Cache Redis, Memcached 0.5–5ms Session data, computed aggregates, database query results
Database Cache Query cache, buffer pool 0.1–2ms Repeated identical queries, hot table pages

Implementing an Application Cache with Redis

Here's a realistic cache-aside pattern in Node.js with Redis. The key insight is that the application code is responsible for populating and invalidating the cache.

import { createClient } from 'redis';
import { pool } from './database';

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

async function getProductById(productId: string): Promise<Product | null> {
  const cacheKey = `product:${productId}`;
  const ttlSeconds = 300; // 5 minutes

  // 1. Check cache first
  const cached = await redis.get(cacheKey);
  if (cached) {
    console.log(`[CACHE HIT] ${cacheKey}`);
    return JSON.parse(cached);
  }

  // 2. Cache miss — query the database
  console.log(`[CACHE MISS] ${cacheKey}`);
  const result = await pool.query(
    'SELECT * FROM products WHERE id = $1 AND deleted_at IS NULL',
    [productId]
  );

  const product = result.rows[0] ?? null;
  if (product) {
    // 3. Populate the cache for future requests
    await redis.setEx(cacheKey, ttlSeconds, JSON.stringify(product));
  }

  return product;
}

Part 3: Cache Invalidation — The Hard Problem

Note: There are only two hard things in computer science: cache invalidation, naming things, and off-by-one errors. — Phil Karlton (extended)

The challenge with caching isn't reading from it — it's keeping it consistent with the source of truth. Stale data in a cache means users see outdated information. Too-aggressive invalidation defeats the purpose of caching.

TTL and Jitter

The simplest invalidation strategy is Time To Live (TTL): every cache entry expires after N seconds. The risk is a thundering herd: if thousands of cache entries for the same data all expire simultaneously, you get a flood of cache misses hitting your database at once.

Jitter solves this by adding randomness to TTLs:

function getTtlWithJitter(baseTtlSeconds: number, jitterPercent: number = 0.1): number {
  const maxJitter = baseTtlSeconds * jitterPercent;
  const jitter = Math.random() * maxJitter;
  return Math.floor(baseTtlSeconds + jitter);
}

// Base TTL of 300s, actual TTL will be 300–330s
const ttl = getTtlWithJitter(300);
await redis.setEx(cacheKey, ttl, JSON.stringify(data));

Probabilistic Early Expiration (XFetch)

A smarter approach is Probabilistic Early Expiration, also known as XFetch. Instead of waiting for a cache entry to expire, each request checks probabilistically whether it should recompute the value early — with the probability increasing as the expiration time approaches.

interface CacheEntry<T> {
  value: T;
  computeTime: number; // milliseconds the computation took
  expiresAt: number;   // Unix timestamp in seconds
}

async function fetchWithXFetch<T>(
  cacheKey: string,
  fetchFn: () => Promise<T>,
  ttlSeconds: number,
  beta: number = 1.0
): Promise<T> {
  const raw = await redis.get(cacheKey);

  if (raw) {
    const entry: CacheEntry<T> = JSON.parse(raw);
    const now = Date.now() / 1000;
    // Compute early revalidation probability
    const shouldRecompute =
      now - entry.computeTime * beta * Math.log(Math.random()) >= entry.expiresAt;

    if (!shouldRecompute) {
      return entry.value;
    }
    // Fall through to recompute
  }

  const start = Date.now();
  const value = await fetchFn();
  const computeTime = (Date.now() - start) / 1000;

  const entry: CacheEntry<T> = {
    value,
    computeTime,
    expiresAt: Date.now() / 1000 + ttlSeconds,
  };

  await redis.setEx(cacheKey, ttlSeconds, JSON.stringify(entry));
  return value;
}

Stale-While-Revalidate

The Stale-While-Revalidate pattern serves stale data immediately while refreshing it in the background. The user never waits for recomputation. This trades consistency for latency — perfect for data where a few seconds of staleness is acceptable.

async function getWithSWR<T>(
  cacheKey: string,
  fetchFn: () => Promise<T>,
  staleTtl: number,
  backgroundRefreshTtl: number
): Promise<T> {
  const raw = await redis.get(cacheKey);

  if (raw) {
    const { value, cachedAt } = JSON.parse(raw);
    const ageSeconds = (Date.now() - cachedAt) / 1000;

    if (ageSeconds > staleTtl) {
      // Serve stale, revalidate in background
      fetchFn().then(async (freshValue) => {
        await redis.setEx(
          cacheKey,
          backgroundRefreshTtl,
          JSON.stringify({ value: freshValue, cachedAt: Date.now() })
        );
      }).catch(console.error);
    }

    return value;
  }

  // No cache entry — must wait
  const value = await fetchFn();
  await redis.setEx(
    cacheKey,
    backgroundRefreshTtl,
    JSON.stringify({ value, cachedAt: Date.now() })
  );
  return value;
}

Comparison diagram of cache protection strategies including TTL with jitter, probabilistic early expiration, mutex locking, and stale-while-revalidate

Part 4: Rate Limiting — Protecting Your System

A cache protects you from repeated computation. A rate limiter protects you from abuse, bot traffic, and traffic spikes that would overwhelm your infrastructure even if caching is perfect. Rate limiting is the answer to: "What if someone sends 10,000 requests per second to a single endpoint?"

Common Rate Limiting Algorithms

Fixed Window: Count requests in a fixed time window (e.g., 100 requests per minute). Simple to implement, but has a boundary burst problem — a client can send 100 requests at 0:59 and 100 more at 1:01, for 200 requests in 2 seconds.

Sliding Window: Track the exact timestamp of each request and count only those within the last N seconds. More accurate, more memory intensive.

Token Bucket: A bucket holds N tokens. Each request consumes a token. Tokens refill at a fixed rate. Allows bursting up to the bucket size while enforcing a long-term average rate.

Leaky Bucket: Requests enter a queue (bucket). The queue drains at a fixed rate. Excess requests are dropped. Enforces a smooth output rate, no bursting allowed.

Algorithm Burst Handling Memory Usage Implementation Complexity Best For
Fixed Window Poor (boundary bursts) Low Very simple Internal services, loose limits
Sliding Window Excellent High Moderate Public APIs needing accuracy
Token Bucket Excellent Low Moderate APIs where bursting is acceptable
Leaky Bucket None (by design) Low Moderate Smoothing traffic to downstream services

Implementing a Sliding Window Rate Limiter with Redis

async function slidingWindowRateLimiter(
  identifier: string, // e.g., IP address or user ID
  maxRequests: number,
  windowSeconds: number
): Promise<{ allowed: boolean; remaining: number; resetAt: number }> {
  const now = Date.now();
  const windowStart = now - windowSeconds * 1000;
  const key = `ratelimit:${identifier}`;

  const pipeline = redis.multi();
  // Remove timestamps older than the window
  pipeline.zRemRangeByScore(key, '-inf', windowStart.toString());
  // Add current request timestamp
  pipeline.zAdd(key, [{ score: now, value: now.toString() }]);
  // Count requests in current window
  pipeline.zCard(key);
  // Set key expiration
  pipeline.expire(key, windowSeconds);

  const results = await pipeline.exec();
  const requestCount = results[2] as number;

  const allowed = requestCount <= maxRequests;
  const remaining = Math.max(0, maxRequests - requestCount);
  const resetAt = Math.floor((now + windowSeconds * 1000) / 1000);

  return { allowed, remaining, resetAt };
}

// Express middleware
export function rateLimitMiddleware(
  maxRequests: number = 100,
  windowSeconds: number = 60
) {
  return async (req: Request, res: Response, next: NextFunction) => {
    const identifier = req.ip ?? 'unknown';
    const { allowed, remaining, resetAt } = await slidingWindowRateLimiter(
      identifier, maxRequests, windowSeconds
    );

    res.setHeader('X-RateLimit-Limit', maxRequests);
    res.setHeader('X-RateLimit-Remaining', remaining);
    res.setHeader('X-RateLimit-Reset', resetAt);

    if (!allowed) {
      return res.status(429).json({
        error: 'Too Many Requests',
        message: `Rate limit exceeded. Try again after ${new Date(resetAt * 1000).toISOString()}`,
      });
    }

    next();
  };
}

Tip: Rate limit by user ID when authenticated (not just IP). Behind corporate NAT or a VPN, thousands of users can share a single IP address, causing false positives.

Part 5: Observability — Understanding Your System

You've added caching and rate limiting. Your system handles 10x more traffic. Now comes the question that will determine whether you can sleep at night: Do you know what's happening inside your system?

This is where most teams underinvest. They add logging as an afterthought, set up basic uptime monitoring, and call it done. Then something subtle breaks in production — a cache poisoning bug, a rate limiter mis-configured for one region, a cascade starting in a third-party API — and they spend 6 hours debugging with console.log statements.

Logs → Monitoring → Observability

These three terms are often used interchangeably, but they represent distinct capabilities:

Logging is recording discrete events. A log entry says: "At 14:23:07, user u_8391 requested product p_0042, which took 340ms."

Monitoring is aggregating metrics over time and alerting when they breach thresholds. A monitor says: "Average response time exceeded 500ms for the last 5 minutes. Page the on-call engineer."

Observability is the property of a system that allows you to infer its internal state from its external outputs. An observable system lets you ask: "Why did this specific user's request fail at 14:23?" — and find the answer without deploying new code.

Structured Logging

Unstructured logs (console.log('User 123 failed to checkout')) are a dead end at scale. You can't query them efficiently. Structured logs emit JSON, which log aggregation systems (Datadog, Loki, CloudWatch) can index and query.

import pino from 'pino';

const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  formatters: {
    level: (label) => ({ level: label }),
  },
});

// Good: structured, queryable, consistent field names
logger.info({
  event: 'checkout.initiated',
  userId: 'u_8391',
  cartId: 'cart_7723',
  itemCount: 3,
  totalCents: 4999,
  traceId: req.headers['x-trace-id'],
});

// Good: structured error with context
logger.error({
  event: 'payment.failed',
  userId: 'u_8391',
  cartId: 'cart_7723',
  errorCode: 'INSUFFICIENT_FUNDS',
  provider: 'stripe',
  durationMs: 234,
  traceId: req.headers['x-trace-id'],
});

Key Metrics to Monitor

For any backend service, the RED method captures the most important signals:

  • Rate: Requests per second
  • Errors: Error rate (4xx/5xx as a percentage of total requests)
  • Duration: Latency distribution (p50, p95, p99)

For infrastructure resources, the USE method:

  • Utilization: How busy is the resource? (CPU %, memory %)
  • Saturation: How much work is queued? (queue depth, run queue length)
  • Errors: Error events from the resource (disk errors, network drops)

Introduction to OpenTelemetry

OpenTelemetry (OTel) is the open standard for generating, collecting, and exporting telemetry data: traces, metrics, and logs. Before OTel, you had to instrument your code once for Datadog, once for Jaeger, once for Prometheus. OTel separates instrumentation from the backend — instrument once, export anywhere.

The three signals in OTel:

  • Traces: A tree of spans representing a single request as it flows through multiple services
  • Metrics: Numeric measurements over time (counters, gauges, histograms)
  • Logs: Structured event records, ideally correlated with a trace ID
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: 'checkout-service',
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

With auto-instrumentation, every HTTP request, database query, and Redis call automatically generates spans with timing, status codes, and error information — with zero changes to your business logic.

Now you can ask: "Show me all requests slower than 2 seconds in the last hour, broken down by which downstream service caused the latency." That question is unanswerable with logs alone. With distributed tracing, it takes 30 seconds in your tracing UI.

OpenTelemetry architecture diagram showing traces, metrics, and logs flowing from instrumented services through the OTel collector to observability backends

Part 6: Putting It All Together — Production Architecture

Let's describe how a request flows through a production system that combines all three layers.

flowchart TD
    Client([Client]) --> CDN[CDN / Edge Cache]
    CDN -->|Cache Miss| LB[Load Balancer]
    LB --> RateLimit[Rate Limiter\nRedis Sliding Window]
    RateLimit -->|429 Too Many Requests| Client
    RateLimit -->|Allowed| AppServer[Application Server]
    AppServer --> AppCache[Application Cache\nRedis]
    AppCache -->|Cache Hit| AppServer
    AppCache -->|Cache Miss| DB[(Primary Database)]
    DB --> AppServer
    AppServer --> CDN
    AppServer --> OTel[OpenTelemetry Collector]
    OTel --> Traces[Trace Backend\ne.g. Jaeger]
    OTel --> Metrics[Metrics Backend\ne.g. Prometheus]
    OTel --> Logs[Log Backend\ne.g. Loki]
    Metrics --> Alerting[Alerting\ne.g. PagerDuty]

The Request Lifecycle

  1. Client request hits the CDN. If a valid cached response exists (and the Cache-Control headers allow it), it returns immediately — 0 database queries, 0 application server load.
  2. Cache miss at the CDN passes the request to the load balancer, which distributes it across application server instances.
  3. Rate limiter checks whether the client (by IP or user ID) has exceeded their quota. If yes, returns HTTP 429 immediately without touching the application.
  4. Application server checks its Redis application cache. Cache hit serves the response in ~1ms.
  5. Cache miss triggers a database query. The result is stored in Redis with jitter-adjusted TTL.
  6. Throughout all of this, OpenTelemetry automatically creates spans for each operation. The trace, complete with latency breakdown by layer, appears in your tracing UI within seconds.

Trade-offs and Pitfalls

Concern Description Mitigation
Cache stampede Simultaneous cache misses flood the database Mutex locking or Probabilistic Early Expiration
Stale data Users see outdated information after writes Explicit invalidation on write, short TTLs, Stale-While-Revalidate
Rate limit false positives Legitimate users behind shared IPs get blocked Rate limit by authenticated user ID, not just IP
Distributed tracing overhead Sampling every request adds latency and storage costs Use head-based sampling at 10–20% in production
Cache memory pressure Too many keys exhaust Redis memory Set maxmemory-policy allkeys-lru, monitor eviction rate
Over-caching Caching data that changes frequently adds complexity with no benefit Profile access patterns before caching

Warning: Never cache responses that contain user-specific sensitive data (e.g., account balances, PII) in a shared cache without proper key isolation by user ID. A mis-keyed cache entry can return one user's data to another.

Best Practices

  1. Measure before you cache. Use query profiling and APM to identify your actual hot paths before adding a caching layer. Don't cache everything — cache what's expensive and frequently accessed.

  2. Always set a TTL. A cache entry without a TTL that never gets explicitly invalidated is a time bomb. Even a 24-hour TTL is better than infinity.

  3. Add jitter to all TTLs. This is a 5-minute code change that prevents thundering herd problems at scale. Do it from the start.

  4. Rate limit at multiple layers. Apply coarse limits at the CDN/edge (e.g., 10,000 req/min per IP) and fine-grained limits at the application layer (e.g., 100 req/min per authenticated user per endpoint).

  5. Use structured logging from day one. Switching from unstructured to structured logs in a running production system is painful. Start with JSON logs and a consistent schema.

  6. Adopt OpenTelemetry early. The auto-instrumentation libraries require almost no code changes and provide enormous debugging leverage. The cost is minimal; the benefit compounds over time.

  7. Alert on symptoms, not causes. Alert when your p99 latency exceeds 2 seconds (the user-facing symptom), not when CPU exceeds 70% (which may or may not matter). Track what your users experience.

  8. Test your cache invalidation logic. Write explicit tests that verify that after a write, subsequent reads reflect the updated data. Cache bugs are subtle and often only appear under concurrent load.

Conclusion

Scalability is not a feature you add at the end — it's an architectural property you build in from the beginning, and it has three pillars:

Caching eliminates redundant computation and reduces database load by orders of magnitude. The cache-aside pattern with Redis, jittered TTLs, and Stale-While-Revalidate handles the majority of real-world read-heavy workloads.

Rate limiting is the immune system of your API. A sliding window rate limiter implemented in Redis protects against abuse, bot traffic, and accidental thundering herds from your own clients.

Observability — structured logging, RED/USE metrics, and distributed tracing via OpenTelemetry — transforms debugging from archaeology into engineering. When something breaks at 3 AM, you want to spend 10 minutes finding the root cause, not 6 hours.

None of these require a massive up-front investment. A Redis instance, a rate limiting middleware, and the OpenTelemetry Node.js SDK can be added to an existing application in a day. The compounding returns on that day of work — in reliability, debuggability, and capacity — will be felt for years.

Your next step: pick the weakest link. If your database is the bottleneck, start with caching. If you're seeing abuse traffic, start with rate limiting. If incidents take hours to diagnose, start with structured logging and tracing. The architecture described here is built one piece at a time.

Further Reading

#scalability#caching#rate-limiting#observability#opentelemetry
4 views

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

T

ThitaInfo Blogs

78 posts

Making AI simple, fun, and practical for developers.