Stateless vs. Stateful (The Real-World Architecture)

March 5, 20264 min read
system designhigh level designHLDdistributed systemsscalabilitymicroservicesload balancingcachingdatabase designAPI designsoftware architecture

The Stateful Server Problem

Current design: Servers are both application AND database.

Each server:

  • Runs application code
  • Stores data locally
  • Acts as database

This creates problems.

Problem 1: Resource Coupling 💰

Application servers need:

  • Powerful CPU (processing logic)
  • Large RAM (active requests)
  • Fast network (many connections)
  • Minimal disk (no data storage)

Database servers need:

  • Moderate CPU (query processing)
  • Large RAM (caching)
  • Massive disk (data storage)
  • Moderate network (fewer connections)

Combined server requirements:

  • Best CPU + Best RAM + Best disk + Best network
  • = Extremely expensive

Must over-provision every resource.

Problem 2: Deployment Coupling 🚀

Microsoft: 2,000+ deployments per day

Scenario: Deploy new code to Server A

During deployment:

  • Server A's application updating
  • Server A's data inaccessible (only Server A can access its data)
  • Data unavailable during deployment

Impact:

  • 2,000 deployments/day
  • Servers constantly updating
  • Data frequently unavailable
  • Unacceptable user experience

Problem 3: Developer Complexity 🧠

Developers must:

  • Track sharding logic
  • Know which data lives where
  • Manage data locality in code
  • Handle cross-server queries

High cognitive overhead for business logic development.

The Solution: Decoupled Architecture 🎯

Separate application and database layers:

Clients → App Load Balancer → Application Servers (stateless) ↓ DB Load Balancer → Database Servers (stateful)

Application Layer (Stateless)

Characteristics:

  • No data storage
  • Process requests only
  • All data in database

Load balancing: Round robin

Why round robin works:

  • All app servers identical
  • Any server can handle any request
  • No data locality concerns

Request flow:

  1. User request → Any app server (round robin)
  2. App server fetches data from database
  3. App server processes request
  4. App server returns response

No consistent hashing needed.

Database Layer (Stateful)

Characteristics:

  • Stores all data
  • Sharded using consistent hashing
  • Managed by database team

Load balancing: Consistent hashing

Who implements:

  • MySQL developers
  • MongoDB developers
  • Redis developers
  • PostgreSQL developers

Application developers: Don't worry about it.

Benefits of Decoupling

1. Independent Scaling

Scale separately based on needs:

  • App servers: Add more CPU/network capacity
  • Database servers: Add more disk/storage capacity
  • Optimize costs

2. Deployment Independence

Deploy application code freely:

  • Update app servers 2,000 times/day
  • Zero database downtime
  • Zero data unavailability
  • User experience unaffected

3. Developer Simplicity

Application developers see:

  • "One giant database"
  • No sharding logic in code
  • Clean abstraction

Database handles:

  • All data distribution
  • Consistent hashing
  • Server management

Clear separation of concerns.

4. Specialization

Database experts: Optimize data layer Application developers: Focus on business logic

Each team owns their domain.

Real-World Implementation

Application Load Balancer

Use: Round robin (or weighted round robin)

Configuration:

  • AWS Elastic Load Balancer
  • Nginx
  • HAProxy

Setup: Simple, well-documented

Database Load Balancer

Use: Consistent hashing (built into databases)

Databases with native sharding:

  • MongoDB (automatic)
  • Redis Cluster (automatic)
  • Cassandra (automatic)

SQL databases:

  • PostgreSQL: No native sharding (use managed services)
  • MySQL: No native sharding (use managed services)

Managed services add sharding:

  • AWS RDS
  • Google Cloud SQL
  • Azure Database

Who Uses Consistent Hashing?

Application Developers: Rarely

Use cases:

  • Stateless app servers → Round robin
  • Let database handle distribution
  • Focus on business logic

Database Developers: Always

Use cases:

  • Build consistent hashing into database
  • Manage sharding transparently
  • Handle data distribution

DevOps Engineers: Sometimes

Use cases:

  • Configure app load balancers (round robin)
  • Set up database clusters (database handles sharding)
  • May configure caching layers (consistent hashing)

Exception: Stateful Systems

When you need consistent hashing in application layer:

1. Caching Systems

  • Redis cache clusters
  • Memcached clusters
  • Application-managed caching

2. Session Storage

  • Distributed session stores
  • Real-time collaboration tools

3. WebSocket Servers

  • Long-lived connections
  • User pinned to specific server

These require consistent hashing at application level.

Key Takeaways

  1. Stateful servers couple resources, deployment, and logic
  2. Decoupled architecture separates concerns
  3. Application layer: Stateless, uses round robin
  4. Database layer: Stateful, uses consistent hashing
  5. Application developers rarely implement consistent hashing
  6. Database developers implement it in database internals
  7. Exceptions exist: Caching, sessions, WebSockets
Back to 2. Load Balancing And Consistent Hashing