Data Partitioning

Why Data Distribution Matters for Routing

Once you have multiple servers, a non-obvious problem appears. The load balancer cannot just send any request to any server -- it depends entirely on where the data lives.

Consider a bookmarking app. User A's bookmarks are stored on Server 1. User B's bookmarks are on Server 2. If the load balancer sends User A's request to Server 2, that server has no record of User A. The response comes back empty. From the user's perspective, the app lost all their data.

This means before you can decide how to route requests, you need to decide how to distribute data across servers. These two decisions are coupled. Get the data distribution wrong and routing becomes impossible to do correctly.

Why You Cannot Distribute Data Randomly

The only valid reason to store data across multiple servers is that a single server cannot hold all of it. Disk space, memory, and compute are all finite. So the data must be split.

But you cannot split it randomly. If each piece of data lands on a random server at write time, you have no way to find it at read time. You would have to query every server on every read. That defeats the purpose entirely.

The distribution logic must be deterministic and reproducible: given a user or a record, you can always calculate which server holds its data -- without scanning every machine.

Data Sharding

Distributing data across servers using a consistent, logical method is called data sharding. A shard is a subset of the full dataset. Each server holds one shard. Together, all the shards form the complete data.

The key property of sharding is that you can always determine which shard (and therefore which server) holds a given piece of data, just by running the same logic you used when you stored it.

Before going into how sharding works mechanically, it helps to understand two foundational concepts: vertical partitioning and horizontal partitioning. Sharding is the cross-server version of horizontal partitioning.

Vertical Partitioning (Normalization)

Start with a single users table containing: ID, username, password, name, avatar, gender.

The first three columns are authentication data. The last three are profile data. They are related but separate concerns. You can split this into two tables:

• auth_users: ID, username, password
• profiles: user_id, name, avatar, gender (with user_id as a foreign key to auth_users.ID)

This is normalization, and it is also called vertical partitioning because you are slicing the table by columns -- cutting it vertically.

Normalization eliminates data anomalies. Without it, updating or deleting a row in one place can leave inconsistent copies of the same data elsewhere. The primary reason to normalize is to prevent these anomalies: insert anomalies, update anomalies, and delete anomalies.

Horizontal Partitioning

Now take that same users table and instead of splitting by column, split by row.

You could say: all rows where gender = male go into users_male, and all rows where gender = female go into users_female. The two resulting tables have identical columns but different rows. This is horizontal partitioning.

The column you split on is called the partition key (or shard key). In this example it is gender, but any column can serve as the key -- the choice depends on how you query the data.

ExpandVertical vs Horizontal Partitioning

Why would you partition horizontally?

Better indexing performance. A smaller table is faster to index and query. If you are building a railway booking system like IRCTC, Tatkal tickets (time-critical, high-load queries) can be separated into their own table so indexes on that subset stay small and fast.

Data isolation for multi-tenancy. If you are building a product like Slack where many companies share the same infrastructure, storing all companies' messages in one table creates risk. A bug in a query filter could expose Company A's messages to Company B. Separating tenant data into partitions provides a hard boundary that prevents this class of bug.

From One Server to Many: Microservices and Sharding

Both types of partitioning can happen within a single server or across multiple servers.

Vertical partitioning across servers is essentially microservices. The authentication data lives on the auth service's server, and the profile data lives on the profile service's server. Each service owns its slice of data. This is separation of concerns at the infrastructure level.

Horizontal partitioning across servers is sharding. Instead of both tables living on the same machine, users_male lives on Server 1 and users_female lives on Server 2. This is what sharding actually means -- the same table, split by rows, distributed across different physical machines.

Partitioning: Clarifying the Terminology

The word "partition" is a mathematical term. It means splitting a set into non-overlapping subsets. Partitioning itself has no inherent purpose -- the purpose comes from how and why you split.

The relationship between the terms:

• Sharding is horizontal partitioning across multiple servers
• Microservices data separation is vertical partitioning across multiple servers
• Normalisation is vertical partitioning within a single server

Asking "what is the purpose of partitioning?" without specifying which kind is unanswerable. The purpose depends entirely on which type of split you are performing and why.

When to Use Each Type of Horizontal Partitioning

Within a single server: useful for improving index performance on very large tables. If the table is split into time-based or range-based partitions, each index is smaller and queries that target a specific range only scan the relevant partition.

Across multiple servers (sharding) is appropriate when:

• Data volume exceeds a single server's capacity. A single machine has a ceiling on disk size. When the data grows past that ceiling, it must be distributed.
• Read and write throughput. A single database can handle a limited number of reads and writes per second. Distributing rows across shards distributes the I/O load proportionally.
• Multi-tenancy. Isolating different customers' data onto separate shards provides both performance isolation and a cleaner data boundary.
• The hot shard problem. If a particular subset of rows (a celebrity's account, a viral post) receives disproportionate traffic, it can be isolated onto its own shard with dedicated resources.