Problem Statement

At times, you may notice that load within your Redpanda cluster is not evenly distributed between nodes. This can apply for any metrics, such as CPU utilisation, memory usage, disk usage, etc.

Solution

Although there can be multiple causes of this issue, addressing any leadership imbalance is easy and safe as a first step.

Firstly, ensure that automatic leadership balancing is enabled within the cluster - without this, leadership will not even be balanced across nodes. You can check the current value with:

rpk cluster config get enable_leader_balancer

If this does not return `true`, set this with the following command and monitor the cluster:

rpk cluster config set enable_leader_balancer true

 

If you still see unbalanced load with the leadership balancer enabled, check any high-load topics to see if they have an skew towards leadership on the affected node. This can be checked with:

rpk topic describe -p, for example this topic is skewed entirely to node 1, as denoted by the `LEADER` column:

 

$ rpk topic describe test -p
PARTITION  LEADER  EPOCH  REPLICAS  LOG-START-OFFSET  HIGH-WATERMARK
0          1       2      [0 1 2]   0                 0
1          1       3      [0 1 2]   0                 0
2          1       2      [0 1 2]   0                 0
3          1       2      [0 1 2]   0                 0
4          1       2      [0 1 2]   0                 0
5          1       4      [0 1 2]   0                 0
6          1       4      [0 1 2]   0                 0
7          1       3      [0 1 2]   0                 0
8          1       3      [0 1 2]   0                 0
9          1       4      [0 1 2]   0                 0
10         1       2      [0 1 2]   0                 0
11         1       3      [0 1 2]   0                 0
12         1       4      [0 1 2]   0                 0
13         1       3      [0 1 2]   0                 0
14         1       4      [0 1 2]   0                 0
15         1       2      [0 1 2]   0                 0

 

If you identify any topics which are skewed towards the affected node, you can force movement of these by toggling maintenance mode on the node in question, for example with node `2`:

$ rpk cluster maintenance enable 2 -w
Successfully enabled maintenance mode for node 2
Waiting for node to drain...
NODE-ID DRAINING FINISHED ERRORS PARTITIONS ELIGIBLE TRANSFERRING FAILED
2 false false false 0 0 0 0

...

2 true true false 1 0 0 0

$ rpk cluster maintenance disable 2

Successfully disabled maintenance mode for node 2

 

If, following this, you still see that load in the cluster is not evenly balanced, please open a Support Ticket so we can work with you to resolve this.

 

Notes

Data in a Redpanda cluster is sharded across all nodes, allowing the load on this data to also be distributed. This sharding happens in two ways:

• Partition balancing - this controls which nodes data is actually present on. Each replica of a given partition will be assigned to one node in the cluster, such that each node hosts as close to an even number of these as possible.
• Leadership balancing - for each partition, one replica acts as a leader. It's this leader which takes all client traffic¹ for the partition and replicates it to others, as well as managing things like cloud upload for that partition.

Partition balancing is generally less likely to cause any uneven load provided topics have 3 or more replicas, as any "hot" partitions or topics will have the same effect on both the leader node, and the other replica nodes, as such the load is more evenly distributed within the cluster naturally.

Leadership balancing can more often be the cause of an imbalance in load, particularly if some partitions or topics are "hotter" than others. This can happen because Redpanda attempts to balance leadership across nodes, but not within each topic.

 

¹ An upcoming feature, Follower Fetching, will allow other replicas to serve read only traffic