Client Connection Overload Causing Cluster Instability (leaderless partitions and more)

Issue 

Detailing cluster instability issue due to large/unbounded connections hitting Redpanda Brokers.
 

Detail

Redpanda Cluster was marked as 'unhealthy' via   rpk cluster health and metrics,
with lots of leaderless partitions and under replicated partitions .
( 'leaderless_partitions', 'under_replicated_partitions) 

What was observed in the redpanda logs 

• Lots of  RPC timeouts due to raftgen::append_entries_full_serde , against all brokers/nodes in the cluster.
  
  For example :
  INFO  2026-03-04 11:59:18,170 [shard 2:rs  ] rpc - transport.cc:215 - RPC timeout (3000 ms) to {host: redpanda-0.redpanda-trading.redpanda.svc.cluster.local., port: 33145}, method: raftgen::append_entries_full_serde, correlation id: 11854, 25 in flight, time since: {init: 3008 ms, enqueue: 3008 ms, memory_reserved: 3008 ms dispatch: 3008 ms, written: 3008 ms}, flushed: true
   

   

• lots of  RPC timeouts due to node_status_rpc, against all brokers/nodes in the cluster. 
   
  For example:
  INFO  2026-03-04 12:00:06,911 [shard 0:main] rpc - transport.cc:215 - RPC timeout (100 ms) to {host: redpanda-0.redpanda.svc.cluster.local., port: 33145}, method: node_status_rpc::node_status, correlation id: 8348, 1 in flight, time since: {init: 102 ms, enqueue: 102 ms, memory_reserved: 102 ms dispatch: 102 ms, written: 102 ms}, flushed: true
   
• lots of topics/partitions were stepping down as leaders with a 'reason' of heartbeats_majority
  
  For example:
  INFO  2026-03-04 11:59:59,429 [shard 2:main] raft - [group_id:422, {kafka/humDrumtown/1}] consensus.cc:229 - [heartbeats_majority] Stepping down as leader in term 33, dirty offset 31
   
• messages indicated the internal seastar queue was overloaded 
  
  For example:
  WARN  2026-03-04 11:53:44,348 [shard 2:kafk] seastar - (rate limiting dropped 7 similar messages) Too long queue accumulated for kafka (1237 tasks) 
   

Some of the above log messages can be indicative of a infrastructure issue , either CPU, network or disk...

In this case, the redpanda CPU metric  was not particularly high.

A  self-test was run during this problem state and it reported extremely poor network latency.  
In this case, the self-test results showing poor latency were because it was measuring a degraded system under load (kernel/network stack)  

Looking at the active connections  

via metric redpanda_rpc_active_connections
+ 

rpk cluster connections list

It was clear that the cluster instability was caused by a large amount of connections that were hitting the brokers repeatedly from a problematic client app.  

There were no client connection caps set via [Configure Client Connections] and the infrastructure used in this set up could not handle this load. 
 

Resolution 

Address client behaviour and implement limits for Client Connections 

All connection limit properties are disabled by default ., so they need to be manually enabled.

see Configure Client Connections