Brokers intermittently reported down on VMware vSphere due to automatic VM live migration (vMotion)

Issue

On a self-managed cluster running on vSphere VMs, brokers are intermittently reported down although the guest OS shows no restart, no crash and no resource exhaustion. rpk cluster health may briefly list nodes under Nodes down: or report leaderless partitions, and the broker returns on its own within seconds.

Clients see the same brief disappearance as a metadata change and raise errors naming the broker ID. With the Python kafka client, for example, the producer I/O thread logs a traceback ending in:

kafka.errors.UnknownBrokerIdError: UnknownBrokerIdError: 1

Detail

Each episode lasts a few seconds and produces a dense burst of RPC timeouts from the peers of the affected broker, not from the broker itself. For example:

rpc - transport.cc:215 - RPC timeout (3000 ms) to {host: 10.0.0.4, port: 33145}, method: raftgen::append_entries_full_serde
rpc - transport.cc:215 - RPC timeout (4500 ms) to {host: 10.0.0.4, port: 33145}, method: raftgen::vote
rpc - transport.cc:215 - RPC timeout (100 ms) to {host: 10.0.0.4, port: 33145}, method: node_status_rpc::node_status

The node_status timeout is what surfaces as a broker being down. A vote storm follows, with vote_stm.cc entries showing pre-vote and vote requests rejected across many partitions. On the affected broker, seastar reports that the reactor was not running rather than busy:

Reactor stalled for 140 ms on shard 22, in scheduling group main. Backtrace: ...

Distinguishing this from genuine resource pressure. The same signatures appear under CPU, disk or network problems, so check that guest CPU and reactor utilisation are normal, disk latency on the data volume is normal and there is no packet loss. Two patterns point specifically at the hypervisor: every peer times out against the same broker simultaneously and different brokers are affected over time rather than one consistently.

What to request from vCenter for the same window:

• vMotion task history for the broker VMs, to compare migration completion times against the timeout bursts

• Migration count and recommendation history for DRS (Distributed Resource Scheduler, the vSphere feature that balances load by live-migrating VMs between hosts automatically)

• Host CPU Ready and Co-Stop for the broker VMs, plus total allocated vCPU across the ESXi cluster against physical cores available

Root Cause

A vMotion migration ends with a switchover phase in which the guest is paused while final memory and device state transfer to the destination host. Redpanda is not slow during that window, it is stopped.

Its liveness and Raft deadlines are short by design, so a pause of a few hundred milliseconds breaches them. The values below are Redpanda 26.1 defaults, quoted as an example of the order of magnitude; check them against your version, as defaults change between releases and may be overridden on the cluster.

Peers then declare the broker unreachable, Raft elects replacements for every partition it led and cluster metadata changes. The client errors follow from that metadata change rather than any client fault.

Migration frequency is a separate contributor. Where an ESXi cluster is fully provisioned on vCPU and physical CPU, DRS has no room to balance and moves VMs constantly, turning an occasional interruption into recurring cluster-wide instability. Note also that vSphere cannot see Redpanda's health and that Redpanda's busy-polling thread-per-core reactor looks like sustained high CPU to the hypervisor, which makes DRS more inclined to move the VM rather than less.

Resolution

The fix is on the hypervisor side. No Redpanda setting makes a stopped process respond.

1. Exclude the broker VMs from automatic DRS migrations. In vCenter, under the cluster's VM Overrides, set each broker VM's DRS automation level to Manual or Disabled. Manual keeps recommendations visible so migrations can still be performed deliberately in a maintenance window. Do this before any other remediation, as repairing cluster state while brokers are still being migrated unpredictably gives misleading results.

2. Remove the CPU over-commitment driving the migrations. Add ESXi hosts or physical CPU so the cluster is not fully provisioned and allocated vCPU leaves headroom. Without this, DRS stays under pressure to relocate workloads.

3. Apply the standard low-latency VM configuration: reserve 100% of configured memory, reserve CPU and do not oversubscribe vCPU on hosts carrying brokers, set Latency Sensitivity to High, and keep one broker per VM per Requirements and Recommendations.

What not to do. Raising node_status_interval or raft_heartbeat_timeout_ms to absorb the pause masks the symptom and slows genuine failure detection. If a pause ever exceeds partition_autobalancing_node_availability_timeout_sec, the balancer starts evacuating replicas from a broker it believes has failed, moving real data for no reason.

Monitoring / Validation

After excluding the VMs, confirm the following items:

• redpanda_node_status_rpcs_timed_out falls to zero and stays there

• No further RPC timeouts naming raftgen::append_entries_full_serde, raftgen::vote or node_status_rpc::node_status

• No Reactor stalled warnings without corresponding CPU or disk pressure

• Leadership change rate back to baseline, no transient leaderless partitions

• Client-side broker-ID errors drop to zero. A before-and-after count in the customer's log platform is usually the clearest evidence for them

• No new automatic migrations in the vMotion task history