Active Log Segment Count Spike After Rolling Restart

Last updated: August 25, 2026

Issue

After a rolling restart or upgrade/rollback cycle, customers observe a large spike in active log segment file counts and raise concern about disk health.

The increase is typically proportional to the number of partitions and the number of restarts performed.

Cause

This is expected behavior, not a bug.

Every time a broker restarts, Redpanda must:

  1. Roll the active segment on each partition leader — the in-flight segment cannot be re-used safely across a restart boundary.

  2. Trigger leadership reshuffling across the cluster — each broker that is still running re-rolls segments as leadership moves away and returns.

Because of this, a rolling restart costs approximately ~2 new segments per replica, per restart event, across every broker — not just the one being restarted. The total expected increase for a single rolling restart is:

~2 × (number of partitions) × (replication factor)

The segment count ratchets up on every restart and comes down only slowly (via compaction/reclaim). It is a record of restart history, not a measure of current health.

Reference Measurements

Collected on cluster support-test (AWS, 3 brokers, RF=3, v26.1.14) across two independent rolling restarts.

Note this cluster had an artificially inflated segment count by triggering fake leadership churn by running repeated manual leadership transfers on the partitions

Run 1 — 2026-08-21, 09:05–09:17 UTC, 1,896 partitions. Segment increase

Broker

Before

After

Increase

Per replica

blue-a-0

126,941

130,962

+4,021

2.12

blue-a-1

126,169

130,438

+4,269

2.25

blue-a-2

125,709

130,084

+4,375

2.31

Cluster

378,819

391,484

+12,665

2.23

Run 2 — 2026-08-25, 17:43–17:51 UTC, 2,006 partitions. Segment increase

Broker

Before

After

Increase

Per replica

blue-a-0

132,663

137,092

+4,429

2.21

blue-a-1

132,403

136,800

+4,397

2.19

blue-a-2

132,954

137,114

+4,160

2.07

Cluster

398,020

411,006

+12,986

2.16

ℹ Per-broker spread is ±4% around the mean. The broker that gains the most segments does not consistently correlate with restart order — do not read significance into which broker gained most. Use the cluster-level total.

How to Verify

Run the following PromQL queries to assess segment health:

Current active segments per broker:

sum by (pod) (vectorized_storage_log_log_segments_active)

Rate of new segment creation (last 10 min):

sum(rate(vectorized_storage_log_log_segments_created[10m])) * 60

Segments removed (reclaim activity):

sum by (pod) (vectorized_storage_log_log_segments_removed)

When This IS a Problem

Investigate further if any of the following are true:

Signal

What to check

Leadership changes do not settle to ~0 within a few minutes of last broker becoming ready

Election instability — check Redpanda logs

Increase far exceeds ~2.2 × partitions × RF

Repeated or crash-looping restarts — kubectl get pods -n redpanda (check RESTARTS column)

Segments accumulating with no restart

Compare segment creation rate against a quiet baseline

Actual disk pressure observed

Check disk utilization metrics and consider retention tuning

Resolution

If the spike matches the expected formula (~2 × partitions × RF per restart):

✅ No action required. This is normal post-restart behavior.

Segment counts will gradually decrease over time as compaction and log reclaim run.

Note: For topics that are defined with cleanup.policy=compact, in current releases of Redpanda compaction does not compact across raft terms. So if there was a scenario of many restarts/leadership storms you could end up with lots of small segment files which will not be merged. There is an enhancement request to allow compaction to merge across raft terms in future releases (CORE-16877)

If the spike exceeds the expected range or disk pressure is observed:

  • Check for crash-looping brokers: kubectl get pods -n redpanda

  • Check for election instability in broker logs

  • Review retention and compaction settings