TSB 2023-06 A single CreateTopic request that creates a high number of partitions can trigger a crash.

 

Executive Summary

A scenario exists in which when several topics are created concurrently (either with a single CreateTopic or multiple concurrent CreateTopic requests) the result would be more than the allowed number of total partitions in a cluster. This scenario causes a rollback that can cause a crash, and depending on your configuration this can result in a crash loop.

 

The precise number of partitions that triggers this limit is related to the size of the cluster, the number of cores and the number of partitions already on the cluster. There is a configurable limit topic_partitions_per_shard which must not be exceeded; if the limit is exceeded in a single CreateTopic request, the crash loop scenario can be triggered.

Severity

High

RedPanda products affected

Redpanda Community, Redpanda Enterprise, Redpanda Cloud 

Releases affected: All 23.2.x up to 23.2.12, all 23.1.x up to 23.1.19, and all 22.3.x up to 22.3.24

Impact

When impacted, the cluster will crash and become unusable.

Immediate action required: Ensure that a single CreateTopic request does not exceed the configured topic_partitions_per_shard partition/core limit and does not contain multiple topics.

Under normal circumstances this would occur only when using a tool that is likely to bulk create topics such as MirrorMaker – for example, when replicating a large number of topics at once, or having the topic list set to .* regex.

If you are concerned about potentially hitting this limit and are unable to upgrade to one of the versions below, please contact Redpanda Support.

Action required

Review usage (including use of MirrorMaker) and upgrade to one of the versions below if required. 

Addressed in (RP Version)

23.2.13, 23.1.20 and 22.3.25

References

https://github.com/redpanda-data/redpanda/pull/14375 

Questions?

If you have any questions on this TSB, or need further guidance, please contact customer-success@redpanda.com / support@redpanda.com