Summary: 

Fast Commission and Decommission is a feature in Redpanda that leverages Tiered Storage to speed up broker addition or removal by minimizing data transfer, using already-uploaded object storage data to initialize replicas instead of copying all local data between brokers. There is a bug in the Fast Commission process where data can be lost from a topic that previously was configured for Tiered Storage and was subsequently configured to have Tiered Storage disabled.

The bug manifests in the following scenario.

1. A topic has Tiered Storage enabled.
2. Data is written to Tiered Storage for that topic.
3. Tiered Storage is then disabled for that topic.
4. A node is decommissioned, and a new node is added while using Fast Decommission and Commission.

Note: Internal topics (i.e. those prefixed with __ , for example __consumer_offsets) are not impacted by this bug).

Severity: 

High

Redpanda Products Affected:

• Redpanda Self-Managed - Enterprise
• Redpanda Self-Managed - Community
• Redpanda Cloud -BYOC
• Redpanda Cloud - Dedicated

Release Affected:

Redpanda versions 23.3.1 - 24.2.26

Redpanda versions 24.3.1 - 24.3.17

Redpanda versions 25.1.1-25.1.7

Identification:

Topics that have never written to tiered storage are not impacted.  Only topics using fast partition commissioning/decommissioning, and who are using tiered storage and have it disabled could be impacted. 

If you are on an impacted version listed above, have a topic with Tiered Storage explicitly disabled, and use Fast Decommission and Commission you could be impacted.

To check if you have a topic with Tiered Storage disabled, you can run the following and look for any topic with remote.write or remote.read set to false

rpk topic describe -r "*"

To check if you are using Fast Partition Movement you can check at the cluster level and also the topic level.  If these configs have values other than null you have Fast Decommission and Commission enabled.

Cluster Level

• rpk cluster config get initial_retention_local_target_bytes_default
• rpk cluster config get initial_retention_local_target_ms_default

Topic Level - Review the output of rpk topic describe -r "*" looking at the settings below.

• initial.retention.local.target.bytes

• initial.retention.local.target.ms

If you believe you meet all of these criteria, please reach out to Redpanda Support

Impact: 

Data from the impacted topic or topics could be lost during the node decommission process.

Action required: 

Customers on an impacted version should refrain from setting remote.read and remote.write to false for user topics.

If it is critical to have Tiered Storage disabled for a topic you will need to also disable Fast Decommission and Commission, with the command below.  If you have already disabled Tiered Storage, do not re-enable it, instead turn off fast partition movement for that topic, and reach out to Redpanda Support.

rpk topic alter-config --set initial.retention.local.target.bytes=-1 --set initial.retention.local.target.ms=-1

You can also set this at the cluster level, however note that topic level properties will take precedence here.

• rpk cluster config set initial_retention_local_target_bytes_default null
• rpk cluster config set initial_retention_local_target_ms_default null

Redpanda recommends upgrading to the versions containing the fix at the earliest possible opportunity.

Redpanda Cloud

Redpanda Cloud customers clusters will be upgraded automatically to a Patched version, and Fast Decommission and Commission has been temporarily disabled on clusters with topics that have Tiered Storage disabled.  Customers with topics that have tiered storage disabled will be contacted by their Customer Success team. 

Self Hosted Redpanda

Redpanda Self Hosted customers should upgrade to one of the following versions 

25.1.8 - Released 

24.3.18 - Releasing end of week of 07/15/2025 - Track here

24.2.27 - Releasing end of week of 07/15/2025 - Track here 

Questions? If you have any questions on this TSB, or need further guidance, please contact Redpanda Support