Summary: 

In Redpanda versions 26.1.1 through 26.1.12, the Cloud Topics housekeeping loop calculated retention and advanced the log start offset without checking the topic's cleanup policy. On a topic with cleanup.policy=compact, retention was applied even though compaction is meant to retain the latest record for every key. 

All compacted topics created with the cloud storage mode are at risk, whether set explicitly with the redpanda.storage.mode topic property or by using the default_redpanda_storage_mode cluster property.

Special attention is needed for the _schemas topic, which backs the Schema Registry. If this topic was created as with the cloud storage mode, a broker restart after triggering this bug can result in the Schema Registry failing and Kafka applications being unable to use schemas.

Clusters that have no compacted Cloud Topics and have not set the default_redpanda_storage_mode property to cloud are not affected.

Severity: 

High

Redpanda Products Affected:

• Redpanda Self-Managed - Enterprise

• Redpanda Self-Managed - Community

Release Affected:

Redpanda versions 26.1.1 - 26.1.12

Fixed in Redpanda version 26.1.13 and all later 26.1 releases. Redpanda 26.2 releases do not contain the bug.

Identification:

Check your Redpanda version and whether Cloud Topics is in use:

rpk cluster config get cloud_topics_enabled 

If you observe cloud_topics_enabled is false, you are not impacted. 

You are at risk if you are running 26.1.1 through 26.1.12 and any topic with cleanup.policy=compact is in the cloud storage mode, whether set explicitly with a topic property or by using the default_redpanda_storage_mode cluster property.

rpk cluster config get default_redpanda_storage_mode

If this returns cloud, it was likely set at cluster creation or through automation, and it is likely all of your topics have their storage mode set to cloud, and all new topics will as well, you will need to continue checking topics, and update default_redpanda_storage_mode per the Action section.

Since the Schema Registry relies on the internal _schemas compacted topic, check it first:

rpk topic describe _schemas -c

You are at risk if the output shows both of the following:

cleanup.policy compact 

redpanda.storage.mode cloud

To check whether records have already been removed, look at the partition offsets:

rpk topic describe _schemas -p

On a healthy topic, LOG-START-OFFSET is 0. A LOG-START-OFFSET greater than 0 on a compacted topic means retention has already been applied and records have been deleted.

Perform the same checks on other topics to identify user-created Kafka topics:

rpk topic describe <my-compacted-topic> -c

rpk topic describe <my-compacted-topic> -p

Impact: 

Records are permanently deleted from compacted Cloud Topics. 

For the _schemas topic, the consequences are cluster-wide:

• Schema Registry does not become ready after a broker restart and returns errors such as 40002 "Fetch returned with error" or 50002 "No records returned in range [0, N)".

• Registered schemas, subjects, versions, and compatibility settings are lost.

• Producers and consumers that depend on schema IDs fail to encode or decode. Data written during the outage may be rejected or routed to a dead letter queue.

• Re-registering schemas assigns new schema IDs, so previously written messages that embed the original IDs cannot be decoded until the original IDs are restored.

• The same data loss applies to any other compacted topic in the cloud storage mode.

Action required: 

Upgrade

Upgrade to Redpanda 26.1.13 or later. This stops any further deletion of records on compacted Cloud Topics. Upgrading does not restore records that have already been deleted.

If you are on 26.1.1 through 26.1.12 and have set default_redpanda_storage_mode=cloud, do not restart and contact Redpanda Support

Mitigation - If you cannot upgrade immediately

Keep compacted topics off the Cloud Topics path by setting the cluster default storage mode to tiered so that newly created and re-created internal topics do not inherit cloud:

rpk cluster config set default_redpanda_storage_mode tiered

Retention can be blocked entirely by setting infinite retention on the topic:

rpk topic alter-config <topic name> --set retention.ms=-1 --set retention.bytes=-1

This prevents retention from being applied to Cloud Topics, but it also blocks retention for Tiered Storage on the same cluster and will cause storage growth. Use it only as a short term measure until you can upgrade, and revert it afterwards.

If you have already been impacted

Do not restart or attempt to upgrade, instead reach out to Redpanda Support and we will assist you.

Questions? If you have any questions on this TSB, or need further guidance, please contact Redpanda Support