TSB-2023-02 Tiered Storage segment upload issue

Summary

Redpanda identified a bug when Tiered Storage is enabled that may result in data loss under certain circumstances. All managed cloud clusters have been upgraded and are safe. We found a single instance in our managed fleet. Redpanda released a fix for the tiered storage segment reupload bug 24 hours after detection. We urge customers running on the affected versions listed below to either implement the short-term mitigation described further down to reduce the possibility of encountering this bug, or to upgrade to versions 22.3.15 or 23.1.3 as soon as possible.

Data loss will only be noticed if a consumer requests data that is only available on tiered storage - i.e. data that has been aged out of local storage.

 

Severity

High

 

Redpanda Products affected

This issue only affects Redpanda clusters running with Tiered Storage enabled.

 

Releases affected

• 22.3.14 or earlier 22.3.x versions
• 23.1.1 or 23.1.2

Impact

Redpanda’s Tiered Storage solution uploads data segments to cloud storage periodically. In certain error conditions, primarily associated with timeouts when applying updates across brokers, a given segment may get re-uploaded to cloud storage. When this happens, a logical error ends up marking the segment as being a duplicate and ready to be claimed by a housecleaning loop that deletes duplicate data, resulting in that data segment being lost. This data loss will be noticed if data is returned from beyond the local retention settings, as any data on the local disk will still be available.

 

Immediate Action required

Yes

 

Action required

Identifying if the problem has occurred in your environment

 

In order to identify if the problem has occurred in your environment you will need to inspect the broker logs as follows. 

 

$ cat  | grep "new remote segment added" | awk -F'INFO' '{print $2}' | awk '{print $8,$17, $19, $21 }' | sort | uniq -cd

 

If the command does not return any output then your cluster is not affected.

 

If the command returns output, for example as follows, please contact Redpanda support immediately to discuss remediation.

 

Sample output

2 {kafka/by_production_receipt_event_avro_0/37} {6425299-3-v1.log}, 6425299 6465082),

 

Short-term mitigation

To reduce the likelihood of the bug on versions prior to the fixes please adjust the following.

 

For Redpanda 22.3.x and 23.1.x:

• Set cloud_storage_metadata_sync_timeout_ms to 60 seconds (default 10 seconds).
• Set cloud_storage_manifest_upload_timeout_ms to 60 seconds (default 10 seconds)
• If running 23.1.x, set cloud_storage_enable_segment_merging to false (default true)

Note: These settings will require restarting the Redpanda cluster, although this can be a rolling restart. 

 

Full fix

To prevent this problem from occurring we strongly recommend you upgrade to Redpanda 22.3.15 or 23.1.3 at the earliest opportunity.

 

To perform an upgrade of Redpanda please follow the steps at https://docs.redpanda.com/docs/manage/cluster-maintenance/rolling-upgrade/. 

 

Addressed in (Redpanda Version)

22.3.15 and 23.1.3

 

References

Questions?: If you have any questions on this TSB, or need further guidance, please contact the Customer Success Team or raise a support case (support@redpanda.com)