TSB 2023-07 Cloud storage metadata integrity checks may result in excessive cloud storage API calls
Last updated: August 11, 2026
Executive Summary
Redpanda 23.2 introduced metadata safety checks on Tiered Storage uploads. However on clusters with metadata errors Redpanda will repeatedly make calls to the Cloud Storage API. This excessive polling may result in additional cloud storage charges for some customers (i.e.. PutObject API calls on S3/GCS)
Severity
Medium
Redpanda products affected
Redpanda Enterprise, Redpanda Cloud BYOC deployment
Releases affected
23.2.x prior to 23.2.16
Impact
Excessive polling of cloud storage APIs, potentially resulting in additional cloud storage charges.
Immediate action required
Customers who are running Tiered Storage should review log files for the line “Delta offset of the uploaded segment X doesn't match with expected value of {Y}”. Customers should also review their cloud billing console for excessive charges related to S3/ABFS/GCS. Concerned customers should contact Redpanda Support immediately.
To check the log files:
# journalctl -u redpanda.service | grep 'Delta offset of the uploaded segment [0-9]* doesn.t match with expected value of [0-9{}]*'
Any log entries such as the following could be indicative of a problem:
2023-11-07 17:06:39,426 [shard 2] archival - [fiber223 kafka/_schemas/0] - ntp_archiver_service.cc:1759 - Delta offset of the uploaded segment 298 doesn't match with expected value of {277}
Customers who would like to examine their cloud billing consoles should look for an uptick in PutObject API calls over the past few weeks as this may be indicative of this problem. For further clarification or questions, please contact Redpanda support.
Action required
Affected customers should upgrade to 23.2.16 and contact Redpanda Support to receive assistance in addressing the metadata problems.
Addressed in (RP Version)
23.2.16
Questions?
If you have any questions on this TSB, or need further guidance, please contact customer-success@redpanda.com / support@redpanda.com