TSB-2023-04 Possible Inability to Read Committed Offsets

Executive Summary:

A race condition has been identified that manifests when a segment is rolled due to the segment.ms property, there's a chance that incorrect offset accounting happens. This will result in some offsets that are not consumable by clients (even though the data is present in the cluster)

 

Note: Redpanda has not identified any CloudV2 clusters affected by this bug.

Severity

High

 

RedPanda Products affected

Redpanda Enterprise, Redpanda Community

 

Releases affected

v23.1.1 through v23.1.13. v22.3 and v23.2 are unaffected.

 

Impact

Consumers cannot read data that has been committed. This only applies to topics where segment.ms is not disabled.

 

To identify if you have been affected by this condition, you can examine the Redpanda logs for lines similar to the following:

 

WARN  2023-06-13 16:52:46,967 [shard 0] kafka - connection_context.cc:451 - Error processing request: std::runtime_error (ntp {kafka/foo/0}: log offset 9 is outside the translation range (starting at 10))

You can identify these lines as follows:

journalctl -u redpanda.service | grep 'log offset [0-9]* is outside the translation range (starting at [0-9]*)'

Or

grep 'log offset [0-9]* is outside the translation range (starting at [0-9]*)' /var/log/redpanda.log

 

If you identify that you have been affected by this, please contact Redpanda support.

 

Immediate Action required

Yes

Action required

If you are running an affected version, upgrade to v23.1.14 or later or v23.2.x or later.

Addressed in (RP Version): v23.1.14

 

References

If you have any questions on this TSB, or need further guidance, please contact customer-success@redpanda.com / support@redpanda.com