Summary: 

Redpanda uses an internal compression implementation in topic compaction when batches are decompressed and recompressed.

In redpanda’s implementation of snappy compression (snappy_java_compressor.cc) , the version and compatible_version fields in the snappy header were erroneously encoded using little-endian format.

Kafka uses a library called snappy-java for its snappy compression, which uses big-endian format to encode and decode these fields. 

Certain Kafka clients also expect these fields to be encoded using big-endian. Kafla clients using these fields for validation may crash due to decompression errors.  

Currently this has only been reported by users of the Kafka-python client, but our testing has also found the Kafka-rust client also appears to be impacted. 

The Github issue  in the kafka-python repo is located here, and Redpanda’s github issue is here.

Severity: 

Medium

Redpanda Products Affected: 

• All Redpanda products 
  • v24.3.x, less than v24.3.6
  • v24.2.x. less than v24.2.19
  • v24.1.x, less than v24.1.19

Impact

Kafka clients using these fields for validation may crash due to decompression errors,this data is not readable via the affected client.  An example error is shown below.

snappy.UncompressError: Error while decompressing: invalid input

Mitigation

If you are experiencing decompression issues with snappy, please report to Redpanda Support  which client, client library, and client version you see the problem with. 

If it is kafka-python, you can upgrade to a version 2.0.4 which contains a fix for this issue.

If you are unable to upgrade, it is recommended to turn off Producer compression of new messages into the affected topics or configure an alternative compression algorithm on the client. This will temporarily prevent the issue of invalid data being published into the topic. 

Action required: 

This issue is patched in current redpanda versions, 24.3.6, 24.2.19 and 24.1.19 and all future versions.

• Community and Self Hosted Enterprise customers will want to upgrade to one of these versions immediately if they are impacted, or as part of regular maintenance if not impacted.
• BYOC, Dedicated Cloud, and Serverless customers will be automatically upgraded to the patched versions during their normal maintenance windows and will not need to do anything.

*This fix does not guarantee success with the decompression of historical batches that were compressed with a Redpanda broker using snappy compression that pre-dates these versions.

Questions? If you have any questions on this TSB, or need further guidance, please contact support@redpanda.com