Problem Statement

When Follower Fetching is enabled and if auto.offset.reset is set to earliest or latest, a partition leadership change may result in consumer offset being reset or forwarded to a certain point in time unexpectedly. The auto.offset.reset properties and the corresponding impacts are as follows:

• auto.offset.reset=earliest - the offset is reset back to the earliest point, hence duplicate data being retrieved by consumers and also a lag being observed
• auto.offset.reset=latest - the offset is forwarded to the latest, hence skipping a certain range of offsets

This effect will be visible by analyzing the redpanda_kafka_consumer_group_committed_offset metric.

 

Affected releases are v23.2.x prior to v23.2.18.

 

Solution

Upgrade your cluster to v23.2.18 or higher. Otherwise, disable Follower Fetching.

If your consumer application uses an implementation that supports KIP-842, use "nearest" for auto.offset.reset.on.invalid.offset. For example, franz-go v1.15.3 and higher supports the feature.

 

Notes

Issue details: https://github.com/redpanda-data/redpanda/pull/15357