Problem Statement

An old consumer group has consumed old topics already. Now, the consumer group state is DEAD. That means the consumer group was gone. If you create the same consumer group, it will consume from the first offset. If you don't want to consume the same messages again, how can you create the same consumer group and consume from the latest messages?

Solution

We can use rpk group seek with --to endoption.

1. Make sure the consumer group does not exist.
   rpk group list to check the existing consumer groups
2. Run rpk group seek --to end --topics --allow-new-topics 
3. Check rpk group describe
   1. You can find an Emtpy state and CURRENT-OFFSET and LOG-END-OFFSET are the same (Lag = 0).
4. Add consumer group members. 
   1. The consumers will start consuming the new messages and not consuming the existing ones.

Notes

Without --to endoption, we can create an Empty consumer group and consume from the start offset.
For more information about using the rpk command, please refer to the Redpanda doc rpk group seek.

 

If you have further questions or need additional guidance please reach out to Redpanda Support.