How to manage consumer group offsets in Redpanda

Overview

Consumer offsets used by Consumer Groups are are stored in internal topic
__consumer_offsets

further details are in the following  Redpanda Documentation : 
https://docs.redpanda.com/current/develop/consume-data/consumer-offsets/

There may be different requirements depending on use case as to how you want to manage older offsets that are no longer needed. 

For example,  if a deployment had lots of consumer groups which were used for testing, where the tests are now complete and the offsets/groups will not be need in the future.

This article details two different options for managing consumer offsets.

One is to let Redpanda  automatically handle the deletion of consumer offsets for empty groups
Note : this is the default for new clusters created  on Redpanda version 23.1 or higher 

The other is to manage them manually.

These options are outlined below. 
(Please note if you are a Redpanda Cloud customer and need to adjust any of the parameters listed in options below , please log a Support Ticket referencing this article and what you would like to change)

 

Detail

1. Redpanda remove consumer offsets for empty groups automatically 

For Redpanda environments created on release 23.1 or higher..
the default behaviour is for Redpanda to delete consumer offsets for Consumer Groups that are in an  Empty state (the group has no no members)  after the retention time specified in parameter 
group_offset_retention_sec

The parameter that controls how often Redpanda checks for consumer  offsets which are eligible for removing is:  group_offset_retention_check_ms

For Redpanda environments that have been upgraded to 23.1 from earlier versions,
(22.3.x for example) then automatic deletion of consumer offsets for empty groups 
is not enabled by default.

To enable automatic consumer offset deletion,  will need to set this parameter to true 
legacy_group_offset_retention_enabled
e.g 

rpk cluster config set legacy_group_offset_retention_enabled true

2. Manually remove consumer offsets for empty groups as required

If you do not want consumer offsets to be removed automatically by Redpanda, then set
parameter -   group_offset_retention_sec
to a value of  'null'

e.g

rpk cluster config set group_offset_retention_sec null

 

It is possible to manually delete offsets from an empty Consumer group  via command :

rpk group-offset delete

If the empty consumer group is no longer required at all then you can delete via  command:

rpk group delete