Problem Statement

You may see old records are consumable from a topic whereas they were ingested older than the configured value in retention.ms, 604800000 (7 days) by default.

Solution

One possibility is events have timestamps which are based on dates in the future. This is discussed further in :   How to deal with bogus retention timestamps

Another common cause is that you aren't producing enough data to roll the active segments and thereby make them ready to be evicted.  The solutions for this scenario are detailed below : 

1. Produce more data to the topic that triggers rolling the active segment.
2. Manually trigger rolling the active segment. It is rolled under the following conditions:
   1. The segment size reaches the configured threshold as specified by segment.bytes or segment.ms (available from 23.1).
   2. The partition leadership changes.

For solution 2-1 above, use the following command to change segment.bytes or segment.ms:

rpk topic alter-config  -c segment.bytes=

rpk topic alter-config  -c segment.ms=

 

For solution 2-2, partition leadership change is triggered when:

a. The broker that serves the partition stops

b. The broker that serves the partition enters maintenance mode

c. Leadership transfer request is explicitly invoked

 

For the condition c above, in order to invoke leadership transfer manually, use the following command:

# Run the command on the current leader host
curl -X POST http://localhost:9644/v1/partitions/kafka///transfer_le

 

Notes

The active segment still carries the records in question, which isn't taken into account in the data eviction.

Redpanda's retention policy works at a segment file level, not at a record level. Neither retention.ms nor retention.bytes guarantees a record is actually evicted as soon as a given record is beyond the retention policy.

Run the following command to identify the start offset carried by the current active segment:

ls -lrt /var/lib/redpanda/data/kafka//_/

The newest file is the active segment. If the file name looks like 7-3-v1.log, it carries from offset 7 therefore offsets greater than 7 won't be deleted until the segment is rolled.