How do I count the number of events in a topic?

Issue

 

It is possible to get an estimate on the number of events for a topic using a calculation on the sum of (HighWaterMark - LogStartOffset ) across partitions.



for example from..  

rpk topic describe   -p 

 

This is the calculation used when displaying the number of messages in  Redpanda Console.



It is important to note that this  calculation is only precise if there are no "gaps" between offset numbers.

There are at least the following scenarios where the calculation may not be accurate:

-Compact topics. 

Topics defined with a cleanup.policy of compact*

can have gaps in offsets which are created as part of the compaction process.

This is discussed in detail here : Redpanda/kafka log compaction  

 

- Events produced via transactions. 

These can generate internal control events, which are not read by standard Kafka/Redpanda clients and therefore appear as offset "gaps"

 

Solution

Currently the only way to obtain an accurate count of number of events in a topic is to consume them all while keeping a tally.

rpk topic consume  has a handy  formatting option of %i which will display the actual record count..  This count is accurate regardless of number of partitions, cleanup.policy, offset-gaps etc.

 

Below is an example, where the total number of events consume from this topic is 93027

rpk topic consume redpanda-topic1 -f "count=%i\n"
......
count=93026
count=93027

 

A feature request has been logged to maintain an accurate count without needing to consume all the data :-  #17983