How do consumers work when assigned to multiple partitions?

 

The partition assignment strategy, is focused on how partitions are distributed among multiple consumers, but not how those consumers will actually consume the messages. 

 

The consumer will send a fetch request to Redpanda for all the partitions it is responsible for at the same time and then the broker will reply with some data from all the parititons based on the client settings (i.e. fetch.message.max.bytes, fetch.max.wait.ms, etc..). It is then up to the client to determine which order to handle those messages.

 

Most will first hand out all the batches from one partition, then from the other, but that is only in the context of one fetch request. Each fetch request might have retrieved only a few messages from one partition or none at all.

 

For example, if you are seeing the single consumer read all the messages from one partition, before moving on to the next partition then it is most likely to do with the message size and the settings of the client. 

 

It is important to keep in mind that ordering guarantees between different partitions is not possible, So it does depend on what you are looking to accomplish. If it is the goal to have the consumer read less messages from one partition, before it moves on to the next partition - then it might be recommended to look at your average message size and adjust the fetch.message.max.bytes setting appropriately.

 

For additional details, the following article, Kafka Consumer Config provides some great insights on consumer configuration options.