Minor Not leader for partition Errors

Problem Statement

Low occurrence of leadership changes and errors similar to "Cannot get source metadata for pipeline. Failed to get watermark offsets from Kafka with error Broker: Not leader for partition"

Explanation:

If leadership changes are low in occurrence, this can be a normal operation of the system, expected and not a great cause for concern.

Leadership changes can occur due to slight changes in latency between nodes and other factors. This is a normal part of the system's operation and design to maintain overall balance in the cluster as long as the occurrence is low and not high volume.

When a leadership change occurs, some clients may log errors when they are notified of the change, this is because the broker that they were previously interacting with is no longer the leader for that topic partition.

This doesn't in general mean a functional issue with the system, Client libraries are normally designed to handle such scenarios by transparently retrying operations. Despite the logged errors, if infrequent, with a properly operating client library in use data operations should continue without interruption. The client should request new metadata.

Example error logging:

"Cannot get source metadata for pipeline. Failed to get watermark offsets from Kafka with error Broker: Not leader for partition"

 

Note: Cluster health should also be checked and the rate of the errors. If you notice a significant increase in high number of leadership changes or Not leader for partition errors, it is something that would benefit further investigation and possibly contacting support.


Using RPK to check Cluster health for Self-Hosted Customers:
https://docs.redpanda.com/current/reference/rpk/rpk-cluster/rpk-cluster-health/

If you have further questions or need additional guidance please reach out to Redpanda Support.