Problem Statement

When a client fails a kafka authentication, Redpanda logs record the following WARN message. 

WARN 2024-09-30 01:16:54,959 [shard 0:main] kafka - connection_context.cc:882 - Error
 processing request: kafka::kafka_authentication_exception (requests.cc:176 - 3: 
Unexpected request during authentication)

How can we find a source IP address to fix the authentication failure? 

Solution

1. Enable trace log-level for kafka subsystem on the brokers where the WARN messages were observed for enough time to capture the error
 - rpk redpanda admin config log-level set kafka -l traace -e 120
 -  Please use the smallest period as kafka subsystem may create many log messages. 

2. Here is an example of trace log messages

TRACE 2024-09-30 01:16:54,957 [shard 0:main] kafka - server.cc:273 - kafka rpc protocol
 - Incoming connection from 192.168.207.1:61743 on "OUTSIDE"
WARN 2024-09-30 01:16:54,959 [shard 0:main] kafka - connection_context.cc:882 - Error 
processing request: kafka::kafka_authentication_exception (requests.cc:176 - 3: 
Unexpected request during authentication)
DEBUG 2024-09-30 01:16:54,960 [shard 0:main] kafka - connection_context.cc:192 - 
Connection input_shutdown; aborting operations for 192.168.207.1:61743

3. Review the IP address and determine where the source IP address originated.
 - Note: Depending on the network flow between the originated source and the Redpanda cluster, the source IP in the trace and debug logs may differ from the originated client IP address. So, you may need to understand the network architecture to identify the source that originated.

Notes

The information, such as a request or correlation ID, would not be available in the logs, as authentication is required before retrieving this info. 

If you have any further questions or need further guidance, please visit support.redpanda.com