BYOC Client Authentication Failures with Healthy Broker Monitoring

Issue

In BYOC deployments, clients may experience connection failures while Redpanda's broker-side monitoring shows all nodes healthy. This occurs when the failure happens at the GCP infrastructure layer between client pods and Redpanda brokers, a layer Redpanda cannot directly observe.

Symptoms

Client-side logs show:

- Error message: "Connection to node X terminated during authentication"
- Errors may reference specific broker hostnames and IPs
- Clients report timeouts or inability to produce/consume

Redpanda monitoring shows healthy:

- rpk redpanda admin brokers list reports all brokers with IS-ALIVE=true
- Broker logs contain no errors during the incident window
- Kafka RPC connection metrics remain stable
- BYOC agent logs show no infrastructure failures

Root Cause

Authentication failures occur at the GCP infrastructure layer (Load Balancer, TLS termination, network policies) that sits between client applications and Redpanda brokers. This layer exists in the customer's GCP project and is not directly observable by Redpanda's standard monitoring.

Redpanda's health checks (internal Raft heartbeats, Admin API queries, Kubernetes probes) validate broker process health, but do not exercise the same infrastructure path that client connections use.

Resolution

1. Check GCP Load Balancer logs

Access your GCP project's Load Balancer logs for the affected time window. Filter for the specific backend IP/port shown in client error messages. Look for:

- Backend health check failures
- SSL/TLS handshake errors
- Backend state transitions (HEALTHY → UNHEALTHY → HEALTHY)

2. Verify certificate status

Check your GCP project for certificate-related events during the incident:

- Load Balancer SSL certificate rotations or updates
- GCP-managed certificate renewal operations
- Certificate expiration warnings

3. Review network policy changes

Check GCP VPC/firewall rule change logs:

- Network policy modifications during the incident window
- Rules that may treat health check traffic differently from client traffic

4. Collect K8s events (optional)

Contact Redpanda Support to retrieve Kubernetes events for the affected broker pod during the incident window via the BYOC agent.