Datadog Alerting for Redpanda Cluster Health

Last updated: August 10, 2026

This document provides alert configurations for monitoring Redpanda cluster health using the Datadog integration.

Core Health Metrics

The following metrics are used to determine cluster health:

  • Broker/node availability - redpanda.cluster.brokers
  • Leaderless partitions - redpanda.cluster.unavailable_partitions
  • Under-replicated partitions - redpanda.kafka.under_replicated_replicas

All three metrics are part of the default metric groups and are automatically collected once the integration is configured.

Alert Configurations

1. Broker/Node Down Detection

Metric: redpanda.cluster.brokers

Alert Query:

max(last_5m):sum:redpanda.cluster.brokers{cluster:your-cluster} <

Details:

  • Type: Gauge metric showing configured brokers in the cluster
  • Threshold: Set to your expected broker count (e.g., if you have 3 brokers, alert when < 3)
  • Tags: Use cluster, env, or other custom tags to scope alerts
  • Enable no-data alerts: Yes (10 minute window)

Example Monitor (JSON):

{
 "name": "Redpanda - Broker Down",
 "type": "metric alert",
 "query": "max(last_5m):sum:redpanda.cluster.brokers{cluster:prod} < 3",
 "message": "{{#is_alert}}Redpanda cluster has fewer than expected brokers!\nCurrent: {{value}}\nExpected: 3\n\nCheck:\n- Node health: `rpk cluster health`\n- Systemd status: `systemctl status redpanda`\n{{/is_alert}}",
 "options": {
   "thresholds": {
     "critical": 3
   },
   "notify_no_data": true,
   "no_data_timeframe": 10
 }
}

2. Leaderless Partitions

Metric: redpanda.cluster.unavailable_partitions

Alert Query:
 

max(last_5m):sum:redpanda.cluster.unavailable_partitions{cluster:your-cluster} > 0

Details:

  • Type: Gauge
  • Description: Number of partitions that lack quorum among replicants
  • Critical threshold: > 0 (any leaderless partition is a problem)
  • Severity: Critical - these partitions cannot serve reads/writes
  • Enable no-data alerts: No

Example Monitor (JSON):

{
 "name": "Redpanda - Leaderless Partitions Detected",
 "type": "metric alert",
 "query": "max(last_5m):sum:redpanda.cluster.unavailable_partitions{cluster:prod} > 0",
 "message": "{{#is_alert}}CRITICAL: Redpanda has {{value}} unavailable partitions!\n\nThese partitions lack quorum and cannot serve reads/writes.\n\nInvestigate:\n- `rpk cluster health`\n- `rpk topic describe `\n- Check for network partitions or crashed brokers\n{{/is_alert}}",
 "options": {
   "thresholds": {
     "critical": 0
   },
   "notify_no_data": false
 }
}

3. Under-Replicated Partitions

Metric: redpanda.kafka.under_replicated_replicas

Alert Query:

max(last_15m):sum:redpanda.kafka.under_replicated_replicas{cluster:your-cluster} > 10

Details:
 

  • Type: Gauge per partition
  • Description: Number of under-replicated replicas (replicas that are live but not at the latest offset)
  • Available tags: redpanda_namespace, redpanda_partition, redpanda_topic
  • Recommended thresholds:
    • Warning: > 5 for more than 5 minutes
    • Critical: > 10 or sustained for > 15 minutes
  • Note: Temporary under-replication is normal during partition rebalancing

Example Monitor (JSON):
 

{
 "name": "Redpanda - Under-Replicated Partitions",
 "type": "metric alert",
 "query": "max(last_15m):sum:redpanda.kafka.under_replicated_replicas{cluster:prod} > 10",
 "message": "{{#is_alert}}WARNING: Redpanda has {{value}} under-replicated replicas.\n\nPartitions are not fully replicated. This may be temporary during rebalancing.\n\nCheck:\n- Recent partition movements\n- Disk I/O performance\n- Network connectivity between brokers\n\nQuery per-topic breakdown:\n`sum:redpanda.kafka.under_replicated_replicas{cluster:prod} by {redpanda_topic}`\n{{/is_alert}}",
 "options": {
   "thresholds": {
     "warning": 5,
     "critical": 10
   },
   "evaluation_delay": 300
 }
}

Additional Recommended Health Alerts

Disk Space Alert

Metric: redpanda.storage.disk_free_space_alert

Alert Query (Warning):

max(last_5m):max:redpanda.storage.disk_free_space_alert{cluster:your-cluster} by {host} >= 1

Alert Query (Critical):

max(last_5m):max:redpanda.storage.disk_free_space_alert{cluster:your-cluster} by {host} >= 2

Details:

  • Type: Gauge
  • Values:
    • 0 = OK (normal disk space availability)
    • 1 = Low Space (soft limit reached - alert/warning state)
    • 2 = Degraded (hard limit reached - write rejection state)

Redpanda Configuration:

Soft Limit (Triggers Value 1):

  • Configured by: storage_space_alert_free_threshold_bytes or storage_space_alert_free_threshold_percent
  • Effect: Logs error messages, updates metric to 1
  • Redpanda continues normal operation (no write rejection)

Hard Limit (Triggers Value 2):

  • Configured by: storage_min_free_bytes (default: 5 GiB)
  • Effect: Redpanda rejects writes to ALL brokers in the cluster
  • Cluster enters degraded performance state

Important Notes:
 

  • When ANY broker reaches storage_min_free_bytes, writes are rejected on all brokers cluster-wide
  • Default hard limit of 5 GiB is very low for production - consider increasing based on disk size
  • Additional property disk_reservation_percent (default 25%) reserves disk space to avoid I/O blocking
  • You can disable one threshold by setting it to zero

Recommended Thresholds:

  • Warning (value = 1): Plan capacity remediation, review retention policies
  • Critical (value = 2): Immediate action required - cluster is rejecting writes

Excessive Leadership Changes

Metric: redpanda.raft.leadership_changes

Alert Query:

sum(last_15m):sum:redpanda.raft.leadership_changes{cluster:your-cluster}.as_rate() > 5

Details:

  • High leadership change rate may indicate cluster instability
  • Consider combining with broker count alerts

Inter-Broker Communication Failures

Metric: redpanda.rpc.request_errors

Alert Query:

sum(last_10m):sum:redpanda.rpc.request_errors{cluster:your-cluster}.as_count() > 100

Details:
 

  • Indicates network issues or broker overload
  • May precede partition unavailability

Low Memory Condition

Metric: redpanda.memory.available_memory

Alert Query:

min(last_10m):avg:redpanda.memory.available_memory{cluster:your-cluster} by {host} < 1073741824

Details:

  • Alert when available memory drops below 1GB (1073741824 bytes)
  • Adjust threshold based on your cluster size and workload

Creating Alerts in Datadog

Via Web UI

  1. Navigate to Monitors → New Monitor → Metric
  2. Select the metric (e.g., redpanda.cluster.unavailable_partitions)
  3. Define the evaluation window and thresholds
  4. Configure notification channels (@pagerduty, @slack, @email, etc.)
  5. Add monitor tags for organization (e.g., service:redpanda, team:platform)
  6. Save the monitor

Via Terraform

resource "datadog_monitor" "redpanda_unavailable_partitions" {
 name    = "Redpanda - Leaderless Partitions"
 type    = "metric alert"
 message = "Redpanda has unavailable partitions @pagerduty-critical @slack-ops"
 query   = "max(last_5m):sum:redpanda.cluster.unavailable_partitions{cluster:prod} > 0"
 monitor_thresholds {
   critical = 0
 }
 notify_no_data    = false
 renotify_interval = 60
 priority          = 1
 tags = ["service:redpanda", "team:platform"]
}
resource "datadog_monitor" "redpanda_broker_down" {
 name    = "Redpanda - Broker Down"
 type    = "metric alert"
 message = "Redpanda cluster has fewer brokers than expected @pagerduty-critical @slack-ops"
 query   = "max(last_5m):sum:redpanda.cluster.brokers{cluster:prod} < 3"
 monitor_thresholds {
   critical = 3
 }
 notify_no_data    = true
 no_data_timeframe = 10
 renotify_interval = 30
 priority          = 1
 tags = ["service:redpanda", "team:platform"]
}
resource "datadog_monitor" "redpanda_under_replicated" {
 name    = "Redpanda - Under-Replicated Partitions"
 type    = "metric alert"
 message = "Redpanda has under-replicated partitions @slack-ops"
 query   = "max(last_15m):sum:redpanda.kafka.under_replicated_replicas{cluster:prod} > 10"
 monitor_thresholds {
   warning  = 5
   critical = 10
 }
 notify_no_data    = false
 evaluation_delay  = 300
 renotify_interval = 60
 priority          = 3
 tags = ["service:redpanda", "team:platform"]
}

Via Datadog API

 

curl -X POST "https://api.datadoghq.com/api/v1/monitor" \
 -H "Content-Type: application/json" \
 -H "DD-API-KEY: ${DD_API_KEY}" \
 -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
 -d '{
   "name": "Redpanda - Leaderless Partitions",
   "type": "metric alert",
   "query": "max(last_5m):sum:redpanda.cluster.unavailable_partitions{cluster:prod} > 0",
   "message": "Redpanda has unavailable partitions @pagerduty-critical",
   "options": {
     "thresholds": {
       "critical": 0
     },
     "notify_no_data": false
   }
 }'

Alert Best Practices

 

  1. Scope alerts appropriately - Use tags like cluster, env to avoid alert fatigue
  2. Set appropriate evaluation windows - Balance between quick detection and false positives
  3. Configure renotification - For critical alerts, renotify every 30-60 minutes
  4. Use composite monitors - Combine multiple conditions (e.g., brokers down AND high error rate)
  5. Test your alerts - Simulate failures in a test environment to validate alert behavior
  6. Document runbooks - Include investigation steps in alert messages
  7. Use monitor tags - Organize alerts with tags like service:redpanda, severity:critical
  8. Set up alert dependencies - Mute under-replication alerts when brokers are down

Troubleshooting Alert Issues

Metrics Not Appearing

 

  1. Verify integration is installed: datadog-agent integration install -t datadog-redpanda==
  2. Check Agent status: sudo datadog-agent status | grep -A 10 redpanda
  3. Verify endpoint is accessible: curl http://redpanda-host:9644/public_metrics
  4. Review Agent logs: sudo tail -f /var/log/datadog/agent.log | grep redpanda

False Positives on Under-Replicated Partitions

  • Increase evaluation window to 15+ minutes
  • Set higher thresholds during known rebalancing operations
  • Use anomaly detection instead of static thresholds
  • Consider creating separate alerts for planned vs unplanned events

No Data Alerts Firing Incorrectly

 

  • Ensure notify_no_data is only enabled for critical infrastructure metrics
  • Set appropriate no_data_timeframe (typically 10-15 minutes)
  • Check that the metric is part of default groups (no additional configuration needed)

Related Dashboards

The Redpanda integration includes two default dashboards:

  • Redpanda Overview - General cluster health and performance
  • Consumer Lag - Consumer group metrics and lag tracking

Import these dashboards to visualize the metrics used in these alerts and aid in troubleshooting.