Resolving "No Disk Space" Errors in K8s Redpanda While Producing

Last updated: August 10, 2026

Problem Description

Producers are unable to publish messages to topics due to a "no disk space" error, despite apparent available disk space.

Symptoms

  • Error messages in logs similar to: WARN [shard 0:main] kafka - rejecting produce request: no disk space; bytes free less than configurable threshold
  • Inability to produce messages to topics
  • High disk usage on /var/lib/redpanda/data/cloud_storage_cache

Root Cause Analysis

The issue is related to the cloud storage cache filling up, triggering a disk space alert even when the main data partition has available space.

Resolution Steps

Option 1 - Cache directory on a different disk than the data directory. 

This is likely resolved in with the Github PR 24436. This is resolved in Redpanda 24.1.17, 24.2.14, and 24.3.3

If you are still seeing this issue after upgrading please reach out to Redpanda Support. 

Option 2 - Cache directory on the same disk as data directory. 

1. Cluster Restart

Perform a rolling restart of the cluster:

2. Additional Remediation (if issue persists)

Below are two options to try and increase disk space, one or both of them can be attempted.

  • Modify the Kubernetes configuration to allocate more space for /var/lib/redpanda/data/cloud_storage_cache
    • This gives more overall disk space.
  • Adjust cloud_storage_cache_size configuration: rpk cluster config set cloud_storage_cache_size
    • Reducing this will reduce the size of the cache increasing overall disk space.

3. Post-Resolution Monitoring

After making changes, monitor the cluster to ensure the issue is resolved.

4. Diagnostic Data Collection (If needed)

If remediation failed, please reach out to the support team with the following diagnostics.

  1. Increase tracing for diagnosis: rpk redpanda admin config log-level set cluster resource_mgmt storage storage-gc -l trace -e 120 - --host localhost:9644
  2. Run rpk debug bundle on each broker

Preventive Measures

  • Regularly monitor disk usage, especially for the cloud storage cache
  • Set up alerts for disk usage thresholds
  • Consider implementing auto-scaling for storage in your Kubernetes environment

If you have further questions or continue to see issues, please reach out to Redpanda Support.