How to Enable Spillover for Self Hosted Clusters

Last updated: August 11, 2026

Redpanda v23.2 introduced a new tiered storage mechanism called spillover. When the size of the in memory cloud storage metadata grows past a certain configurable size for a given partition, the oldest metadata is truncated and uploaded to cloud storage.

This document is intended for Self Hosted customers as this is managed internally for our BYOC and Dedicated Cloud clusters.

Benefits

Enabling spillover in RedPanda can be beneficial when the size of the in-memory cloud storage metadata grows past a certain configurable size for a given partition. When this happens, the oldest metadata is truncated and uploaded to cloud storage, which can help manage memory usage and prevent out-of-memory (OOM) errors.

Here are some indicators that you might need to enable spillover on versions greater than 23.2.x

1. High memory usage: If you're noticing high memory usage related to RedPanda, enabling spillover can help manage this.

2. Large number of segments: If your system has accumulated more than 128KiB of metadata per partition (approximately 6000 segments per partition), enabling spillover should occur immediately.

3. OOM errors: If you're experiencing OOM errors, enabling spillover can help prevent these by reducing the memory footprint of RedPanda.

 

Considerations

1. Performance Impact: While spillover can help manage memory usage, it might have an impact on performance. When spillover is enabled, some metadata is offloaded to cloud storage and retrieved on demand. This could potentially introduce latency, especially if the cloud storage is slow or the network connection to the cloud storage is not optimal.

2. Increased Cloud Storage Usage: Enabling spillover will increase the usage of cloud storage as more metadata is stored there. This could potentially increase costs if you're using a cloud provider that charges based on storage usage.

3. Testing: Enabling spillover can be an impactful change to your cluster and you should test this in a lower environment before making these changes to a production cluster. 

 

Enablement

Cloud storage got a number of large changes in v23.2 (new manifest format, chunked reads, spillover, etc.). To limit the risk, it was decided that spillover will be enabled gradually, so it is disabled by default.

In order to enable it, one needs to set the following cluster configs as below.

  • cloud_storage_spillover_manifest_size=65536 - when the size of the in memory metadata per partition exceeds 128KiB (double the config value), the older half of the metadata will be spilled to cloud storage

  • cloud_storage_manifest_cache_size=16777216 - sets the maximum size of the per shard cache which stores spillover manifests to 16MiB

Rollback

Spillover can be disabled by setting cloud_storage_spillover_manifest_size back to its default value (null). This will prevent future spillovers events from happening. However, resetting the config does not “undo” the spillovers that have already been done.

If the partition already has a lot of uploaded segments enabling spillover may trigger creation of large number of spillover manifests. This should be non impactful, but you can look at the redpanda_cloud_storage_spillover_manifest_uploads_total metric. It will have a spike right after the spillover is enabled if the cluster has enough data. The amount of data uploaded in this case will be small and wouldn’t interfere with anything else.

Observability

There’s a number of metrics which track spillover related events. 

  • redpanda_cloud_storage_spillover_manifest_uploads_total: number of successful manifest uploads. If it’s not equal to zero then spillover happened since last restart. Can be used as an indicator that it’s working and doing something.

  • redpanda_cloud_storage_spillover_manifests_materialized_count: non-zero value means that the feature is in use and some metadata from spillover manifests is cached in memory for fast access.

  • redpanda_cloud_storage_spillover_manifests_materialized_bytes: same above but expressed in bytes.

If you have questions or need assistance please reach out to Redpanda Support.