In general, topic compaction works the same way whether Tiered Storage offloading is enabled or not. The contents of a topic are retained until either the retention policy is reached or compaction is triggered. Retention policy settings will trigger the deletion of older messages when a partition reaches a particular size or message age. The policy can also combine both and deletion will occur for whichever threshold is crossed first.

Retention is sometimes not sufficient to maintain topic size, especially if publishing multiple updates to a particular message key. When using this message pattern, you may need to enable message compaction. This process will attempt to flag older message versions for deletion, retaining the most recent version within that partition. 

It is important to note that compacting a topic does not guarantee that Redpanda will completely de-duplicate and delete older message versions within that topic. Redpanda's compaction will only attempt to compact multiple versions of a message that were produced to a partition's log segments created within a single Raft leadership term. This means that messages would need to be sufficiently adjacent to one another in the timeline for Redpanda to compact to the latest version for that period in that partition. 

Note: if you change the producer's key assignment mechanism such that existing keys in partition A now end up getting written to partition B, compaction does not attempt to do cross-partition cleanup.  

 

Configuring a topic's cleanup.policy

There are three available configurations for cleanup.policy

1. cleanup.policy=delete - this is the default and relies on topic retention policy to delete older messages. For example, deleting messages older than 7 days.
2. cleanup.policy=compact - this only triggers cleanup of messages that have multiple versions. 
3. cleanup.policy=compact,delete - this combines retention cleanup, as well as compaction of multiple versions of messages.

While compaction can be enabled at any point in a topic's lifetime, we recommend doing it at topic creation time. Because compaction works via a sliding window mechanism, we only start compacting from that point forward. If you enable compaction some time after topic creation, there may exist segments that never run through the compaction process. This is similar to Kafka behavior. 

 

Configuration 1, delete, is straight forward. Messages age out when a time expires or when the topic partition size reaches a specific cap.

Configuration 2 and 3, compact and compact,delete have some caveats, as listed below.

 

Redpanda (and Kafka) does not do perfect (or complete) compaction of a topic.

1. Compaction does not guarantee that a Redpanda topic will be perfectly de-duplicated in any scenario right now (and Apache Kafka doesn't guarantee this either). There could still be duplicates or additional versions of keyed messages, ordered by time. In the current Redpanda compaction design, this feature is intended as an additional mechanism beyond retention to manage storage utilization. Consumers, especially those treating a topic as a key-value store, must handle the probability of seeing multiple versions of a message for a key if consuming far enough back in the partition.
   
   
2. Compaction only happens on segments created within a Raft term (meaning, a leadership election period). So if we create messages at a particular key, and then update it within the same period, we’ll attempt to compact it. But if those messages were produced in different terms, we won’t. Compaction isn’t a complete topic operation. It’s an operation for things created within a specific period of time.
   
   
3. The closer a key's updates are in time, the higher the likelihood we can compact them. If you produce an update to a key days, weeks, or months after it was originally published and uploaded to object store, it's not likely that we're going to compact it. This means there's still a chance your downstream consumers could have to deal with duplicates (or reassembling message order to get the current snapshot of your data.
   
   

Compaction with Tiered Storage

4. Compaction only happens on local storage segments. We do not currently pull segments back from Tiered Storage to attempt compaction.
   
   
5. When Tiered Storage is enabled, we opportunistically offload closed segments to object store as soon as possible, but ... we generally do not delete segments from local storage until they're older than the retention.local.target.* size/time settings. In 23.2 and later, local retention is handled more dynamically, where we will opportunistically delete segments early if the cluster is undergoing disk capacity pressure.
   
   During that window, if a message update comes in for a key still on the local disk, we will do a compaction, and that compacted segment will get uploaded to the object store.
   
   
6. While segments are on local disk, we may compact multiple times if there are multiple updates to a key, if we have not previously uploaded that compacted segment. Once we finally upload that compacted segment, for that range of messages, we don’t attempt to re-compact and reupload if an update happens to come in while these segments still live on local storage.
   
   
7. Generally, we will clean up an uncompacted segment in the object store if we upload a compacted segment covering that range. Once a compacted segment is uploaded to tiered storage, the compaction ratio for that segment is never improved because Redpanda does not make additional recompaction attempts on uploaded segments.
   
   But, remember that deduplication of keys only happens for messages published within the same Raft term. So you could still end up with multiple message versions if they appear in separate uploaded segments over time. So you get some cleanup, but not complete cleanup. 
   
   
8. Currently, the only way multiple messages with the same key in object store get fully cleaned up is if you utilize the topic retention settings (e.g., compact,delete). But this is only useful in cases where you know you can tolerate messages having a finite life span. 

Frequently Asked Questions on Tiered Storage with Compacted Topics

Is it safe to turn off compaction at any time?

Yes. A topic's compaction policy can be turned off at any time. When you turn off compaction, you may experience increased disk usage over time because Redpanda will no longer deduplicate messages for a given key.

To turn compaction off on a topic, set the topic-level cleanup.policy property to delete.

 

I'm treating my topic like a key/value store; How do I handle this?

Use cases like this typically expect that the latest version of a key is what's presented to the consumer. That's not how Redpanda (and Kafka) are designed. The Kafka protocol is a time-ordered log of messages where the consumer reads back from a requested log offset. The consumer may still run across different versions of a message as it reads forward through the log and must handle this situation accordingly.

If your goal is to have a consumer always get the latest version of a key without having to process the full topic history, neither Kafka nor Redpanda really support this. A topic is a time (offset) oriented log and the consumer must make the decision at read-time on how key versions are de-deduplicated when reading from offsets other than latest. 

In effect, this use pattern is looking for a present-time snapshot of the latest value for each key in the topic. If your consumers require this behavior, you may need to work around this by building the snapshot behavior yourself, having a consumer that is periodically querying the entire topic to reassemble the "current" version, then publishing to a new topic. This can be time consuming and computationally expensive for large topics.

How do I manage storage if I need every key and can't use retention cleanup?

One of the challenges of using a cleanup.policy of compact,delete is that retention operations could delete the only copy of a particular key if it ages out because of a size- or time-based retention policy. This is not ideal, especially if the expectation is that at least one copy of a message key is available somewhere in the topic at all times.

If using cleanup.policy of compact, expect additional storage usage over time for the reasons previously mentioned. Compaction works to reduce storage utilization by dealing with multiple versions of a message produced to a partition within a single raft term. 

In some cases, where you might be periodically re-publishing a complete topic update for every key, you might consider combining compaction with administratively triggered trim operations. Redpanda allows the administrator to trim the oldest segment offsets out using rpk topic prefix trim. This allows you to delete the beginning of the log up to a particular offset for one or more topic partitions. For this pattern, you need to set the cleanup.policy to compact,delete, and ensure that retention.ms and retention.bytes are both set to infinite. Once this is configured, you can treat the topic as a compacted topic where you administratively control when and where to delete up to.

Is compaction coordinated across brokers?

The brokers do not coordinate  with each other.

 

Brokers manage compaction of segments that live on local disk, independent of partition leadership. That means a segment in an RF=3 partition is technically compacted 3 times, once on each broker. Once compacted, we make guarantees that the latest versions of messages are logically available but because of timing and other considerations (like when compaction fired off on each broker), the underlying compacted segment _files_ could have different record boundaries across the brokers in the replica set. For purposes of compacted segment reuploads, the leader is responsible for uploading its version of the compacted segment to the object store.

Will Redpanda eventually support compaction across Raft terms?

Redpanda continues to evaluate improvements to the compaction system while maintaining appropriate performance characteristics for a workload. 

 

Cross-term compaction has the potential to be expensive resource-wise, especially when considering segments uploaded to Tiered Storage. For example, compacting the entire lifetime of a partition, including all segments uploaded to object store, would likely require pulling back all segments to local storage and re-uploading a full compacted set of segments. As message count in that partition goes up, so does the time and effort required to keep doing this. 

 

Please reach out to your Redpanda Customer Success partner to discuss your requirements for this.