Overview

As Redpanda is Kafka API compatible, existing tools in the broarder Kafka ecosystem will work with Redpanda. One such tool, the Terraform provider produced by Mongey can be used to automate management of Redpanda (Kafka) topics and resources.

We are aware however of a small number of edge cases where interactions between this provider and Redpanda can result in slightly different behaviour from other Kafka implementations.

 

Detail

In particular, issues can arise when defining topics in Terraform around the use of the cleanup.policy property. Depending on versions of the provider and Redpanda in use, the Terraform plan generated may unexpectedly consider this value to either need resetting, or to be missing after being set. This behaviour results from the way Kafka topic properties are exposed to clients - these can be tagged as default values, or topic specific values (and subcategories of these), which may not align with what the provider expects for a given configuration.

This article doesn't provide an explicit listing of versions and a matrix of potential config values as this would not be feasible, but instead aims to highlight that this specific property may cause unexpected behaviour.

 

Mitigation

If you encounter any errors from Terraform around the cleanup.policy property when using the Mongey Terraform provider, it is worth reviewing exactly how this is being defined in your config - if explicitly specifing a value which is already default, you may need to remove this and rely on the implicit default being used. Alternatively, if on a version combination where the opposite is true, you may need to specify the value explicitly to match the default.

We are working to ensure that any differences between Redpanda and other Kafka implementations is resolved, and changes have also been merged to the Mongey provider in this area. We also (at time of publishing for this article) have a Redpanda specific Terraform provider in beta to improve this experience. Please speak to your Redpanda Account Team if you're interested in trying this.