Migrating from Kafka to Redpanda FAQ
Last updated: August 11, 2026
General Information and Planning
Introduction
List of commonly encountered questions for managing migrations from legacy Kafka systems to Redpanda.
General Information and Planning
Can we migrate from XYZ distribution of Apache Kafka?
Redpanda is a Kafka-compatible drop-in replacement for Apache Kafka. As long as the distribution is generally based on upstream Apache Kafka, we should be able to migrate from it.
For example:
- Apache Kafka
- Confluent Platform
- Confluent Cloud
- AWS MSK
- Aiven for Apache Kafka
- Instaclustr for Apache Kafka
How long does it take to do a migration?
Migration operations have varied timelines. It could be as short as a few days for small environments, to as long as several weeks or months depending on the complexity of the workloads, testing and validation, or amount of data moving (and being continuously created).
Data movement tends to be the short pole in the scope of work. This is entirely dependent on available network bandwidth/capacity and related resources. A good rule of thumb is that, on a single 10 gigabit network link between two systems, you should be able to move approximately 85 terabytes of data per day. To put this in perspective, producing one gigabyte per second of data to a Kafka cluster results in about 85 terabytes of data being created per day.
Do you have example plans and scopes of work?
Yes. Please reach out to your Redpanda Account or Customer Success teams. The scope of work generally follows the same process:
Phase 1: Pre-flight evaluation
Phase 2: Pre-flight cluster setup
Phase 3: Data replication
Phase 4: Data validation
Phase 5: Application reconfiguration
Phase 6: Application, load, and security evaluation
Phase 7: Decommission
Additional detail is described in our Migration Report.
Do I have to take any extended downtime for the migration?
Generally, no. The process is designed to minimize outages for producers and consumers. There may be brief outages to account for client restarts required for configuration updates, such as changing broker lists.
We're moving to Dedicated/BYOC. Do I need to set up my own Kafka Connect?
No. Each Dedicated or BYOC cluster includes a managed Kafka Connect cluster where you can configure the MirrorMaker2 components to facilitate your migration. You will need to ensure that your legacy Kafka cluster is accessible to the Kafka Connect cluster and that appropriate user credentials and ACLs are configured to allow MirrorMaker2 to connect and copy data.
Do you have example configurations for standalone MirrorMaker2?
Please contact your Redpanda Customer Success team. We can provide you with assistance in crafting these configurations.
If you're coming from AWS MSK, an example configuration is described in our blog, How to migration from Amazon MSK to Redpanda.
Clients
Do I need to change clients or code to move to Redpanda?
There is generally no need or requirement to make code changes or swap out to different client libraries.
Redpanda is a Kafka API compatible drop-in replacement for Apache Kafka. In some cases, we may make recommendations for configuration changes on the client to improve performance or behavior so you can get the most out of your Redpanda installation.
Do I need to change client configurations to move to Redpanda?
Minimal configuration changes will be required. These are usually focused on configuration properties involving hostnames. For example, updating broker or seed lists, schema registry endpoints, and so on.
If you have dependencies on specific hostnames, please reach out and we can discuss additional options.
I depend on the Kafka command line scripts in my operations. Can I still use them?
The Kafka command line scripts, such as kafka-topics.sh and kafka-acls.sh are compatible with Redpanda. You can continue to use them to facilitate migrating over. We do recommend that you consider updating your tooling over time to use Redpanda's rpk command line tool. This is a unified tool that interfaces with many of the Admin and Kafka APIs available within the product. This is a lightweight tool developed in Go and is available for various operating systems and architectures, including MacOS and Linux on ARM and x86.
One additional benefit of using Redpanda provided tooling for development and operations is that you'll no longer have to keep the Apache Kafka or Java installed on your systems.
Please see our blog Getting Started with RPK: Redpanda's dev-friendly CLI or rpk documentation for more detail.
Moving Data
What is used to move data?
Redpanda currently recommends using Kafka Connect with MirrorMaker2 to copy data between two clusters. This is the Kafka ecosystem standard at this time.
Can I copy the raw Kafka data files to Redpanda?
No. Redpanda uses its own data format to achieve high performance, scalability, and resilience. These formats are incompatible. The mechanism to get data to Redpanda in a migration is via consuming it from one cluster and producing to Redpanda.
How are consumer group offsets handled?
MirrorMaker2 has a feature for ensuring that offsets published in one cluster are copied and translated to another. We heavily utilize this feature to ensure that consumer groups can more seamlessly transition between clusters.
Do I need to run my own Kafka Connect cluster for migration?
A Kafka Connect cluster is required because MirrorMaker2 depends on it.
In some cases, such as migrating to Redpanda Cloud with a Dedicated or BYOC cluster, we provide a managed Kafka Connect setup for you. For self-managed setups, we have a community-supported Kafka Connect container that can run under Docker or Kubernetes. You may also run the upstream Kafka Connect implementation provided in the Apache Kafka tarball.
MirrorMaker2 doesn't seem to copy fast enough. What do I do?
There are several things to look at when trying to improve data copy performance with MirrorMaker2.
- Is there enough bandwidth between the source cluster and Kafka connect cluster?
- Is there enough bandwidth between the Kafka connect cluster and Redpanda?
- Is the Kafka Connect cluster large enough to handle the parallelization required?
- Have the MirrorMaker2 connectors been configured with enough task.max parallelism?
Keep in mind that your MirrorMaker2 environment needs to be large enough to handle the rate of new data creation in the legacy cluster, as well as keeping up with pushing that data to the Redpanda cluster. For example, if the source cluster is getting 1 gigabyte/sec of new data, MirrorMaker2 will need to handle 2 gigabytes/sec of data movement: 1 gigabyte/sec for consuming from the source cluster, and 1gigabyte/sec for producing to Redpanda.
It's also best to size the MirrorMaker2 environment with spare capacity so it can catch up if errors occur or lag starts to happen.
Schema Registry
Do you support moving Schema Registry data?
We have two mechanisms for moving Schema metadata from a legacy Kafka system into Redpanda.
- Batch style using an export/import process
- Continuous style using MirrorMaker2 with a Single Message Transform
Batch style is intended as a single-shot migration mechanism for schemas and only requires access to the REST interface for the source registry.
Continuous style is intended for more extended migrations where the source and destination clusters must coexist with production use cases for a time. In the continuous style, you would continue publishing schema updates to the source cluster until all migration work has completed. This will replicate newly created schemas or schema updates to Redpanda.
Note: During the migration period, you should not directly publish updates to Redpanda's Schema Registry as that could create Schema clashes or cause schemas to get overwritten unexpectedly.
Please see the Schema Migration project on the Redpanda Github page:
https://github.com/redpanda-data/schema-migration
MirrorMaker2
How do I prevent duplicate or missing messages during migration?
Duplicates and gaps/missing messages happen when certain conditions occur. These tend to be misconfigurations across competing MirrorMaker2 setups or when MirrorMaker2 is configured to reset to earliest offset when an error is encountered.
In the first type, ensure that you don't have overlapping topic name configurations if you're using multiple MIrrorMaker2 setups (such as multiple MirrorSourceConnectors). Keep an eye out for how regular expressions are used when selecting topics.
In the second type, ensure that you have auto.offset.reset set to none. If you set this to earliest, you will get duplicates. If you set this to latest, you will find gaps can result in your replicated data.
How do I improve transfer performance with MirrorMaker2
First, verify you have a sufficient number of workers in the Kafka Connect cluster and available bandwidth capacity between the legacy Kafka cluster, Kafka Connect cluster, and Redpanda.
If you believe you've resolved that part and you still see poor performance, check the configuration of tasks.max for the MirrorMaker2 MirrorSourceConnector. This controls the parallelism of the work when partitions are assigned to the Source connector's consumer threads.
It's important to note two things:
- Don't configure this to exceed the number of partitions in the source cluster. Those additional tasks will go unused.
- Your KC cluster memory and cpu dictate how many parallel tasks you can reliably run. Each task tasks cpu and memory (java heap). Allocate about 1 core and 500MB of heap for every two tasks.
If using MirrorMaker with Dedicated or BYOC clusters, please remember that these clusters have a defined network throughput capacity, so take that into account when you're sizing your workloads and migration plan.