Problem Statement

MirrorMaker2 Source Connector(MM2) is a Redpanda-managed Kafka connector in the Redpanda Cloud service. When there are many partitions to be replicated, some topics may not be replicated intermittently. Or, you may feel like the replication was stopped. There is no error in the connector logs. The throughput has not reached the limit. You may also observe that replication only works immediately after the connector is restarted. 

Solution

The connector's configuration includes an option to set the Maximum Number of tasks. This controls the parallelism of the work when partitions are assigned to the Source connector's consumer threads.  

Increasing this number may reduce the delay of replications. 

Notes

It's important to note:

• You can generally expect around 55 partitions per task. However, the number of partitions per task depends on the level of activity in the partitions and the amount of CPU and memory required. Based on this number, Redpanda Cloud service will allocate more resources to the connector.  Please monitor the connector metrics for memory and CPUs. Allocating too many tasks may hit the limit of the VM instances. 
• Don't configure this to exceed the number of partitions in the source cluster. Those additional tasks will go unused.
• A Redpanda Cloud Cluster has a defined network throughput capacity. Please monitor your sizing of workloads and migration plan.

References:

- Doc - Creating MM2 Connector: link

- Doc - Connector Metrics: link

- KB - Migrating from Kafka to Redpanda:  link 

 

If you have any questions on this notice or need further guidance, please contact Redpanda Support