Partition errors when migrating data to Redpanda.

 

Problem Description

During the process of migrating data, for example with MM2, from an external environment to a Redpanda cluster, you may encounter issues such as missing topics or Consumer groups not being imported.

 

Error Analysis

One possible cause for these issues could be the number of partitions on the external system exceeding the maximum limit allowed by the Redpanda cluster. For instance, if the number of partitions on an external Kafka environment exceeds 1k, some topic creations may fail on the Redpanda side due to a limit of 1k partitions on certain tiers of Redpanda Cloud clusters.

The error messages you encounter may indicate an InvalidPartitionsException with the message Can not increase partition count due to core limit". This error is thrown when the number of partitions for a topic is increased beyond the maximum limit allowed by the system.

 

Suggested Resolution

To resolve this issue, consider the following steps:

1. Check the limits/sizings on the Redpanda cluster: If all those topics/partitions on the Kafka side need to be imported, it might be necessary to adjust the limits/sizings on the Redpanda cluster to accommodate them.

2. Reduce the number of partitions being migrated: Evaluate if all of the partitions being migrated need to be migrated and either reduce the number of partitions in use, or the number being migrated. 

 

If you continue to see issues after sending a new invite please reach out to our Support Team or refer to the Redpanda Documentation.