Problem Statement

A topic with replication factor of 1 (RF=1), can have partitions become leaderless when a broker that the RF=1 partitions' leaders were allocated to is removed without a proper decommission process.

As a result, a client may receive an error like "LeaderNotAvailableException." Here is an example of the client error message.

Error: org.apache.kafka.common.errors.LeaderNotAvailableException: Unable to get end offsets during leader election for topic" error

A proper decommission process will move the partitions with RF=1 out of the decommissioning broker. However, when the decommissioning broker was removed without a proper decommission process, the partitions with RF=1 could cause data loss. Please check rpk topic list -di output to confirm that partitions with RF=1 contain the removed broker ID as a replica.

Solution

RF=1 topics are inherently risky, but you can potentially recover the data using the remote topic recovery feature, if Tiered Storage is enabled. Please note that there may still be data loss if some data were not uploaded to the tiered storage before removing the broker.


Recovery Process

• We recommend testing this in a non-production environment before executing this in a production environment.
• For more details, please refer to the remote topic recovery documentation.

1. Disable deleting the topic from the Tired Storage
   rpk topic alter-config -s redpanda.remote.delete=false
2. Delete the topic
   rpk topic delete
3. Create the same topic with the remote recovery
   rpk topic create -c redpanda.remote.recovery=true -c remote.read=true

Another option is to use Forced Partition Recover(link available in v24.2.x). This may lose a partition data. However, you do not need to delete all partitions of an affected topic. 

Please reach out to Redpanda Support if you need additional assitance.