rpk "unsafe-recover" command

 

In this Article lets go into a little bit more detail about what is happening when running the unsafe-recover command and why you might want to run it in the first place. Please note, that there are additional details about the command here, Unsafe-Recover Docs.

 

 

What does the "unsafe-recover" command do?

rpk cluster partitions unsafe-recover will forcefully recover data adversely affected by irrecoverable nodes. To put that in context, the base case here is explicitly focusing on 1 or more nodes you know are "dead" (not the entire cluster), but still have partitions associated with the "dead" nodes.

 

When should I consider using this command?

This command is useful when partitions need to be moved from a non node that is no longer active or working to another active node. This could also include nodes that do not exists, but are showing up as part of your cluster when running rpk commands.

 

Example

For some additional context, here is an example of an output prior to running unsafe-recover and after. Nodes 0 and 5 are old ID's and 6 and 7 are the new ones.

 

CLUSTER
=======
redpanda.1234-456-789-22131-1235

BROKERS
=======
ID    HOST                                             PORT
0     redpanda-0.redpanda.redpanda.svc.cluster.local.  9093
1*    redpanda-1.redpanda.redpanda.svc.cluster.local.  9093
2     redpanda-2.redpanda.redpanda.svc.cluster.local.  9093
3     redpanda-3.redpanda.redpanda.svc.cluster.local.  9093
4     redpanda-4.redpanda.redpanda.svc.cluster.local.  9093
5     redpanda-5.redpanda.redpanda.svc.cluster.local.  9093
6     redpanda-0.redpanda.redpanda.svc.cluster.local.  9093
7     redpanda-5.redpanda.redpanda.svc.cluster.local.  9093

 

Then after running rpk cluster partitions unsafe-recover --from-nodes 0,5

rpk cluster info -b

ID    HOST                                             PORT

1     redpanda-1.redpanda.redpanda.svc.cluster.local.  9093
2     redpanda-2.redpanda.redpanda.svc.cluster.local.  9093
3     redpanda-3.redpanda.redpanda.svc.cluster.local.  9093
4*    redpanda-4.redpanda.redpanda.svc.cluster.local.  9093
6     redpanda-0.redpanda.redpanda.svc.cluster.local.  9093
7     redpanda-5.redpanda.redpanda.svc.cluster.local.  9093

redpanda@redpanda-0:/$ rpk redpanda admin brokers list
NODE-ID NUM-CORES MEMBERSHIP-STATUS IS-ALIVE BROKER-VERSION
1       6         active            true     v23.3.1 - ce417018bfd6a2c4a953117b4c080190d474d648
2       6         active            true     v23.3.1 - ce417018bfd6a2c4a953117b4c080190d474d648
3       6         active            true     v23.3.1 - ce417018bfd6a2c4a953117b4c080190d474d648
4       6         active            true     v23.3.1 - ce417018bfd6a2c4a953117b4c080190d474d648
6       6         active            true     v23.3.1 - ce417018bfd6a2c4a953117b4c080190d474d648
7       6         active            true     v23.3.1 - ce417018bfd6a2c4a953117b4c080190d474d648