BYOC Cluster stuck in DELETING_AGENT state

When attempting to destroy a cloud cluster, it is import to preserve the Terraform State.

 

When the Terraform state bucket is deleted, then the cluster cannot be properly removed. When this happens, your cluster state will be stuck in "DELETING_AGENT" state. You'll need to manually remove the resources within your cloud provider related to this cluster.

 

Once this has been completed on your end, then the agent can finally clean everything up and remove it from your redpanda cluster.

 

Bellow is an example error you might see when attempting to use the rpk command to remove a cluster without a valid terraform state.

 

$ rpk cloud byoc gcp destroy --redpanda-id '' --project-id '' 

2024-02-22T13:42:28.608Z INFO .rpk.managed-byoc cli/cli.go:222 Running destroy {"provisioner": "agent"} 2024-02-22T13:42:29.051Z ERROR .rpk.managed-byoc cli/cli.go:225 Failed to destroy provisioner {"provisioner": "agent", "error": "failed running terraform init: exit status 1
Error: Failed to get existing workspaces: querying Cloud Storage failed: storage: bucket doesn't exist"} 
Failed to destroy provisioners: failed to destroy provisioner agent: failed running terraform init: exit status 1 Error: Failed to get existing workspaces: querying Cloud Storage failed: storage: bucket doesn't exist