Issue / Scenario

 rpk cluster health  output shows the cluster marked as not healthy , 
with no_elected_controller reported  as a reason + a negative value for Controller ID:

For example :

CLUSTER HEALTH OVERVIEW
=======================
Healthy:                          false
Unhealthy reasons:                [leaderless_partitions no_elected_controller nodes_down]
Controller ID:                    -1
All nodes:                        [0 1 2 3]
Nodes down:                       [1 2]
Nodes in recovery mode:           []
Leaderless partitions (1):        [redpanda/controller/0]
Under-replicated partitions (0):  []
Cluster UUID:                     90423fd3-b10b-4d4c-83ae-90a9ef58db2a

Additionally errors such as this from returned from API calls / in the Redpanda logs ..

- Currently there is no leader controller elected in the cluster
- Service Unavailable, Leader not available

redpanda admin brokers list will typically report multiple nodes with IS-ALIVE=false

ID    HOST                                               PORT   RACK  CORES  MEMBERSHIP  IS-ALIVE  VERSION    UUID
0     redpanda-0.redpanda.humdrumtown.svc.cluster.local.  33145  -     1      active      true      25.3.1   20c08752-c6d2-4582-a79e-032f85443906
1     redpanda-1.redpanda.humdrumtown.svc.cluster.local.  33145  -     1      active      false     -        45bcf4de-215c-42ed-a77d-ff00b080ad52
2     redpanda-2.redpanda.humdrumtown.svc.cluster.local.  33145  -     1      active      false     -        4a910f3b-14a4-422f-939c-30fdda37cc35
3     redpanda-2.redpanda.humdrumtown.svc.cluster.local.  33145  -     1      active      true      25.3.1   12834a5b-0f96-478a-b955-4391c74b8aa0

Cause 

no_elected_controller  happens when the cluster cannot get quorum and is unable to elect a leader controller 

One cause of this could be if multiple brokers are down  (for whatever reason ) .. for example 3 brokers down in a 5 broker cluster.

Another scenario is after disk crashes/replacements on multiple nodes , where additional nodes (new node-ids) are started up and are unable to join the cluster leaving a node count where the number of failed/down nodes compared to running  nodes is less than the required raft quorum .

Action to take 

If you encounter a no_elected_controller scenario please log a ticket with Redpanda Support  (severity 1 if a production environment  ) so we can provide steps to fix your cluster ..
 

Provided the following in information on the ticket to ensure the quickest resolution for this scenario...

• Detailed information on what lead to this problem state.  For example was there any disk/node crashes or node upgrades for example. 
• Logs from each broker . Ideally from Redpanda startup.

• Output of Redpanda commands
  Please note that it's important to specify the correct flags to access your cluster in the rpk command - these will be different for each environment, but would generally include:

  -X user=my_username

  -X pass=mY_p4sSw0rD

  -X sasl.mechanism=SCRAM-SHA-256  (or SCRAM-SHA-512 )

  -X tls.enabled=true 
 -X admin.tls.enabled if configured on the cluster
  -X admin.tls.insecure_skip_verify=true (to bypass certificate checks) 
  
   Or you can set up an rpk profile to pass all of these for you automatically.
   

rpk cluster health;
rpk redpanda admin brokers list --include-decommissioned ;
rpk topic describe -r '*';

# on a redpanda node that has IS-ALIVE=true in 'rpk redpanda admin brokers list'
rpk debug bundle 
# end points (change to https:// if admin api not tls enabled ) 
curl -k https://localhost:9644/v1/broker_uuids #(to capture all UUIDs)
curl -k https://localhost:9644/v1/brokers #(to capture core count)

• for Kubernetes/K8s 
  
  kubectl -n get pv,pvc,po;
  kubectl -n describe pods;
  kubectl get events --namespace --sort-by='.metadata.creationTimestamp'