TSB-2023-01 RPC TLS Settings Not Correctly Applied

Last updated: August 11, 2026

Summary:

When attempting to configure RPC TLS (used for inter-broker communications), previous versions of rpk rewrite the Redpanda configuration in such a way that RPC TLS does not become enabled and therefore inter-broker communications remain unencrypted. This is fixed in the Redpanda versions listed below, however manual action will be needed to re-enable RPC TLS as this cannot be done without a full cluster restart. 

 

Severity: Medium

 

Redpanda Products affected:
Redpanda, RPK

 

Releases affected:
v22.2.1-v22.2.10, v22.3.1-v22.3.13, v23.1.1

 

Impact:
RPC TLS was not enabled despite the configuration appearing the suggest that it would have been

 

Immediate Action required:

Yes


Action required:


To identify whether RPC TLS was intended to be enabled please run the following command on one or more Redpanda brokers:

$ grep -zo 'rpc_server_tls:\n\s*-.*' /etc/redpanda/redpanda.yaml

 

Sample output:

rpc_server_tls:

  - cert_file: /etc/redpanda/c.cert

 

If no output is observed from this command then you are not affected.

Affected customers should upgrade to v22.2.11+, v22.3.15+ or v23.1.3+ (Redpanda recommends the latest maintenance release for your current major version), however it is not possible to effect a rolling restart when enabling RPC TLS, therefore the cluster will need to be fully shut-down as part of the process to enable RPC TLS. Note: you can apply the upgrade to the latest version without downtime, however this will leave RPC TLS disabled.

In order to configure RPC TLS the following procedure should be followed:

  1. Complete shutdown of all nodes in the cluster.
  2. On each node:
    1. Remove the - character from the rpc_server_tls element. For example:
 rpc_server_tls:

    enabled: true

    require_client_auth: false

    cert_file: "/etc/redpanda/certs/node.key"

    key_file: "/etc/redpanda/certs/node.crt"

    truststore_file: "/etc/redpanda/certs/truststore.pem"
  1. Verify the certificate configuration is correct.
# /etc/redpanda/certs# openssl rsa -modulus -noout -in node.key | openssl md5

(stdin)= 07a970aa9eeb11cc55241c9518484b8e

# /etc/redpanda/certs# openssl x509 -modulus -noout -in node.crt | openssl md5

(stdin)= 07a970aa9eeb11cc55241c9518484b8e # check this matches above

 

# openssl verify -trusted /etc/redpanda/certs/truststore.pem \ 

/etc/redpanda/certs/node.crt

/etc/redpanda/certs/node.crt: OK

 

  1. Start the cluster node and check that the RPC server is accepting connections.
# systemctl start redpanda.service

# journalctl -u redpanda.service | grep 'Started RPC server listening'

Mar 17 14:37:00 ip-10-0-0-110 rpk[20683]: INFO  2023-03-17 14:37:00,034 [shard  0] main - application.cc:1756 - Started RPC server listening at {host: 10.0.0.110, port: 33145}

# journalctl -u redpanda.service | grep redpanda.rpc_server_tls

Mar 17 14:47:42 ip-10-0-0-110 rpk[24539]: INFO  2023-03-17 14:47:42,905 [shard  0] main - application.cc:614 - redpanda.rpc_server_tls:{ enabled: 1 key/cert files: {{ key_file: /etc/redpanda/certs/node.key cert_file: /etc/redpanda/certs/node.crt }} ca file: {/etc/redpanda/certs/truststore.pem} client_auth_required: 0 }        - TLS configuration for RPC server




Kubernetes with Helm

Because of the aforementioned problem, Redpanda clusters deployed using Helm will need to be manually edited before upgrading. If have TLS enabled in your cluster (which is the default), you must do one of the following to upgrade:

  • Disable TLS for the RPC listener by setting listeners.rpc.tls.enabled to false.

  • Enable TLS for the RPC listener by running the upgrade with --set force=true. Then, while Helm is upgrading the release, manually delete all the pods:


kubectl -n rn delete pod -l app.kubernetes.io/component=redpanda-statefulset


Warning: If you use emptyDir volumes, your brokers will lose all their data and start with an empty data directory. Consider backing up your brokers' data directories before upgrading.


Addressed in (RP Version):
v22.2.11+, v22.3.14+ or v23.1.2+; Helm chart 4.0+

References:

Questions?:
If you have any questions on this TSB, or need further guidance, please contact customer-success@redpanda.com / support@redpanda.com (TBC)