TSB-2023-03 Tiered & Local Storage - Corrupt metadata risk

Executive Summary

Through monitoring of the Redpanda Cloud estate, Redpanda has identified on one cluster, that in memory constrained environments, Redpanda 22.x might potentially fail in ways that could cause data corruption in tiered storage and in local storage. This behaviour is not seen in Redpanda 23.1 because 23.1 has a different approach to error handling that prevents such occurrences.

 

Severity

High

 

Redpanda Products affected

Redpanda

 

Releases affected: 22.2.x, 22.3.x  

Impact: If memory exhaustion occurs in Redpanda 22.2 or 22.3, Redpanda might potentially corrupt the on-disk metadata in either local or cloud storage, leading to unavailability of segments.

 

Immediate Action required

Yes

 

Action required

 

Redpanda recommends either:

• Upgrade to 23.1. Note: Users on 22.2 will need to upgrade to 22.3 before proceeding to 23.1.
• Set the --abort-on-seastar-bad-alloc startup flag by modifying /etc/redpanda/redpanda.yaml and perform a rolling restart of the cluster. Absence of the flag could mean that in the event of memory exhaustion in-memory and on-disk state of the system could become inconsistent. Ensuring the flag is turned on allows Redpanda to restart on failure-to-alloc-memory allowing for consistent recovery and continued predictable operation

Note: The --abort-on-seastar-bad-alloc  flag means that any in-flight operations that occur when memory exhaustion occurs will be terminated, resulting in predictable failure scenarios.

 

 

Identifying if you have been affected by possible data corruption:

 

Redpanda recommends analysing the Redpanda logs to look for occurrences of the phrase “bad alloc”. If this term has been observed, please contact Redpanda Support.

 

Setting startup parameters

Setting startup parameters can be done by editing the /etc/redpanda/redpanda.yaml file to include the following section under the rpk element:

 

rpk:

 additional_start_flags:

   - "--abort-on-seastar-bad-alloc"

Perform a restart of the node:

1. Identify the node_id: 

$ journalctl -u redpanda.service | grep 'Starting Redpanda with node_id' | tail -1
Jun 13 13:31:47 ip-172-31-29-70 rpk[254398]: INFO  2023-06-13 13:31:47,454 [shard 0] main - application.cc:1819 - Starting Redpanda with node_id 0, cluster UUID

2. Put the node in maintenance mode: 

$ rpk cluster maintenance enable -w
Successfully enabled maintenance mode for node 0
Waiting for node to drain...
NODE-ID  DRAINING  FINISHED  ERRORS  PARTITIONS  ELIGIBLE  TRANSFERRING  FAILED
0        false     false     false   0           0         0             0
0     false  false  false  0     0     0     0
0     false  false  false  0     0     0     0
0     false  false  false  0     0     0     0
0     true  true  false  27    0     11    0

       3. Restart the node:

 

$ systemctl restart redpanda

 

   4. Check that the flag has been correctly set:

 

$ journalctl -u redpanda.service | grep seastar-bad-alloc
Jun 13 13:42:09 ip-172-31-29-70 rpk[254613]: /opt/redpanda/bin/redpanda redpanda --redpanda-cfg /etc/redpanda/redpanda.yaml --abort-on-seastar-bad-alloc --lock-memory=false --io-properties={"disks":[{"mountpoint":"/var/lib/redpanda/data","read_iops":411200,"read_bandwidth":2015342735,"write_iops":181500,"write_bandwidth":808775652}]}

        5. Remove the node from maintenance mode:

 

$ rpk cluster maintenance disable
Successfully disabled maintenance mode for node 0

Addressed in (RP Version): 23.1.x

References

Questions?: If you have any questions on this TSB, or need further guidance, please contact customer-success@redpanda.com / support@redpanda.com