Large number of ACL's can cause memory crashes

Issue

In releases :
- 26.1.7 and lower
- 23.3.14 and lower 
if your Redpanda deployment has a large number of ACLS (8k + for example ) .. then there is a chance of 
memory allocation related crashes even if it appears your  Redpanda broker has plenty of free emory

To check if how many ACL's you have :      rpk security acl list|wc -l 
Anything 8k or over you are at risk of this bug occurring 

Detail

Seastar, the I/O framework Redpanda utilises,  manages memory per shard using a buddy allocator. Each shard's memory arena may have plenty of free memory in aggregate, but that memory is fragmented into small regions by normal broker operation.

When handling ACL's the memory allocation size is approximately:
peak allocation ≈ (ACL count) × 168 bytes × 2
 
For example, with 9,350 ACLs the peak reallocation request is approximately 1.5 MB, which requires a 2 MB contiguous span. If the 2 MB span pool on the shard is exhausted, the broker crashes.

Example crash sequence  (values may vary):
06:43:14.787  Recorded crash reason to crash file on shard 3 (OOM)
06:43:14.796  [ERROR] seastar_memory — Dumping seastar memory diagnostics
06:43:14.796  Free memory:  2917M
06:43:14.796  Used memory:  1179M
06:43:14.797  [ERROR] seastar — Failed to allocate 1572864 bytes
06:43:14.797  Total memory: 4G   |  Hard failures: 1
06:43:14.797  Aborting on shard 3, in scheduling group kafka.
06:43:14.797  Segmentation fault: si_code: 4294967290, ip: 00007f3e837109fc
06:43:14.797  Segmentation fault on shard 3, in scheduling group kafka.
06:43:14.797  Segmentation fault: resolved ip: 0x00000000000969fc
              in /opt/redpanda/lib/libc.so.6[0x00007f3e8367a000+0x2261a5]
 

In the memory diagnostics from the crash .. in this case where the alloc size was around 1.5MB .. the 2MB span size was exhausted , hence the memory crash 

index  span-size   free    used   spans  
─────────────────
 0       4K       22M    190M    54k   │
 1       8K       22M     24M     6k   │
 2      16K       14M     53M     4k   │
 3      32K      295M    143M    14k   │  large free pool
 4      64K      516M    220M    12k   │  large free pool
 5     128K      778M    386M     9k   │  large free pool
 6     256K      757M      5M     3k   │  large free pool
 7     512K      398M      0B    795   │  large free pool
 8       1M      115M      1M    116   │  some free
 9       2M        0B     22M     11   │  *** EMPTY — this is the cause of crash ***
10       4M        0B      8M      2   │  EMPTY
11       8M        0B     16M      2   │  EMPTY
12      16M        0B     48M      3   │  EMPTY
13      32M        0B      0B      0   │
14      64M        0B     64M      1   │
15+    ≥128M       0B      0B      0   │

Resolution 

The fix (CORE-14944) replaces `std::vector` with `chunked_vector` — Seastar's non-contiguous vector implementation — so the ACL result set is built in fixed-size chunks that always fit within the small-pool allocation range, regardless of ACL table size.

The bug is fixed in the following releases :

26.1.8 and higher via PR  30401
25.3.15 and higher  via PR 30389