Problem Statement

Several users reported that Redpanda stopped with std::__1::system_error (error system:24, could not create inotify instance: Too many open files). How can we fix this?

Solution

The error was from the OS system. The error indicated the value of fs.inotify.max_user_instances was too low. Please check the settings and increase it to 1024 or 8192.

- How to check the current value

$ cat /proc/sys/fs/inotify/max_user_instances
128

- How to change the value temporarily. This will not persist after restarting the system.

$ sudo sysctl fs.inotify.max_user_instances=8192
$ sudo sysctl -p

- How to set the value persistently 

$ tee /etc/sysctl.d/9999-redpanda.conf <fs.inotify.max_user_instances = 8192
EOF

 

Notes

rpk redpanda tune does not change this setting.

 

If you have further questions or continue to see issues, please reach out to Redpanda Support.