How to solve too many files open issue: file and process limits guide
If you are running an operating system such as Linux, macOS, Ubuntu or CentOS, you may experience errors such as "too many files open" crashes due to the limits set on the number of files and processes that can be open at the same time. This limit is a system default that protects the system from being overrun, but it is usually set too low. In this article, we will discuss how to increase file and process limits on Linux, macOS, Ubuntu and CentOS.
Check current limits
To check the current limit of file descriptors, you can use the "ulimit" command which is common among Linux flavors. Simply run:
ulimit -n
This command will output a number, such as "256" or "10032". On macOS, the maximum number that can be specified is 12288. To check the current limit of processes, use the command:
ulimit -u
This command will output a number, such as "1418".
MacOS
On macOS, you can obtain the current limit by using the command:
launchctl limit maxfiles
The response output will show numbers like this: "maxfiles 65536 200000", where the first number is the "soft" limit and the second number is the "hard" limit.
Configuration changes
If lower numbers are displayed, such as "maxfiles 256 unlimited", you need to make some configuration changes. If the soft limit is too low (such as 256), set the current session to:
sudo launchctl limit maxfiles 65536 200000
Some set it to 1048576 (over a million). Since sudo is needed, you will be prompted for a password. To make this setting permanent, add this command to your ~/.bash_profile. Alternatively, install Facebook’s Watchman utility which watches and adjusts automatically.
To detect issues, it is recommended to take both a full/complete backup and run a benchmark performance measurement before and after changing the configuration. Over several versions, Apple has been changing the way system-wide open file limits can be set upon restart.
For older macOS versions like Yosemite, increase the limits and edit (or create) file /etc/launchd.conf to contain:
limit maxfiles 65536 200000
For newer macOS versions like Sierra and High Sierra, you need to add two plist files into the /Library/LaunchDaemons/ folder from a GitHub repository. The plist files are:
- limit.maxfiles.plist at 524288
- limit.maxproc.plist at 2048
After copying these files, run:
sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist
sudo launchctl load -w /Library/LaunchDaemons/limit.maxproc.plist
Make sure the files' ownership is changed to "root:wheel" and their permissions are "-rw-r--r--", set by sudo chmod 644.
CentOS & Ubuntu
CentOS and Ubuntu, popular Linux distributions, is widely used in server environments due to its stability, reliability, and long-term support. However, its default settings limit the number of files and processes that can be opened, which can lead to performance issues, especially during peak loads.
To set a new open file limit, you can use the ulimit command followed by the -n option and the desired limit value. For example, to set the limit to 5000, run the following command:
ulimit -n 5000
Note that this will only set the limit for the current session. To make the limit permanent, you need to modify the system configuration.
To modify the system configuration, you need to edit the /etc/security/limits.conf file. This file contains the system-wide limits for various resources, including open files.
To set a new open file limit for all users, add the following lines to the end of the file:
* soft nofile 5000
* hard nofile 5000
This will set the soft and hard limits to 5000 for all users. The soft limit is the current limit, while the hard limit is the maximum limit that can be set by a process with the necessary privileges.
Save the file and reboot the system for the changes to take effect.
sudo shutdown -r now
If the limits have been increased to 5000, you're good to go.
By optimizing CentOS and Ubuntu for maximum file and process limits, you can ensure that your server can handle high loads without crashing or experiencing performance issues. With these simple steps, you can boost your server's performance and improve your overall system stability.
Conclusion
Configuring file and process limits on your operating system can help prevent system crashes and ensure optimal performance. By default, the limits are set too low, but you can adjust them to match your system's needs.
On Linux and macOS, the ulimit command is used to control the resources available to the shell and the processes it starts. You can use the ulimit -n command to obtain the current limit of file descriptors and the ulimit -u command to obtain the current limit of processes.
On macOS, you can also use the launchctl limit maxfiles command to obtain the current limit of file descriptors. If the soft limit is too low, you can set the current session using sudo launchctl limit maxfiles 65536 200000. You can also install Facebook's Watchman utility to watch and adjust the limits automatically.
If you're using an older version of macOS like Yosemite, you can edit or create the /etc/launchd.conf file to set the limits permanently. However, newer versions of macOS like Sierra require you to add two plist files to the /Library/LaunchDaemons/ folder and invoke them using the sudo launchctl load -w command.
On Ubuntu, RedHat and CentOS, you can edit or create the /etc/security/limits.conf file to enable user limits and specify a higher value for maximum open files.
Remember to take a full backup and run a benchmark performance measurement before and after changing the configuration to detect issues. By following these steps, you can ensure that your system is protected and optimized for maximum performance.