We'd been seeing graphs like this:
for listener number statistics (using Shoutstats) on our Icecast streaming server at radiosai.org. PB correctly diagnosed this as being due to the server not accepting new connections after 1000 connections were reached. The server limits in the icecast xml file were set to 2048. PB correctly pointed out that this was due to a user limit issue.
Checked the error log and found lot messages like:
accept() failed with error 24: Too many open files
We have to set ulimt (user limits) as suggested here
Currently it is set to 1024 on our server.
But after setting the limit to 4096 and restarting the server, still the problem persisted:
Checked the icecast limit as suggested here
It still shows Max open files as 1024
I tried following http://askubuntu.com/
questions/162229/how-do-i- increase-the-open-files-limit- for-a-non-root-user
which says to use ulimit -S 4096 as the relevant user to change per session. But doing that and running the script to start our icecast using nohup still did not solve the issue. Max open files still reported as 1024 when doing cat /proc/pid_of_icecast/limits
Finally PB found the solution: which was, edit /etc/security/limits.conf by adding the lines
relevant_username hard nofile 65536
relevant_username soft nofile 65536
and also editing /etc/pam.d/common-session to add the line
session required pam_limits.so
at the end. And then, after a reboot, instead of directly logging in as relevant_username, either log in as root and then do su relevant_username, or do an su to become root, and then su relevant_username. After doing that, ulimit -n returns 4096, and starting our icecast now, it can open more than 1024 files.