Friday, April 18, 2014

file limit workaround on Ubuntu 10.04 Lucid

We'd been seeing graphs like this:

for listener number statistics (using Shoutstats) on our Icecast streaming server at PB correctly diagnosed this as being due to the server not accepting new connections after 1000 connections were reached. The server limits in the icecast xml file were set to 2048. PB correctly pointed out that this was due to a user limit issue.

Checked the error log and found lot messages like:

WARN connection/_accept_connection
accept() failed with error 24: Too many open files

We have to set ulimt (user limits) as suggested here

Currently it is set to 1024 on our server.

But after setting the limit to 4096 and restarting the server, still the problem persisted:

Checked the icecast limit as suggested here

It still shows Max open files as 1024

which says to use ulimit -S 4096 as the relevant user to change per session. But doing that and running the script to start our icecast using nohup still did not solve the issue. Max open files still reported as 1024 when doing cat /proc/pid_of_icecast/limits

Finally PB found the solution: which was, edit /etc/security/limits.conf by adding the lines 
relevant_username         hard    nofile      65536
relevant_username         soft    nofile      65536
and also editing /etc/pam.d/common-session to add the line 
session required 
at the end. And then, after a reboot, instead of directly logging in as relevant_username, either log in as root and then do su relevant_username, or do an su to become root, and then su relevant_username. After doing that, ulimit -n returns 4096, and starting our icecast now, it can open more than 1024 files. 

No comments:

Post a Comment