Wednesday, March 31, 2010

free space on colinux and hard disk performance

Over the last two days, there were several interruptions on our streams. Problem seems to be excessive Hard Disk usage, since copying files was taking a very long time. But at a time when the system was behaving normally, trying out the counters
1. PhysicalDisk: Avg. Read Queue Length Should be less than 2
2. PhysicalDisk: Avg. Write Queue Length Should be less than 2
3. PhysicalDisk: % Disk Time more than 50% indicates a bottleneck
as given here indicates that the disk is fine. That page says
to ensure that it was not a processor or memory bottleneck, I also recorded % processor time and available bytes. As you can see from Diagram 1, the processor's average was below 30%. If the processor were the bottleneck the trace would be over 80%. On the other hand, if there was a memory shortage, available bytes should drop below 10MB.

And our processor usage was < 20%, with > 200 MB of memory also available.

Maybe I should try the same checks when the system is crawling. If there is some indication of what application is causing the hard disk usage in some log somewhere, it would have been useful. I suspect IIS, since SSSBPT and the audio downloads are also on this machine.

Checking the colinux virtual machine, found only 100 MB free, 10%. Cleaned up to make it around 40%, 400 MB or so, by the following.

apt-get clean cleaning out /var/cache/apt

As given here, postsuper -d ALL
for /var/spool/postfix

from /home/sgh/playlist,
ls ????09* to find the 2009 playlists, removed them and older ones...

Some interesting info:
2007 Feb asia/africa/america started
2004 Oct started web stream

for /home/sgh/log

mv 0*.check ../audio/log_backup 2> /dev/null
mv 1*.check ../audio/log_backup 2> /dev/null
mv 2*.check ../audio/log_backup 2> /dev/null

etc - the errors to dev null, because mv complains that it cannot keep the same user ownership and permissions.

tail -5 logfile > logfile did not work, so had to do
tail ices.log > icesnew.log
mv icesnew.log ices.log

after taking backup.

But tho playout continues, log file is not appended :(
only after playout restarts next day? Anyway, had to restart all playouts by restarting colinux service since the streams started skipping, so the logs have restarted now.

No comments:

Post a Comment