firewall rule to restrict browsing during live times

P wanted some firewall rules on Cyberoam so that the live video broadcasts are not affected by people browsing, downloading, etc.

1. Added a schedule under
Firewall -> Schedule -> Define Schedule
called LiveTimes and added the times 7.50 am to 10 am and 4.25 pm to 6.30 pm.

2. Created a Bandwidth policy under
Policies -> Bandwidth Policy -> Create Policy
called LiveTimesThrottle to throttle bw to 256 kbps up and down.
It says Bandwidth in KB - so we have to enter 256/8 = 32 in the box there.

3. Added the LiveTimes schedule for this policy.

4. Added it to Firewall rule in Firewall -> Manage Firewall -> Rule 1 Any Host to Any Host Accept.
Now I am not sure if the Voxel traffic also will be allowed! So, adding a firewall rule,

5. LAN Any Host -> WAN Any Host, RTMP service, Accept + MASQ, with no time/bandwith limit.

Unfortunately, this had the opposite effect to what was desired - during the live times the bandwidth was unlimited, and at other times the bandwidth policy was enforced! This was because the bandwidth policy page says:
Default Values to be applied all the time....
and the schedule is added under
Add Schedule wise details to override default Bandwidth details

So one way out would have been to reverse the schedule timings. But P wanted an alternate solution, just giving priority to the RTMP streams to and from Voxel instead of throttling all traffic on all days at those times. So, he decided on creating a bandwidth policy, Guaranteed 1 Mbps (128 KB) and Burstable 2 Mbps (256 KB) for RTMP service to voxel, with maximum 0 priority. Now we have to check whether enabling just this bandwidth policy is enough by itself, or we need to explicitly de-prioritize other traffic.

making pretty pdfs of webpages and reports

In my previous post, I mentioned creating pdfs server-side for awstat. But creating pdfs from any web page or report?


One way - view only main frame by right-clicking on required frame and choosing View only this frame, and use that url in Adobe Acrobat Pro,
File -> Create PDF -> From Web Page


If you don't have Adobe Acrobat Pro, this solution looks good: using www.web2pdfconvert.com

configuring voxel server for sgh and h2h - part 3

Continuing my series of posts on server setup.

Wanted the home page to be not cached, since they update it often. Looking into it, found two interesting articles, at stackoverflow and metaskills. The first article gave a tip on how to control browser cache for specific urls - using the LocationMatch tag. The second one recommended expiring all html pages immediately. I followed the second one, since our server is not facing any resource crunch right now.

ExpiresActive On 

ExpiresDefault "access plus 1 seconds" 

ExpiresByType text/html "access plus 1 seconds" 

ExpiresByType image/gif "access plus 120 minutes" 

ExpiresByType image/jpeg "access plus 120 minutes" 

ExpiresByType image/png "access plus 120 minutes" 

ExpiresByType text/css "access plus 60 minutes" 

ExpiresByType text/javascript "access plus 60 minutes" ExpiresByType application/x-javascript "access plus 60 minutes" ExpiresByType text/xml "access plus 60 minutes"
Initially this block in apache2.conf was inside an if tag, IfModule mod_expires.c - but was not working. Then I commented out the IfModule tags - then it started working, since anyway I had enabled the module using a2enmod expires

Next - wanted to enable some sort of automated log parsing. Among free log analyzers, settled on awstats, since someone had made a mod for wowza too. Followed the ubuntu documentation for awstats installation. There, the recommended cron is

0 */3 * * * /usr/lib/cgi-bin/awstats.pl -config=yourdomain.ext -update > /dev/null

This notation */3 means once every three hours.

For installing GeoIP - tried enabling it directly, since it is supposed to be one of the default included modules, but got this error,

Error: Plugin load for plugin 'geoipfree' failed with return code: Error: Can't locate Geo/IPfree.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl . /usr/share/awstats/lib /usr/share/awstats/plugins /usr/lib/cgi-bin/plugins) at (eval 9) line 1.

So now I knew where the plugin directory was - so, could follow the method given here for installing maxmind's geolite databases.

wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
gunzip GeoIP.dat.gz
gunzip GeoLiteCity.dat.gz
gunzip GeoIPASNum.dat.gz

mkdir /usr/local/share/GeoIP/
mv *.dat /usr/local/share/GeoIP/

Tried installing Geo::IP Perl module with
http://www.cpan.org/modules/INSTALL.html

cpan App::cpanminus
cpanm Geo::IP

 FAIL Configure failed for Geo-IP-1.40. See /root/.cpanm/build.log for details

So, followed the steps at
http://antezeta.com/awstats/geoip.html
to install the C module.

wget http://geolite.maxmind.com/download/geoip/api/c/GeoIP.tar.gz
gunzip GeoIP.tar.gz
tar xvf GeoIP.tar.gz
 cd GeoIP-1.4.8
./configure
make
make install

Then,
cpanm Geo::IP worked.

Uncommented
LoadPlugin="geoip GEOIP_STANDARD /usr/share/GeoIP/GeoIP.dat"
changed the path to /usr/local/share/GeoIP/GeoIP.dat

Did not add the other two plugins, for city and ISP, since that would slow down AWstats.


Had to flush newer records, by deleting files in 
DataDir - /var/lib/awstats
in order to process older log files, as given at
http://awstats.sourceforge.net/docs/awstats_faq.html#OLDLOG
Took around 15 minutes to crunch old logs for 15 days, so probably the log for 3 hours will take only half a minute or less.


Next for Wowza. Did not have to modify the logging parameters as given at
http://www.wmconsulting.info/awstats-mod-for-process-wowza-stats/
since most of the parameters were already set. But had to change the LogFormat drastically, since my Wowza version is different - 3.03. Also, one point which he makes is to log only the destroy stream event. Not doing that, logging everything as we are doing, was causing over-counting of bytes sent. So, changed the LogFile parameter to read
LogFile=”/usr/local/awstats-wmc/tools/logresolvemerge.pl /usr/local/WowzaMediaServer/logs/*access* | grep destroy |” 


And my LogFormat parameter is
LogFormat="date time %other %other %other %other %code %other %other %other %virtualname %other %other %other %other %url c-ip %method %referer %ua %other %other sc-bytes %other %other %other %other %other %other %other %other %other %other %other %other %other %other %other" 


Using this method, bandwidth is slightly under-reported - by some 10%. But cheaper than Sawmill.


To make pdf reports, on the server, the buildstaticpages tool needed htmldoc, which I installed with apt-get install htmldoc

UPS test

Tested the APC Smart-UPS 5000VA with our projector and sound system load. Lasted for 45 minutes with 2 battery LEDs remaining, out of 5. Charging without load, with output switched off, came back to 5 LEDs within 8 hours.

configuring voxel server for sgh and h2h - part 2

Continuing my earlier post.

This part of the configuration, for the audio streams, was a series of experiments with Shoutcast v1 and v2 and Icecast2 - from Dec 15 2011 till now. Duplicating our Shoutcast server setup using v1 server as on our earlier server could have been done, but P was keen on improving things, especially the recorded to live (and back) transitions.

Shoutcast2 was our first trial, since using it would help us to restore our multiple streams on shoutcast.com directory. Tried Shoutcast2's multiple mount strategy. The Shoutcast2 source sc_trans does re-encoding, so we don't want that. Then we found that as this forum post indicates, sc_serv2 does not work with 1.x source like ices - so we need to have a Shoutcast1 server between ices and Shoutcast2. And when that happens, we can't use many of Shoutcast2's features. Also, if we wanted it running on multiple ports, we would need unique mount points on each port - all the ports show all the mount points. Then came the killer - Shoutcast2 was dumping listeners at live/recorded changeover even with backupfile set, and autodumpusers=0 , when relaying the Shoutcast1 streams. So that ended our Shoutcast2 experiment.

Icecast2 had similar features, like multiple mount points etc. And it was compatible with ices as a source. Only needs a change in the ices config file, instead of icy protocol, it has to use http protocol. And it had the nifty feature of moving listeners from one mount point to another using the admin interface, which caught P's eye. And fallback mounts, using which what we wanted to do could be accomplished. Basic idea being:
/mount-to-which-listeners-connect
      -has fallback mount-> /livemount
            -has fallback mount-> /recordedstream

When fallback override is set to true, whenever a source connects to /livemount, the listeners are automatically moved from /recordedstream to /livemount. And automatically moved back when the source disconnects. Very nice.

But preliminary exploration gave some disappointments - multiple ports still showed up all the mounts on all the ports, and the fallback mounts did not work for shoutcast relays. Also, there were some funny (buggy?) aspects, like mount points with relays not showing up on the admin interface till a (non-relay) source was connected. Also, how to configure the fallback mechanism for TeluguStream was not clear, since we would not be connecting a source to it directly. The fallback update mechanism given in the icecast admin section did not work, since there was no source directly connected to that mountpoint. Finally, cobbled together the following method.

• lots of alias directives, one for each port, redirecting the root / to the correct stream for that port. This could be done because aliases can be defined on a per port basis.
  Aliases like alias source="/" port="8008" dest="/discoursestream"
• TeluguStream falls back by default to its recorded stream, and AsiaStream falls back first to the live stream mount, and then to recorded stream - so the live changeover for AsiaStream is taken care of.
• An additional mount was defined as a relay of asiastream, and another icecast config file was written where TeluguStream falls back to this relay instead.
• When TeluguStream has to go live, the icecast config file is switched over, and
   killall -HUP icecast makes icecast reload the config file - so, listeners who connect now fall back to the relay of asiastream .
• Using the admin interface, the listeners who were already connected to the recorded telugustream mount are now moved to the relay mount.
• These steps were written into a shell script, with wget being used to do the admin interface calls - wget takes --user= and --password= parameters.
• A shell script which calls the above remote machine's scripts through ssh can be run from the Mac by double-clicking on the file, by just naming the script file with a .command extension.
• For our website's jplayer-based web player to work, had to change the called URL - for shoutcast it was mp3:"http://stream.radiosai.net:8002/;stream/1" and now for icecast it should be mp3:"http://server1.radiosai.org:8002/"
  The stream suffix was for Shoutcast to serve the stream instead of the status page. Our aliases are configured to redirect requests from root to the appropriate stream, so we can just request the root, /
• Just a caveat - icecast by default has very small numbers in threadpool and source connections. Have given 50 and 20 for now.
• Made icecast run like a daemon by calling it with nohup:
  nohup icecast -c /path/to/icecast_config_file.xml > sh.out 2> sh.err < /dev/null &

Then came the monitoring part. P pointed to amptracker.com with nice graphs. But they wanted $5 per month for the graphs. So, went back to shoutstats, which now has support for icecast too. But after install, found it did not work. Troubleshooting:

• First, apache was serving the file index.php like html, without rendering the php parts. Reason was, php short tags were turned off in our server, as found with phpinfo(). Enabled them in /etc/php5/apache2/php.ini
  and restarted apache2, the page displayed now. But shoutstats itself was not showing any graphs, even after I updated manually by accessing update.php manually.
• Tested rrdtool with http://oss.oetiker.ch/rrdtool/tut/rrdtutorial.en.html
  and working fine.
• Enabling print statements for debugging inside the php pages, found that though rrdtool was being called, it was not creating the .rrd file. File permissions were 777 for the relevant directories. Finally, printing out the exact system() call pointed out the error - there was a missing space after $rddtool in the system call.
   ' create '.$rddfile.'DS:audience...
  should have been
   ' create '.$rddfile.' '.'DS:audience
  Mentioned it in the github page, fixed it, shoutstats working after correct cron job entered.
• Had to remove the hidden status of the mounts - otherwise shoutstats would not work out of the box, would have to change the parsing of status2.xsl. So, made all the mounts visible - hidden = 0.

Most probably this finishes the server configuration posts. Possible tweaks needed for icecast would be to make the logs smaller - maybe - and maybe some tweaking needed for thread management under load and so on.

apache2 server redirect

Struggled for a while with mod-rewrite and all that to redirect one page to another page on the apache server. Finally solution turned out to be simple, as referred to in one line in the page above,
"Note that this can also be handled using the RedirectMatch directive". Syntax was, in relevant virtual host file, within the Directory tag,
RedirectMatch path/to/url.html http://server.path/to/redirected/file.htm

configuring voxel server for sgh and h2h

This is an ongoing process, started on 10 Dec 2011.

1. adduser username
2. apt-get install vsftpd
3. Disabled ssh with password using the procedure in the earlier post, copying required keys to .ssh/authorized_keys
4. Started copy process with sftp  from colinux, running under screen from saiwaves.
5. Enabled vsftpd -
   
   service vsftpd start
   To make it useful, had to enable ftp write by
   vim /etc/vsftpd.conf
   and un-commenting the line
   write_enable = YES
6. Also, added username1 user to /etc/ftpusers - so username1 is not allowed to ftp (only username2 is allowed to ftp)
7. Copied over the entire web folder from old server. Tried this first with Windows Explorer's built-in ftp, but was taking too long - several days - and breaking too often. So did it with Filezilla and completed in 10-15 hours.
8. Made postfix work: it was giving the error 
   send-mail: fatal: open /etc/postfix/main.cf: No such file or directory
   Can't send mail: sendmail process failed with error code 75
   Solution was, as given here, dpkg-reconfigure postfix
9. Fixed the ices error Cannot use config file (no XML support). by doing
   apt-get install libxml*and then recompiling ices
10. Worked out why and how the metadata is not sent for our shoutcast streams by ices: the song info is not displayed if the SrcIP=ANY and DestIP=127.0.0.1 and the ices connects on an address other than 127.0.0.1. Basically, song metadata is not sent if the ip on which ices connects does not have a route to one of the ips in DestIP. 
11. For ices not to show the filenames for icecast server, the above trick does not work. Also, the method mentioned here and here, of setting
    mp3-metadata-interval to 0
    also does not work. So, fiddled with the ices source code and recompiled. Just commented out most of the code in metadata_update function. No more metadata!
12. Enabled virtual hosts as given here, first making a suitable file sitename.com in
    /etc/apache2/sites-available
    and then
    a2ensite sitename.com
    service apache2 reload
13. Replaced asp redirects with php redirects for index pages etc.
    header( 'Location: http://www.site.com/new_page.html' ) ;
14. Our windows web server was ignoring case in filenames - uppercase or lowercase did not matter. With the new Linux server, case does matter, and many many links break. To fix, loading mod_speling. It was already available in /etc/apache2/mods-available. So only had to do
    a2enmod speling
    (a2enmod = apache2 enable module script available on debian/ubuntu)
    Then, had to add CheckSpelling on in /etc/apache2/apache2.conf
    (This part is not well documented!)
15. Had a large number of Main.asp files (for each old journal issue cover page). Actually these files were almost completely html. So, took the short-cut of serving these files as html. Edited the relevant file in sites-available to add
    
    
16. Then added DirectoryIndex Main.asp
    This caused major breakage, till I added all the other default index pages!
    DirectoryIndex Main.asp index.htm index.html index.php
17. Apache was giving a 403 forbidden for most pages. Turns out it needs execute permissions also on the directories. This page discusses the same problem and the solution. So, set
    local_umask=022
    file_open_mode=0777
    in vsftpd.conf

The configuration for the radio streams is yet to be finalized, since the icecast configuation has to be tested.

ebook reader on iphone

I'm back to the iPhone - the farewell did not work out! Selling it on ebay did not work so well, as the highest bidders seems to be phoneys who bid higher than the Buy Now price, and then did not pay up. Also, the touchscreen on the windows mobile stopped working, making it lose a lot of functionality, like zooming pdfs and images. Showing IRCTC and KSRTC tickets on the mobile then becomes difficult :)
So, back to iPhone.

Unfortunately, I overwrote Stanza with the newer version, which crashes for old versions of iOS. Installing old versions of Kindle and Stanza with iFunBox did not work - the apps always crashed. Researched and found:

PDF Reader Lite - reflow doesn't work very well, very slow, but works.

ePagine Reader - did not crash, but could not see the sideloaded epub books.

iFlow Reader - need to rename the epubs to the same name as one of the three supplied epub books!

Bluefire reader - works like a charm for epub.

making DVD player region-free

A wanted to make their DVD player region-free. Googled to get
http://www.videohelp.com/dvdhacks

saving youtube video to ipad

B wanted a method to save youtube video to ipad. Googled up this:
http://www.ipadtabletcentral.com/?p=2252
Basically using the TagDisk HD app.