more on overlay

After my earlier post on using AviSynth and virtualdub to create an overlaid video, found that the framerate of the clips had been wrong. The first version of the script had
bg=Imagesource("bk3.jpg", end = 16200, fps=30, use_DevIL=true)
top = BicubicResize(Crop(DirectShowSource("1.mp4"),40,6,-36,-4),320,240,0,0.75)

audio=DirectShowSource("1.mp4")
video1=Audiodub(Overlay(bg, top, 834, 771), audio)

Overlay uses the background clip's length to create the framerate. In this case, the clip used for "top" became speeded up.
So, modified the script, and also took into account that some of the clips were PAL and some NTSC - converted the PAL to NTSC using Bob and Weave using the recipe given here.

bg1=Imagesource("bk3.jpg", end = 15732, fps=30)

# changed the number of frames here
top = BicubicResize(Crop(DirectShowSource("1.mp4"),40,6,-36,-4),320,240,0,0.75)
audio=DirectShowSource("1.mp4")
video1=Audiodub(Overlay(bg1, top, 834, 771), audio)
# this was NTSC video, did not have to change framerate

 bg2=Imagesource("bk3.jpg", end = 17264, fps=30)
# changed the number of frames here to match number of frames
#  in 30 fps version of ntscvid2 below

palvid2=DirectShowSource("2.mp4")
palvid2=Bob(palvid2) # Separate fields and interpolate them to full height.
palvid2 = ChangeFPS(palvid2, 60000, 1001)
# Convert field rate to NTSC, by duplicating fields.


palvid2 = SeparateFields(palvid2)
palvid2 = SelectEvery(palvid2,4,0,3)
# Undo Bob, even field first. Use SelectEvery(4,1,2) for odd field first.
ntscvid2= Weave(palvid2)
top2 = BicubicResize(ntscvid2,320,240,0,0.75)
audio2=DirectShowSource("2.mp4")
video2=Audiodub(Overlay(bg2, top2, 834, 771), audio2)
# and so on

recovering a Windows XP laptop

V's Windows XP laptop - this old Acer machine, repurposed as an internet radio player - was not booting. Not even in safe mode command prompt only. As recommended by PB, tried booting from Windows XP boot disk, recovery console. That worked. Then tried chkdsk in recovery console, rebooted, chose "Last known good configuration", worked fine.

Did a Scandisk thorough after that also, just to be safe. Seems to work fine now.

abandoned attempts to play shows on Toshiba AC100

With mvideoplayer app on the Toshiba AC100, was able to play files sequentially from a folder - that seemed to be its playlist functionality as mentioned briefly earlier. With that, tried to play today's Planetarium show using the Toshiba AC100. Made multiple copies of the intro avi files in each relevant folder using hard links, with the SD card formatted as ext3 on Linux to save space on the SD card.
ln /full/path/to/sourcefile /full/path/to/destinationfile

And so on.

Unfortunately, when the video stutters a bit - and the Toshiba has that problem, with a few frame drops here and there - the HDMI input on the projector blacks out for 5 seconds or so. This happened 2-3 times during the show. Clearly this is unacceptable, so gave up playing Planetarium shows on the Toshiba AC100 for now.

The HDMI input does make a difference, with the video being much crisper than with the VGA output from the laptop. Maybe a standalone HD player might work out.

importing files without playing them - revisited

After I had used this technique, we have made some changes, ie. adding DiscourseStream to the mkcsv scripts, with AsiaStar being dropped. So, had to do an additional step of creating some dummy records for DiscourseStream - finding the fileid 695 for bcid 1314 (Signature tune for AsiaStar), changing it to bcid 9 (DiscourseStream) and changing the time from 6.30 am IST to 7.00 am IST (changeover time for DiscourseStream). Then changing bcid for a couple more dummy records which have timestamp after 7 am. This way, imported a lot of files like Polish Veda Pushpanjali, Satyopanishad in Mandarin, etc to our audio archive / audio search.

avisynth and virtualdub redux

Last time I had used a transform in virtualdub in order to create a video with predominantly white background and a window with a slideshow. This time, did all the processing in avisynth and used virtualdub only to save as avi. Used virtualdub's audio filters for the first time, and had to see this article in order to make them work - we need to add the input and output filters in addition to any other filter we add, and then link them together in a meaningful way.

The main avisynth filter I used was Overlay. Also, Crop to remove unnecessary black bars from one of the videos. The script is as follows, concatenating together three videos with a background pic which has a nice frame on a predominantly white background, as below.

top = BicubicResize(Crop(DirectShowSource("1.mp4"),40,6,-36,-4),320,240,0,0.75)
audio=DirectShowSource("1.mp4")
video1=Audiodub(Overlay(bg, top, 834, 771), audio)

top2 = BicubicResize(DirectShowSource("2.mp4"),320,240,0,0.75)
audio2=DirectShowSource("2.mp4")
video2=Audiodub(Overlay(bg, top2, 834, 771), audio2)

top3 = BicubicResize(DirectShowSource("3.mp4"),320,240,0,0.75)
audio3=DirectShowSource("3.mp4")
video3=Audiodub(Overlay(bg, top3, 834, 771), audio3)

video1 ++ video2 ++ video3

extracting audio from DVD

The older post on this topic mentioned DVD Decrypter and others. This time, for some reason, that method did not work. Perhaps because of some mistake in the sequence of steps, I suppose, the resulting file was a VOB file and not an mp2 file. Anyway, the file was not being opened by any of the AC3 decoding methods like WinAmp plugin etc.

So, this time, I tried just copied the VOB files to the hard disk and opening them with VirtualDubMod, extracting the audio stream using Streams -> Stream List -> Save wav. That wav was opened using Soundforge, using AC3 codec. Slightly buggy, since each VOB file's wav data had the duration info of the entire program chain, and so had some sort of repeating audio after the first 25 minutes or so which had to be cut out. But otherwise OK.

installing Perl module on dreamhost

K wanted me to include mp3 tags on the downloads at dl.radiosai.org. Googled installing Perl Module without root access, and found lots of people doing it, so went on to install MP3::Tag module without root access.

First I tried the procedure at
http://perl.jonallen.info/writing/articles/install-perl-modules-without-root
but the perl process kept getting killed as given at
http://wiki.dreamhost.com/Killed

Dreamhost recommends installing via the CPAN command, as given at
http://wiki.dreamhost.com/CPAN
and then the process doesn't get killed.

But the wiki had some bugs. Got errors like

Cannot gzopen $ENV{HOME}/.cpan/sources/authors etc. Googled it and found
http://discussion.dreamhost.com/thread-87436.html
where the user k2S explains that he did two things differently from the wiki page
http://wiki.dreamhost.com/CPAN
that is

**don't forget to add the suggested lines into your '~/.bash_profile' - that is, not ~/.profile

 ** test command 'echo $PERL5LIB', it should return the correct path

* *I had to edit also ~/.cpan/CPAN/MyConfig.pm and replace all '$ENV{HOME}' with the correct path

Once I did this, the install worked fine, with perl -MMP3::Tag -e 1 not returning any errors.
 (I had actually done the manual config of the MyConfig.pm  as given in the wiki, too.)

After the module installation, some trial scripts still gave errors. Trying out a basic script from

http://search.cpan.org/~ilyaz/MP3-Tag-0.9709/Tag.pm
got errors like
Can't call method "get_tags" on an undefined value

Found this explanation, that it is because the file has no tag to begin with:
http://www.linuxquestions.org/questions/programming-9/cant-call-method-title-on-undefined-title-in-perl-script-396858/

Then tried tagging the file with no tag to start with, and got errors like:
Can't call method "new_tag" on unblessed reference at ./mp3tagger.pl line 6.

Problem was - the filename has to have the full path, and not start with ~/
Once that was done, the script worked. Now to work out some sort of tagging scheme, after which the tagging can be done.

Cyberoam and ftps

The guys at the studio had some trouble connecting to secure ftp. Their newly commissioned server would accept connections only over secure FTP on port 21, as below,

I was able to connect from BSNL network as well as from the Planetarium. But when they tried from the studio, they were getting timed out at "Initializing TLS". I replied with the following:

When the connection is initialized, a dialog box pops up, talking about the certificate, and whether we want to accept the certificate - attached. Only after you say yes does it connect. So, there are several possibilities - either your machine is not getting the certificate due to cyberoam, or somewhere in settings you have set it to not ask, or deny the certificate.

My guess is cyberoam. Cyberoam is set to proxy ftp connections. So, ftps may need some settings. You may want to ask the cyberoam support about this.
http://kb.cyberoam.com/print.asp?id=36&Lang=1&SID=

"Cyberoam parses SSL handshake (SSLv2, SSLv3, and TLS) and extracts “Common Name” (CN) from the certificate. It applies control filters on common name. Based on the outcome of filters, user is either served the page or the connection is terminated."

So probably somewhere in cyberoam you will need to add the certificate of this server, saying accept this certificate. Only then, I am guessing, will cyberoam connect to the server....

They contacted cyberoam support, and as per their recommendation added a MASQ rule so that this connection goes directly instead of through cyberoam's transparent proxy. Then it worked fine.
 

Prayaya

A sent me this link,
http://en.prayaya.com/index.html
for software which can create a portable subset of your desktop with the software you need, which you can then run from USB key or other portable storage on any Windows computer. Basically making any app portable. Interesting concept.

Google Sky Map

I remember seeing a device on Amazon.com which you could point to the sky, and it would label the objects seen. Now, an app - free to boot - does the same job. Google Sky Map.

finally disabled old server

When the SaiUniti FS3 was restarted, our old Colinux service as well as the ShoutCast servers started running on it again, as I found with the Colinux auto email on reboot, since they were set to automatic startup. So, logged in to the server, disabled the automatic startup of all the shoutcast services + colinux on that server.

regressive feature on Mac OS X Lion

Yesterday DSP came with his MacBook to reinstall Microsoft Office to solve a bug as instructed by "experts" at Applecare Australia. Long procedure to uninstall, and after reinstalling, a longer time to download and install all updates. And finally, at the end, his problem remained. His issue was that even after he releases the SHIFT key, the next letter would be capitalized. Now, this is actually a "feature" of Mac OS X Lion, carried over from iOS - demonstrated by typing something in TextEdit. But it does create problems for those who type moderately fast.

monitoring apache server

A couple of days back, we were caught left-footed when our website slowed down to a crawl. The "standard practice" of restarting the process, restarting networking, even rebooting the server, did not help. Later, found that the problems were due to a 500 MB + flipbook uploaded the previous night, with a link on the home page.

Voxel support had pointed to
apache2ctl -t
as a possible first test, to see if apache was able to start with the config given. Did some googling and found this optimization and tuning page, which has some sections relevant for our static website also. In particular, the part where he says use
apache2ctl status
and
top d 1
to monitor apache's health.

These discussions also provide some tips on monitoring swapping, disk thrashing, io bottlenecks etc using vmstat, top, iotop and so on. This page has a good description of vmstat and free, what the displays mean etc.

preventing saiwaves disk from filling up

PB reports:

automysqlbackup script runs on Krishna and takes a backup into ~/dbBackups on krishna
The cron on saiwaves takes a copy of the backup into saiwaves through dbdumpsftp.sh

Changed the scp to copy only the daily folder instead of the entire backups folder. Also, changed destination to /local instead of ~

/local has about 200+ GB free

Hope this resolves the issue.

error running our import script

While running our file import script, got an error,

error while loading shared libraries: libgnomeui.so.32: cannot open shared object file: No such file or directory

Sorry response not recoginsed...

 

Turned out it was because there was a wav file in the import folder. Removed the wav file, the import script ran fine.

procedure for Mandir live broadcast

Documenting the Mandir live broadcast procedure.

1. Power on procedure is:
   (a) UPS input on
   (b) UPS on
   (c) Extension box power on (right-hand side of equipment box)
   (d) Tascam US-1641 power on
   (e) MacBook power on
   Power down procedure follows in the reverse order.
2. In the MacBook, the programs to be run are in right-to-left order in the Dock. We start by powering up the Tascam Control Panel app.

3. In the Tascam Control Panel, choose the dropdown to change the Digital output channels from line out 1 & 2 to line out 3 & 4.

4. Next, fire up the Jack Pilot app.

5. Press the start button in Jack Pilot to start Jack. Wait till Jack starts up, with a wait dialog box indicating the starting of Jack.

6. Next, start the BUTT app.

7. And finally start Reaper.

8. Now set up routing in Jack for broadcast. For this, click on the Routing button in Jack Pilot, and choose File -> Load Studio Setup, choosing the broadcast_setup.jks file.

9. The setup can be tested by playing the output of reaper. Reaper channels 1-8 are mapped to sliders 1-8 on the Korg NanoKontrol. Channels 9-14 are mapped to knobs 1-6.
   
   Channel 12 (knob 4) has instrumental music and
   Channel 13 (knob 5) has the pre-live announcement pre-loaded.
   Channel 14 (knob 6) has the full mix output of PA mixer coming in.
   Pressing play on the Korg NanoKontrol allows you to test whether the BUTT gets signal when knob 5 is opened, as seen by the lit up LED-style indicators on BUTT.
10. Start playback of pre-live Announcement in Reaper approximately 3 minutes before desired start of live broadcast.
11. Press the 'Play' button in BUTT to start live broadcast on AsiaStream.
12. 
    Double-click the "startlive.command" shortcut on the Desktop to start live broadcast on TeluguStream.
13. 
    
    
    Fade out knob 5 and allow knob 6 to maintain around -12 dB at the output when the program starts. Playback can be stopped with the stop button on the NanoKontrol.
14. Recording of programs is mapped to Channel 15 which is record enabled. The spacebar is the shortcut key.
15. If recording is in progress, if a filler instrumental is to be added, BI.mp3 can be dragged from desktop to Channel 12 (knob 4) which has record enable DISABLED - the dull red ar button.
    Unless the track has record enable Disabled, playback will not occur from the track.
16. 
    The same procedure as above can be used to drag and drop the ANN_END_LIVE end announcement file for playout at the end if no one is available for live announcement.
17. The microphone with stand available can be connected to the XLR input Channel 8 on the Tascam, and this can be used for Live announcements.
18. In case there is live video streaming, the network cable tagged "Studio" has to be disconnected from the Macbook and connected to one of the BLUE coloured LAN ports of the router. The router has to be plugged into the UPS. The lan cable from the router has to be plugged into the MacBook. This setup has to be restored at the end of the live broadcast by unplugging the router and restoring the "Studio" tagged network cable, connecting it back to the Macbook.

creating closed caption subtitles for youtube videos

Youtube gives the option for closed captioning, which can be used to provide subtitles for videos. The part 1 of this post details the creation of the subtitle .srt file, which can be used for viewing subtitles on desktop players also. Part 2 walks through the process of adding the .srt file to youtube. The workflow follows this youtube tutorial, http://www.youtube.com/watch?v=e9t6cvEm600

Part 1 - creating the SRT file.

1. Download and install DivXLand Media Subtitler from http://www.divxland.org/subtitler.php
2. Create subtitles in notepad, use | to delineate multiline titles, else one line per subtitle.
3. File -> Open Text or Subtitle to open the text file create above,
    File -> Open video - rename flv video to mp4, if not recognized by the player.
4. Click apply and hold for as long as you want the title to appear.
5. Repeat till end of video, deleting duplicate last title if any.
    File -> Save As SRT

Part 2 - applying to youtube video
1. Upload video to youtube.
2. Choose Video Manager in your youtube account dropdown,

3. In the Edit dropdown, choose Captions,

4. Select Add New Caption 

 5. Choose this SRT file you have created using the browse button, and click the upload file button.

Edit: If you want youtube to show subtitles or captions ON by default, you must add the following tag to the video - yt:cc=on

Yamaha audiogram and "noise in recording"

I got a call from SK saying they are trying to start work at the newly inaugurated Studio, with a Yamaha Audiogram 6, but the recordings turn out to have some noise. My reply:

Some of the probable reasons are given in the forum posts below:
http://forum.recordingreview.com/f23/yamaha-audiogram-6-cubase-ai4-18362/

1. Upgrade to Windows XP SP3
2. Make sure you have connected to a USB2 port and not to USB1
3. Make sure no other device (like mouse) are connected to the same bus (using device manager, you can check this.)
   
4. Increase the buffer as given in the forum posts above

save a particular web page at predefined intervals in a specific folder

NA had this query - how to automate saving  a particular  web page ( Specified  url in the settings )  at  predefined intervals  in a specific folder.

My reply:

I assume you want to do this on a Windows machine. On a Linux machine this is quite easy.

For Windows:

write a vscript file to save the webpage as given in
http://www.computerhope.com/forum/index.php?topic=110781.0

Schedule it to run periodically using Windows task scheduler as given at
http://www.iopus.com/guides/winscheduler.htm

Follow-up about making the filename unique for each grab:
filename changing can be managed by massaging the vbs file, appending the date-time to the filename or something like that. Since I'm not a vbs guru, I'm not spending time on it to modify the file for you. :)

Also, making Windows Scheduled tasks repeat tasks every few minutes:
http://support.microsoft.com/kb/226795

NA reported that he has successfully implemented this.

Laptop DVD drive troubleshooting

Guest post from DJ:

Platform: 

Dell Vostro 1400

Windows XP SP3

Problem: For over a year the CD/DVD RW player did not perform properly. Jittery playback of Audio CD and most commercial DVD discs were not read. Writing was producing errors and most DVD R blanks were not recognized. 

Analysis 1: Drive lens problem. Cleaned it a couple of times - No positive impact

Analysis 2: Drive hardware problem. - Result out of laziness did not bother to fix and planned to buy external USB drive.

Analysis 3: (After over an year of sadness and lack of enthusiasm). Lens and hardware at fault. 

Took out my camera lens cleaner kit and carefully cleaned the lens. Still there was jitter. But all discs and DVD's were recognized. This gave me hope. Also I noticed that during DVD playback the computer response was sluggish. This prompted to think of DMA. In Device manager I found that the mode was sett to PIO and DMA was not an option. This was quite strange. Out of curiosity I checked the Dell website for any available updates. 

There was a firmware update available for the DVD drive. Carried out firmware upgrade for the DVD drive. Voila - Now it works like a charm. All DVD's are playing smoothly. Have not yet tried burning any discs yet.

I still cannot figure out how the firmware got corrupted - the DVD drive used to work fine earlier!

order mixup on schedule page

Our schedule page appeared to have some sort of bug where the order of some files would get mixed up in the "Listen Now" or downloads listings. The reason seemed to be that the files were being shown in the order of the playlist based on timestamp, but the timestamp only had hours and minutes, and not seconds, in the csv file generated from our local database and uploaded to the server. So, those files which had durations of less than a minute would have the same timestamp as the next file, and these two files would be interchanged during the display. So, today, added seconds to the csv, and the bug seems to have gone away. B tested the import before and after, etc.

bug-fixing the downloads links in search results

After the javascript for sorting the table was added to the audio search results page, there was a bug introduced, brought to our notice by a listener. Due to the way in which the javascript call was written, sorting the table caused the wrong links to appear. B, who had written the original code, fixed it by changing the javascript calls to explicitly reference the fids.

screenshots of entire webpages

Useful tool - Fireshot - to create snapshots of entire (multipage) web-pages.
Can annotate the page, too. Works with IE also.

https://addons.mozilla.org/en-US/firefox/addon/fireshot/

Connecting iPhone to ad-hoc network - without access point

Connecting iPhone to ad-hoc network - without access point - as requested by A:
for a Mac,
https://culturedcode.com/things/wiki/index.php/Setting_up_an_Ad-hoc_Wi-Fi_network
or
http://geeknizer.com/how-to-connect-iphone-to-mac-adhoc-wireless-without-router-reverse-tether/

picture in picture with avisynth

Here is a nice recipe for doing p-in-p using avisynth:
http://www.fauskes.net/nb/avisynth/#picinpic
Basically using the Overlay function.

Using this for creating intro video, a white background with a picture frame of 1080p resolution around a 320x240 video.

unlimited bandwidth hosting options

Interesting thread on slashdot,
http://ask.slashdot.org/story/12/02/28/0029218/suggestions-for-music-hosting

1. http://www.fdcservers.net/ [fdcservers.net]
FDCservers has had great support. They are absolutely the cheapest per-megabit in North America, with rates of <$150 for 100Mbps unlimited bw, which you can easily push with even a mediocre server. Their plans also don't require you to build or buy your own servers, which is great.

2. http://www.leaseweb.com/ [leaseweb.com]
In Europe, you can look into Leaseweb. Based in Amsterdam, they have great rates (<$150 too for 100Mbps unlimited bw) for their dedicated servers, slightly worse support than FDCservers, but still pretty good service. They're based in Amsterdam, and have pretty decent North American data transfer rates despite their location.

etc

Toshiba AC100 trials

Got a Toshiba AC100 for just Rs. 10,000, wanted to try and play the Planetarium shows from it since it had an HDMI out and is supposed to be HD capable. It played the 6 Mbps Xvid videos without hiccups, but the HDMI out was not stable. Pausing the video would cause the HDMI to be interrupted, so that the projector would show a black screen. Also, there seemed to be no way to play a playlist of multiple videos one after the other.

Moboplayer was supposed to be one of the players for Android which supports playlists. Since the Toshiba AC100 does not have the Android Market, downloaded and installed it from apktops.com. But it appears that the playlist support needs the menu, which needs to be accessed by a swipe on the touchscreen, and the AC 100 does not have a touchscreen! Foiled!

Tried installing Linux on the device instead of Android. But the players in Linux were disappointing. The default movie player in Ubuntu and vlc both were quite unable to play the 1080p 6 Mbps movie files - played like a slide show, only updating the frame when seeking etc. mplayer was better, but was still dropping frames. So, restored Android.

Tried an android port of mplayer. Unfortunately it skips frames when playing the 6 Mbps 1080p XVid movie files.

Then tried mvideoplayer, downloading from some download site - I think mediafire. It works reasonably well, playing all the files in one folder sequentially like a playlist. It is supposed to support playlists, but I have not been able to make playlists with it yet, it says "No playlists found" even when m3u pls and asx files are in the usb disk.

Sigil for hand-crafted EPUBs

H introduced me to Sigil - the wysiwyg ebook creation tool for EPUB.

Installing and running Linux on the Toshiba AC100

Since my main purpose was to use the Toshiba AC100 for screening 1080p HD movies at 6 Mbps or more, I did not do too many benchmarks or other tests. But the installation procedure was as follows.

This link provided tips on taking a backup of the factory boot-loader. Since I run Kubuntu as a dual boot on my laptop, I could install nvflash as given here, with dpkg.

After installing nvflash,
whereis nvflash gave the location  /usr/lib/nvflash for fastboot.bin which was the included bootloader. So, the command I used for backup of the Toshiba's bootloader was
nvflash --bl  /usr/lib/nvflash/fastboot.bin --read 6 partition_6_backup.bin --go

When I tried it the first time, I had got the path wrong, a typo. After retyping the correct path, it took a very long time, and didn't seem to be doing anything. Just for testing out, tried putting off the device, and again putting it in recovery mode with Ctrl-Esc pressed down while power on. Then tried
nvflash --bl /usr/lib/nvflash/fastboot.bin --go
nvflash -r --getpartitiontable partitiontable.txt

nvflash -r --read 2 part-2.img

etc

as suggested at http://gitorious.org/ac100/pages/Installation , worked in seconds. So, made a backup of partition 6 as given above.

Then followed the instructions at https://wiki.edubuntu.org/ARM/TEGRA/AC100 for the install.

The system was fairly OK with Ubuntu 11.10, though a bit sluggish with its 512 MB RAM. Firefox opened in 10 seconds. Boot time was around a minute. xubuntu loaded with apt-get, and with an xfce desktop, it felt more responsive.

Anyway, since the video was not playing without frame-drops, went back to Android by restoring the boot loader,
nvflash --wait --bl /usr/lib/nvflash/fastboot.bin --download 6 partition_6_backup.bin --go

why did the server crash

There was an outage for a few hours yesterday. Reason was that the server had stopped responding. All services seemed to have crashed. Audio streams from around 11 am, Apache from around 3.50 pm. No ssh also. But a remote reboot worked and solved the issue.

Checking out logs, Apache's access log and error log both stop at around 3.50 pm till the restart around 50 minutes later. Not much info except some lines in Apache logs which say possible SYN flooding, which may or may not be a real DoS attack. If there is a tuning issue with SYN flooding, this might be a possible solution,
http://labs.creativecommons.org/2010/12/08/tuning-tcp-on-ccs-servers/

ebook readers on iphone redux

In my previous post, I mentioned that Bluefire reader is working fine. But... while using, found that sometimes it becomes very slow. Slow to load, 2-3 minutes to start; slow to change pages, sometimes taking 2-3 seconds or more. Looking around for other options, found this post, and also this comparison wikipedia page.

Tried out BeyondPrint, ShuBook and uBooks.

BeyondPrint - interface looks very similar to BlueFire reader, has similar limitations. But also supports PDF.

ShuBook - is quick and responsive, but can only load books from the online sources built-in with the free version. The paid version has the option for us to load our own books etc.

uBooks - Can upload books with wifi using its own web interface. Web interface doesn't work with Firefox which complains about some incompatibility, but works fine with IE. Has a slightly different interface - not the paginated interface of other ebook readers, but allows continuous scrolling. Also jumps to "next page" with a tap on rhs of screen.

So, as of now, using ShuBook for books from Gutenberg et al, and uBooks for others.

PHPlist - issues with spaces in words

PB found a possible solution for the problem where the SI messages when sent by email had spaces inside random words. Basically it seems to be due to the MTA adding carriage returns or line feeds or CRLF in the middle of large lines.
https://mantis.phplist.com/view.php?id=15603

For the time being, the SI people are using a workaround, of pasting the message into the PHPlist compose box again after copy-pasting the entire html message. Perhaps that adds CRLF and has the same end result.

firewall rule to restrict browsing during live times

P wanted some firewall rules on Cyberoam so that the live video broadcasts are not affected by people browsing, downloading, etc.

1. Added a schedule under
Firewall -> Schedule -> Define Schedule
called LiveTimes and added the times 7.50 am to 10 am and 4.25 pm to 6.30 pm.

2. Created a Bandwidth policy under
Policies -> Bandwidth Policy -> Create Policy
called LiveTimesThrottle to throttle bw to 256 kbps up and down.
It says Bandwidth in KB - so we have to enter 256/8 = 32 in the box there.

3. Added the LiveTimes schedule for this policy.

4. Added it to Firewall rule in Firewall -> Manage Firewall -> Rule 1 Any Host to Any Host Accept.
Now I am not sure if the Voxel traffic also will be allowed! So, adding a firewall rule,

5. LAN Any Host -> WAN Any Host, RTMP service, Accept + MASQ, with no time/bandwith limit.

Unfortunately, this had the opposite effect to what was desired - during the live times the bandwidth was unlimited, and at other times the bandwidth policy was enforced! This was because the bandwidth policy page says:
Default Values to be applied all the time....
and the schedule is added under
Add Schedule wise details to override default Bandwidth details

So one way out would have been to reverse the schedule timings. But P wanted an alternate solution, just giving priority to the RTMP streams to and from Voxel instead of throttling all traffic on all days at those times. So, he decided on creating a bandwidth policy, Guaranteed 1 Mbps (128 KB) and Burstable 2 Mbps (256 KB) for RTMP service to voxel, with maximum 0 priority. Now we have to check whether enabling just this bandwidth policy is enough by itself, or we need to explicitly de-prioritize other traffic.

making pretty pdfs of webpages and reports

In my previous post, I mentioned creating pdfs server-side for awstat. But creating pdfs from any web page or report?


One way - view only main frame by right-clicking on required frame and choosing View only this frame, and use that url in Adobe Acrobat Pro,
File -> Create PDF -> From Web Page


If you don't have Adobe Acrobat Pro, this solution looks good: using www.web2pdfconvert.com

configuring voxel server for sgh and h2h - part 3

Continuing my series of posts on server setup.

Wanted the home page to be not cached, since they update it often. Looking into it, found two interesting articles, at stackoverflow and metaskills. The first article gave a tip on how to control browser cache for specific urls - using the LocationMatch tag. The second one recommended expiring all html pages immediately. I followed the second one, since our server is not facing any resource crunch right now.

ExpiresActive On 

ExpiresDefault "access plus 1 seconds" 

ExpiresByType text/html "access plus 1 seconds" 

ExpiresByType image/gif "access plus 120 minutes" 

ExpiresByType image/jpeg "access plus 120 minutes" 

ExpiresByType image/png "access plus 120 minutes" 

ExpiresByType text/css "access plus 60 minutes" 

ExpiresByType text/javascript "access plus 60 minutes" ExpiresByType application/x-javascript "access plus 60 minutes" ExpiresByType text/xml "access plus 60 minutes"
Initially this block in apache2.conf was inside an if tag, IfModule mod_expires.c - but was not working. Then I commented out the IfModule tags - then it started working, since anyway I had enabled the module using a2enmod expires

Next - wanted to enable some sort of automated log parsing. Among free log analyzers, settled on awstats, since someone had made a mod for wowza too. Followed the ubuntu documentation for awstats installation. There, the recommended cron is

0 */3 * * * /usr/lib/cgi-bin/awstats.pl -config=yourdomain.ext -update > /dev/null

This notation */3 means once every three hours.

For installing GeoIP - tried enabling it directly, since it is supposed to be one of the default included modules, but got this error,

Error: Plugin load for plugin 'geoipfree' failed with return code: Error: Can't locate Geo/IPfree.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl . /usr/share/awstats/lib /usr/share/awstats/plugins /usr/lib/cgi-bin/plugins) at (eval 9) line 1.

So now I knew where the plugin directory was - so, could follow the method given here for installing maxmind's geolite databases.

wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
gunzip GeoIP.dat.gz
gunzip GeoLiteCity.dat.gz
gunzip GeoIPASNum.dat.gz

mkdir /usr/local/share/GeoIP/
mv *.dat /usr/local/share/GeoIP/

Tried installing Geo::IP Perl module with
http://www.cpan.org/modules/INSTALL.html

cpan App::cpanminus
cpanm Geo::IP

 FAIL Configure failed for Geo-IP-1.40. See /root/.cpanm/build.log for details

So, followed the steps at
http://antezeta.com/awstats/geoip.html
to install the C module.

wget http://geolite.maxmind.com/download/geoip/api/c/GeoIP.tar.gz
gunzip GeoIP.tar.gz
tar xvf GeoIP.tar.gz
 cd GeoIP-1.4.8
./configure
make
make install

Then,
cpanm Geo::IP worked.

Uncommented
LoadPlugin="geoip GEOIP_STANDARD /usr/share/GeoIP/GeoIP.dat"
changed the path to /usr/local/share/GeoIP/GeoIP.dat

Did not add the other two plugins, for city and ISP, since that would slow down AWstats.


Had to flush newer records, by deleting files in 
DataDir - /var/lib/awstats
in order to process older log files, as given at
http://awstats.sourceforge.net/docs/awstats_faq.html#OLDLOG
Took around 15 minutes to crunch old logs for 15 days, so probably the log for 3 hours will take only half a minute or less.


Next for Wowza. Did not have to modify the logging parameters as given at
http://www.wmconsulting.info/awstats-mod-for-process-wowza-stats/
since most of the parameters were already set. But had to change the LogFormat drastically, since my Wowza version is different - 3.03. Also, one point which he makes is to log only the destroy stream event. Not doing that, logging everything as we are doing, was causing over-counting of bytes sent. So, changed the LogFile parameter to read
LogFile=”/usr/local/awstats-wmc/tools/logresolvemerge.pl /usr/local/WowzaMediaServer/logs/*access* | grep destroy |” 


And my LogFormat parameter is
LogFormat="date time %other %other %other %other %code %other %other %other %virtualname %other %other %other %other %url c-ip %method %referer %ua %other %other sc-bytes %other %other %other %other %other %other %other %other %other %other %other %other %other %other %other" 


Using this method, bandwidth is slightly under-reported - by some 10%. But cheaper than Sawmill.


To make pdf reports, on the server, the buildstaticpages tool needed htmldoc, which I installed with apt-get install htmldoc

UPS test

Tested the APC Smart-UPS 5000VA with our projector and sound system load. Lasted for 45 minutes with 2 battery LEDs remaining, out of 5. Charging without load, with output switched off, came back to 5 LEDs within 8 hours.

configuring voxel server for sgh and h2h - part 2

Continuing my earlier post.

This part of the configuration, for the audio streams, was a series of experiments with Shoutcast v1 and v2 and Icecast2 - from Dec 15 2011 till now. Duplicating our Shoutcast server setup using v1 server as on our earlier server could have been done, but P was keen on improving things, especially the recorded to live (and back) transitions.

Shoutcast2 was our first trial, since using it would help us to restore our multiple streams on shoutcast.com directory. Tried Shoutcast2's multiple mount strategy. The Shoutcast2 source sc_trans does re-encoding, so we don't want that. Then we found that as this forum post indicates, sc_serv2 does not work with 1.x source like ices - so we need to have a Shoutcast1 server between ices and Shoutcast2. And when that happens, we can't use many of Shoutcast2's features. Also, if we wanted it running on multiple ports, we would need unique mount points on each port - all the ports show all the mount points. Then came the killer - Shoutcast2 was dumping listeners at live/recorded changeover even with backupfile set, and autodumpusers=0 , when relaying the Shoutcast1 streams. So that ended our Shoutcast2 experiment.

Icecast2 had similar features, like multiple mount points etc. And it was compatible with ices as a source. Only needs a change in the ices config file, instead of icy protocol, it has to use http protocol. And it had the nifty feature of moving listeners from one mount point to another using the admin interface, which caught P's eye. And fallback mounts, using which what we wanted to do could be accomplished. Basic idea being:
/mount-to-which-listeners-connect
      -has fallback mount-> /livemount
            -has fallback mount-> /recordedstream

When fallback override is set to true, whenever a source connects to /livemount, the listeners are automatically moved from /recordedstream to /livemount. And automatically moved back when the source disconnects. Very nice.

But preliminary exploration gave some disappointments - multiple ports still showed up all the mounts on all the ports, and the fallback mounts did not work for shoutcast relays. Also, there were some funny (buggy?) aspects, like mount points with relays not showing up on the admin interface till a (non-relay) source was connected. Also, how to configure the fallback mechanism for TeluguStream was not clear, since we would not be connecting a source to it directly. The fallback update mechanism given in the icecast admin section did not work, since there was no source directly connected to that mountpoint. Finally, cobbled together the following method.

• lots of alias directives, one for each port, redirecting the root / to the correct stream for that port. This could be done because aliases can be defined on a per port basis.
  Aliases like alias source="/" port="8008" dest="/discoursestream"
• TeluguStream falls back by default to its recorded stream, and AsiaStream falls back first to the live stream mount, and then to recorded stream - so the live changeover for AsiaStream is taken care of.
• An additional mount was defined as a relay of asiastream, and another icecast config file was written where TeluguStream falls back to this relay instead.
• When TeluguStream has to go live, the icecast config file is switched over, and
   killall -HUP icecast makes icecast reload the config file - so, listeners who connect now fall back to the relay of asiastream .
• Using the admin interface, the listeners who were already connected to the recorded telugustream mount are now moved to the relay mount.
• These steps were written into a shell script, with wget being used to do the admin interface calls - wget takes --user= and --password= parameters.
• A shell script which calls the above remote machine's scripts through ssh can be run from the Mac by double-clicking on the file, by just naming the script file with a .command extension.
• For our website's jplayer-based web player to work, had to change the called URL - for shoutcast it was mp3:"http://stream.radiosai.net:8002/;stream/1" and now for icecast it should be mp3:"http://server1.radiosai.org:8002/"
  The stream suffix was for Shoutcast to serve the stream instead of the status page. Our aliases are configured to redirect requests from root to the appropriate stream, so we can just request the root, /
• Just a caveat - icecast by default has very small numbers in threadpool and source connections. Have given 50 and 20 for now.
• Made icecast run like a daemon by calling it with nohup:
  nohup icecast -c /path/to/icecast_config_file.xml > sh.out 2> sh.err < /dev/null &

Then came the monitoring part. P pointed to amptracker.com with nice graphs. But they wanted $5 per month for the graphs. So, went back to shoutstats, which now has support for icecast too. But after install, found it did not work. Troubleshooting:

• First, apache was serving the file index.php like html, without rendering the php parts. Reason was, php short tags were turned off in our server, as found with phpinfo(). Enabled them in /etc/php5/apache2/php.ini
  and restarted apache2, the page displayed now. But shoutstats itself was not showing any graphs, even after I updated manually by accessing update.php manually.
• Tested rrdtool with http://oss.oetiker.ch/rrdtool/tut/rrdtutorial.en.html
  and working fine.
• Enabling print statements for debugging inside the php pages, found that though rrdtool was being called, it was not creating the .rrd file. File permissions were 777 for the relevant directories. Finally, printing out the exact system() call pointed out the error - there was a missing space after $rddtool in the system call.
   ' create '.$rddfile.'DS:audience...
  should have been
   ' create '.$rddfile.' '.'DS:audience
  Mentioned it in the github page, fixed it, shoutstats working after correct cron job entered.
• Had to remove the hidden status of the mounts - otherwise shoutstats would not work out of the box, would have to change the parsing of status2.xsl. So, made all the mounts visible - hidden = 0.

Most probably this finishes the server configuration posts. Possible tweaks needed for icecast would be to make the logs smaller - maybe - and maybe some tweaking needed for thread management under load and so on.

apache2 server redirect

Struggled for a while with mod-rewrite and all that to redirect one page to another page on the apache server. Finally solution turned out to be simple, as referred to in one line in the page above,
"Note that this can also be handled using the RedirectMatch directive". Syntax was, in relevant virtual host file, within the Directory tag,
RedirectMatch path/to/url.html http://server.path/to/redirected/file.htm

configuring voxel server for sgh and h2h

This is an ongoing process, started on 10 Dec 2011.

1. adduser username
2. apt-get install vsftpd
3. Disabled ssh with password using the procedure in the earlier post, copying required keys to .ssh/authorized_keys
4. Started copy process with sftp  from colinux, running under screen from saiwaves.
5. Enabled vsftpd -
   
   service vsftpd start
   To make it useful, had to enable ftp write by
   vim /etc/vsftpd.conf
   and un-commenting the line
   write_enable = YES
6. Also, added username1 user to /etc/ftpusers - so username1 is not allowed to ftp (only username2 is allowed to ftp)
7. Copied over the entire web folder from old server. Tried this first with Windows Explorer's built-in ftp, but was taking too long - several days - and breaking too often. So did it with Filezilla and completed in 10-15 hours.
8. Made postfix work: it was giving the error 
   send-mail: fatal: open /etc/postfix/main.cf: No such file or directory
   Can't send mail: sendmail process failed with error code 75
   Solution was, as given here, dpkg-reconfigure postfix
9. Fixed the ices error Cannot use config file (no XML support). by doing
   apt-get install libxml*and then recompiling ices
10. Worked out why and how the metadata is not sent for our shoutcast streams by ices: the song info is not displayed if the SrcIP=ANY and DestIP=127.0.0.1 and the ices connects on an address other than 127.0.0.1. Basically, song metadata is not sent if the ip on which ices connects does not have a route to one of the ips in DestIP. 
11. For ices not to show the filenames for icecast server, the above trick does not work. Also, the method mentioned here and here, of setting
    mp3-metadata-interval to 0
    also does not work. So, fiddled with the ices source code and recompiled. Just commented out most of the code in metadata_update function. No more metadata!
12. Enabled virtual hosts as given here, first making a suitable file sitename.com in
    /etc/apache2/sites-available
    and then
    a2ensite sitename.com
    service apache2 reload
13. Replaced asp redirects with php redirects for index pages etc.
    header( 'Location: http://www.site.com/new_page.html' ) ;
14. Our windows web server was ignoring case in filenames - uppercase or lowercase did not matter. With the new Linux server, case does matter, and many many links break. To fix, loading mod_speling. It was already available in /etc/apache2/mods-available. So only had to do
    a2enmod speling
    (a2enmod = apache2 enable module script available on debian/ubuntu)
    Then, had to add CheckSpelling on in /etc/apache2/apache2.conf
    (This part is not well documented!)
15. Had a large number of Main.asp files (for each old journal issue cover page). Actually these files were almost completely html. So, took the short-cut of serving these files as html. Edited the relevant file in sites-available to add
    
    
16. Then added DirectoryIndex Main.asp
    This caused major breakage, till I added all the other default index pages!
    DirectoryIndex Main.asp index.htm index.html index.php
17. Apache was giving a 403 forbidden for most pages. Turns out it needs execute permissions also on the directories. This page discusses the same problem and the solution. So, set
    local_umask=022
    file_open_mode=0777
    in vsftpd.conf

The configuration for the radio streams is yet to be finalized, since the icecast configuation has to be tested.

ebook reader on iphone

I'm back to the iPhone - the farewell did not work out! Selling it on ebay did not work so well, as the highest bidders seems to be phoneys who bid higher than the Buy Now price, and then did not pay up. Also, the touchscreen on the windows mobile stopped working, making it lose a lot of functionality, like zooming pdfs and images. Showing IRCTC and KSRTC tickets on the mobile then becomes difficult :)
So, back to iPhone.

Unfortunately, I overwrote Stanza with the newer version, which crashes for old versions of iOS. Installing old versions of Kindle and Stanza with iFunBox did not work - the apps always crashed. Researched and found:

PDF Reader Lite - reflow doesn't work very well, very slow, but works.

ePagine Reader - did not crash, but could not see the sideloaded epub books.

iFlow Reader - need to rename the epubs to the same name as one of the three supplied epub books!

Bluefire reader - works like a charm for epub.

making DVD player region-free

A wanted to make their DVD player region-free. Googled to get
http://www.videohelp.com/dvdhacks

saving youtube video to ipad

B wanted a method to save youtube video to ipad. Googled up this:
http://www.ipadtabletcentral.com/?p=2252
Basically using the TagDisk HD app.

CSV list for phplist mailing list

H wanted a new mailing list, for which he supplied email addresses in an excel sheet. The easiest way to import these into phplist, since he did not want any customization, was as a simple list of emails, one per line - phplist can import such a list with minimum of fuss. Unfortunately, even that took a while - the list had many typos, so phplist was barfing it up. After H did cleanup using some tool, tried the 18,000 email list at once, phplist did not respond. Then cut it up into smaller pieces - upto 6000 emails per file was imported without any problem.

More CDNs

A follow-up to my CDN roundup.

Akamai offered minimum commit of Rs. 1.5 lakhs per month.
$2000 for 2000 GB, around $1 per GB till 10 TB.

Internap offered a free unlimited streaming account till Dec 30th, but pricing was minimum commit $1000 with traffic charged at $2 per GB. Also, using their test account, found:

• the sample flash player provided by internap was crashing in Safari
• the sample flash player also did not allow the hiding of controls - controls were always visible onscreen
• we were unsuccessful in getting jwplayer to work with the internap stream, even after the suggested setting of rtmp.subscribe flashvar to true.

converting stereo wav files to mono

A wanted some advice on converting interview audio files recorded in stereo to mono.

In Sound Forge: To mute only one channel, move the cursor to the top quarter (or bottom quarter) of the screen. The cursor will get a small subscript of L or R (for left or right channel). Select the area you want to mute - only the L (or R) side will get selected. Now if you choose Process -> Mute, only that channel will get muted.

After going through this process of muting out unnecessary parts in L and R channels separately, you can bring the whole wav back to mono using Process -> Channel Convertor. You should use the built-in preset
Stereo to Mono use both channels 50% (no faders)
in the Channel convertor drop-down list for this.

shutting down a Windows server remotely

H & N wanted a method of shutting down their server from the client machine at the counter. Client running Windows XP, server running Win2003 server. From Microsoft documentation, an example batch file (contents shown below), which can be put on the desktop, and double-clicked to shutdown the server. You will need to change the servername in the batchfile for this to work, and of course rename it as something.bat .
type "Are you sure you want to shut down the server? Press any key to continue"

Pause
shutdown -m \\servername -t 10 -c "Shutting down server in 10 seconds"


To start up the server, I mentioned Wake up On Lan to them, but their BIOS had another option for timed startup, so that is what they are using - starts up at a particular time.

counting number of files in folder

While doing server migration, needed a method for counting number of files in folder from the Linux command-line. Found this post, where in the comments this method was listed:
ls -lR | grep ^- | wc -l

capturing DV on old PowerMac G5

Resurrected a retired PowerMac G5 (1.9 Ghz, 3 GB RAM) for playback of Planetarium video. It was not fast enough to play full HD movies at 30 fps - coming to only 15 fps for a 12 Mbps HD video. So, relegated it to DV capturing duties for archive work - Hi8 to DV using a Sony TRV330E Digital8 camera.

iMovie for MacOS X 10.4 Tiger - iMovie HD (obtained with a torrent) seemed to be the last working version - iMovie 8 did not run on this machine.

iMovie was capturing the video as .dv files, and the files were being hidden inside the project "file" which was actually a "Package" - so in Finder you right-click and "Show Package Contents" to see the Media folder inside. To convert .dv to .mov without re-encoding, used MPEG Streamclip.

Edit: Finally this solution was abandoned - the .mov produced in this manner still needed rendering on FCP at the Studio. Then tried capturing in an old version of FCP, that worked without rendering.

stream not working with jplayer

After doing some tests for changing the default player on our website to jplayer, AfriStream stopped working with the jplayer based player, all over the site! But only from my work computer. So, guessed it was some cache problem. Cleared the cache, and it started playing again.