ebook readers on iphone redux

In my previous post, I mentioned that Bluefire reader is working fine. But... while using, found that sometimes it becomes very slow. Slow to load, 2-3 minutes to start; slow to change pages, sometimes taking 2-3 seconds or more. Looking around for other options, found this post, and also this comparison wikipedia page.

Tried out BeyondPrint, ShuBook and uBooks.

BeyondPrint - interface looks very similar to BlueFire reader, has similar limitations. But also supports PDF.

ShuBook - is quick and responsive, but can only load books from the online sources built-in with the free version. The paid version has the option for us to load our own books etc.

uBooks - Can upload books with wifi using its own web interface. Web interface doesn't work with Firefox which complains about some incompatibility, but works fine with IE. Has a slightly different interface - not the paginated interface of other ebook readers, but allows continuous scrolling. Also jumps to "next page" with a tap on rhs of screen.

So, as of now, using ShuBook for books from Gutenberg et al, and uBooks for others.

PHPlist - issues with spaces in words

PB found a possible solution for the problem where the SI messages when sent by email had spaces inside random words. Basically it seems to be due to the MTA adding carriage returns or line feeds or CRLF in the middle of large lines.
https://mantis.phplist.com/view.php?id=15603

For the time being, the SI people are using a workaround, of pasting the message into the PHPlist compose box again after copy-pasting the entire html message. Perhaps that adds CRLF and has the same end result.

firewall rule to restrict browsing during live times

P wanted some firewall rules on Cyberoam so that the live video broadcasts are not affected by people browsing, downloading, etc.

1. Added a schedule under
Firewall -> Schedule -> Define Schedule
called LiveTimes and added the times 7.50 am to 10 am and 4.25 pm to 6.30 pm.

2. Created a Bandwidth policy under
Policies -> Bandwidth Policy -> Create Policy
called LiveTimesThrottle to throttle bw to 256 kbps up and down.
It says Bandwidth in KB - so we have to enter 256/8 = 32 in the box there.

3. Added the LiveTimes schedule for this policy.

4. Added it to Firewall rule in Firewall -> Manage Firewall -> Rule 1 Any Host to Any Host Accept.
Now I am not sure if the Voxel traffic also will be allowed! So, adding a firewall rule,

5. LAN Any Host -> WAN Any Host, RTMP service, Accept + MASQ, with no time/bandwith limit.

Unfortunately, this had the opposite effect to what was desired - during the live times the bandwidth was unlimited, and at other times the bandwidth policy was enforced! This was because the bandwidth policy page says:
Default Values to be applied all the time....
and the schedule is added under
Add Schedule wise details to override default Bandwidth details

So one way out would have been to reverse the schedule timings. But P wanted an alternate solution, just giving priority to the RTMP streams to and from Voxel instead of throttling all traffic on all days at those times. So, he decided on creating a bandwidth policy, Guaranteed 1 Mbps (128 KB) and Burstable 2 Mbps (256 KB) for RTMP service to voxel, with maximum 0 priority. Now we have to check whether enabling just this bandwidth policy is enough by itself, or we need to explicitly de-prioritize other traffic.

making pretty pdfs of webpages and reports

In my previous post, I mentioned creating pdfs server-side for awstat. But creating pdfs from any web page or report?


One way - view only main frame by right-clicking on required frame and choosing View only this frame, and use that url in Adobe Acrobat Pro,
File -> Create PDF -> From Web Page


If you don't have Adobe Acrobat Pro, this solution looks good: using www.web2pdfconvert.com

configuring voxel server for sgh and h2h - part 3

Continuing my series of posts on server setup.

Wanted the home page to be not cached, since they update it often. Looking into it, found two interesting articles, at stackoverflow and metaskills. The first article gave a tip on how to control browser cache for specific urls - using the LocationMatch tag. The second one recommended expiring all html pages immediately. I followed the second one, since our server is not facing any resource crunch right now.

ExpiresActive On 

ExpiresDefault "access plus 1 seconds" 

ExpiresByType text/html "access plus 1 seconds" 

ExpiresByType image/gif "access plus 120 minutes" 

ExpiresByType image/jpeg "access plus 120 minutes" 

ExpiresByType image/png "access plus 120 minutes" 

ExpiresByType text/css "access plus 60 minutes" 

ExpiresByType text/javascript "access plus 60 minutes" ExpiresByType application/x-javascript "access plus 60 minutes" ExpiresByType text/xml "access plus 60 minutes"
Initially this block in apache2.conf was inside an if tag, IfModule mod_expires.c - but was not working. Then I commented out the IfModule tags - then it started working, since anyway I had enabled the module using a2enmod expires

Next - wanted to enable some sort of automated log parsing. Among free log analyzers, settled on awstats, since someone had made a mod for wowza too. Followed the ubuntu documentation for awstats installation. There, the recommended cron is

0 */3 * * * /usr/lib/cgi-bin/awstats.pl -config=yourdomain.ext -update > /dev/null

This notation */3 means once every three hours.

For installing GeoIP - tried enabling it directly, since it is supposed to be one of the default included modules, but got this error,

Error: Plugin load for plugin 'geoipfree' failed with return code: Error: Can't locate Geo/IPfree.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl . /usr/share/awstats/lib /usr/share/awstats/plugins /usr/lib/cgi-bin/plugins) at (eval 9) line 1.

So now I knew where the plugin directory was - so, could follow the method given here for installing maxmind's geolite databases.

wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
gunzip GeoIP.dat.gz
gunzip GeoLiteCity.dat.gz
gunzip GeoIPASNum.dat.gz

mkdir /usr/local/share/GeoIP/
mv *.dat /usr/local/share/GeoIP/

Tried installing Geo::IP Perl module with
http://www.cpan.org/modules/INSTALL.html

cpan App::cpanminus
cpanm Geo::IP

 FAIL Configure failed for Geo-IP-1.40. See /root/.cpanm/build.log for details

So, followed the steps at
http://antezeta.com/awstats/geoip.html
to install the C module.

wget http://geolite.maxmind.com/download/geoip/api/c/GeoIP.tar.gz
gunzip GeoIP.tar.gz
tar xvf GeoIP.tar.gz
 cd GeoIP-1.4.8
./configure
make
make install

Then,
cpanm Geo::IP worked.

Uncommented
LoadPlugin="geoip GEOIP_STANDARD /usr/share/GeoIP/GeoIP.dat"
changed the path to /usr/local/share/GeoIP/GeoIP.dat

Did not add the other two plugins, for city and ISP, since that would slow down AWstats.


Had to flush newer records, by deleting files in 
DataDir - /var/lib/awstats
in order to process older log files, as given at
http://awstats.sourceforge.net/docs/awstats_faq.html#OLDLOG
Took around 15 minutes to crunch old logs for 15 days, so probably the log for 3 hours will take only half a minute or less.


Next for Wowza. Did not have to modify the logging parameters as given at
http://www.wmconsulting.info/awstats-mod-for-process-wowza-stats/
since most of the parameters were already set. But had to change the LogFormat drastically, since my Wowza version is different - 3.03. Also, one point which he makes is to log only the destroy stream event. Not doing that, logging everything as we are doing, was causing over-counting of bytes sent. So, changed the LogFile parameter to read
LogFile=”/usr/local/awstats-wmc/tools/logresolvemerge.pl /usr/local/WowzaMediaServer/logs/*access* | grep destroy |” 


And my LogFormat parameter is
LogFormat="date time %other %other %other %other %code %other %other %other %virtualname %other %other %other %other %url c-ip %method %referer %ua %other %other sc-bytes %other %other %other %other %other %other %other %other %other %other %other %other %other %other %other" 


Using this method, bandwidth is slightly under-reported - by some 10%. But cheaper than Sawmill.


To make pdf reports, on the server, the buildstaticpages tool needed htmldoc, which I installed with apt-get install htmldoc

UPS test

Tested the APC Smart-UPS 5000VA with our projector and sound system load. Lasted for 45 minutes with 2 battery LEDs remaining, out of 5. Charging without load, with output switched off, came back to 5 LEDs within 8 hours.

configuring voxel server for sgh and h2h - part 2

Continuing my earlier post.

This part of the configuration, for the audio streams, was a series of experiments with Shoutcast v1 and v2 and Icecast2 - from Dec 15 2011 till now. Duplicating our Shoutcast server setup using v1 server as on our earlier server could have been done, but P was keen on improving things, especially the recorded to live (and back) transitions.

Shoutcast2 was our first trial, since using it would help us to restore our multiple streams on shoutcast.com directory. Tried Shoutcast2's multiple mount strategy. The Shoutcast2 source sc_trans does re-encoding, so we don't want that. Then we found that as this forum post indicates, sc_serv2 does not work with 1.x source like ices - so we need to have a Shoutcast1 server between ices and Shoutcast2. And when that happens, we can't use many of Shoutcast2's features. Also, if we wanted it running on multiple ports, we would need unique mount points on each port - all the ports show all the mount points. Then came the killer - Shoutcast2 was dumping listeners at live/recorded changeover even with backupfile set, and autodumpusers=0 , when relaying the Shoutcast1 streams. So that ended our Shoutcast2 experiment.

Icecast2 had similar features, like multiple mount points etc. And it was compatible with ices as a source. Only needs a change in the ices config file, instead of icy protocol, it has to use http protocol. And it had the nifty feature of moving listeners from one mount point to another using the admin interface, which caught P's eye. And fallback mounts, using which what we wanted to do could be accomplished. Basic idea being:
/mount-to-which-listeners-connect
      -has fallback mount-> /livemount
            -has fallback mount-> /recordedstream

When fallback override is set to true, whenever a source connects to /livemount, the listeners are automatically moved from /recordedstream to /livemount. And automatically moved back when the source disconnects. Very nice.

But preliminary exploration gave some disappointments - multiple ports still showed up all the mounts on all the ports, and the fallback mounts did not work for shoutcast relays. Also, there were some funny (buggy?) aspects, like mount points with relays not showing up on the admin interface till a (non-relay) source was connected. Also, how to configure the fallback mechanism for TeluguStream was not clear, since we would not be connecting a source to it directly. The fallback update mechanism given in the icecast admin section did not work, since there was no source directly connected to that mountpoint. Finally, cobbled together the following method.

• lots of alias directives, one for each port, redirecting the root / to the correct stream for that port. This could be done because aliases can be defined on a per port basis.
  Aliases like alias source="/" port="8008" dest="/discoursestream"
• TeluguStream falls back by default to its recorded stream, and AsiaStream falls back first to the live stream mount, and then to recorded stream - so the live changeover for AsiaStream is taken care of.
• An additional mount was defined as a relay of asiastream, and another icecast config file was written where TeluguStream falls back to this relay instead.
• When TeluguStream has to go live, the icecast config file is switched over, and
   killall -HUP icecast makes icecast reload the config file - so, listeners who connect now fall back to the relay of asiastream .
• Using the admin interface, the listeners who were already connected to the recorded telugustream mount are now moved to the relay mount.
• These steps were written into a shell script, with wget being used to do the admin interface calls - wget takes --user= and --password= parameters.
• A shell script which calls the above remote machine's scripts through ssh can be run from the Mac by double-clicking on the file, by just naming the script file with a .command extension.
• For our website's jplayer-based web player to work, had to change the called URL - for shoutcast it was mp3:"http://stream.radiosai.net:8002/;stream/1" and now for icecast it should be mp3:"http://server1.radiosai.org:8002/"
  The stream suffix was for Shoutcast to serve the stream instead of the status page. Our aliases are configured to redirect requests from root to the appropriate stream, so we can just request the root, /
• Just a caveat - icecast by default has very small numbers in threadpool and source connections. Have given 50 and 20 for now.
• Made icecast run like a daemon by calling it with nohup:
  nohup icecast -c /path/to/icecast_config_file.xml > sh.out 2> sh.err < /dev/null &

Then came the monitoring part. P pointed to amptracker.com with nice graphs. But they wanted $5 per month for the graphs. So, went back to shoutstats, which now has support for icecast too. But after install, found it did not work. Troubleshooting:

• First, apache was serving the file index.php like html, without rendering the php parts. Reason was, php short tags were turned off in our server, as found with phpinfo(). Enabled them in /etc/php5/apache2/php.ini
  and restarted apache2, the page displayed now. But shoutstats itself was not showing any graphs, even after I updated manually by accessing update.php manually.
• Tested rrdtool with http://oss.oetiker.ch/rrdtool/tut/rrdtutorial.en.html
  and working fine.
• Enabling print statements for debugging inside the php pages, found that though rrdtool was being called, it was not creating the .rrd file. File permissions were 777 for the relevant directories. Finally, printing out the exact system() call pointed out the error - there was a missing space after $rddtool in the system call.
   ' create '.$rddfile.'DS:audience...
  should have been
   ' create '.$rddfile.' '.'DS:audience
  Mentioned it in the github page, fixed it, shoutstats working after correct cron job entered.
• Had to remove the hidden status of the mounts - otherwise shoutstats would not work out of the box, would have to change the parsing of status2.xsl. So, made all the mounts visible - hidden = 0.

Most probably this finishes the server configuration posts. Possible tweaks needed for icecast would be to make the logs smaller - maybe - and maybe some tweaking needed for thread management under load and so on.

apache2 server redirect

Struggled for a while with mod-rewrite and all that to redirect one page to another page on the apache server. Finally solution turned out to be simple, as referred to in one line in the page above,
"Note that this can also be handled using the RedirectMatch directive". Syntax was, in relevant virtual host file, within the Directory tag,
RedirectMatch path/to/url.html http://server.path/to/redirected/file.htm

configuring voxel server for sgh and h2h

This is an ongoing process, started on 10 Dec 2011.

1. adduser username
2. apt-get install vsftpd
3. Disabled ssh with password using the procedure in the earlier post, copying required keys to .ssh/authorized_keys
4. Started copy process with sftp  from colinux, running under screen from saiwaves.
5. Enabled vsftpd -
   
   service vsftpd start
   To make it useful, had to enable ftp write by
   vim /etc/vsftpd.conf
   and un-commenting the line
   write_enable = YES
6. Also, added username1 user to /etc/ftpusers - so username1 is not allowed to ftp (only username2 is allowed to ftp)
7. Copied over the entire web folder from old server. Tried this first with Windows Explorer's built-in ftp, but was taking too long - several days - and breaking too often. So did it with Filezilla and completed in 10-15 hours.
8. Made postfix work: it was giving the error 
   send-mail: fatal: open /etc/postfix/main.cf: No such file or directory
   Can't send mail: sendmail process failed with error code 75
   Solution was, as given here, dpkg-reconfigure postfix
9. Fixed the ices error Cannot use config file (no XML support). by doing
   apt-get install libxml*and then recompiling ices
10. Worked out why and how the metadata is not sent for our shoutcast streams by ices: the song info is not displayed if the SrcIP=ANY and DestIP=127.0.0.1 and the ices connects on an address other than 127.0.0.1. Basically, song metadata is not sent if the ip on which ices connects does not have a route to one of the ips in DestIP. 
11. For ices not to show the filenames for icecast server, the above trick does not work. Also, the method mentioned here and here, of setting
    mp3-metadata-interval to 0
    also does not work. So, fiddled with the ices source code and recompiled. Just commented out most of the code in metadata_update function. No more metadata!
12. Enabled virtual hosts as given here, first making a suitable file sitename.com in
    /etc/apache2/sites-available
    and then
    a2ensite sitename.com
    service apache2 reload
13. Replaced asp redirects with php redirects for index pages etc.
    header( 'Location: http://www.site.com/new_page.html' ) ;
14. Our windows web server was ignoring case in filenames - uppercase or lowercase did not matter. With the new Linux server, case does matter, and many many links break. To fix, loading mod_speling. It was already available in /etc/apache2/mods-available. So only had to do
    a2enmod speling
    (a2enmod = apache2 enable module script available on debian/ubuntu)
    Then, had to add CheckSpelling on in /etc/apache2/apache2.conf
    (This part is not well documented!)
15. Had a large number of Main.asp files (for each old journal issue cover page). Actually these files were almost completely html. So, took the short-cut of serving these files as html. Edited the relevant file in sites-available to add
    
    
16. Then added DirectoryIndex Main.asp
    This caused major breakage, till I added all the other default index pages!
    DirectoryIndex Main.asp index.htm index.html index.php
17. Apache was giving a 403 forbidden for most pages. Turns out it needs execute permissions also on the directories. This page discusses the same problem and the solution. So, set
    local_umask=022
    file_open_mode=0777
    in vsftpd.conf

The configuration for the radio streams is yet to be finalized, since the icecast configuation has to be tested.

ebook reader on iphone

I'm back to the iPhone - the farewell did not work out! Selling it on ebay did not work so well, as the highest bidders seems to be phoneys who bid higher than the Buy Now price, and then did not pay up. Also, the touchscreen on the windows mobile stopped working, making it lose a lot of functionality, like zooming pdfs and images. Showing IRCTC and KSRTC tickets on the mobile then becomes difficult :)
So, back to iPhone.

Unfortunately, I overwrote Stanza with the newer version, which crashes for old versions of iOS. Installing old versions of Kindle and Stanza with iFunBox did not work - the apps always crashed. Researched and found:

PDF Reader Lite - reflow doesn't work very well, very slow, but works.

ePagine Reader - did not crash, but could not see the sideloaded epub books.

iFlow Reader - need to rename the epubs to the same name as one of the three supplied epub books!

Bluefire reader - works like a charm for epub.

making DVD player region-free

A wanted to make their DVD player region-free. Googled to get
http://www.videohelp.com/dvdhacks

saving youtube video to ipad

B wanted a method to save youtube video to ipad. Googled up this:
http://www.ipadtabletcentral.com/?p=2252
Basically using the TagDisk HD app.

CSV list for phplist mailing list

H wanted a new mailing list, for which he supplied email addresses in an excel sheet. The easiest way to import these into phplist, since he did not want any customization, was as a simple list of emails, one per line - phplist can import such a list with minimum of fuss. Unfortunately, even that took a while - the list had many typos, so phplist was barfing it up. After H did cleanup using some tool, tried the 18,000 email list at once, phplist did not respond. Then cut it up into smaller pieces - upto 6000 emails per file was imported without any problem.

More CDNs

A follow-up to my CDN roundup.

Akamai offered minimum commit of Rs. 1.5 lakhs per month.
$2000 for 2000 GB, around $1 per GB till 10 TB.

Internap offered a free unlimited streaming account till Dec 30th, but pricing was minimum commit $1000 with traffic charged at $2 per GB. Also, using their test account, found:

• the sample flash player provided by internap was crashing in Safari
• the sample flash player also did not allow the hiding of controls - controls were always visible onscreen
• we were unsuccessful in getting jwplayer to work with the internap stream, even after the suggested setting of rtmp.subscribe flashvar to true.

converting stereo wav files to mono

A wanted some advice on converting interview audio files recorded in stereo to mono.

In Sound Forge: To mute only one channel, move the cursor to the top quarter (or bottom quarter) of the screen. The cursor will get a small subscript of L or R (for left or right channel). Select the area you want to mute - only the L (or R) side will get selected. Now if you choose Process -> Mute, only that channel will get muted.

After going through this process of muting out unnecessary parts in L and R channels separately, you can bring the whole wav back to mono using Process -> Channel Convertor. You should use the built-in preset
Stereo to Mono use both channels 50% (no faders)
in the Channel convertor drop-down list for this.

shutting down a Windows server remotely

H & N wanted a method of shutting down their server from the client machine at the counter. Client running Windows XP, server running Win2003 server. From Microsoft documentation, an example batch file (contents shown below), which can be put on the desktop, and double-clicked to shutdown the server. You will need to change the servername in the batchfile for this to work, and of course rename it as something.bat .
type "Are you sure you want to shut down the server? Press any key to continue"

Pause
shutdown -m \\servername -t 10 -c "Shutting down server in 10 seconds"


To start up the server, I mentioned Wake up On Lan to them, but their BIOS had another option for timed startup, so that is what they are using - starts up at a particular time.

counting number of files in folder

While doing server migration, needed a method for counting number of files in folder from the Linux command-line. Found this post, where in the comments this method was listed:
ls -lR | grep ^- | wc -l

capturing DV on old PowerMac G5

Resurrected a retired PowerMac G5 (1.9 Ghz, 3 GB RAM) for playback of Planetarium video. It was not fast enough to play full HD movies at 30 fps - coming to only 15 fps for a 12 Mbps HD video. So, relegated it to DV capturing duties for archive work - Hi8 to DV using a Sony TRV330E Digital8 camera.

iMovie for MacOS X 10.4 Tiger - iMovie HD (obtained with a torrent) seemed to be the last working version - iMovie 8 did not run on this machine.

iMovie was capturing the video as .dv files, and the files were being hidden inside the project "file" which was actually a "Package" - so in Finder you right-click and "Show Package Contents" to see the Media folder inside. To convert .dv to .mov without re-encoding, used MPEG Streamclip.

Edit: Finally this solution was abandoned - the .mov produced in this manner still needed rendering on FCP at the Studio. Then tried capturing in an old version of FCP, that worked without rendering.

stream not working with jplayer

After doing some tests for changing the default player on our website to jplayer, AfriStream stopped working with the jplayer based player, all over the site! But only from my work computer. So, guessed it was some cache problem. Cleared the cache, and it started playing again.

changing the default player on radiosai.org

On the home page of radiosai.org the links to the streams use a flash player from WaveStreaming.com. Trying to substitute it seamlessly using jplayer, found that modifying the size of the player skin requires more CSS skills than I have at the moment. Will think of something later.

Edit: Most problems have a brute-force solution! Resized the images by a factor of half, halved all the pixel values in the css, et voila - a half-sized player! That was too small, so multiplied again by 1.4 to get the size we want.

setting up live video streaming - resource usage

This will be a series of posts about setting up video streaming on our new dedicated server from Voxel.

When the Wowza streaming server was serving at around 600 Mbps, that is around a 1000 clients, top reported the following usage of memory and cpu:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           

18825 root    20   0 2009m 1.3g 7892 S  113 17.1   1064:58 java  

Since this is a quad-core machine, probably top reports 100% as maxing out a single core. So, we have plenty of cpu to spare, and could easily max-out the 1 Gbps port. (And that's what we did .... running video at 800 kbps per instance.)

disk full on saiwaves

While uploading files to dl.radiosai.org saiwaves reported disk full - unable to write to disk etc. As given at this post, removed some 3 GB from Trash by doing

rm -Rf ~/.local/share/Trash/*

Later, P removed around 70 GB (!) of old database backups.

mysql database error

Our schedule page and audio search page stopped working - sometimes the pages would respond with "Could not connect to database" and sometimes they would show 0 records returned. Trying to connect with the phpMyAdmin using Plesk gave the error message

MySQL said: Documentation
#1 - Can\'t create/write to file \'#sql_abc_0.MYD\' (Errcode: 17)

Reported it to nettigritty support, they restarted the service, everything became fine.

setting up live video streaming - aspect ratio and timezone

This will be a series of posts about setting up video streaming on our new dedicated server from Voxel.

 P wanted the timezone on the server to be set to IST, for ease of recognizing recorded files etc. Did it with dpkg-reconfigure tzdata
which is the Ubuntu way of doing it at present.

Then: the flv video which we saved on the server seemed to have an NTSC flag set somewhere. Probably originating from the Adobe FMLE (Flash Media Live Encoder). So, the video was being seen with a skewed aspect ratio even though it was square pixels 400x224 25 fps. Finally solved it at the client side using stretching:"exactfit" as a parameter for the jwplayer.

setting up live video streaming - recording on the server

This will be a series of posts about setting up video streaming on our new dedicated server from Voxel.


There are different ways of recording.
1. using live-record as stream type in Wowza. This does not give us control over when recording starts - recording starts as soon as the source gets connected.
2. using the LiveStreamRecord addon module, we can choose when to start recording and when to stop.

So, we used the latter, set up a password protected page on our website for the record enable form which is included with the module. We set it up not to overwrite (so it would create a new file like myStream.sdp_2.flv etc increasing the number to _3, _4 etc.) Also set it to start on keyframe and record data. (Metadata, I suppose.)

First we tried setting up http pseudo-streaming with mp4 on Apache. After setting up the module for streaming video the seeking was not working properly for parts of the video which had not yet been downloaded - jwplayer was playing the beginning of the video after the seek.

Then the audio and video were coming out of sync on the recorded mp4 file. It looks like many people have faced this issue.

So we tried recording as flv files instead. With that, the audio-video sync problems were solved. But since the module for flv seek over http uses Lighttpd - setting it up was a bit of a pain - we fell back to providing the video over rtmp using Wowza's vod application. That, of course, worked perfectly.

Argument list too long

Doing ls or mv invariably leads to Argument list too long errors. A nice discussion about ways to get around this:
http://www.linuxjournal.com/article/6060

setting up live video streaming - download links

This will be a series of posts about setting up video streaming on our new dedicated server from Voxel.

Setting up a download link - I tried to do this by adding the .htaccess file like in dl.radiosai.org with the lines


  ForceType application/octet-stream
  Header set Content-Disposition attachment

But this did not work, even when I modified the config file again, as in
https://help.ubuntu.com/community/EnablingUseOfApacheHtaccessFiles
adding the line

AllowOverride All

So, did it by editing the config file:
http://www.jtricks.com/howto/content_disposition/apache2.html

       

                Options FollowSymLinks
                AllowOverride All
               
                    ForceType application/octet-stream
                    Header set Content-Disposition attachment
               
       

 

Then, files can be either copied or linked into the download directory, and accessing them automatically gets the "Save as" dialog box for the user.

setting up live video streaming - iPad and iPhone

This will be a series of posts about setting up video streaming on our new dedicated server from Voxel.

After setting up Wowza server as per my earlier post, tried to get the live stream to work on iOS devices - iPad, iPhone, iPod touch etc.

To make the stream work with iphone, the encoder settings have to be tweaked. Format has to be H.264 and AAC audio. Otherwise the device gives the message, "This video cannot be played."

AAC settings should be low complexity - I found 22050 Hz, mono, 32 Kbps to work OK.
H.264 settings have to be Baseline 3.0
(set by clicking on the spanner icon next to Format: H.264 in Adobe Flash Media Live Encoder)

When this is done, the following url works for iOS devices, when entered into Safari on iPhone, for eg.:
http://server1.radiosai.org:1935/live/myStream.sdp/playlist.m3u8
(entering this URL on a normal computer does not work).

In order to make this work with the jwplayer, which we can embed in a page which will work for desktop browsers as well as iOS devices, the player had to be updated to the latest version - 5.8. Then the sample code for embedding worked fine.

local recording along with live streaming

S was looking for solutions for local recording along with live streaming, for upload to vimeo later. He wanted to explore Quicktime broadcaster for this.

There are several possible solutions, which we can try out.

1. Use a DV hub, record DV on another computer, encode offline after the event, then upload. This will result in best quality.

2. Record on DV tape on the DV VCR which you are using as source, capture and encode after the event. The tape has to start before the stream starts, and the tape has to last for the entire event, because stopping the tape causes the stream to restart.

3. Recording on the same machine which is doing the broadcasting - seems to require some funky workaround like
http://forums.adobe.com/thread/657963
but may be possible. To open multiple instances of FMLE you have to use this technique:
http://osxdaily.com/2011/05/11/multiple-instances-application-mac/

4. Using Quicktime broadcaster instead of FMLE is also probably possible, wowza server supports QT broadcaster also as a source. But broadcaster also seems to have the same limitations - setting up the entire chain for it separately seems to be a waste of effort.


(Finally the solution chosen was 1.)

setting up live video streaming - C++ RTMP server

This will be a series of posts about setting up video streaming on our new dedicated server from Voxel.

First attempt was to replicate my earlier test, and set up the C++ RTMP server. That involved re-compiling from source, since the binary did not seem to run on the Ubuntu machine.

1. Checking cpu type:
cat /proc/cpuinfo shows it is
Intel(R) Xeon(R) CPU X3330 @ 2.66GHz
2. so trying crtmpserver-690-Ubuntu-10.10-i386.tar.gz
3. Procedure as in my earlier test
4. Gave error "no such file or directory" when tried to run the executable.
ia32-libs recommeded by this launchpad post.
5. apt-get install ia32-libs and then tried. Then the executable runs, but seems to have some error, since the console doesn't come up. Tried with the 64 bit binary also, still no go. So, tried the sources from http://www.rtmpd.com/downloads/

6. For that, while following the instructions to install at the rtmpd site.... had to install lots of stuff.

set CMAKE_CXX_COMPILER to a valid compiler path or name -> apt-get install gcc

and

apt-get install g++

OPENSSL_INCLUDE_PATH-NOTFOUND -> as per this forum post,
apt-cache search libssl
grep SSL

Then I installed apt-get install libssl-dev

Finally worked. Binary is in /root/crtmpserver-690/builders/cmake/crtmpserver folder.

But....
C++ RTMP server does not have the nifty features of Wowza, does not automatically make the http  pseudo-streaming chunks for streaming to iPad etc. So, decided to go for Wowza server after all.

google analytics code

Upgrading to the new template on blogger had removed the GA code. Added it back today, going to Edit HTML in the template editing section, and adding the code just above the /body end tag.

Actually a related topic, checking if the analytics code is working:
http://www.rachaelarnold.com/dev/archive/how-to-check-if-google-analytics-click-tracking-is-triggered
this method uses  Firebug, with the Net panel enabled. We should be able to see requests starting with _utm.gif? - and the params tab gives the info being sent to Google.

streaming video and quality

Just a small study I did about streaming video and quality.

Ted talk quality, for eg.
www.ted.com/talks/ken_robinson_says_schools_kill_creativity.html

- wireshark clocked it at 0.415 Mbps - let's say 400 kbps.

(Ted uses variable bit rate streaming - bitrate changes with scene as well as with bandwidth availability)

Youtube video at 360p -
http://www.youtube.com/watch?v=I1Tbn1GtIGE&feature=grec_index

21.6 MB for 8 minutes 45 sec
21.6 MB/525 seconds
= 0.0411 MB/s
= 0.329 Mbps, close to 300 kbps.

But these are all offline encoded videos. For live encoder, quality will be somewhat lower for the same bitrates.

TFD internal error

The Thought for the Day page was giving an Internal Server error if the particular day's entry had not been made. And interestingly enough, putting an if statement to check if the record exists was preventing the TFD from being displayed at all, even if it exists. Nz solved the mystery: the text only version, which showed the TFD even though it had the if statement, had some extra type information when the recordset was being declared. Did the same for the former page, and got it to work. The
rsObject.Open strQuery,objConn,adOpenKeyset,adLockOptimistic
The last two, adOpenKeyset,adLockOptimistic were the ones added to the line.

gimp batch process gui

This makeuseof page links to David's Batch Processor.

Allows operations on the entire image, like blur etc. Quite fast, even for large images. Is perhaps faster than doing it in Irfanview - though I did not try a comparison. But not really a replacement for Actions on Photoshop. For that, scripting is required, I suppose. Perl, anyone?

iPhone 3G jailbreak and unlock

PB got an iPhone 3G for unlocking.

Version : 4.2.1 (8C148)
Modem Firmware:  05.15.04
Carrier : AT&T 9.0
Model : MB702LL
 
Jailbreakme.com does not work with 3G - only with 3GS and above, I think. I gave him a link to pwnagetool, but he did it with redsn0w instead. He noted the following:

1. Could not download - redsn0w 0.9.6b6 managed with redsn0w_win_0.9.6rc16
2. http://www.idownloadblog.com/iphone-downloads/ gave the link to this version of the restore, which was
from apple's site.
3. redsn0w_win_0.9.6rc16 kept crashing. This forum post  helped with the following suggestion, which made it work.

After couple of minutes Redsnow just shuts down (disappears)
what cause this problem ?
first install only cydia. don't tick the update baseband

after jailbreaking then you can fetch baseband tick only update baseband.

4. Could not add source http://repo.ultrasn0w.com to cydia
followed the alternate repository given at  http://nabtron.com/ultrasn0w-repo-sources-for-cydia/3567/
 

finding the first played date and displaying it

A small gotcha in implementing the "First broadcast on ...." feature on the radiosai schedule page - our local database has the first played values per bcid. So, the first of all those values has to be selected. After some trial and error with group by - which would fail -

finally used this technique,

SELECT firstplayed FROM play_history_tablename WHERE filename = NameofFile.mp3 order by firstplayed  limit 1

This query conveniently gets the single result which we want, which is the first played date on any stream. Then incorporated this into our csv export script, added an extra field on the web-server's database, and added code to the schedule and search pages over the course of a week. On the search page, sorting by first played adds more functionality. So, added the code for that, and made the table clickable to enable sort by Description field also, using the sorrtable javascript library from kryogenix.

Made search results sortable

Added first played info to our audio search results page.
Changed default to show newest files instead of oldest results.

Made search results sortable using javascript, by using the sorttable javascript library from
http://www.kryogenix.org/code/browser/sorttable

email not reaching Google apps mailbox

Our German program producers reported bouncing email from office@radiosai.org:

A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed: office@radiosai.org 

SMTP error from remote mail server after RCPT TO:: host aspmx.l.google.com [74.125.39.27]: 550-4.2.1 The user you are trying to contact is receiving mail at a rate that 450-4.2.1 prevents additional messages from being delivered. Please resend your 450-4.2.1 message at a later time. 

If the user is able to receive mail at that 450-4.2.1 time, your message will be delivered. For more information, please 450 4.2.1 visit http://mail.google.com/support/bin/answer.py?answer=6592 q11si2325486fah.22

Replied as follows:

Checking the GA help center as suggested in that page, this page seems to indicate that signing out of the account instead of remaining signed in all the time solves the issue. We'll try doing that.

This could also be an issue with freenet.de - because the other complaints listed on Google's forums are also from freenet.de customers. So if you get a bounce message like that, you could try sending the email from another email id instead of the freenet.de id.

exporting mobile contacts and upgrading .NET

I wanted to export the contacts from my Windows Mobile 5 phone. At first I looked at TBexport as given here. But TBexport needed a newer version of .NET Compact Framework. Checked out the procedure given here. That is, basically install it, restart phone, and then go to HKLM\Software\Microsoft\.NETCompactFramework and change the value of the 3.5 version from 0 to 1 and changing the old one's value from 1 to 0. And then a restart of the phone. That worked, but was not needed - PPCPIMbackup itself could do the job. The pib file is in zip format, so 7-zip etc can unzip it and extract the csc file from it, which is actually a csv file.

Blender QA with Ron Proctor

Ron Proctor is a respected planetarian, and Blender enthusiast, releasing Blender-produced free planetarium shows like Sizing up Space, Expanded View, etc. I asked him some questions over email, and he graciously replied promptly.

Hello Ron,

I'm just starting out with Blender for show production, and tried out your ""Starball"" and fisheye rig at http://webersci.org/blendheads/?p=13

Just wondering - in Blender 2.4, there is an option in Camera settings for Panorama. Is this there somewhere in 2.5 also? Can this be used instead of your fisheye rig?

Thanks for your time.

The panorama setting was moved to the camera properties.

Unfortunately, it does not produce correct fisheye images. A native fisheye camera would be great.

Presently, the only ways to get fisheye out of Blender are:

- Fisheye Camera Rig (Reflective Hemisphere)

- 5 (or 6) camera frustum + external stitcher (such as Paul Bourke's cube2dome).

- Third party renderer (I have no experience with these).

You are welcome to use our fisheye camera rig under the terms of the provided license.

Good luck in your work.

Thanks for the quick reply, Ron.

Are the 3D models of objects like Chandra / Hubble / Voyager spacecraft etc available somewhere? I suppose we could roll our own Planets using spheres and textures.....

You might start at the NASA 3D Repository. Celestia Motherlode also has some resources, but make sure the source data are ok for use.

Can I ask a few more questions please?

I've tried to put background images to the rendered scenes using the Blender World -> Texture , choosing Global Co-ordinates for the World Texture.
(details and pictures are at https://hnsws.blogspot.com/2011/10/blurry-background-for-video-on-fulldome.html )
Do you have any other preferred way to add background images to a scene rendered with your fisheye camera rig?

Also - the Blender Game Engine seems to have a Dome mode, with a Fisheye dome camera,
http://wiki.blender.org/index.php/Dev:Source/GameEngine/2.49/Fisheye_Dome_Camera

Since I'm using exactly the same technique from Paul Bourke
( http://hnsws.blogspot.com/search?q=mirrordome )
was wondering if there is some way in which I can get these warped dome images directly rendered to files from the Game Engine and not just interactively displayed using the game engine?

Thanks in advance.

No problem.

Q: I've tried to put background images to the rendered scenes using the Blender World -> Texture , choosing Global Co-ordinates for the World Texture.

A. This should work, but I don't use it.

Q: Do you have any other preferred way to add background images to a scene rendered with your fisheye camera rig?

A. I prefer to use a big UV sphere that surrounds the scene. It gives more intuitive control, especially when animating things like daily motion.

Q: Also - the Blender Game Engine seems to have a Dome mode, with a Fisheye dome camera,

http://wiki.blender.org/index.php/Dev:Source/GameEngine/2.49/Fisheye_Dome_Camera

A. Yes. It's pretty cool. Unfortunately, I don't think it's supported in Blender v2.5x yet.

Q: Since I'm using exactly the same technique from Paul Bourke

( http://hnsws.blogspot.com/search?q=mirrordome )

was wondering if there is some way in which I can get these warped dome images directly rendered to files from the Game Engineand not just interactively displayed using the game engine?

A. This has been done, but I don't know if it will work with the fisheye camera. See item #6 on this page: http://www.blendernation.com/2006/12/03/using-blenders-game-engine-for-more-than-just-games/

I hope that helps. Have a good day and happy Blending!

Thanks Ron. I'll take some time to understand the Blender Game Engine file which you pointed out, but basically  I gather that it is possible to take screenshots from the Game Engine with Python.

I prefer to use a big UV sphere

Your "starball"! I should have thought of that. Somehow, I thought I should use the starball only when there are no other objects, thinking that other objects would collide with it... Now I've tried it, found that it is large enough to not interfere with other objects, and the result is great.

I'm also looking to map a hi-def 2d movie, something like the Known Universe movie, onto the dome. To map it to a large area of the dome, I've tried texturing it on a plane and keeping it very close to the fisheye camera, slightly angled since we have unidirectional seating, like below. But is quite distorted. Maybe acceptable for some parts of the video. Any comments, suggestions? Did you use something like this for your Expanded View show, or did you have some other technique?

It depends on the situation, but for regular "flat screen" video, I usually go with something like this (45º above the front horizon, directly facing the camera):

In Expanded View, AmyJo subdivided image planes in edit mode and used proportional editing to achieve curved shapes.

adding a background image for fulldome images in Blender

This turns out to be not so simple. My first try was using a texture for the World in Blender. Using either Global or AngMap Co-ordinates, and making sure the Influence -> Horizon is checked. Result is not too bad, except that the background image is a bit zoomed in.

This was the background image, and after using it, the result was like this:

Well, not too bad. Perhaps if I fiddled around with the background image, making it bigger, it might improve things? Anyway, the method of using the Alpha channel as given in the tutorial here doesn't work too well because of the fisheye camera rig. When rendered with the fisheye camera, the alpha channel includes the whole circle - since that is what is being reflected on the fisheye rig's mirror. So, the scene when rendered will not include the background image inside the circle at all - only in the corners, which is not useful at all.

One of the comments in the above tutorial mentioned simply using a paper sky in World settings, and then enable Horizon in Influence settings. That works well with normal cameras, but had a problem with the reflection from the fisheye rig - with a ring around the horizon of the fulldome - it looked like this:

Anyway, going through the Blendercookie/CGcookie tutorial was a good learning experience, especially since it took me to his previous tutorial, and I learnt the reason for the scale - dimension problem I noted earlier. Apparently if you change the dimensions of an object in Object mode, the scale goes out of whack. To correct it, he does Ctrl A, and chooses Scale. Then - et voila - the scale goes back to 1.0 and the dimensions are preserved.

blurry background for video on fulldome using Blender

As mentioned in my last post, tried to get a blurry background effect for the video. Putting a video texture in the Blender World seemed to be the easiest option. As given in this tutorial, the way to do it is to click on World, then on Texture, and then set it to the file which you need. The crucial step is to check all the boxes under Influence:

And to get a blurred effect, the simplest way seemed to be to use the same video, but sampled down to 32x24 pixels, as the World texture.

HD video file on fulldome with fisheye camera in Blender

Tried out the HD flat version of "The Known Universe"  rendering it to the dome using Ron Proctor's fisheye camera. 

(

Edit 2022-09-01 - The webersci link above is broken, not available in archive.org either. A google search turned up https://groups.google.com/g/blendertarium/c/_4n-AwZ6Bs4?pli=1 

Uploading the camera blend file to github, in case that link also goes away - 

https://github.com/hn-88/Blender-files/blob/main/fisheyeCamera.blend

And also, now there seems to be Fisheye camera support directly in Blender - 
https://blender.stackexchange.com/questions/174750/how-to-make-fisheye-lens-for-camera

)

Brought up the flat video real close to the camera, and tilted it so that the video can cover more of the dome, as below.

The co-ordinates of the plane were
Rotation - X = 125, Y = 0, Z = 0
Location - X = 0, Y = 0, Z = 1.3

This gave an effect similar to the video part in the TSPOG show - probably I need to add some blurring on the blank black area outside the video window like in that show. Rendering took around 1 minute per frame with two threads, and 2 minutes per frame with a single thread. Sample fulldome frame was like below.

tried out firessh

FireSSH and FireFTP - Firefox plugins for SSH and FTP/SFTP right inside firefox. May be useful sometime, when installation is too much of a bother. FireSSH has support for key-based auth as well.

ebook using instapaper

Instapaper provides a read-later bookmarklet, and using that, a series of web-pages can be made into an ebook fairly easily. Instapaper gives download options in epub, mobi and "printable" (from which one can print to pdf). The thing to note is that the page you want as the first page of your ebook should be the last you click "Read later" on. That is to say, last in first out. So, you have to click "Read later" on the last page of your ebook first, and go on to the beginning, and then export from instapaper.

rudimentary planet in Blender, also video texture

Today I made my first Planet. Instead of creating three spheres etc as given here, (which seems to be the same tutorial as given in Issue 9 of BlenderArt magazine) I made a single sphere, just to see how it comes out. With the default 32 segment 32 ring UV sphere, the aliasing was obvious. I bumped it up to 128 x 128 and then used the Earth texture from here, and the result was not bad for what I want to do, which is a flyby.

Things to remember:

• UV mapping means wrapping the texture around the object - U and V are used to describe the 2D mesh since X and Y are already taken!
• For the planet, the mapping used was Co-ordinates = Object, Object = Sphere, Projection = Sphere. (Not Co-ordinates = UV!)
• Smooth shading: as given in Ron Proctor's tip here, using smooth shading makes the sphere look much better. Accessed from Object tools - T after selecting the object.

Also animated a plane with a video texture. For this,

• The plane could be of the same dimensions as the movie, so for a 4:3 movie, a plane of dimensions x=4 and y=3 is good.
• In such a case, the texture mapping to use is Co-ordinates = Generated, Projection = Flat, and Extension = Clip.
• For ease of placement of camera(s) and the plane, the N key which brings up the properties allows numerical entry of values.
• As mentioned before, Emit=1 and Specular=0 are good for video textures.
• In some tutorials like this one, the rotation animation is done by entering a value like 3600 degrees for 10 turns. But when I tried entering a value of 360 for rotating the camera a full circle, it did nothing. When I tried 340 or 370, the camera moved only by 20 or 10 degrees respectively. So I settled for 180 degrees in my test.

NASA models and more in Blender

Ron Proctor of the Ott Planetarium very helpfully pointed me to the NASA 3D repository. Checking out the Hubble Space telescope model from there, learnt the following:

• Textures need to be in UV mapping mode for them to be displayed properly in this model - Mapping -> Co-ordinates -> UV - and I had to manually map the texture bitmaps to the relevant materials in the model.
• Blender's Textured Viewport shading only works if object has a UV mapped texture (in general).
• Imported model was extremely large as seen in default Blender view. Scaling it down was easier than moving the camera and lights out.
• Ctrl Alt Numpad 0 puts Camera to current view.

getting started with Blender

A few days ago I had done a trial with the Starball file from Weber U's Ott Planetarium. Rendering a single frame at 2k, 2048x2048, took around 2-3 minutes on my machine, running AMD X2 Dual core at 1.9 Mhz with Win XP 32 bit. When that was warped and projected on our dome, it looked pretty good.

Today, I dove a bit deeper, to take the first baby steps towards making our own fulldome show. Printed out the cheat sheet from qubodup and went through the first few getting started tutorials again to refresh my memory.

• Right-click to select an object.
• Middle-click and drag to move around in the view-port, with Shift and Ctrl as modifiers for pan and zoom respectively.
• Shift+A to add an object
• N to numerically adjust location and size
• S R and G for Scale, Rotate and Grab, with X Y Z for constrained motion along those axes.
• TAB to toggle Edit mode and Object mode
• Shift+F10 Image Editor (where rendered output is seen)
• Shift+F5 3D viewport
• Numpad 7 1 3 for Top Front Right views, 4 6 8 2 for rotate, 5 for toggling perspective and orthogonal views.
• Numpad . for Zoom to object.  

Also learnt from this video texture guide to use Emit=1 and Specular = 0 for a video texture. And from this tutorial, learnt to use Projection = Cube instead of Flat - or else the rendered surface may be only on the Z axis, and you may not see it! Rendering with a video texture to 1080p HD - 1920x1080 - went at around 10 seconds per frame.

Problems faced:

• Extrude tool seems to Extrude only along the natural face of the object - some sort of constraint somewhere.
• When I change dimensions of an object using N and then render using F12, it is going back to its original dimensions!

Next steps:

Would probably want to make a set of stills or videos move around the dome as in the IBEX show.

using Digital Universe to create fulldome content

The first step might be to export frames from Digital Universe - this seems to be clunky, and seems to need linux async as given at this page which has a perl script, which links to this more manual method.

But maybe instead I can use the video they give for download.