new streaming server live

Changed TTL for our streaming server to 10 minutes, and some time later, changed the DNS A record. Tweeted that users should clear their browser caches in case of problems. After half an hour or so, many users were still connected to the old server. Manually kicked them, then many started connecting to the new server. After one day, only a handful were on the old server, so shut down icecast.

changes to our icecast setup

Continuing my earlier post about changes while migrating, we made some changes to the icecast setup - using the on-connect tag to run our live begin and end scripts. There were some small hiccups while doing this - since the she-bang #!/bin/bash was not present on our scripts, they were not being executed. PB corrected this, and then giving the full path in the on-connect directive does the trick.

Also, instead of changing the alias of port 8020 to a different mount-point as earlier, we've now made the fallback-mount change in the mount-point telugustream during the live session.

(Our attempts to avoid the live begin and end scripts did not work. The fallback-mount directive for relay does not seem to work. So, if livemandirasia is a relay for livemandir, and livemandir does not have a fallback, then if asiastream falls back to livemandirasia, then if livemandir is not live, listeners get a 404 file not found. And if connect when live and live disconnects, listeners are kicked.)

In the web player at our website, the streams were linked to server1.radiosai.org:8000 etc - changed to stream.radiosai.org:8000 etc. and purged the cloudflare cache for those files. Other places have stream.radiosai.net, which is fine since it is a cname for stream.radiosai.org.

Then, PB found a nice way to make the icecast daemon start as our user (and not root) on reboot - the @reboot directive in crontab. So, now no manual intervention is required for starting the streams. Also, when done this way, the ulimit workaround we had to implement is automatically taken care of - the cron is run as root su'ed as our user.

key-based rsa auth - a caution

When doing key-based auth for ssh, a caveat - if the .ssh/authorized_keys or .ssh/id_rsa files are being modified, the permissions need to be set to not allow read or write by any other user for the authentication to work - chmod 600 needs to be done. 

changing servers with an upgrade

Migrating data from Internap (formerly Voxel) to Codero, we found that the upgrade from Ubuntu 10.04 to 14.04 breaks many of our configuration files. Apache and vsftpd in particular. Icecast remains at the same version, but our icecast scripts which use small binary files needed re-compiling, of course.

Apache:

• Merely creating symlinks did not seem to work - we had to run a2enmod and a2ensite for them to work properly. This included a2enmod cgi , without which the awstats perl scripts were being shown as plain text :)
• The default website name has changed from default to 000-default - without that, requests to the server using the ip address or using other names will not work. 
• The configuration files in sites-available needed to have .conf at the end.
• The old method of
  Order Allow, Deny
  Allow from all
  had to be changed as per this post to
  Require all granted
• Many more changes in apache2.conf, so we just modified the newer apache2.conf step-by-step with our old requirements. 

Vsftpd:

This needed additional config lines as per this post, done by PB, added the lines 

chroot_local_user=YES

allow_writeable_chroot=YES

and restarted vsftpd to fix the problem with not being able to write using ftp. 

Icecast:

We created symlinks to mp3info, mp3splt etc since the paths to these given in the old scripts had changed. Also recompiled the "CCodes" files. cc filename.c -o filename_of_executable

We tried to run icecast from init scripts, but just changing the USERID and GROUPID in the icecast2 init script was not enough. Perhaps we have to add chroot and run as user parameters also in the xml config file - we have to explore this later. 

We also tried some techniques to try and get Telugustream to mirror the live from mandir stream only during live times, but the methods we tried did not work.

livefrommandirfortelugustream as an alias for livemandir, and having autotelugu as fallback-mount for livefrommandirfortelugustream did not work in that the alias does not work in cascaded fashion - we already have :8020/ aliased to telugustream, but this does not follow the alias to livemandir. 

Also, trying to make livefrommandirfortelugustream a relay of livemandir works, but when livemandir ends, it has the fallback-mount of autoasia, so livefrommandirfortelugustream continues to get the relay audio instead of falling back to autotelugu. 

So we have to retain our method of switching config files using a shell script and manually dumping telugustream users to the relevant mountpoint using icecast's /admin/moveclients.xsl called using wget. 

video setup troubleshooting

The video team had a couple of problems. After changing video switchers from their older Datavideo DV switcher to the newer Panasonic AG-HMX100 HD mixer, they were not able to get a composite analog video out, or IEEE 1394 (firewire) DV out for streaming, and had rigged up a camera facing a screen for getting a DV feed. Apparently the Aja Ki Pro had earlier given them a composite out, but now was not giving it. Checked, found the manual mentions that the component out has to be set to SD in order for the composite out to be available. V did it, and it worked. That composite out is given to a DVCAM tape deck, which in turn gives a firewire out to the streaming server.

The next issue was regarding sourcing of PC playback video to the video mixer. Their setup had VGA out from PC going to a video doubler connected to the display screens in the hall, and a HDMI out from the PC going to a screen with a HDMI -> DVI cable. It so happened that the HD video switcher had a DVI input, only the resolutions supported were 1920x1080, 1280x768 etc. After changing the resolution to 1280x768 (1080p not supported by graphics card) and choosing the same format 1280x768 @ 60 Hz in the mixer settings, it could recognize the PC input as one of the valid sources. 

instapaper epubs on kobo reader

NNR gave me his Kobo Vox ereader to use. The epubs available for download from instapaper do not open in the Kobo's built-in library, but open in FBReader which I installed. Converting a mobi downloaded from instapaper to epub using Calibre could make that file recognizable by Kobo's library app. But with any method, the Feynman lectures' equations don't render correctly in epub. Printing to pdf using PDF995 and reading with Adobe Reader works nicely.  

documenting my fulldome process

Answer to a query about how we make movies from domemasters (still images of single frames):

To make the images into a movie, I usually do my own technical work. My workflow is on Windows. So, I generally follow this method.

Software used:

Avisynth - http://avisynth.nl/index.php/Main_Page

Virtualdub - http://www.virtualdub.org/

After installing the above software, I create an avisynth file, which is just a text file with .avs extension.

Opening this file in virtualdub, I choose
 Video -> Compression Xvid codec, maximum bitrate 9000 (kbps or so)
Audio -> Audio from other file -> choose the audio file
and then File -> Save as avi

Edit 19 Oct 2015: Adding a picture:

data analysis with ROOT - redux

In my previous post about data analysis with ROOT, I had used the data from a single gamma. Extending it to multiple gammas - 1500 - and calculating the energy weighted position, found a bug in my code.

Apparently the Fill() function does not fill if the indices passed to it are 0. That is,
h2->Fill(i,j,energy2D[i][j])
will not work for i=0 or j=0. Since our energy2D C array has a range 0-7 for i & j, this Fill() had to be re-written as
h2->Fill(i,j,energy2D[i-1][j-1])
with the i & j range 1-8.

I also changed my definition of the histogram to
TH2I *h2 = new TH2I("h2","energy distribution",8,1,9, 8,1,9);
instead of
TH2I *h2 = new TH2I("h2","energy distribution",8,0,8, 8,0,8);

With the resulting root file, energy-weighted position analysis could be done quite quickly, almost instantaneously, with

// loading the data in root file
    TFile f("multifile.root");
    f.ls();
    TH2I* h = (TH2I*)f.Get("h2"); 

for (int i=1;i<9;i++)
    {     
    for (int j=1;j<9;j++)
      {
      bincontent=h->GetBinContent(i,j);
      ewposx += posx[i-1][j-1] * bincontent ;
      ewposy += posy[i-1][j-1] * bincontent ;
      totalcount +=  bincontent ;      
      }
    }
    ewposx /= totalcount;
    ewposy /= totalcount;
    printf("ewposx & y   = %f, %f\n", ewposx  , ewposy  );

Changing the position of the gamma was fairly straightforward in LXePrimaryGeneratorAction.cc

The calculation of energy weighted position in the LXe example seems to have broken somewhere in development - people have reported non-zero positions in 2010 and so on, but when it is run now, it returns only zero, since the GetPos() for the pmt hit collection class returns zero. This seems to have causes in the class definitions and so on - LXePMTHit.cc doesn't mention fPos private variable in the over-ride function for = operator and so on. Instead of trying to fix that, I just re-entered the position data in root, as

for (int i=0;i<8;i++)
    {
    y = -22.75;
    for (int j=0;j<8;j++)
      {
      posx[i][j]= x ;
      posy[i][j]= y ;

      //printf("pos2D[%d,%d]=%f,%f\n",i,j,posx[i][j] , posy[i][j] ); 
      y+=6.5;
      }
    x+=6.5;
    }

blogger limitations

Blogger is still raw around the edges! Surprising, even after so many years.

• A post I did with email - the one just before this one - became mangled somehow, and made any page containing that post to break the browser so that links were disabled. Deleted it and reposted it from blogger compose window in order to fix it. Not sure what exactly was the issue, perhaps links are the issue in emailed posts.
• Any < or > anywhere in the post has to be manually escaped, written as & lt ; and & gt ; respectively (without the spaces, of course). Otherwise blogger creates all sorts of problems, misrecognising them as html tags.

three interesting fulldome content-creation links

Rendering of 3D fractals, Mandelbulber - via LR.

Rendering of molecules and other datasets, Chimera - via fulldome yahoo group.

Rendering of circular symmetric shapes with polar co-ordinates in AfterEffects - via Newton's Cannon blog.

opening multiple files in ROOT

If the filenames (or pathnames in this case) are sequential, this method in the ROOT forums would work.

while (i1<50) {
   
 sprintf(filename1,"pathnameTemplete_%d.txt",i1);
    f1 = new TFile(filename1,"READONLY");

and so on.

Otherwise this glob based solution might be useful.

ROOT canvas save problem

At first I could not save the ROOT plots, error mesg
Error in <tunixsystem::dynamicpathname>: Postscript[.so | .dll | .dylib | .sl | .dl | .a] does not exist in <tunixsystem::dynamicpathname> ...

Then found this discussion about this very same issue.

sudo apt-get install libroot-graf2d-postscript5.34

Solved.

Now I can save as png or pdf / ps etc. But not gif. Error,
Error in <tasimage::writeimage>: error writing file

But pdf and png are fine for now :)

data analysis with ROOT

Got similar results to my previous post with Octave, using ROOT.

Had some hiccups. Initial testing of ROOT C macro failed due to that particular sample data file being empty! The file input was based on this basic.C example. After getting the samples into the ntuple,
while (1) {
      in >> runID >> sID >> energy >> pixelID;
      if (!in.good()) break;
      ntuple->Fill(runID,sID,energy,pixelID);
      nlines++;
   }
I could do a simple linear plot just to check,
ntuple->Draw("pixelID","energy");

Then I created the same sort of 2D matrix as in octave - perhaps this is not necessary, but this is the way I did it :)

 for (int i=0;i<8;i++)
    for (int j=0;j<8;j++)
     {
     energy2D[i][j]=0;    // initializing to zero, since 
                          // some i,j are not mentioned in the next loop
     }

for (int i=0;i<nlines;i++)     {
     ntuple->GetEntry(i);
    //pmtNumber pmtx is an 8x8 array,
    // assumed numbered rowwise.
    //for i=0:7, j=0:7,
    //i*8+j=pmtx. So
    jj= int(pmtx) % 8; //here we need to cast the float pmtx as an int
    ii= (pmtx -jj) / 8;
    energy2D[ii][jj]=enx   
     }

for (int i=0;i<8;i++)
 for (int j=0;j<8;j++)      {
     h2->Fill(i,j,energy2D[i][j]);
     }
h2->Draw("lego2");

vGate 3.0 ROOT issues

First there were some issues with not being to type quotes inside vGate. Fixed that by going to Ubuntu's System Settings -- Keyboard -- Layout Settings, adding English US there and removing the English US extended international layout.

But ROOT on vGate 3.0 seems to be broken. Screenshot below. libjpeg.so.62 is not found in this case. Trying TBrowser b; also causes "segmentation violation" with some xpm files not found and so on.

ROOT font issue

On a fresh install of ROOT with Ubuntu Software Center on Ubuntu 14.04, this error on starting root

Couldn't find font "-adobe-helvetica-medium-r-*-*-10-*-*-*-*-*-iso8859-1",
trying "fixed". Please fix your system so helvetica can be found,
this font typically is in the rpm (or pkg equivalent) package
XFree86-[75,100]dpi-fonts or fonts-xorg-[75,100]dpi.

ROOT continues to run without problems. This issue is mentioned in this discussion, but xfs is not an available package for Ubuntu 14.04. Ignoring for now.

root -l doesn't bring up this error, nor the splash screen. Perhaps this font is used in the splash screen.

rough draft of output with Octave

Imported the ascii file output by the modified GEANT4 example I'm working with into a google spreadsheet. Manually sorted it by PMTNumber, manually added 0 photon counts for the missing PMTNumbers, exported the resulting photon counts into Octave as:

data = load ("Run-PMTHitOrder-PhotonCount-PMTNumber.csv")
for i = 1:8
  for j = 1:8
  matrixed_data(i,j)=data((i-1)*8+j,3);
  endfor
endfor
  my_bar3(matrixed_data)

plotted it using this octave version of bar3:

G4BestUnit

In order to check if the GetPMTPos() is working in the modified GEANT4 example I'm working with but returning 0,0,0 due to some units problem, tried to include G4BestUnit in the cout. As an example, LXeSteppingVerbose.cc has
std::setw(6) << G4BestUnit(fTrack->GetPosition().x(),"Length")

For this to work, the include file needed is
#include "G4UnitsTable.hh"

But even with this, GetPMTPos() returns zeros - units set to femto-metres! So, we have to work with GetPMTNumber() instead, since that is working fine. 

using vGATE 3.0 - a virtual machine with GATE, GEANT4, ROOT

vGATE is a virtualbox virtual machine image of Ubuntu 64 bit running GATE and associated tools. Tried it out. Running it on an Ubuntu 13.10 host, would not boot with the default virtualbox settings. On Windows XP, virtualbox was very flakey. The virtualbox control panel would come up only right after an install if "Run virtualbox" was ticked in the last step of the Install wizard. At other times, it would silently fail to start - the process would be seen in task manager for a few seconds before it vanished. Running the virtualbox image directly after a first run, by double-clicking (on Windows XP)

C:\Documents and Settings\Username\VirtualBox VMs\VGATE3VM\VGATE3VM.vbox

would work sometimes, and not work sometimes. Flakey. Also to be noted, the various issues of running Ubuntu with a non-3D-accelerated VM display - becomes very slow.

So I converted the virtualbox image to a VMWare image using the tool supplied with virtualbox,
C:\Full-path-to\VBoxManage clonehd --format VMDK vGATE_v3.0.vdi vGATE_v3.0.vmdk
The process took around half an hour for the 7 GB file on an external USB hard disk.

Running on VMWare Player was much more stable - at least for me on a WinXP host. Enabled 3D acceleration before starting the VM, of course.

Speeds - using a VM with 1.2 G ram, 1 processor,

SPECT benchmark - each acquisition of 37.5 sec was taking around 2 minutes to run.

Running the LXe example of GEANT4, from command-line the virtual machine was not too much slower than the bare-metal Ubuntu install. But with graphical display, was 3x slower.

VM - ./LXe LXe.in

Run Summary
  Number of events processed : 3
  User=0.92s Real=1.17s Sys=0.09s

Ubuntu 13.10 bare metal - ./LXe LXe.in

Run Summary

  Number of events processed : 3

  User=1.05s Real=1.06s Sys=0s

Using GUI, running .LXe and then running the macro with /control/execute LXe.in inside the LXe application, 

VM on VMWarePlayer, WinXP Host - 

Run Summary

Number of events processed : 3

User=1s Real=6.7s Sys=1.57s

Ubuntu 13.10 bare metal - 

Run Summary

Number of events processed : 3

User=1.44s Real=2.11s Sys=0.07s

Some notes on the modified version of Geant4 LXe example

Some graduate students have worked on modifying the LXe example supplied with the geant4 package, here are my notes on understanding and troubleshooting the modifications.

• cout needs << endl ; at the end, for flushing the buffer, especially when called inside a loop. 
• "Find in Files" from commandline -
    grep "string to search for" /path/to/directory -R -n
• the G4cout statements in the LXe example have an if statement before them, so the macro statement /LXe/eventVerbose 1 should be called if those are to be executed. 
• the supplied macros in LXe example also have errors -
  COMMAND NOT FOUND

-  this is due to the modification noted in the Changes file, in 2013, November 22, 2013 P. Gumplinger (LXe-V09-06-12) - replace Update method and commands with ReinitializeGeometry. So we should just comment out the update lines in the macros. Four cc files have been modified in src - LXeMainVolume.cc, LXeDetectorConstruction.cc,  LXeEventAction.cc and LXePrimaryGeneratorAction.cc.

• LXePrimaryGeneratorAction.cc sets the gamma energy to be 140 keV.
• pmt positioning is done originally in LXeMainVolume.cc inside G4PVPlacement function -
  
  There, the PlacePMTs function calls have been commented out except in the xy plane.
• G4double lxe_Energy[lxenum]    = { 3.001*eV , 3.000*eV, 2.999*eV, 2.998*eV };
• is defined in LXeDetectorConstruction.cc - also
  
  fLXe_mt->AddConstProperty("SCINTILLATIONYIELD",63000./MeV);
  
  G4double vacuum_RIND[lxenum]={1.,1.,1.,1.};
  (changed from 3 values in orig to 4 values here. seems to be a mistake. - refractive index should be a 3D vector, not 4D.
  
  In SetDefaults, sizes and numbers defined as
  
  fScint_x = 5.2*cm;
  fScint_y = 5.2*cm;
  fScint_z = 1*cm;
  fNx = 8;
  fNy = 8;
  fNz = 0;
• LXeEventAction.cc has the added opfile <<  statements.
  Most of the G4couts in the original are after an if(fVerbose>0)
  that is, LXe/eventVerbose 1 must be set while running LXe for them to execute.
• vis.mac the default visualization macro has
  /run/verbose 2
  /control/verbose 2
  setting these to 0 reduces some clutter.
• Somewhere the process verbose is being set to 1, can't seem to override. So all hadron processes etc are listed.
• 140 keV being deposited in modified version as against 511 in orig.
• Energy weighted position of hits in LXe is being shown as non-zero in new and old, but in both cases, reconstructed position is 0,0,0

some Ubuntu tweaks for general working

To add Open in Terminal to the right-click context menu for folders displayed in Ubuntu, followed this page
http://askubuntu.com/questions/207442/how-to-add-open-terminal-here-to-nautilus-context-menu
and installed with
sudo apt-get install nautilus-open-terminal

To install Adoble Flash Plugin, Software Center had Adobe Flash Plugin installer. But even after that, radiosai streams did not work - apparently that was due to absence of mp3 codec. Installing GStreamer plugins for mp3 and so on from Software Center solved that issue.

To avoid having to adjust the brightness every time on reboot, followed this workaround.
cat /sys/class/backlight/acpi_video0/max_brightness
to get the value for max brightness, and then put that or a corresponding value - I used 8 - at the end of /etc/rc.local as
echo 8 > /sys/class/backlight/acpi_video0/brightness

Apparently the correct solution is to fix it in grub, as given here.