ERROR: duplicate key value

PB being out of action, making playlists and doing Telugu database work etc. So, while importing Telugu files, came across

ERROR: duplicate key value violates unique constraint "download_filename_file_id_ukey"

This causes some files not to be entered into database, so when importing playlist, it complains that some fids are NULL

Solution was to import files one by one till the offending file was found - a file which had the same name as an existing file, but with space in the filename replaced with underscore. Removed the old entry from database - rather, changed space to underscore for its filename.

setting up vnc

While setting up vnc for doing the Telugu files on the telugubc server, found that it was installed but not running. Using the same script as on saiwaves did the trick, only thing was that it needed vncpasswd to be run before running the script to start vnc, so that we set the password to something we know. And editing .vnc/xstartup is required if you want kde - I changed the default twm window manager to kde. Last line becomes startkde & instead of twm &

procedure for telugu broadcasts

1. Go to ~/new_telugu and copy mp3 files from relevant folder to ~/ftpsent

2. Copy relevant m3u file to ~/playlist directory.

3. cd to playlist directory and run the sed script modifying the commmandline to relevant date, which converts the extended m3u to simple m3u and appends the audio directory path.

4. run ~/scr*/lame_encoder2422.sh

5. Open playlist from step 2 in kwrite and change / to \, save as m3u

6. Open the m3u playlist in Winamp after lame finishes, to check that all files are present - total duration should be 24 hours and a bit, and it should not show a + symbol after the total duration.

If playlist generation is desired, additional steps:

7. Run scr*/main_ftp.sh which takes files from ~/ftp folder

8. upload files from ~/ftp folder to dl.radiosai.org

9. Run scr*/playlist_import.sh

10. Update missing descriptions from localhost/SGHDescription as with sgh database, if necessary copy-pasting from sgh database

11. Make csv and upload from localhost/mkcsv

bug in our playlist import script

Faced the same problem as given in this earlier post, with the schedule csv/email creating script complaining that SIGNATURE TUNE not found, "PROGRAM NOT FINALISED". The solution is to just search for entries in the playlist table where timestamp is 2011-08-16 06:30:00+05:30 (or whatever date). This can occur at midnight also - as PB pointed out, a bug in the script changes a timestamp of 2011-08-16 00:00:00+05:30
to 2011-08-16 06:30:00+05:30 during the import process. So, verify if the correct timestamp should be 00:00:00+05:30 and then edit the entry manually from 06:30:00+05:30 to 00:00:00+05:30 .

The point to be noted is that the problematic entry may not be adjacent to the actual 06:30 signature tune file, it could be at midnight also.

updating MBV download filenames

The MBV files were imported into Telugu stream database without suitably modifying the download filenames. This caused quite a bit of confusion. PB contributed the following post :

Steps done for rectification:

1. In sgh database on saiwaves run the following query:

select 'UPDATE file_id SET download_filename=' || '"' ||download_filename|| '"' || ' WHERE filename="' || filename || '"' from file_id where filename ilike 'MBV%' \t \o update_mandirbhajans.sql ;

First row of the result set looks like:

UPDATE file_id SET download_filename="MBV_04_RAMA_OM_SRI_RAM.mp3" WHERE filename="MBV_04_SUJITH_RAMA_OM SRI RAM.mp3"

2. Save the result set to a file update_mandirbhajans.sql

3. Replace double quotes with single quotes in update_mandirbhajans.sql

4. In sghtelugu database on saiwaves execute the saved update_mandirbhajans.sql (through phpPgAdmin interface)

5. upload the updated csv files / run the update sql statements on the schedule database through the cpanel

stellarium night sky on our mirrordome

Trying out Stellarium's Mirrordome distortion mode. On the laptop I use, Stellarium cannot use the full screen when in dual-screen mode with different resolutions on each screen, and also refuses overlays like the toolbars on the secondary screen. So, used the Dual-screen clone mode with both displays set to 1280x800. With the default settings, the stars don't look so good on our dome, but some tweaks make the stars look better.

View Settings Window -> Sky tab -> Absolute scale = 2 makes everything bigger and hence brighter, and Relative scale = 0.2 or so makes the stars uniform in size instead of brighter stars being as big as oranges! This makes the night sky much better to look at, more like a "starball" effect.

viewing panoramas on fulldome - simple technique

To make a panorama viewable on the fulldome, tried the following method as given at this page as a shortcut instead of going through Blender or buying commercial software. At the bottom of this page, the type of fulldome image we need is shown. Basic method is simple - make the pan into a square image by resizing, then convert to polar co-ords. In Photoshop it is Filter -> Distort -> Polar Co-ordinates and in the Gimp it is Filters -> Distorts -> Polar Co-ordinates. This produces a fulldome master which I can later distort with tgawarper for display on our dome.

When I tried it with a pan of the SSSIHMS, a hi-res version of this picture, found that it looks squished on the dome - need original to be more squarish? Also, building is too far above the horizon. Need more sky and less ground. So, probably the solution would be to take a pan with the interesting parts near the bottom edge of the pic, and clone the sky to make it square instead of distorting it by simply resizing.

download users as csv in Google Apps

CV had a problem downloading list of Google Apps users as CSV - the option was grayed out in the control panel.

This post seemed to have the solution:

http://www.google.com/support/forum/p/Google+Apps/thread?tid=5a8c21e76144b065&hl=en

So, sign in, and then use the following url for the csv file:

https://www.google.com/a/cpanel/urdomain.com/DownloadUserData/UserData-urdomain.com-20110727.csv

where you must change the date 20110727 for the relevant date. We also have to remember that google servers use US time, so today is still 28 July for them and not 29 July.

Telugu stream playlist - Extended m3u to simple m3u

PB passed on the following info.

cat 26th_JULY.m3u | sed /^#/d | tr '\\' '\n' | tr '\r' '\n' | sed '/\(.mp3\)\|\(.MP3\)\|\(.Mp3\)/!d' | sed 's/^/\/home\/sgh\/audio\//' > 260711.playlist.telugu


sed /^#/d Remove the lines starting with #

tr '\\' '\n' The windows path backslash (\) is changed to line feed

tr '\r' '\n' The Carriage return is changed to line feed

sed '/\(.mp3\)\|\(.MP3\)\|\(.Mp3\)/!d' Remove the lines not having .mp3 or .MP3 or .Mp3

sed 's/^/\/home\/sgh\/audio\//' Add the path /home/sgh/audio at the beginning of each line.




Added tr '\r' '\n' on 26th July 2011.
The line used to end with both \r\n which made ices fail to start the playlist.

\r - Carriage Return - 0D in Hex
\n - Line feed - 0A in Hex

problem playing file from finder for live broadcast

There was some difficulty with using the technique I had used in April for playing an announcement file directly from Finder into the live broadcast. Going through the steps, found that the following sequence is required: First change system audio out to Jack device using Audio MIDI setup utility. Then the Jack setup file which I had created earlier can be loaded.

But in general, this is a bit risky - a better way to do announcement insertion would be to drag and drop the relevant files into the timeline on Reaper. This can be done even when Reaper is recording. The track into which the file is dropped should not be armed for recording - only then will the file play back if Reaper is recording on another track.

more ports on server

We wanted some more ports open on the server firewall, for future expansion if required. The service provider said they have opened the ports, but ShoutCast complained that it could not bind to those ports.

Did some troubleshooting by starting the server from the command line.

Found that the reason it gives for the other ports not to work is:
error opening client socket! FATAL ERROR! Some other process is using this port!

So it is not the service provider's fault.

When I test with telnet, I find (after stopping shoutcast)

telnet stream.radiosai.org 8021
Trying 66.249.27.181...
Connected to stream.radiosai.org.
Escape character is '^]'.
200 1500
GET / HTTP/1.1
fxÚ
ÍËÎË/ÏSHÎÏÍMÌKÑ5

Yesterday when I tested, ports 8023, 24 and 25 were also probably occupied by the other process. Today, those ports were open, and shoutcast could use those ports.

Using
Netstat -a -b -n -o > c:\netstatnames.txt

and searching for 8021 in that file,
found that the process is
G6FTPSERVER.EXE

So, requested that the Gene ftp be tweaked so that it uses only ports above 8100 for listening, like 8121 etc. That solved the issue.

procedure to change over to live broadcast

On stream.radiosai.org I have created services 8002, 8002live, 8020 and 8020live in addition to the services for the other streams. Batch files

c:\startlive.bat and c:\stoplive.bat do the work of

net stop 8002

sleep 5

net start 8002live

etc

(similarly for the 8020 stream, and similarly for stoplive)

Shortcuts on desktop for these bat files have been created. So, one minute before the live broadcast is to start, like 4.29 pm for the evening programme, just login with Remote Desktop and double-click on the respective icon.

To stop live, wait 70 seconds after the ending point - usually the Jaikar after Arati - and double-click the stoplive bat file. The BUTT has to be running during these 70 seconds too. This is because of buffering of the stream by the Shoutcast servers.

Edit: Initially we experimented with using Scheduled Tasks to automate the changeover, at least for the startlive which was a fixed time. But Scheduled Tasks don't run if the password is changed, and then we would also have issues when DST kicked in. So, decided manual change-over was better. And the next day the manual change-over came in useful when we had a long power outage at the Studio and could not change to live till 50 minutes into the morning session - recorded filler was played on air at that time.

importing SGH M CDs into local database

Found that only upto SGH M 244 has been imported into SGH database. PB reported that after shifting to the Dell servers, the script that imports the cd into the database failed as it had some gui components in it and not just command line. Checking the code, found the following and modified:

1. The script mount_cd.sh had a call to the binary message

which basically just displays a message to insert the cd. I have made this into an echo statement.

2. After that, main.sh had a call to the binary question

That also I turned into an echo statement.

3. Next, check_if_all_mp3.sh called the binary error

That also I changed to a simple echo statement. Then a call to question, which I turned to echo and read.

4. Next call to error and question come from main.sh which I again changed to echo and read.

5, Next came check_if_cd_entered.sh which had calls to error which I changed to echo.

6. Next, check_bc_entry.sh had a call to broadcastcenter, but it had already been commented out.

Importing worked after this.

colinux instabilities and D drive

There were some issues with our streams skipping files, and the problem seemed to be on the filesystem of stream.radiosai.org. Stopping the webserver on stream.radiosai.org which referenced files on the D drive seemed to solve the issue. PB did some research, and said the following:

Yes, the issue could be the windows file system size or its way of dealing with long file names. But the failure in reading might even be due to Colinux. Windows has never complained in sending the data to Colinux but CoLinux always complains in not able to read the data. The Colinux change log does mention about I/O errors.


Was going through this link on Working with File Systems which says:

If you use a large number of files in an NTFS folder (300,000 or more), disable short-file name generation, especially if the first six characters of the long file names are similar.

Every time you create a file with a long file name, NTFS creates a second file entry that has a similar 8.3 short file name. A file with an 8.3 short file name has a file name containing 1 to 8 characters and a file name extension containing 1 to 3 characters. The file name and file name extension are separated by a period.

If you have a large number of files (300,000 or more) in a folder, and the files have long file names with the same initial characters, the time required to create the files increases. The increase occurs because NTFS bases the short file name on the first six characters of the long file name. In folders with more than 300,000 files, the short file names start to conflict after NTFS uses all the 8.3 names that are similar to the long file names. Repeated conflicts between a generated short file name and existing short file names cause NTFS to regenerate the short file name from 6 to 8 times.

To reduce the time required to create files, use the fsutil behavior set disable8dot3 command to disable the creation of 8.3 short file names. (You must restart your computer for this setting to take effect.) For more information about disabling 8.3 short file names, see “MS-DOS-Readable File Names on NTFS Volumes” later in this chapter.

If you want NTFS to generate 8.3 names, improve performance by using a naming scheme in which long file names differ at the beginning of the name instead of at the end.

But again, disabling 8.3 filenames systemwide seemed a bit drastic, and could lead to other unexpected problems elsewhere. So, for now we have not done the disabling.

wifi radios for radiosai

Since WorldSpace broadcasting stopped, a lot of queries about ways to listen without a computer. Found that wifi radios are quite cheap in US and UK, but almost double the price in India.

This one, for eg. is around Rs. 4.5k in the UK while in India, the listings start from 10k. Other cheaper options in India other than mobile phones are products like this Asus box for around 6k. What you have to look for in products which allow you to listen to Internet radio stations is: do they allow you to enter custom stations - if yes, then you are safe. Otherwise, do they use the Reciva online database - if yes, then also it is safe, since Sai Global Harmony all streams are listed in Reciva. If the player uses a custom database and does not allow you to enter your own stations, then you may not be able to listen to SGH with such a player.

buffering on server

Was listening to radiosai on my mobile today with wifi for long periods while lying on bed, made the following interesting observations.

1. The changeover actually happened at around 6.32 as heard on the stream, but the shoutcast server went into the interruption music for more than ten minutes more, till the signature tune finally came, around 6.44. Meaning the stream is buffered by 14 minutes due to the two servers?

2. By afternoon, the delay was around 7 minutes - The dd which was supposed to get over at 2.41 was heard to end at 2.48. So somehow the buffering delay was reduced ?!

BSNL online directory enquiry

BSNL online directory enquiry has become centralized, has changed location and needs City to be entered. For Puttaparthi, City needs to be Penukonda. Not too intuitive!

adjustments for mirrordome

We had some minor adjustments done to improve the looks of our fulldome projection.

• Moving the mirror up by a few inches more now allows us to bring the horizon a bit lower in the front area of the tilted dome of the theatre.


• Moving the projector laterally left-right corrected the tilt in the reflected image, where earlier the left side of the reflected image was around a foot higher than the right side - for this, we moved the projector physically to the right.


• Moving the projector towards and away from the mirror, and adjusting the zoom setting allowed us to reduce the shadow area right above the mirror. Covering a larger area on the mirror makes the shadow smaller and vice-versa.


• Using the Bright mode instead of Custom mode in the projector, the colours are a bit better though the brightness levels are quite lower. This has to be further tuned.

couple of queries of email migration

Today, V had a question: if he deletes a group in Google Apps, will it take 5 days for him to create a user id with the same name. It turns out the answer is no.

http://www.google.com/support/forum/p/Google+Apps/thread?tid=31213ea23239c903&hl=en

When you delete the group, you should be able to recreate a user of the same name immediately. The 5 day wait period only occurs when you delete a user and try to recreate a user of the same name. Deleting a group or email alias does not lock out that address for 5 days like it does when deleting a user.

Also, Google Apps seemed to have removed the online IMAP migration tool from within the Domain control panel. So the email migration has to be done using the Google Apps Migration for Microsoft Exchange utility .

Once you install it, the setup is like the screenshot for migration from gmail:



In case it asks you for changing the consumer key, you have to log in to google apps as webmaster and copy-paste the new key.

The users csv file should be of the form
user1@domain.in#user1password_of_domain.in, google_apps_user1

(Please note that the @google.apps.domain is not required for the second field.)

Unfortunately, this tool takes a lot of bandwidth, needs the computer to be on all the time it runs, and is much slower for gmail to gapps migrations than the online IMAP migration tool. A sample run is:

Info:Start local time is: 2011:06:01 13:56:12
Info:Migration start local time is: 2011:06:01 13:56:12
Start: Now migrating user=**********
Start: Migrating Email for store=**********
Start: Migrating 105 messages from Folder [Gmail]/All Mail for store=**********
In progress: Migrated 100 out of 105 messages from folder [Gmail]/All Mail for user.
Success: Migrated 105 messages from folder [Gmail]/All Mail for store=

snip....

Info:Migration end local time is: 2011:06:01 14:25:19
Info:End local time is: 2011:06:01 14:25:19

So it took half an hour for 100 mails.

upload problems persist

As noted at the end of this previous post, upload problems persist with sftp. When using GUI tools like filezilla and gftp, we can see that sometimes the server drops the connection, then the tool reconnects and resumes the upload. filezilla is much faster than gftp, but has support for key-based auth under Settings only in later versions which are not available for the particular OS on saiwaves. Runs OK on WinXP.

Sometimes, when using sftp and it stalls, if we kill the process, then the file at stream.radiosai.org becomes orphaned or something like that. If we delete the file, then also the file appears in ls. After a day or two it disappears on its own - probably when the process times out or something like that.