Wednesday, November 24, 2010

live streaming with stock laptop

Used the daily-use machine for live streaming on 23 November, with just its built-in soundcard. Reaper was running as something for me to monitor the waveform and also as an archive copy of the stream. Srv ran the mix through his Yamaha mixer from Chennai and used the Alesis compressor from the Studio to give me a compressed mix. My nominal input level for 0 dB for -10 dBV signal was as shown.

I rode the rec-level fader since Srv was monitoring before the compressor. Maybe next time a better choice of compressor settings can avoid my having to ride the faders. Anyway, output level was respectable, near -12 dB RMS, after the first few minutes when I was not riding the faders close enough and the compressor was not set on the output.

Saturday, November 20, 2010

direct download to kindle

Playing around with the Kindle got for A - Latest Generation Wifi. Can download directly to the kindle by browsing But can't load to the kindle from email attachments. I think it's because the kindle's experimental browser does not support redirects and most webmail services have a redirect for attachments. Pointing the kindle to the direct link after deciphering the redirect's URL encoding, it loads the content with no issues. This works with the content emailed by too.

Sunday, November 14, 2010

smtp timeout errors

After introducing the Cyberoam 50ia as the gateway at the studio, there were some teething troubles. When one of the two redundant links went down, the device would refuse incoming connections on the other link also. The management services would then have to be restarted from the console. Also, occasionally krishna had some smtp timeout issues even after the links were up. These were solved by restarting krishna.

Friday, November 12, 2010

qshape for checking postfix queue

One of the tools for checking the sending of our emails - qshape.
Some results from 5 days back when the problem was first detected:

sudo qshape -s incoming active deferred
T 5 10 20 40 80 160 320 640 1280 1280+
TOTAL 33626 0 0 0 0 0 0 0 4 9645 23977 32886 0 0 0 0 0 0 0 4 9645 23237 740 0 0 0 0 0 0 0 0 0 740
Out of these mails, most of them were yahoo.

sudo qshape incoming active deferred
T 5 10 20 40 80 160 320 640 1280 1280+
TOTAL 33623 0 0 0 0 0 0 0 4 9645 23974 24602 0 0 0 0 0 0 0 3 6365 18234 7809 0 0 0 0 0 0 0 1 2903 4905 154 0 0 0 0 0 0 0 0 30 124 127 0 0 0 0 0 0 0 0 72 55 82 0 0 0 0 0 0 0 0 20 62 67 0 0 0 0 0 0 0 0 13 54
.... snip .....

Wednesday, November 10, 2010

bug - some files not imported into database

When there is a file with the same download file name (spaces replaced by underscores) being imported, the import script silently fails. The workaround would be to change the download filename of the old file in the database and then do the import.

importing files to downloads without playing them

1. Create playlists for the required files. Don't forget Signature tune. Choose a date in 1985 or something like that.
2. Enter suitable date in /home/sgh/scripts/
3. Then we can run to get the csv to the database
4. Then go to http://localhost/mkcsv/csv.php?date=01&month=01&year=1985 (or whatever date) to upload csv
6. Then go to and import the csv of the correct date.

bounce processing for yahoo and new ip address

Found a lot of delays from yahoo mail servers, deferred connections and so on. At first I thought it was just due to the new ip address - found that it was listed in Spamhaus, and sent in a request for removal from Spamhaus.
Contacted yahoo using their bulk mail contact form. Their reply said
After looking into the issue, it appears that your emails may have become de-prioritized due to a number of potential issues with your mailings. Please visit the following URL for information on improving list maintenance:
The last point was about too many non-existent email addresses - and found lots of such errors in mail.log

Digging deeper, this is what I replied to yahoo:
Going through your checklist, and checking out our bounce processing, found that the bounces from yahoo were not being processed! The reason being that yahoo mail servers don't send the bounce messages, our mailer daemon has to send the bounce message, and this was not being sent to user noreply - user did not exist! This meant that more than 100 - 200 bad yahoo accounts have been sent mail from our server for the past several months! We're weeding out the bad ids - done some manually, rest will be processed by the bounce processing mechanism of phplist over the next week.
Actually there were around 400 bad mails of yahoo domains and a few from other domains. Unsubscribed the yahoo ones manually by digging through the bounces one by one using the gmail interface. The others will be caught by the bounce processing in due course.

For getting the bounces to the right place, I just had to create a noreply user, and put the following lines in a file /home/noreply/.procmailrc
MAILDIR=/var/spool/mail/noreply #The location of mail directory

After I sent the email to yahoo, they replied, saying in part,
Yahoo! Mail accepts a maximum of 20 messages per SMTP connection. We encourage you to cap the number of messages you send to Yahoo! Mail to fall within this per-connection limit.

My reply to that was:
We're running postfix > 2.3 which does not have a direct way to limit number of mails sent per connection.

Hopefully our server will reconnect OK with yahoo servers on dropping connections - our logs don't show any problems with that. We're still in the process of weeding out bad addresses. Once we finish that, hopefully, we won't irritate the yahoo MX servers as much!

In addition, I have now put
smtp_defer_if_no_mx_address_found = yes
in the /etc/postfix/ - Default was to bounce mail if MX record not found, i.e. if DNS fails. Now changed to wait and retry for upto 5 days or whatever message age time is set.

Friday, November 05, 2010

download link from vimeo

Initially there was a requirement that we make the videos of MoL downloadable. P and I checked out various ways to do it from vimeo itself, since uploading duplicate copies to would be not only tedious, but also tough on the server! Guided by this shell script, where he uses curl or wget to retrieve${VIMEO_ID}/${REQUEST_SIGNATURE}/${REQUEST_SIGNATURE_EXPIRES}/?q=sd" > ${VIMEO_ID}.flv

REQUEST_SIGNATURE=`echo $VIDEO_XML sed -e 's/^.*\([^<]*\)<.*$/\1/g'`
REQUEST_SIGNATURE_EXPIRES=`echo $VIDEO_XML sed -e 's/^.*\([^<]*\)<.*$/\1/g'`
P made a very elegant PHP5 script based on
$xml = simplexml_load_file("$id");
$request_signature = $xml->request_signature;
$request_signature_expires = $xml->request_signature_expires;
$downloadurl = "$id/$request_signature/$request_signature_expires/?q=sd";
Unfortunately our server did not have PHP5, and PHP4.x did not have the simplexml functions. So I modified it to use curl using the code example from this page and using the value_in() example from this page.
$xml = get_html("$id");
$request_signature = value_in("request_signature", $xml) ;
$request_signature_expires = value_in("request_signature_expires", $xml) ;
$downloadurl = "$id/$request_signature/$request_signature_expires/?q=sd";
echo "<p align = \"center\">";
echo "<a href=\"";
echo $downloadurl;
echo "\">Right-click here and choose 'Save Link As...' or 'Save Target As...'</a>";
Attempls to avoid the 'right-click instruction' did not work, since the vimeo server sends the video as mp4 without a Content-Disposition attachment header, and redirects don't conserve headers. Finally, of course, the script was not used due to non-technical reasons....