Friday, April 05, 2019

creating an AppImage to distribute Linux binaries

Some notes of my AppImage creation process.


  1. First tried creating manually, using
    https://github.com/AppImage/docs.appimage.org/blob/master/source/packaging-guide/manual.rst and the AppRunx86-64 from https://github.com/AppImage/AppImageKit/releases
  2. Found and copied the libs it uses by doing the following to copy all shared libraries -https://www.commandlinefu.com/commands/view/10238/copy-all-shared-libraries-for-a-binary-to-directory
    ldd file | grep "=> /" | awk '{print $3}' | xargs -I '{}' cp -v '{}' /destination
  3. Need to use the environment variable $OWD (Original Working Directory) to get files from the OWD, like our ini file. This code snippet was a good template, https://stackoverflow.com/questions/40306012/c-using-environment-variables-for-paths
        char* pPath;
        pPath = getenv("PATH");
        if (pPath)
          std::cout << "Path =" << pPath << std::endl;
        return 0;
  4. Many libs copied with the manual method had to be excluded using the excludelist at
    https://github.com/AppImage/pkg2appimage/blob/master/excludelist
  5. Still did not work on newer distros. In this thread, probono recommended using trusty on TravisCI for building, and linuxdeployqt instead of manual creation of AppDir. Did that, and it worked. Far fewer libs are bundled by linuxdeployqt. Lists of the two bundles are available at this issue thread.
  6. The download link from transfer.sh can be seen if the entire log is downloaded from TravisCI.
  7. The travis yml file gives the outline of the build and bundling process.

Thursday, April 04, 2019

USB drive reported as read only - solution

Found my USB drive reported as read-only on Linux Mint 18.1.

Corruption could cause this, so tried
sudo dosfsck -a /dev/sdc1

That did not help. Though the CLI is able to write to it, Nemo still complains it is read-only.

Apaprently the fix is at https://forums.linuxmint.com/viewtopic.php?t=176546

So everything needed to fix the issue is to execute:
sudo chmod 777 /media/USERNAME

And/or maybe a reboot?

When I faced this issue today, I rebooted the system, and then the USB drive was mounted read/write.

Wednesday, April 03, 2019

transparent proxy over ssh - sshuttle

Sometimes the network to which I am connected fails, while the LAN over fiber to Studio is active, and Studio's backup network is functional. Then, an elegant way to use the Studio network, without having to install any proxy server software, is using sshuttle.

sshuttle --dns -r username@ip.address.of.machine 0.0.0.0/0

will transparently proxy all dns requests as well as all traffic through the remote machine. Only python needs to be installed on the remote machine. 

Tuesday, March 19, 2019

Fedex (India) - expectation vs reality

Documenting my experience with Fedex. Plenty of lessons here for everyone, I think. My takeaway was, avoid Fedex (at least in India) whenever possible, if you are an individual, and not a corporate with a Fedex account. Perhaps they treat corporates better - who knows. But the level of miscommunication and lack of communication would indicate otherwise. Even our University had requested us a couple of years ago to avoid Fedex when importing.

1. I get a phone call on Saturday, 23 Feb 2019, that my shipment is due to arrive the next day, and I need to submit the required documents. Since I had already submitted my "one time only" Know Your Customer (KYC) documents last year, I thought this would be a breeze. I was wrong.

2. The form I had to fill in this year was different from last year's. (My printer was on the blink, so I had to trek all the way to another office, 15 minutes away, in order to print all the new documentation that had been requested.) The documents requested would probably change again, so it may not be worthwhile documenting them, but just for completeness:


·        KYC Form along with supporting documents – DULY SELF ATTESTED {signed & stamped}  (KYC is mandatory and is a 1 time requirement)

·         PAN copy,

·         Aadhar Card & Voter ID card

·         Address proof. (Telephone /Electricity Bill/Bank Statement/ Rental Agreement )

·         AD code ( Authorized dealer code from Bank)

·         DUMMY IEC LETTER

·         Gatt and DGFT declaration ( Attached )

·         Technical write up & INDIVIDUAL NET WEIGHT FOR ALL ITEM to prepare Advance checklist

3. At first, the Fedex rep on the phone told me that "since the item has come in cargo mode" it was mandatory that I have an IEC code - which is an Import-Export code for importers. She told me to find out from my organisation if I can import under their IEC, saying that individuals cannot import. Though I tried telling her that I had imported items in the past, she said it was impossible. Luckily, as it turns out, there was a delay in getting hold of the head of operations at our institution. In the meantime, Fedex called me again, saying that it would be possible, using something known as a dummy IEC, the request for which was listed in the above list.

4. Then there was Sunday - a day of rest - and no communications from Fedex.

5. On Monday, there were a flurry of 17 emails. Fedex did not accept my Voter ID as address proof. Only one of the items listed above. And AD code - I thought that since I am not an authorized dealer, I need not submit. But I was wrong. Apparently this is the AD code of the forex section of the bank. Even the bank manager did not know this, and this led to another day of delays.
Expectation - if someone from Fedex had explained this to me, I would have done it immediately.
Reality - I got the same "form" email from Fedex which I received earlier, so I just wrote back saying that I have already submitted the documents. So, this led to a one-day delay in filing the Bill of Entry (BoE) and thus a Rs. 5000 fine. Which has to be paid by me, of course. And even after receiving the AD code, there was some confusion - I got an email saying that the AD code is not valid. I again contacted the bank. The bank said it was valid. Then Fedex confirmed that the AD code was valid, it was the wait for permission to use dummy IEC which was causing the delay.

6. On Tuesday, I got an email with approval list for BoE filing, from jeena.co.in, Fedex India's clearing agents. I filled it up and returned it within 15 minutes. Then came the long wait, 26 Feb to 1 Mar. No intimation from Fedex or Jeena as to what is happening. I tried calling all the numbers available. No one picks up. Calling the toll-free numbers gets me an agent who speaks very soothingly, saying that Mr. (name) is in charge of clearing this package, they will inform him to get in touch with me. But Mr. (name) does not contact me. At all. Ever.
Expectation - if someone from Fedex or Jeena had explained that it would take a week or two for customs to clear it, I would have waited patiently. At the very least, someone should pick up the phone.
Reality - Either the staff are so over-worked and harassed by customers that they cannot bear to pick up calls, or they just stopped caring long ago.

7. On 1st March, I was asked to submit any or all of the following, for value evidence:
1) Internet price
2) Supplier price list
3) Previous BOE,which cleared earlier with same Item
4) If payment is made Bank remittance.

I did so.

8. Again a looong silence. On Mar 7, late in the evening, I get an email, asking for technical writeup. Technical writeup had been submitted in the first batch of documents. Anyway, I again edit that writeup, add a couple of lines, and submit along with all the SLED documentation in the supplier's website.

9. Silence again, till 13 Mar. Now, I am asked to pay the customs duty directly using the ICEGATE website, and to provide "E-way bill". Since I am an individual and not a company, I was not sure if I could do these things. Called up the toll-free. They said they will ask their people to call me and guide me. They never called. I googled, found the ICEGATE website, paid using the details like Challan number supplied by Jeena. But according to various internet sources, individuals need not file E-way bill unless they are transporting high-value goods in their own vehicle. Again write to Fedex, no reply. Again call toll-free. Again they tell me that they will ask their rep to contact me. They do not do so.
Expectation - when the customer asks for help, someone tries to help.
Reality - The toll-free operators either are not allowed to, or not able to, help. The customs clearance people are either overworked or not bothered.

10. 14 Mar - called up Jeena - the executive picked up! That itself was half the battle won. I mentioned that I'm an individual, not a company. Immediately he said, in that case, just send me a letter saying so, no E-way bill required. So, again I sent an email saying just that. And that evening, I get the automated Fedex tracking email, shipment released for delivery.

11. I thought I had paid all that was necessary, but wanted to make sure. Called Fedex toll-free to check. On 15th morning, they said please call back by 15th evening, because some documents may take time to be uploaded into their system. On 15th evening, I call, they say nothing to be paid. On 16th morning, I get a call from Fedex delivery agent, saying I need to pay Rs. 7000 for delivery. Again I call the toll-free number. They tell me that though it is not yet mentioned on their system, it is possible that the clearance charges are what I am expected to pay.
Expectation - clear intimation of what is to be paid, with an emailed invoice in advance.
Reality - confused signals. Nobody knows anything.

12. Part of the problem is the expectation. With services like India Post, you know that there is no one to call to ask for updates, you expect delays, you don't expect any email updates. But then Fedex is more than twice as expensive, so maybe the expectations are warranted. Especially when they put up pages like
http://www.fedex.com/in/why-ship-with-fedex/money-back-guarantee.html
and then completely renege on the delivery date. Like what happened last night - item still not delivered, so delivery status was shifted from "by 8.00 pm" to "pending".

Edit: The item was finally delivered on 20th March, after being shipped on 22 Feb. Even if you discount all the customs clearance delays, it took 8 days. With "Fedex International Priority" shipping. Now compare this with another similar package, sent via Royal Mail and India Post, which took a total of 10 days to reach me from Aberdeen, Scotland.  

Monday, March 18, 2019

different kinds of hash verifications, and writing boot disk iso to USB

Fedora - the hash is in a file called NameofISO-CHECKSUM:
sha256sum -c *CHECKSUM
OpenSuse is similar - the hash is in a file called NameofISO.iso.sha256:

Ubuntu - has sha sums as well as  md5sums, so can use
md5sum -c MD5SUMS 
also.

Then, writing - for Ubuntu based distros, can use the "make startup disk" tool. If the iso is from a different distro, and it is known to be compatible with usb boot, then something like (after verifying the device name - sdd or whatever - very important!)

sudo umount /dev/sdd*
sudo dd if=Fedora-Workstation-Live-x86_64-24-1.2.iso of=/dev/sdd bs=8M status=progress oflag=direct

The above method is known to work for Fedora 24 to 29, and OpenSuse 13.2.

If it is a DVD iso and not meant for USB boot, maybe unetbootin is required. 

Friday, March 15, 2019

burning in subtitles with VLC

From https://www.youtube.com/watch?v=G0NzMlhnzQQ -

Burning in subtitles with VLC needs a bit of tweaking -

1. Media -> Stream, choose the video file. Don't enable the subtitle checkbox on that screen. Click the Stream button.
2. In the Stream Media Wizard, choose the destination as File. Click Add and choose a filename for the destination.
3. In the next Wizard screen, make sure activate transcoding is checked. Click on the settings button, in the subtitles tab, make sure the 'enable subtitles', and 'overlay subtitles on the video' are checked. Codec can remain T.140.
4. In the next screen, in the Generated Stream Output String, we need to edit out (delete) the part of the string which says scodec=t140,
5. Click stream to transcode. 

Saturday, March 02, 2019

F&D smart TV - removing the ads from home screen

We wanted the F&D smart TVs in our lobby to start up with "last selected input" instead of the Home screen with all the youtube "recommendations" or ads. I tried out
Setting -> Common -> System Recovery

(To be updated.)

For Sony TVs, https://www.sony.com/electronics/support/articles/00257833 - Quick settings - Power on behaviour - Last input

There are many posts about using third-party home screen apps etc, but I'd rather use "Last input". Will check on this later. 

Monday, February 25, 2019

Sunday, February 24, 2019

easy way to copy ssh keys to remote machines

ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub user@host


This copies the public key to the authorized_keys file on the remote machine.

Saturday, February 23, 2019

stabilizing cycling videos

In the process of making videos of my morning cycling jaunts, wanted to stabilize the video and also speed it up. Found that Microsoft Hyperlapse Mobile was doing a reasonable job, though it is limited to saving in 720p on my LG Q6 mobile. Very blocky output (probably a low-bitrate codec?) and the video goes blurry once in a while, but stability is good. Compared to kdenlive's vidstab stabilization in the video below.

Bugs and workarounds -

1. My phone doesn't show all the videos all the time, either to Hyperlapse when importing video, or via MTP to Linux Mint 18.1 on USB for taking backups. Workaround, as suggested here, was to just restart the phone.

2. Initially, I was working with kdenlive 15 which came with Linux Mint 18.1, and faced the problem of tooltips being unreadable, having white text on yellow background as mentioned in this post. Then I started using the version 18.12 appimage - that does not have this issue, and also is less prone to random crashes.

3. My equipment is the LG Q6 mobile mounted on the handlebar using a cheap mount from Amazon,


The bracket is not tight enough, and I keep needing to push it up 5-6 times per ride (if the ride is not very smooth) to prevent the phone from tilting down and capturing only the ground right in front of me. Maybe a different way of mounting the phone may solve this - will try out different orientations. 

Time taken for processing

Hyperlapse took around 10 minutes per video to import, and 2 minutes to export as 4x sped up 720p video. Input videos were 1080p 30 fps captured by OpenCamera with the phone's default bitrate and codec, 17 Mbps mp4, file size limited to 1 GB. The 1 GB had around 8 minutes of footage. So, overall speed of Hyperlapse was 12 minutes for 8 minutes of footage, 1.5x real-time.

Kdenlive + vidstab was much slower. Around 2.5x real-time for just rendering mp4 to mp4. This is at a higher bitrate, but on an i7 Macbook. And around 50 minutes for an 8 minute clip stabilization. So, for speeding up 4x and then stabilizing, it would take around 35 minutes in a two-step process, or 4.4x real-time.  

So, I'm going with Hyperlapse. Maybe my colleagues with better stabilization software would prefer the raw footage, so I'm archiving the raw footage separately. 

Finally, here's the comparison video






Friday, February 22, 2019

options when running out of space in legacy GSuite free

Copy-pasting from an email I wrote to a colleague - Regarding email - currently, our domain uses GSuite legacy free plan and does not allow expansion of the existing amount of space. So, the only way is to delete large attachments. In many cases, the attachments are required and cannot be deleted. This can be handled in many ways.


1. Forward them themewise to specially created gmail ids and then delete
2. archive them locally by storing on a local hard disk and then delete
3. automate the local archiving using some tool like

Wednesday, February 20, 2019

videos for waiting area

We did a first test of a video screening solution for our waiting area at the planetarium. The idea is to have videos playing on the screen, with some onscreen text indicating the probable time of the next show. Raspberry Pis to play out on old surplus VGA or TV screens. After a bit of trial and error, the method used was to play videos using VLC called from a script,
cvlc -I http --http-port 43210 --http-password MyPassword  --fullscreen --no-video-title --sub-source="marq{file=/pathto/marq.txt,position=10}" /home/pi/playlist.xspf

Documentation for the marq subfilter is at https://wiki.videolan.org/Documentation:Modules/marq
and the idea came from this discussion.

Auto starting using a .desktop file put in .config/autostart directory as described at
https://www.raspberrypi.org/forums/viewtopic.php?t=17051
with the contents of the .desktop file being
[Desktop Entry] 
Type=Application
Exec=/path/to/my/shellscript.sh

Modifying the marq.txt is currently by ssh, need to make some better solution, perhaps using a web server and PHP or something like that. Some images below of our test setup -


Saturday, February 16, 2019

making a flower timelapse

Timelapses shot on 13 and 16 Feb 2019
using OpenCamera app on an LG Q6 mobile phone.

Settings:
Repeat - Unlimited
Repeat Mode Interval - 10 seconds.

So, playback at 30 fps made it 300x real-time.

Imported into kdenlive as an image sequence and then exported with pan and zoom effect applied.
Bug and workaround - kdenlive 15.12.3 which comes with Linux Mint 18.1 has an issue with importing image sequences when the numbering is not sequential, or jumps. For example, the image sequence shot by OpenCamera had filenames like
Although kdenlive would see all 463 files in the folder, the images after 85953 would be ignored, and 85953 would be repeated for those frames. Workaround was to put the 8xxx sequence in one folder, 9xxx sequence in a separate folder, etc, and then import as image sequence.

kdenlive 18.12.1 64 bit appimage seems to have mitigated this bug by just importing those frames in the sequence it recognizes, without causing any of the repeat frames. But in the clip properties, it still erroneously shows the larger number of frames found etc. So, the workaround above is recommended.


https://www.youtube.com/watch?v=tD0L7H1jbWo




Thursday, February 14, 2019

TVS MSP345 printer with Epson drivers

Using the TVS MSP345 printer with Epson 24-pin generic drivers on Linux, we need to use the resolution 360x180 - lower than that, the print quality is terrible, and higher than that, it just prints the print codes.

Monday, February 11, 2019

sending and receiving emails with domain aliases on GSuite

One of our units had two domains, longname.tld and shortname.tld - they asked if they would receive emails sent to user@shortname.tld which had been set up as a domain alias in GSuite. My reply - 

The above holds good for receiving emails from username@shortname.tld

If you wish to send emails from username@shortname.tld (as against username@longname.tld), you need to follow the procedure below by logging on to username@longname.tld 


  1. On your computer, open Gmail logged in as username@longname.tld .
  2. In the top right, click Settings Settings and then Settings.
  3. Click the Accounts and import or Accounts tab.
  4. In the "Send mail as" section, click Add another email address.
  5. There, enter username@sssvv.org
You also have an option to make whichever email you wish to have as the default sending email, just below that section.

Tuesday, February 05, 2019

modus operandi for processing audio files for rebroadcast

Made this list for the people at Studio, since I would no longer be processing mandir audio for rebroadcast -

1. Listen to the whole thing (if feasible at 2x speed, or at mandir itself, while the program is going on), mark problematic parts with markers.
2. If the program is a drama that contains copyrighted songs, edit out those songs within only 10-20 seconds.
3. If the program is of poor quality, like songs which are completely out of tune, do not process for broadcast.
4. If the program has speeches which are not of general interest, or which are not suitable for broadcast due to poor delivery, do not process for broadcast. If the speech is good, but marred by repetitive words like Ummmm / Aaaah / "You know" / "Like" etc, edit those out. Also edit out any gaps like speaker turning pages, speaker drinking water, etc. 
5. If it is a music program, try to cut into individual songs/announcements if possible. Filenames similar to historical filenames which you can see under MSV* and SV* in our local cdsearch page.
6. For speeches, I have been processing to remove noise as per this:
http://hnsws.blogspot.com/2010/01/tips-for-clean-audio.html or
http://hnsws.blogspot.com/2013/11/another-audio-cleanup-tool-chain-reaper.html

7. I have been skipping processing Telugu dramas for AsiaStream once TeluguStream started, so that if the Telugu team is interested, they can use for TeluguStream. Similarly now with Hindi team. 

Sunday, January 27, 2019

paying income tax on tax demand

I had made a mistake in income tax calculation and payment, and in the intimation sent by the Income Tax department, there was a demand to pay Rs. 230 as tax. The clickable link on the pdf did not work for me (since I was using Linux) with pdf readers Xreader and Google Chrome. So, googled and got the website
where there is a Pay Taxes online link.

The next links were deciphered using the Challan included in the pdf. Challan No. 280, so the relevant link for 280. Address as per PAN. Details of payment as per the Challan again, (no surcharges or penalties for me.) And as soon as the confirmation link is clicked, we get a downloadable receipt. Done. Painless.

Friday, January 11, 2019

malware/adware on our home page with BSNL router

When I opened our home page now from my room, I got an advertisement popup. 

Whichever link on our website I click, a pop-up appears.

No such popup on other sites. No such popup if I click on direct links, popups only if I click on javascript links like on our home page.

Also, no such popups when I use my phone as a hotspot and don't use the BSNL router.

There was recently an email from a user who also experienced this.

According to this blog post, this is some issue with BSNL?
As mentioned by this person, I put off and put on the BSNL router (reset the router), and the popups stopped.

Apparently, this malware/adware is being injected into any and all non-encrypted traffic. So, a good solution would be to upgrade to https.