Work Stuff

Sunday, January 12, 2020

warping on Linux - filter on avisynth and virtualdub - OK

While exploring ffmpeg and the Remap filter for warping videos, noticed that this filter is not GPU optimized, it does not use OpenGL. So, if avisynth works on Linux, even the avisynth filter made for warping should do the job just as well.
http://www.avisynth.nl/users/vcmohan/Planetarium/Planetarium.html
http://www.avisynth.nl/users/vcmohan/index.html

For this, would need to get avisynth working on Linux under wine.

This link
https://forums.opensuse.org/showthread.php/429428-howto-install-virtualdub-under-wine-deshaker-plugin
gave hope for getting Virtualdub working under Wine.

In my case, I downloaded
VirtualDub-1.10.4,
ffdshow_beta7_rev3154_20091209.exe
(the latest one, ffdshow_rev4532_20140717_clsid did not work)
Installed winetricks with apt and mfc42 using winetricks
vcredist_x86.exe
Xvid-1.3.2-20110601 (Xvid-1.3.5-20171208 did not work)
installing ffdshow, the vcredist and xvid using
wine ffdshow_beta7_rev3154_20091209.exe
and so on. Mostly choosing the defaults. In my case, running the setup for vfw, video and audio after installing ffdshow, I ran into errors in the MS runtimes. But things seem to work anyway.

Avisynth works, except that DirectShowSource doesn't work, we need to use FfMpegSource.

Avisynth can be used with mencoder

https://ubuntuforums.org/archive/index.php/t-1333264.html

and avs2yuv may be an easier way to do piping.

https://web.archive.org/web/20060430184405/http://students.washington.edu/lorenm/src/avisynth/avs2yuv/avs2yuv.exe

"There is no function called FFMS2" - found that I was running Avisynth 2.5, while the plugin was was Avisynth 2.6 - "This is not an avisynth 2.5 plugin".

https://github.com/FFMS/ffms2/issues/278

Had to install older version, 2.1 of ffms2 - trying 2.18rc1 worked. Installation is by copying the dll to the plugins folder.

Syntax for old version is different from the latest version -

Loadplugin("/home/myusername/.wine/drive_c/Program Files/AviSynth 2.5/plugins/ffms2.dll")
FFvideosource("in.mp4")

If I make any changes to codec settings inside Virtualdub, then rendering crashes. Must close and open vdub again. Also, need to set DirectX rendering Off, otherwise does not proceed beyond the first few frames, and garbage is rendered. This is in VirtualDub Options -> Preferences

Also, vdub on Wine does not work as frameserver.

My machine renders Xvid videos with simple profile, as needed by my BluRay player, at around 20 fps, and drops to 8 fps when using the avisynth warping filter Planetarium.dll. The rendering only works for me with the ffvfw MPEG-4 codec and not with the Xvid codec - probably some installation glitch. Settings for simple profile and high quality like the screenshot below.

Testing of avisynth scripts is easily done in Virtualdub. As in the link mentioned above, can explore mencoder and or other rendering options using pipes.

Saturday, January 11, 2020

Blender Fulldome plugin and OSL

A couple of links -

someone trying out Open Shading Language (OSL) scripts with 360 images and fulldome - https://blenderartists.org/t/osl-for-fulldome/570991

and a Blender plugin to "quickly create and preview domemasters for planetariums" - https://github.com/johnroper100/Fulldome-Pro

Friday, January 10, 2020

Gui to encapsulate the OCT software

A possible way to encapsulate the OCT software with a GUI on Windows might be using pywinauto -
https://github.com/pywinauto/pywinauto#ms-ui-automation-example

Wednesday, January 08, 2020

faster rendering of kdenlive projects with melt

I had made a 107 minute video with kdenlive - exporting slides from a Google Slides presentation, and syncing with audio. While rendering it, found this link,
https://amp.reddit.com/r/commandline/comments/7mpiox/the_fantastic_melt_package_and_rendering_a/

melt /path/to/project.kdenlive -consumer avformat:output.mp4 vcodec=libx264 b=5000k acodec=aac ab=128k
went at 1.6x - 3600 frames in 72 sec
375% CPU with top. The entire render took approx 75 min.

Kdenlive was going at 0.6x - the 107 minute video showed 2:45, i.e. 165 minutes to complete.
250% CPU with top.

So, rendering with melt is more than twice as fast as rendering in kdenlive. This was with kdenlive-18.12.1b-x86_64.appimage and melt -version shows 6.0.0.

Tuesday, January 07, 2020

working with multi-track single wav files on Reaper and creating 'minus' tracks

To split a single multitrack file into separate mono tracks in Reaper (at least in Reaper 5), menu item
Item -> Item processing -> Explode multichannel item to one channel items

Then, single tracks can be solo'd till we hear only music, if the narration track is localized to one or two tracks. Muting does not seem to work, but solo seems to work. Then the file can be rendered into a 'minus' track, without the narrator.

Monday, January 06, 2020

enabling https on apache

Wanted to make our websites running Apache to be SSL enabled. Since we use Cloudflare, and Cloudflare has an option by which we can use self-signed certificates on the origin web-server, decided to try self-signed certificates first.

Followed this tutorial on digitalocean, which is concise, complete and to the point.

1. Our current servers use su and not sudo.

2. Creating a key pair with

openssl req -x509 -nodes -days 365 -newkey rsa:2048 
-keyout /etc/ssl/private/apache-selfsigned.key 
-out /etc/ssl/certs/apache-selfsigned.crt

3. Creating ssl-params.conf in /etc/apache2/conf-available with the following text:

SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder On
# Disable preloading HSTS for now.  You can use the commented out header line that includes
# the "preload" directive if you understand the implications.
# Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
Header always set X-Frame-Options DENY
Header always set X-Content-Type-Options nosniff
# Requires Apache >= 2.4
SSLCompression off
SSLUseStapling on
SSLStaplingCache "shmcb:logs/stapling-cache(150000)"
# Requires Apache >= 2.4.11, so commented out
#SSLSessionTickets Off

4. Modifying /etc/apache2/sites-available/default-ssl.conf by adding the correct contact info, and the correct key-pair paths. In our case, I had to copy the directives in default.conf, since there were a lot of changes to be done.

5. For finding Apache version,

apache2 -version

and enabling all the modules, configs and sites by

a2enmod ssl

a2enmod headers

a2ensite default-ssl

a2enconf ssl-params

apache2ctl configtest

service apache2 restart

6. In our case, cloudflare could do the https redirecting. But redirecting all traffic to https breaks our pages which have POST links, since those are not redirected. (Edit 21 Feb - after the POST links were updated to https, turned "Always use HTTPS" on. No issues.)

7. Found that this procedure can reasonably be easily replicated with a certificate from letsencrypt -
https://letsencrypt.org/getting-started/
https://certbot.eff.org/
In our case, perhaps doing a cert only cron job with certbot would be desirable.

certbot-auto certonly --server https://acme-v02.api.letsencrypt.org/directory --manual 
--preferred-challenges dns -d 'ourdomain.tld,*.ourdomain.tld'

I did try it out manually. For automating it, we would need to put the Cloudflare API key on the server, or the newer token with more limited powers, for automatically updating the DNS TXT record used for validation by certbot and letsencrypt. Or perhaps even the simpler option given at the certbot website might be enough.
certbot-auto certonly --apache

In this case, it takes a bit of time, and asks which domain we want the certificate for. After that, it works, with http authentication. May need some tweaking to automate this, since there are multiple domains involved. But renewal seems to be simple. As the closing notes say,

Performing the following challenges:
http-01 challenge for our.domain.tld
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/our.domain.tld/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/our.domain.tld/privkey.pem
   Your cert will expire on 2020-04-05. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot-auto
   again. To non-interactively renew *all* of your certificates, run
   "certbot-auto renew"

Thursday, January 02, 2020

Reset Raspberry Pi password

Forgot password - reset it by booting into single user mode as given at http://mapledyne.com/ideas/2015/8/4/reset-lost-admin-password-for-raspberry-pi

Take out the SD card after powering down the Pi
Edit the cmdline.txt and add to the end init=/bin/sh
Once the Pi boots into single user mode, just su to become root, no password, and then do passwd pi to reset the pi user's password.
Then again take out the SD card after powering down, remove the init phrase from cmdline.txt

The link above also has troubleshooting tips like mount -o remount,rw /dev/mmcblk0p2 / etc. Also, if root pw is set, , replace contents of /etc/shadow with * to make it blank as above.

SD card was read-only with the laptop's built in reader. If an external card-reader was used, it could be written to. The issue was something else, not a chown issue or a hardware switch as in the following: https://ubuntuforums.org/showthread.php?t=2400119

Also, changed the auto login by editing /etc/lightdm/lightdm.conf

Raspberry Pi wifi not working for new Pi 3B+

Apparently the wifi is disabled for regulatory reasons till country code is set - https://www.raspberrypi.org/documentation/configuration/wireless/desktop.md - "To set the country code, open the Raspberry Pi Configuration application from the Preferences Menu, select Localisation and set the appropriate code."

and then it probably needs a restart.

Tuesday, November 19, 2019

brief review of machine transcription - old recording to text

Tried out automated speech recognition from Google. Though there is a lot of improvement over the last couple of decades, still a long way to go.

1. Google's paid product is priced at around Rs. 100 per hour, with the first hour free. https://cloud.google.com/speech-to-text/
Google's version first:

Followed by my transcript.

You can see that it has made quite a few mistakes, and more importantly, missing entire sections. The skipping of sections would be the big no-no for me. (I take around 3 minutes to transcribe one minute of English from this old recording).

2. Using Google Docs "voice typing" free feature -

Tuesday, November 12, 2019

Node.js related link dumps

VulcanJS - Install the latest (LTS) node.js using nvm as given at https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-16-04

Apache and node - https://www.ionos.com/community/server-cloud-infrastructure/nodejs/set-up-a-nodejs-app-for-a-website-with-apache-on-ubuntu-1604/

https://forums.meteor.com/t/npm-bcrypt-and-meteor-npm-bcrypt-madness/28714/21

trying

meteor add npm-bcrypt
meteor add accounts-password

Did not work, abandoning.

Remove node - https://stackoverflow.com/questions/47240567/uninstalling-the-currently-active-version-of-node-js-with-nvm/47251263

Monday, November 11, 2019

some simple light-weight webservers

On linux - lighttpd is available widely, for example in ubuntu's repos. Some examples given at https://unix.stackexchange.com/questions/32182/simple-command-line-http-server are SimpleHTTPServer in Python, node.js (lightweight compared to apache) and many many others. And for windows, this page lists tinyweb which can even run cgi.

trial embed of audio and pdf

Testing an embedded audio file with two embedded pdfs. All hosted on Google Drive, with embed code obtained from More items -> Open in New Window -> More items -> Embed.

Indic input enabled using Google Acccount Settings -> Language -> Input Tools -> Edit
https://www.google.com/inputtools/services/products/account-central.html
Then choose File -> Language to choose the required language within Google Docs.

warping on Linux - initial ffmpeg trials without a final solution

I still use the tool made in 2014 to pre-warp fulldome planetarium shows for our planetarium's Blu-ray-player-based playout system. But that tool's main limitation seems to be that it works only on Windows XP. It does not seem to work even on Windows 7 - I tested with RS's laptop.

In the search for alternatives for future-proofing, I had a look at ffmpeg's Remap filter. If a suitable map is made, it should be possible to use this filter to do the warping in a platform independent way.

https://trac.ffmpeg.org/wiki/RemapFilter

https://stackoverflow.com/questions/30832248/is-there-a-way-to-remove-gopro-fisheye-using-ffmpeg

http://paulbourke.net/miscellaneous/ffmpeg/

https://stackoverflow.com/questions/38360837/ffmpeg-remap-filter-not-found

ffmpeg -i /path/input.mp4 -i front_kodak_sp360_1440_xmap.pgm -i front_kodak_sp360_1440_ymap.pgm -lavfi remap out.mp4

Preliminary test went at around 8 fps, 1 minute video took 4 minutes to encode.

out.mp4 was of a similar format to input file, but not exactly.

mediainfo out.mp4

Testing to see if opengl will be faster, tried with just re-encoding to divx,

ffmpeg -i /path/input.mp4 -c:v libxvid out.mp4

did the 1 minute video in 2 minutes, but quite low quality for the highly changing scenes. That would be 15 fps.

ffmpeg -i /path/input.mp4 -c:v libxvid -qscale:v 100 out.mp4

did not make much of a difference.

Some info about OpenGL filters with ffmpeg

https://nervous.io/ffmpeg/opengl/2017/05/15/ffmpeg-pbo-yuv/

https://nervous.io/ffmpeg/opengl/2017/01/31/ffmpeg-opengl/

In case conversion has a problem,

https://stackoverflow.com/questions/53357909/ffmpeg-remap-with-video

Piping to screen works, albeit not in realtime, 0.25x perhaps.

ffmpeg -i /path/input.mp4 -i front_kodak_sp360_1440_xmap.pgm -i front_kodak_sp360_1440_ymap.pgm -lavfi remap -f matroska - | ffplay -

inspired by

https://superuser.com/questions/322216/how-can-i-pipe-output-of-ffmpeg-to-ffplay

https://superuser.com/questions/685022/how-can-i-pipe-data-losslessly-to-and-from-ffmpeg

Using rawvideo, (raw RGB is not supported in matroshka, so using NUT), went to 0.68x, but in black and white!

ffmpeg -i /path/input.mp4 -i front_kodak_sp360_1440_xmap.pgm -i front_kodak_sp360_1440_ymap.pgm -lavfi remap -c:v rawvideo -f NUT - | ffplay -

with AVI instead of NUT, colours are mismatched and not antialiased?!

Later, went to try avisynth instead. That is described in another post.

Friday, November 08, 2019

shared folder new behaviour

Now when I move something out of a shared windows folder from our Linux server, I get this message:

mv: listing attributes of ‘THE_FILE_NAME.mp3’: Numerical result out of range

After the file is moved to a local drive on the Linux server, moving it elsewhere doesn't give this error. Maybe this behaviour is something to do with this,

https://wiki.archlinux.org/index.php/File_permissions_and_attributes

"mv silently discards extended attributes when the target file system does not support them."

Edit: Most probably this message is due to group name or user name not being mapped to GID or UID properly, I guess - as discussed at https://serverfault.com/questions/811997/mounting-a-cifs-share-fails-with-numerical-result-out-of-range

Thursday, November 07, 2019

link dump - possible ways to port to GL_warp2Avi

Exploring ways to port my old tool GL_warp2Avi to cross-platform code -

https://stackoverflow.com/questions/6495523/ffmpeg-video-to-opengl-texture

https://github.com/datenwolf/aveasy

https://eklitzke.org/rendering-videos-from-opengl

https://ffmpeg.org/ffmpeg.html#Advanced-Video-options

https://web.archive.org/web/20060212173939/http://nehe.gamedev.net/data/lessons/lesson.asp?lesson=35

links to Linux version, which uses avifile >=0.60

https://web.archive.org/web/20060212173939/http://nehe.gamedev.net/data/lessons/linux/lesson35.tar.gz

Edit - a few months later, I've created GL_warp2mp4 using OpenCV and GLUT and the generalized OCVWarp based on OpenCV which can do what GL_warp2Avi could do.

virtual tours with 360 photos, and panoramas

Thought about creating a virtual tour of the planetarium with the Insta360 Air camera. This camera works reasonably well in good light, but cannot capture in low light as is the case inside the planetarium theatre - severely pixelated with the lights on, and completely dark when trying to capture the live planetarium show.

Saw some tutorials using cupix and google tour creator, then made one with google tour creator,
https://poly.google.com/view/6fr4ibdUbv0

Added the same images (after resizing to 4K) to google street view using the street view app. To change the directions in which the 360 images are pointed, under Edit location, the photo can be rotated by swiping while keeping another finger on the directions (N, NE, E etc.) shown below, to keep the directions steady. A bit fiddly. Positioning was also a bit fiddly, since the zoom level on the positioning map was not sufficient in my case.

Made a couple more panoramas, both using the Panorama feature in the stock camera app on my LG Q6 Android phone and using the street view app. The street view app tries to stitch it into a 360 image, so requires a very large number of photos.

Shot with the Google Street View App, which needs you to align the phone to a shown circle and click repeatedly.

Shot with the Panorama feature of the phone's camera App, which needs you to just wave your phone from left to right or vice versa.

Also shot with the Panorama feature on the phone camera. A passerby created an artifact, so repeated below.

stackoverflow gyaan on when to use GPU

https://dsp.stackexchange.com/questions/37013/good-heuristic-point-where-sending-fft-to-gpu-is-computationally-worth-it

https://stackoverflow.com/questions/8158632/is-it-worth-offloading-fft-computation-to-an-embedded-gpu

getting a human being with HDFC phonebanking

It seems a bit difficult to get a human being using the HDFC bank tollfree number. One possible workaround is - choose the "Loans" option. If that option does not appear on the first menu, try the "Report stolen card" option, and when they ask for confirmation, choose "2 for any other services" and then "Loans". After a human being answers, you could tell them what you want, and if they say that they need to transfer your call, agree.

Thursday, October 31, 2019

GPU and CUDA explorations with OpenCV

Successfully installed the Zotac GeForce GTX 1050 Ti graphics card in the Dell Optiplex 3020 desktop.
Installed CUDA SDK using web installer, https://developer.nvidia.com/cuda-downloads
Tried following https://jamesbowley.co.uk/build-opencv-4-0-0-with-cuda-10-0-and-intel-mkl-tbb-in-windows/ for building OpenCV. Configure kept failing, saying fseeko not found. So, decided to use the pre-compiled library from https://jamesbowley.co.uk/downloads/
https://jamesbowley.co.uk/opencv-3-4-gpu-cuda-performance-comparison-nvidia-vs-intel/ gives an interesting test comparison.
For comparing my code, which reads from disk and then does computations for 50 images, tried a test with the computation part commented out.

Bscancompute

While loop 9.01159 sec.

While loop 12.5376 sec.

While loop 13.7008 sec.

2nd run

While loop 9.03921 sec.

While loop 8.88572 sec.

While loop 8.32227 sec.

Test wo FFT,

Before while loop 0.0226007 sec.

While loop 3.82644 sec.

While loop 3.6609 sec.

While loop 3.07903 sec.

A simple sample program, to perform a 2D DFT and inverse, gave the following results on running more than once -
samplecudadft,

480x360 fft and inv fft

DFT and inverse, with upload/dl 0.255261 sec.

(after running several times).

2400x360,

DFT and inverse, with upload/dl 0.261917 sec.

CPU

E:\OCT\opencvcuda\OpencvCuda\x64\Release>samplecudadft
DFT and inverse, on CPU 0.0144597 sec.
DFT and inverse, with upload/dl 0.260429 sec.
bottleneck seems to be FFT "planning" and not upload download.

E:\OCT\opencvcuda\OpencvCuda\x64\Release>samplecudadftDFT and inverse, on CPU 0.0146046 sec.
DFT and inverse, with upload/dl 0.263986 sec.
DFT and inverse, without upload/dl 0.260521 sec.

and running it a 2nd time,

DFT and inverse, 2nd time without upload/dl 0.00618453 sec.
Even when called as a function, quite fast after the first time.

DFT and inverse, as a function 0.00678208 sec.
DFT and inverse, as a function 0.00605592 sec.
DFT and inverse, as a function 0.00654344 sec.
DFT and inverse, as a function 0.00608254 sec.
DFT and inverse, as a function 0.00651554 sec.
DFT and inverse, as a function 0.00623649 sec.
DFT and inverse, as a function 0.00607195 sec.
DFT and inverse, as a function 0.0063154 sec.
DFT and inverse, as a function 0.00597894 sec.
DFT and inverse, as a function 0.00685649 sec.
DFT and inverse, as a function 0.00608895 sec.
Will probably need to optimize based on

https://docs.opencv.org/master/dd/d3d/tutorial_gpu_basics_similarity.html
Currently, with upload / download and variable assignment not optimized,
on cpu,

While loop 9.15065 sec.

on gpu,

While loop 9.34765 sec.

where variables were initialized, 64 to 32 conversion and back were included.

More important, the Bscan image shows up a bug.

Friday, October 25, 2019

a workflow for equal loudness RMS normalization on Adobe Audition

Sat with the Telugu team and worked out the following procedure for increasing and standardizing the RMS volume of their audio to -12 dB in Adobe Audition.

Match the volumes in the relevant regions to around -12 dB peak.
Do Noise Reduction by selecting a region of audio without noise, and then using that as the noise to be removed, around 70% or 80%.
Noise Gate at -60 dB - the threshold to be tweaked depending on audio's background noise level.
Choose either L or R track and mute the other track if relevant audio is only on one track, then convert to Mono.
Multiband compression using the Broadcast preset, and decreasing the thresholds by -20 dB, with Brickwall On.
Reverb filter if required.
Total RMS to -18 dB.
Hard Limiter at -6 dB.
Normalize peaks to 100%.

Tuesday, October 22, 2019

UTF-16 LE with BOM and compiler errors, warning: null character(s) ignored

By some fluke, the source code which I had edited on Windows got saved on github as UTF-16 LE with BOM (as reported by Geany). This was not compiling on Linux, which gave lots of warnings,

warning: null character(s) ignored

Googling got the UTF to UTF convertor on www.fileformat.info which converted the file properly to UTF-8. Problem solved.

Saturday, October 12, 2019

clearing up GMail used space

Change of 16.21 GB to 16.11 GB by deleting 8893 conversations from Trash, most of them short text-only email notifications.

100 MB for 10,000 short text emails, or around 10 kB per email.

So, probably it does not make sense to delete small emails - deleting attachments would clear up much more space.

Monday, October 07, 2019

X windows issues after CUDA install, resolved

Since my machine is supposed to have a GeForce GT 330M according to lspci, I thought I could try out the old version of CUDA SDK - apparently version 6.5 is the last one to support the GT 330M. The deb was supposed to be for Ubuntu 14.04, and I was running Linux Mint 18.1 based on Ubuntu 16.04. The deb and the apt-get install ran without any complaints. But ...

The build of examples worked, but the the cuda compatible nvidia card was not detected. And, after a restart, X did not come up.

Possible culprit was the NVidia driver for the older 14.04 kernel, which probably needed to be removed and so on. After some googling and some trial and error, the following method worked -

sudo apt purge nvidia*

sudo apt-get update

sudo apt-get autoremove

sudo apt-get upgrade

sudo apt-get install --reinstall xserver-xorg-core

sudo apt-get install --reinstall xserver-xorg-video-nouveau

and a reboot.

Sunday, September 22, 2019

WhatsApp notification issue and solution

On my LG Q6 phone, running Android 8, there was a problem of WhatsApp notifications arriving only after I opened WhatsApp. WhatsApp voice call notifications were also not coming through. After some digging, found the setting which made the difference - under Battery -> Power saving exclusions, allowed WhatsApp as an exclusion. Problem seems to be solved.

Sunday, September 15, 2019

ways to increase OpenCV fps

One way to increase the fps of the OCT software might be via multi-threading - using a separate thread to poll the camera as in
https://www.pyimagesearch.com/2015/12/21/increasing-webcam-fps-with-python-and-opencv/

Another way might be the cufft library?
https://www.researchgate.net/post/How_efficient_are_the_parallel_2D_FFT_implementations

Saturday, September 14, 2019

tutorial on 360 to dome with Vuo on Mac

In the second half of this tutorial, Paul talks about warping 360 videos to dome -http://paulbourke.net/dome/vuo/

Edit: Can now use OCVWarp and do this ...

Thursday, September 12, 2019

system program problem

Popup on startup - System Program Problem detected. The solution was to clear /var/crash as per this post.

Saturday, September 07, 2019

increasing font size in Octave legend

Some figures created in Octave had very small font sizes when added to the LaTEX manuscript. Trying to fix the text with Inkscape led to kerning issues similar to this post. So, found these solutions at
https://stackoverflow.com/questions/1532355/increase-font-size-in-octave-legend

Used commands like
print ("j0exp.eps", "-depsc", "-F:20")

Tuesday, September 03, 2019

starting offline processing in a new thread - C++

Tried various options to begin processing in a new thread for BscanFFT, without locking the UI -

Just
std::system(offlinetoolpath);
// - this causes the BscanFFT program to wait for the offline tool to finish

Then,

pid = fork();

if (pid == 0)

{

// child process

std::system(offlinetoolpath);

}

else if (pid > 0)

{

// parent process

}

gives the error

[xcb] Unknown sequence number while processing queue

[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called

[xcb] Aborting, sorry about that.

BscanFFTspinj.bin: ../../src/xcb_io.c:259: poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed.

Apparently, system specific is safer, http://www.cplusplus.com/forum/beginner/13886/

"Actually, the most important disadvantage of system() is that it introduces security vulnerabilities. Using a system-dependent method (that skips the shell) is always safer.

In Windows, it is CreateProcess(), which is really straight-forward to use.

In POSIX (Linux, etc) it is a combination of fork() and one of the exec() functions.

http://linux.die.net/man/2/fork

http://linux.die.net/man/2/execve

http://linux.die.net/man/3/execv

These are also easy to use, but there are some caveats. Follow the example codes."

execve with the example code gave
(show:9323): Gtk-WARNING **: 10:47:16.601: cannot open display:

https://stackoverflow.com/questions/2437819/gtk-warning-cannot-open-display-when-using-execve-to-launch-a-gtk-progra/2437850#2437850

suggested using execv instead. This worked.

external hard disk prices

From amazon.in, current external USB hard disk prices which I collected for N -

1 TB = Rs. 3500 - per TB cost 3500

2 TB = Rs. 5100 - per TB cost 2550

3 TB = Rs. 6200 - per TB cost 2067

4 TB = Rs. 8100 - per TB cost 2025

The 3TB drives seem to be attractively priced due to the sharp drop in per TB price as compared to 2TB drives.

Monday, September 02, 2019

gimp 16-bit caution

Gimp 2.8 opens 16-bit png files, but saves as 8-bit tif or png. Octave 4.2.2 on the other hand, does write 16-bit png with imwrite. Octave's behaviour depends on shared libraries, too, of course.

On Linux,
file filename.png
gives information about the file, like bit-depth and so on.

Sunday, August 25, 2019

smorgasbord of programming tips and ideas - OpenCV etc

Here is a collection of links for various implementations of OpenCV-based medical imaging software -

CUDA and OpenCV - just need to compile OpenCV on a machine with a Cuda compatible graphics card. My old machines have cards which are only supported by older versions of the CUDA SDK. Can expect a 180% speedup as per some papers.
https://answers.opencv.org/question/62955/cudadft-speed-issues-too-slow/
indicates that the Matlab times were wrong.

https://opencv.org/cuda/
https://developer.nvidia.com/cuda-gpus and legacy gpus at
https://developer.nvidia.com/cuda-legacy-gpus
https://developer.nvidia.com/cuda-toolkit-archive and supported GPUs at
https://en.wikipedia.org/wiki/CUDA#GPUs_supported
A nice class to capture from Spinnaker SDK to OpenCV Mat, running in a separate thread - https://github.molgen.mpg.de/MPIBR/SpinnakerCapture
Writing an OpenCV Mat file to a binary format on disk - Lots of options mentioned at
https://answers.opencv.org/question/6414/cvmat-serialization-to-binary-file/
I had a thought of trying DICOM writing. But it turned out to be quite complicated to write DICOM, since so many parameters need to be initialized.
https://stackoverflow.com/questions/36379637/read-dicom-in-c-and-convert-to-opencv
https://github.com/marcinwol/dcmtk-basic-example
https://itk.org/ITKSoftwareGuide/html/Book2/ITKSoftwareGuide-Book2ch1.html#x7-240001.12
https://www.researchgate.net/post/ITK_vs_OpenCV
Some tips on exporting from Octave to OBJ and STL formats - commonly used in 3D printers - https://alexbjorkholm.wordpress.com/2016/10/07/exporting-3d-objects-from-octave/
Plotting using slice in Matlab -
https://stackoverflow.com/questions/35549733/how-can-i-plot-several-2d-image-in-a-stack-style-in-matlab

Ubuntu 18.04 screen issue for File Open dialog boxes, scrollbars [fixed]

After a new install of Ubuntu 18.04, had a funny problem where Geany File Open would lead to the application being greyed out, as if the dialog box had opened, but no dialog box being visible on screen. Googling found the answer in this thread, https://ubuntuforums.org/showthread.php?t=2395250

Apparently, 18.04 configures 2 screens even when there is only one screen. Fix was to go to Settings and change to one screen.

Still there is a problem with Geany's scrollbars - the scrollbars don't redraw after scrolling, leading to the whole scrollbar becoming an orange colour instead of just the orange scrollbar handle. My initial installation was a minimal Ubuntu desktop. Later, installing unity did not help.

Edit: This post seems to have the solution - installing a newer version of Geany from the ppa solved the scrollbar issue.
sudo add-apt-repository ppa:geany-dev/ppa
sudo apt-get update
sudo apt-get install geany geany-plugins-common

Friday, August 23, 2019

Visual Studio specifics of OpenCV project with Spinnaker SDK

This is linked to my previous post about the QHY -> Spinnaker implementation, the nitty-gritties of Visual Studio project changes required under Windows.

The SDK and examples installed by default under Program Files in C:. If Visual Studio opens the examples there, it complains that the directory is not writable. So, make a copy, including the entire spinnaker directory - not just the src folder, since the include paths in the examples are set as ..\..\include and similarly for libs.
In my case, since I had VS2017, I had to install the VS2015 toolset as mentioned in my previous post, and when opening the solution, choose the "No Upgrade" option for these projects.
Once these steps were done, I could compile and run the examples. But for my project, I had to create a new project, choose the V140 toolset, add libraries, add include and lib paths, and also prevent errors due to my use of sprintf.
One way would have been to make a copy of the common properties file which I had made earlier, make modifications to it, and add that to the project. In this case, I just made specific changes to the Release configuration of this particular project.
Additional include dir in C/C++ -> General, Additional Include Directories would be a better place to add it than in this screenshot.
Linker -> Input -> Additional Dependencies
Additional Library Directories.

I would probably need to copy the dlls manually, or follow the post build steps mentioned at this earlier post.

implementing opencv project with Spinnaker SDK for Point Grey FLIR cameras

This post documents my learning curve with migrating code written for the SDK for QHY cameras to the Spinnaker SDK for Point Grey FLIR cameras.

On both Windows and Linux, I had to make changes to my build platform. On Linux, Ubuntu 18.04 was a supported platform, so I installed 18.04 onto a spare partition. On Windows, I was using Visual Studio 2017, and had to install the VS2015 toolset.
On Linux, even though the spinnaker examples had large and complex makefiles, the sdk had put the include and lib files in the default locations, so I just had to modify my CMakeLists.txt file minimally -
On Windows, there was a much more elaborate series of steps, which I will put in a separate post.
The simpler parts of implementation, like changing exposure time, gain etc could be done in a fairly straight-forward way. Unfortunately, there was no example for continuous capture with live display. ptgrey.com being redirected to flir.com, the examples obtained on googling couldn't be directly found. Finally, found this, which indicated that StreamBuffer handling had to be changed. Looking for this specific information, found the documentation at this page, which mentions its potential use in live view scenarios. Putting that in place, then Begin Acquisition and End Acquisition could be implemented outside the capture loop, and the frame rate improved.

Edit: Here is a dump of the nodes I changed, full code is at https://github.com/hn-88/FDOCT/blob/master/BscanFFTspin.cpp (spinnaker documentation talks about the details of the nodes, and the example code helps).

(Gain is in dB)

In terms of nodes,
AcquisitionControl -> AcquisitionMode -> Continuous
AcquisitionControl -> ExposureAuto -> Off
AcquisitionControl -> ExposureTime -> 1000
AnalogControl -> GainAuto -> Off
AnalogControl -> Gain -> 0.0
AnalogControl -> Gamma -> 1.0
ImageFormatControl -> Width -> 1280
ImageFormatControl -> Height -> 960
ImageFormatControl -> OffsetX -> 0
ImageFormatControl -> OffsetY -> 0
ImageFormatControl -> PixelFormat -> Mono8
ImageFormatControl -> PixelFormat -> Mono16
ImageFormatControl -> AdcBitDepth -> Bit12
ImageFormatControl -> AdcBitDepth -> Bit10
BufferHandlingControl -> StreamBufferHandlingMode -> NewestOnly

Using Displayspin, on my machine, I got
71 fps for Bit12 Mono8
34 fps for Bit12 Mono16 (converted to Mono8 to display on screen)
71 fps for Bit10 Mono8

So, probably we can always use Bit12, since it does not seem to have much of a speed penalty.

And a small note on a mistake I made - initially I got the error
Spinnaker: Can't clear a camera because something still holds a reference to the camera [-1004].
Putting pCam = nullptr; before camList.Clear(); removes this error.

Wednesday, August 21, 2019

IP address, MTU setting on Linux for FLIR Point Grey GigE cameras

It looks like my hardware does not support jumbo frames - https://linuxconfig.org/how-to-enable-jumbo-frames-in-linux

Even after setting up a larger MTU with ifconfig,
https://www.flir.com/support-center/iis/machine-vision/knowledge-base/lost-ethernet-data-packets-on-linux-systems/
the MTU reverted back to 1500. So, probably hardware or driver limited.

Also, Network Manager then complains that the interface is not managed,
https://askubuntu.com/questions/71159/network-manager-says-device-not-managed

But the camera works as long as the IP address is set correctly. According to the knowledge base article at
https://www.flir.in/support-center/iis/machine-vision/knowledge-base/setting-an-ip-address-for-a-gige-camera-to-be-recognized-in-linux/

setting it anywhere in the 169.254.x.y range with a 255.255.0.0 netmask should do the trick - link local address (LLA).

More info about our camera is at
https://www.flir.com/products/blackfly-s-gige/?model=BFS-PGE-16S2M-CS

https://www.flir.com/support-center/iis/machine-vision/knowledge-base/technical-documentation-bfs-gige/

Getting Started Manual which talks about Windows installation
Technical Reference which gives frame rates, pin-outs etc

Edit - found this page which has a good explanation of the bandwidth used, effect of lower MTU (slightly higher CPU usage), multiple GigE cameras and so on.

Sunday, August 18, 2019

Remmina, RDP and sound

For convenience, I was trying out using Sound Forge and other audio editing software on my laptop, after logging in with a Remote Desktop Connection using Remmina as the client. With the default setting, Sound Forge etc would complain that there is no suitable audio output device. Googling, found that RDP has various options - taking sound to remote client, leaving the sound on local machine, or no sound at all. Remmina default was no sound. Tried bringing sound to local device, but that did not work for DirectSound, since I'm on Linux Mint. Changing it to leave at local, SF etc would work properly.

tried ubuntu install from mint

https://help.ubuntu.com/lts/installation-guide/i386/apds04.html

but apt update after chroot seemed to be taking the values from sylvia.

So, simpler to just use a bootable usb to create and install the new ubuntu partition.

Wednesday, August 14, 2019

moving mailing list to google groups

The process begun in Dec 2017 was completed now, in mid-Aug. Transitioning our mailing list with 60k subscribers to google groups. Had to move slowly due to the limits set on directly adding users, 100 per day per group, and only in batches of 10 email ids per addition.

Had asked PB to unconfirm users who have been moved to groups instead of unsubscribe, because unsubscribe will send them a mail saying they have been unsubscribed.

Points to note:
(a) any user you add will get an email
(b) for a strictly mailing list only functionality, remove posting permissions for everyone except owner and manager, under Permissions -> Posting Permissions, like
Inline image 1

Google groups does handle bounces, and it is advisable to remove ids which bounce -
https://productforums.google.com/forum/#!msg/apps/5PCc-uDVowo/S4A4YPnnzXcJ

(Automating signup though google docs or google forms or something would be great. An example is given here,
https://sites.google.com/site/sandorexampleclub/how-to/step-11
PS wrote his own implementation which is on script.google.com as a GSuite script, I think.)

Tuesday, August 13, 2019

easy 3D volumetric rendering with imagej

ImageJ, or its full-featured version Fiji, makes it easy to create volumetric renderings and flythroughs using the 3D Viewer plugin. Plugins are installed by just copying the jar file into the plugins folder, found using locate on my system at ~/.imagej/plugins. Multi-layer tifs or gifs can be used as input. Gimp cannot create multilayer tifs, but it can create gif animations, which are sufficient. Open as layers in Gimp to select multiple layer files, save as animated gif, open in ImageJ.

Inside ImageJ, Image -> Scale allows setting of proper scale. Making one axis longer etc. The Image menu also has Crop, Duplicate, Transform and Zoom. Image -> Adjust has Threshold, Brightness/Contrast, etc. Then, Plugins -> 3D Viewer.

The 3D Viewer plugin has nice recording features, with a built-in 'Record 360 degree rotation' under its View menu. It opens another Movie window, which can then be exported using ImageJ's File -> Save as and choosing Gif.

ezgif.com helps to crop and compress gifs easily.

Monday, August 12, 2019

creating animated gif or video from Octave

My Octave install does not have the campos function, but the latest version seems to have it. So, this could be useful in the future.

This post has some good guidelines - using
set(0, 'defaultfigurevisible', 'off');
to speed up things, and using print to save as png. Then using mencoder etc to create the movie.
mencoder mf://output/*.png -mf w=640:h=480:fps=25:type=png -ovc lavc -lavcopts vcodec=mpeg4:mbd=2:trell -oac copy -o output.avi

Sunday, August 11, 2019

bulk renaming from commandline on linux

Using the rename tool,

https://www.tecmint.com/rename-multiple-files-in-linux/

An example is

rename 's/\.html$/\.php/' *.html

From

https://www.gossland.com/perlcourse/matching/substitution

we see that a normal string substitution would be like

s/dog/cat/

rename -n 's/SV_2019_08_10_PM_SOME_LONG_NAME/SV_2019_08_10_SHORTNAME/' SV*mp3

shows what it will do, and -v will actually do it and also display the result.

Tuesday, July 16, 2019

using DMCTK with CMake on Linux

This stackoverflow post mentions an old version of CMake and making on Windows. More googling led to FindDCMTK on cmake.org so that I could just edit my CMakelists.txt to read

...snip...

find_package(USB-1 REQUIRED)
find_package(OpenCV REQUIRED)
find_package(DCMTK REQUIRED)

include_directories($(USB_1_INCLUDE_DIR))
include_directories($(OpenCV_INCLUDE_DIR))
include_directories($(DCMTK_INCLUDE_DIRS))
... snip ...

target_link_libraries(${FILENAME1} ${LIBUSB_1_LIBRARIES} ${OpenCV_LIBS} ${DCMTK_LIBRARIES} )

... snip ...

That worked nicely, without having to manually find the libraries and so on.

Tuesday, July 02, 2019

mounting a windows smb share and automating file moves

mounted //PATH/FOLDER on a linux machine onto a folder /path/to/mountdir using

sudo mount -t cifs //192.168.ip.address/FOLDER path/to/mountdir -o uid=usernameofowner,username=username,password=xxxxx

A cron job was added to automatically move files every 5 minutes, like
*/5 * * * * mv /path/to/mountdir/LIVE_* /path/to/another/dir/

But that caused problems, generating hundreds of emails to usernameofowner, saying mv: cannot stat ‘/path/to/mountdir/LIVE_*’: No such file or directory whenever there was no file there to move. So, PB modified the cron by adding
> /dev/null 2>&1
which seems to have solved the issue, working well.

Wednesday, June 12, 2019

income tax filing quirks for assessment year 2019-20

This year, I thought of filing ITR1, since I only had short-term capital losses and no gains. But on downloading the pre-filled XML and trying to pre-fill the java utility, it gave an error that the xml is not valid. Then tried ITR2 java utility, with ITR2 prefilled XML. This time it mentioned that the assessment year is wrong. Manually edited 2018 to 2019 in the XML, and it pre-filled successfully.

ITR2 needed Basic Salary and DA. So, I computed it from my salary slips, as a percentage of gross salary.

Unfortunately, the CG (Capital Gains) section was not working properly in the java utility (AY2019-20 version 1.2). The Mutual Fund section's form was not being displayed, only the instructions were being displayed. So, I did not put in my losses to carry forward to the next year.

The tax computation is in the tab Part B TTI. When you click on Save, it does validation. This time, there was also a Submit button, which allows you to directly submit the return from the java utility itself. I did that, and the ITR-V was sent to my email. I chose to e-verify via HDFC bank's netbanking like in this earlier post. The first time I tried, I was logged on to the incometax portal when I started the process, so it did not work. I needed to log out of the incometax portal, and then click on the Income tax e-filing link under Requests in HDFC bank's netbanking. Then it worked.

Wednesday, May 29, 2019

ffmpeg commandline for maximum compatibility

An old Raspberry Pi was not playing some videos. Looked around for a transcoding option with ffmpeg, baseline x264 seemed fine, and this example worked for me -

ffmpeg -i input.avi -c:v libx264 -preset slow -crf 22 -c:a copy -profile:v baseline -level 3.0 output.mp4

by looking at

https://trac.ffmpeg.org/wiki/Encode/H.264

Another post has the 2-pass encoding example, where the bitrate is also specified.

Tuesday, May 21, 2019

videos for waiting area - redux

The original idea of having a marquee which would update as in my previous post turned out to be not very workable. Even if the wifi was not flaky, updating the times etc on a computer in the office would be "too much work" for us in the middle of shows. And the wifi signal was being completely lost due to electrical noise from our solar system exhibit's motor and my room's AC compressor motor. So, the idea was adapted to showing a notice in 3 languages, that the average wait time is 30 minutes, every five minutes or so.

Notice made in Google docs and screenshotted, added an "invert" filter scrolling up-down for visibility.

The next change was to use the "smart TV" media playback for the two flat screens instead of buying Raspberry Pis for them. Since anyway the content was going to be static, did not make sense to buy. Idea suggested by S.

The "smart" TVs would only play a single video file, would need intervention after a single video file was played, would not play playlists. So, strung together all the files into a single file, repeated it for 3 hours so that no intervention would be needed. I had used avidemux, vdub, ffmpeg for doing this, but S more comfortable with VSDC on Windows.

Ffmpeg commandline for making files of known size - first calculate required bitrate by size / duration, using kbits and seconds, then something like

for half an hour 500 MB, = 500 * 1000 * 8 kbits

bitrate = file size / duration

=4000000/1800 = 2222 kbps. Let's take as 2200 kbps.

so, 2200k

ffmpeg -y -i input -c:v libx264 -preset medium -b:v 2200k -pass 1 -c:a libmp3lame -b:a 128k -f mp4 /dev/null && \

ffmpeg -i input -c:v libx264 -preset medium -b:v 2200k -pass 2 -c:a libmp3lame -b:a 128k output.mp4

In the ffmpeg example obtained by googling, it was AAC audio. But on my machine,
Unknown encoder 'libfdk_aac'
So, used libmp3lame instead.

Playing the single file on the Pis had some issues. The older Pi would not play, since it did not support the codec - x264 High Profile, I think. The other Pi had vignetting issues, the 16x9 videos were appearing too small on the 4x3 TVs. So, the Pis play playlists, and the "smart" TVs play single video files. The older Pi does not play VP8. Files downloaded from Youtube via desktop (MPEG4 codec) were fine, downloaded via mobile by S were VP8.

Also learned that the Raspberry Pi overscan controls do not work with VLC in fullscreen mode (on RPi3).

Also learned that the Raspberry Pi (2 and 3) video out pinout of the TRRS connector did not have Sleeve as ground! TRRS = L, R, GND, Video! Had to modify the TRRS to RCA cable we bought, inverting GND and VID for one of the RCAs.

For getting composite video out, 0 is NTSC and 2 is PAL,
https://www.raspberrypi.org/documentation/configuration/config-txt/video.md

if only sdtv_out=2 does not work,

https://bhavyanshu.me/tutorials/force-raspberry-pi-output-to-composite-video-instead-of-hdmi/03/03/2014/

And with the remote for shutting down the Pis, they can be operated completely independent of wifi availability.

Sunday, May 12, 2019

Veracrypt fix

A Veracrypt volume on a removable (NTFS) hard disk was not mounting. Veracrypt was complaining of bad password or bad volume. I thought the password was correct. Finally, did

sudo fdisk -l

to find the device name, which was /dev/sdc1 for the removable drive in this case, and then

sudo ntfsfix /dev/sdc1

It finished in less than a second, noting that it had successfully completed. Then, tried mounting with Veracrypt, success!

After mounting, the encrypted volume was visible in Nemo with an icon to unmount it. But unmounting it with Nemo does not really unmount it, probably because Veracrypt uses su privileges to mount - you have to unmount (or Dismount) it via the Veracrypt interface. Only then would the removable drive become free of processes and unmountable via Nemo.

Thursday, May 02, 2019

Octave memory issues and workaround

I had an Octave script which was reading data from files into a variable, doing some processing, then reading some different data into the same variable, and so on. For many hundreds of MB of data, using the load function. But Octave was running out of memory, and finally I had to
killall -9 octave-gui

The workaround was to process part of the data, save the intermediate data to file, exit Octave, reload Octave, resume processing, and so on. Painful, but it worked.