warping with ffmpeg - and some filter_complex examples

Since OCVWarp uses OpenCV remap() and there does exist a remap filter in ffmpeg also, revisited ffmpeg remap in order to make use of the accelerated NVidia codecs, which ffmpeg supports. Asked ChatGPT to generate python code to create the required map files, debugged the code manually, and put the code at

https://github.com/hn-88/OCVWarp/tree/master/python

Currently it's just for the spherical mirror warping of fulldome content, but could potentially be extended to all of OCVWarp's functionality.

This ffmpeg command using the accelerated hevc_nvenc codec runs at 13 fps on a desktop running NVidia RTX 1060 as against 2.5 fps using OCVWarp saving to avc1 codec out.

ffmpeg -i input.mp4 -i map_x_directp2.pgm -i map_y_directp2.pgm -i weight_alpha_mask.png -filter_complex "
  [0:v]scale=3840:2160[scaled];
  [scaled][1:v][2:v]remap[remapped];
  [3:v]format=gray,scale=3840:2160,colorchannelmixer=rr=1:gg=1:bb=1[mask_rgb];
  [remapped][mask_rgb]blend=all_mode=multiply[out]
" -map "[out]" -map 0:a -c:v hevc_nvenc -preset p5 \

-cq 23 -rc vbr -maxrate 15M -bufsize 26M \
-c:a aac -b:a 128k output.mp4

Can potentially even create a UI using Tkinter in python, which would be cross-platform.

Some notes about the ffmpeg commands copy-pasted from the readme there - 

-ss for the start time should come before the input, and 
-t for the duration should come just before the filter_complex - 

For starting at two minutes and stopping after 10 seconds duration,

ffmpeg -ss 00:02:00 -i input.mp4 \

       -i map_x_directp2.pgm \

       -i map_y_directp2.pgm \

        -i weight_alpha_mask.png \

       -t 10 \

-filter_complex "

 [0:v]scale=3840:2160[scaled];

  [scaled][1:v][2:v]remap[remapped];

  [3:v]format=gray,scale=3840:2160,colorchannelmixer=rr=1:gg=1:bb=1[mask_rgb];

  [remapped][mask_rgb]blend=all_mode=multiply[out]

" -map "[out]" -map 0:a -c:v hevc_nvenc -c:a copy output.mp4

And a problematic video was fixed using an initial crop to 4096x4096 (although avidemux could show it as 4096x4096)

ffmpeg  -i "short_nometadata.mp4"        -i map_x_directp2.pgm        -i map_y_directp2.pgm         -i weight_alpha_mask.png      -filter_complex "[0:v]crop=4096:4096[cropped]; [cropped]scale=3840:2160[scaled]; [scaled][1:v][2:v]remap[remapped];[3:v]format=gray,scale=3840:2160,colorchannelmixer=rr=1:gg=1:bb=1[mask_rgb];

  [remapped][mask_rgb]blend=all_mode=multiply[out]

" -map "[out]" -map 0:a -c:v hevc_nvenc -preset p5 -cq 23 -rc vbr -maxrate 15M -bufsize 26M -c:a aac -b:a 128k short_w2.mp4

But unfortunately, the quality of the warped output is slightly lower than what is seen with OCVWarp - slightly less sharp. Perhaps changing the workflow to first remap and then resize (instead of resize to 3840x2160 first and then remap) might improve things.

Edit - 6 October 2025 - Yes - after doing the remap filter first, at the full resolution of the input video, and then the resizing to output resolution brings up the quality to as good or better than OCVWarp. ("Better" when OCVWarp's default avc1 video codec settings don't deliver as good quality as the hevc_nvenc settings above, I guess.) A separate repo has been created for this, incorporating a Python Tkinter UI,

https://github.com/hn-88/python-ffmpeg-warp/

qemu for arm64 on x86_64 Linux machine

This is just documenting the various points to note, trials, errors etc when I tried to run arm64 on x86_64 using qemu. This post does not have a very satisfactory ending - the VM was far too slow for me to try OpenSpace or OCVWarp or anything else for arm64 tests.

Via bing copilot

https://documentation.ubuntu.com/server/how-to/virtualisation/qemu/index.html#boot-arm64-virtual-machines-on-qemu

https://documentation.ubuntu.com/server/how-to/graphics/gpu-virtualization-with-qemu-kvm/#gpu-virtualization-with-qemu-kvm

general qemu setup 

https://christitus.com/vm-setup-in-linux/

from

https://www.youtube.com/watch?v=Kq849CpGd88

qemu-system-aarch64 \

 -m 2048\

 -cpu max \

 -M virt \

 -nographic \

 -drive if=pflash,format=raw,file=efi.img,readonly=on \

 -drive if=pflash,format=raw,file=varstore.img \

 -drive if=none,file=noble-server-cloudimg-arm64.img,id=hd0 \

 -device virtio-blk-device,drive=hd0 \

 -netdev type=tap,id=net0 \

 -device virtio-net-device,netdev=net0

could not find net0, so tried

without network

---------------

qemu-system-aarch64 \

 -m 2048\

 -cpu max \

 -M virt \

 -nographic \

 -drive if=pflash,format=raw,file=efi.img,readonly=on \

 -drive if=pflash,format=raw,file=varstore.img \

 -drive if=none,file=noble-server-cloudimg-arm64.img,id=hd0 \

 -device virtio-blk-device,drive=hd0 

This went into a loop around 10 minutes in,

[FAILED] Failed to start systemd-resolved.service - Network Name Resolution.

See 'systemctl status systemd-resolved.service' for details.

         Starting systemd-resolved.service - Network Name Resolution...

[FAILED] Failed to start systemd-resolved.service - Network Name Resolution.

See 'systemctl status systemd-resolved.service' for details.

         Starting systemd-resolved.service - Network Name Resolution...

[FAILED] Failed to start systemd-resolved.service - Network Name Resolution.

See 'systemctl status systemd-resolved.service' for details.

         Starting systemd-resolved.service - Network Name Resolution...

[    **] (1 of 2) Job apparmor.service/start running (13min 35s / no limit)

When running virt-manager,

"The emulator may not have search permissions for the path"

First run with sudo and then without?

https://askubuntu.com/questions/722034/permission-error-in-virtual-machine-manager

sudo virt-manager

(say yes when it asks to fix the problem, go to next step, then cancel)

then

virt-manager

That worked.

Chrome remote desktop is an option - 

https://medium.com/@selvamraju007/install-and-launch-ubuntu-22-04-desktop-on-google-cloud-1fba8c0f9585

https://remotedesktop.google.com/?pli=1

But not available for arm64?

Login incorrect

ubuntu login: password[  361.700501] cloud-init[1126]: Cloud-init v. 24.4.1-0ubuntu0~24.04.3 running 'modules:final' at Fri, 09 May 2025 07:42:17 +0000. Up 360.36 seconds.

Password: ci-info: no authorized SSH keys fingerprints found for user ubuntu.

Probably must run

virt-sysprep

https://stackoverflow.com/questions/29137679/login-credentials-of-ubuntu-cloud-server-image

--password ubuntu:password:ubuntu

sets the password ubuntu for the user ubuntu.

virt-sysprep [--options] -a disk.img

https://manpages.ubuntu.com/manpages/trusty/man1/virt-sysprep.1.html

virt-sysprep --password ubuntu:password:ubuntu -a ubuntu2404-arm64.img

sudo apt install libguestfs-tools

virt-customize --password ubuntu:password:ubuntu -a ubuntu2404-arm64.img

virt-customize -a ubuntu2404-arm64.img  --password ubuntu:password:ubuntu

Extra parameter on the command line: ‘ --password’.

Try ‘virt-customize --help’ or consult virt-customize(1) for more information

Needed to run as sudo.

sudo virt-customize -v -x -a ubuntu2404-arm64.img --root-password password:password

Boots up in around 5 minutes.

Increase size of disk

https://computingforgeeks.com/how-to-extend-increase-kvm-virtual-machine-disk-size/

sudo qemu-img resize /home/sssvv/Downloads/QEMU/noble-server-cloudimg-arm64.img +20G

qemu-img: Command not found: resize /home/sssvv/Downloads/QEMU/noble-server-cloudimg-arm64.img +20G

maybe needs converting to raw?

https://bobcares.com/blog/qemu-img-error-resizing-image-invalid-argument/

development for Android / iOS - preliminaries

Seeing that the mobile devices were quite competitive with the desktop's processing power in a previous post, thought about working on porting various tools like OCVWarp to Android and iOS if feasible.

OpenCV Android - had earlier cloned / forked NDK samples.

OpenCV for iOS - samples are at https://github.com/opencv/opencv_for_ios_book_samples

https://docs.opencv.org/4.11.0/d3/dc9/tutorial_table_of_content_ios.html

The binaries are available here 

https://sourceforge.net/projects/opencvlibrary/files/opencv-ios/

only up to OpenCV v3.4.3, but iOS framework seems to be available at

https://sourceforge.net/projects/opencvlibrary/

and

https://github.com/opencv/opencv/releases/tag/4.11.0

To check if swift playgrounds is available on 9th gen ipad - https://www.theverge.com/2021/6/15/22534902/ipad-pro-apple-swift-playgrounds-4-wwdc-2021

To check if example opencv apps can run on ipad.

OCVWarp AppImage won't run on Raspberry Pi

The OCVWarp AppImage compiled on github's ARM64 runner will not run on Raspberry Pi 4, error screenshot below.

But OCVWarp runs when compiled on the Pi itself.

( 

In hindsight, 

This might be a problem with 
(a) /tmp not allowing executable permissions
(b) the ld-linux-aarch64.so not having executable permissions
)

Ubuntu using userland on Android

Trying out UserLAnd - https://userland.tech/features

via https://www.geeksforgeeks.org/linux-on-android/

Rs. 190 to buy pro version.

Downloaded 80 MB, on 5G took less than 2 minutes.

Setting up file system - another 2 minutes, then came up without any browser or remote desktop.

Find ip address from Android settings > Wifi > Connection > scroll down to Details.

Port is 5951, Via

https://github.com/CypherpunkArmory/UserLAnd/issues/1133

xtightvncviewer

VNC pw is found via Filesystem > long press on "apps" > Edit 

sudo apt update

Reading package lists... Error!

W: Unable to read /etc/apt/sources.list - RealFileExists (38: Function not implemented)

sudo apt-get update

works

sudo apt-get upgrade

says 204 packages to be upgraded.

Connection failed, since vnc was being upgraded.

ssh with username=userland did not work over the network on port 22, but worked OK on localhost.

Problems when trying to install a browser - 

Preparing to unpack .../firefox_1%3a1snap1-0ubuntu2_arm64.deb ...

debconf: unable to initialize frontend: Dialog

debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)

debconf: falling back to frontend: Readline

=> Installing the firefox snap

==> Checking connectivity with the snap store

===> System doesn't have a working snapd, skipping

Unpacking firefox (1:1snap1-0ubuntu2) ...

Setting up firefox (1:1snap1-0ubuntu2) ...

update-alternatives: using /usr/bin/firefox to provide /usr/bin/gnome-www-browser (gnome-www-browser) in auto mode

update-alternatives: using /usr/bin/firefox to provide /usr/bin/x-www-browser (x-www-browser) in auto mode

Command '/usr/bin/firefox' requires the firefox snap to be installed.

Please install it with:

snap install firefox

 sudo apt-get install snapd

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

snapd is already the newest version (2.67.1+22.04).

 sudo snap install firefox 

error: cannot communicate with server: Post "http://localhost/v2/snaps/firefox": dial unix /run/snapd.socket: connect: no such file or directory

snap install firefox

error: cannot communicate with server: Post "http://localhost/v2/snaps/firefox": dial unix /run/snapd.socket: connect: no such file or directory

via

https://packages.debian.org/sid/arm64/firefox/download

 wget http://http.us.debian.org/debian/pool/main/f/firefox/firefox_137.0.2-1_arm64.deb

sudo dpkg -i firefox*.deb

Selecting previously unselected package firefox.

(Reading database ... 59497 files and directories currently installed.)

Preparing to unpack firefox_137.0.2-1_arm64.deb ...

Unpacking firefox (137.0.2-1) ...

dpkg: dependency problems prevent configuration of firefox:

 firefox depends on libasound2t64 (>= 1.0.16); however:

  Package libasound2t64 is not installed.

 firefox depends on libatk1.0-0t64 (>= 1.12.4); however:

  Package libatk1.0-0t64 is not installed.

 firefox depends on libc6 (>= 2.39); however:

  Version of libc6:arm64 on system is 2.35-0ubuntu3.9.

 firefox depends on libevent-2.1-7t64 (>= 2.1.8-stable); however:

  Package libevent-2.1-7t64 is not installed.

 firefox depends on libglib2.0-0t64 (>= 2.38.0); however:

  Package libglib2.0-0t64 is not installed.

 firefox depends on libgtk-3-0t64 (>= 3.13.7); however:

  Package libgtk-3-0t64 is not installed.

 firefox depends on libnss3 (>= 2:3.109~); however:

  Version of libnss3:arm64 on system is 2:3.98-0ubuntu0.22.04.2.

 firefox depends on libvpx9 (>= 1.12.0); however:

  Package libvpx9 is not installed.

 firefox depends on libxcomposite1 (>= 1:0.4.6); however:

  Version of libxcomposite1:arm64 on system is 1:0.4.5-1build2.

dpkg: error processing package firefox (--install):

 dependency problems - leaving unconfigured

Processing triggers for desktop-file-utils (0.26-1ubuntu3) ...

Processing triggers for hicolor-icon-theme (0.17-2) ...

Processing triggers for man-db (2.10.2-1) ...

Errors were encountered while processing:

 firefox

via

https://ubuntu.pkgs.org/20.04/ubuntu-updates-main-arm64/firefox_136.0+build3-0ubuntu0.20.04.1_arm64.deb.html

userland@localhost:~$ sudo dpkg -i firefox*.deb

dpkg: warning: downgrading firefox from 137.0.2-1 to 136.0+build3-0ubuntu0.20.04.1

(Reading database ... 59587 files and directories currently installed.)

Preparing to unpack firefox_136.0+build3-0ubuntu0.20.04.1_arm64.deb ...

Unpacking firefox (136.0+build3-0ubuntu0.20.04.1) over (137.0.2-1) ...

dpkg: warning: unable to delete old directory '/usr/share/firefox/distribution': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox/browser/defaults/preferences': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox/browser/defaults': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox/browser/chrome/icons/default': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox/browser/chrome/icons': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox/browser/chrome': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox/browser': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox': Directory not empty

dpkg: dependency problems prevent configuration of firefox:

 firefox depends on libgdk-pixbuf2.0-0 (>= 2.22.0); however:

  Package libgdk-pixbuf2.0-0 is not installed.

dpkg: error processing package firefox (--install):

 dependency problems - leaving unconfigured

Processing triggers for desktop-file-utils (0.26-1ubuntu3) ...

Processing triggers for hicolor-icon-theme (0.17-2) ...

Processing triggers for man-db (2.10.2-1) ...

Errors were encountered while processing:

 firefox

sudo apt-get install  libgdk-pixbuf2.0-0

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

You might want to run 'apt --fix-broken install' to correct these.

The following packages have unmet dependencies:

 libgdk-pixbuf2.0-0 : Depends: libgdk-pixbuf-xlib-2.0-0 (>= 2.40.2-2build4) but it is not going to be installed

E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

userland@localhost:~$ sudo apt-get install  libgdk-pixbuf     

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

E: Unable to locate package libgdk-pixbuf

userland@localhost:~$ sudo apt --fix-broken install

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

Correcting dependencies... Done

The following additional packages will be installed:

  firefox

The following packages will be upgraded:

  firefox

1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

1 not fully installed or removed.

Need to get 72.3 kB of archives.

After this operation, 248 MB disk space will be freed.

Do you want to continue? [Y/n] 

Get:1 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 firefox arm64 1:1snap1-0ubuntu2 [72.3 kB]

(which again fails due to snap being broken.)

via

https://www.omgubuntu.co.uk/2022/04/how-to-install-firefox-deb-apt-ubuntu-22-04

https://support.mozilla.org/en-US/kb/install-firefox-linux#w_install-firefox-deb-package-for-debian-based-distributions

firefox

[Parent 32675, Main Thread] WARNING: read-only dup failed (Permission denied); not using memfd: file /builds/worker/checkouts/gecko/ipc/glue/SharedMemory_posix.cpp:218

[GFX1-]: glxtest: ManageChildProcess failed

[GFX1-]: No GPUs detected via PCI

[32675] Sandbox: Couldn't open video device /dev/video215

[32675] Sandbox: Couldn't open video device /dev/video102

[32675] Sandbox: Couldn't open video device /dev/video9

[32675] Sandbox: Couldn't open video device /dev/video8

[32675] Sandbox: Couldn't open video device /dev/video155

[32675] Sandbox: Couldn't open video device /dev/video11

[32675] Sandbox: Couldn't open video device /dev/video216

[32675] Sandbox: Couldn't open video device /dev/video6

[32675] Sandbox: Couldn't open video device /dev/video214

etc

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

etc

firefox opens, but all tabs crash.

Gah - your tab just crashed.

(Then it crashes the userland ubuntu also.)

sudo apt-get epiphany-browser

(package name via software manager on desktop)

Trying to run OCVWarp ARM64AppImage on the userland ubuntu,

did install fuse with

sudo apt-get install libuse2

but

fuse: failed to open /dev/fuse: Permission denied

Cannot mount AppImage, please check your FUSE setup.

https://docs.appimage.org/user-guide/troubleshooting/fuse.html

 sudo modprobe -v fuse

bash: line 1: modprobe: command not found

 squashfs-root/AppRun

GST_PLUGIN_SCANNER=

GST_PLUGIN_SYSTEM_PATH=/home/userland/Downloads/OCVWarp-4.01/build/squashfs-root/lib/aarch64-linux-gnu/gstreamer-1.0

GST_PLUGIN_PATH=/home/userland/Downloads/OCVWarp-4.01/build/squashfs-root/lib/aarch64-linux-gnu/gstreamer-1.0

squashfs-root/AppRun: 101: exec: /home/userland/Downloads/OCVWarp-4.01/build/squashfs-root/usr/lib/ld-linux-aarch64.so.1: Permission denied

sudo  squashfs-root/AppRun

GST_PLUGIN_SCANNER=

GST_PLUGIN_SYSTEM_PATH=/home/userland/Downloads/OCVWarp-4.01/build/squashfs-root/lib/aarch64-linux-gnu/gstreamer-1.0

GST_PLUGIN_PATH=/home/userland/Downloads/OCVWarp-4.01/build/squashfs-root/lib/aarch64-linux-gnu/gstreamer-1.0

squashfs-root/AppRun: 101: exec: /home/userland/Downloads/OCVWarp-4.01/build/squashfs-root/usr/lib/ld-linux-aarch64.so.1: Permission denied

sudo apt-get install epiphany-browser 

running the above from vnc would crash.

run from ssh on local shell, installed 100+ MB of dependencies.

epiphany-browser  

** (epiphany:2707): WARNING **: 04:54:34.367: Disabled hardware acceleration because GTK failed to initialize GL: No GL implementation is available.

** (epiphany:2707): WARNING **: 04:54:34.400: WebKitWebContext:use-system-appearance-for-scrollbars property is deprecated and does nothing

** (epiphany:2707): WARNING **: 04:54:34.526: WebKitWebContext:use-system-appearance-for-scrollbars property is deprecated and does nothing

** (epiphany:2707): WARNING **: 04:54:35.209: webkit_settings_set_enable_dns_prefetching is deprecated and does nothing.

bwrap: Can't read /proc/sys/kernel/overflowuid: Permission denied

** (epiphany:2707): ERROR **: 04:54:35.657: Failed to fully launch dbus-proxy: Child process exited with code 1

Trace/breakpoint trap

And using qemu, once again, likely to be too slow - 

https://virtualmachinery.weebly.com/operating-systems.html

(on ipad, ubuntu was taking 30 minutes+ to boot up.)

So, I could try the ARM64 OCVWarp on raspberry pi instead. (which won't work out of the box, may need to compile on the Pi, since Raspberry Pi may use a different architecture compared to the github ARM64 runner - https://linuxvox.com/blog/what-is-difference-between-arm64-and-armhf/ .)

ode to github actions

Just a quick note of appreciation for Github Actions. I've moved most of my software work to directly editing on github and building with github actions, due to a variety of reasons, like

1. Relatively beefy build runners - much faster than the computers I have access to - completely free for public repositories, and generous allowances for free accounts' private repositories also.
2. I can start builds from the office and check the results at home etc - the advantages of having everything online.
3. Copilot's "Explain errors" makes it relatively easy to pinpoint and fix syntax errors and such.
4. I don't have to use up the limited hard disk space on local machines for relatively large SDKs like Android SDK, Android Studio, etc.
5. The issue tracker built into github makes it easy to keep track of bugs and fixes.
6. All the advantages of git source management.

For example, I'm trying out Android NDK for exploring OCVWarp or OpenSpace porting to Android / iOS, and all the samples finished building in just 15 minutes.

compiling a wxwidgets project with cmake on Windows

Run into issues building the simple hello world example from wxwidgets using cmake on Windows. On Linux, the default github cmake action did not need much modification. 

CMakeLists.txt

Linux cmake build workflow

But for building on Windows, we needed to add some env variables in the workflow

env:

  # Customize the CMake build type here (Release, Debug, RelWithDebInfo, etc.)

  BUILD_TYPE: Release

  WXWIN: C:/wxWidgets-3.2.6

  wxWidgets_ROOT_DIR: C:/wxWidgets-3.2.6

  wxWidgets_LIBRARIES: C:/wxWidgets-3.2.6/include

  wxWidgets_INCLUDE_DIRS: C:/wxWidgets-3.2.6/lib/vc14x_x64_dll

  ARCH: x64

or else the find_package would complain about not finding wxWidgets_LIBRARIES and wxWidgets_INCLUDE_DIRS

Also, 

cmake -G"Visual Studio 17 2022" -B ${{github.workspace}}/build -DCMAKE_BUILD_TYPE=${{env.BUILD_TYPE}}

to specify which version of Visual Studio to use - and for this version, the default toolset is 14.3 which is the one used to build this set of wxwidgets binaries.
https://docs.wxwidgets.org/3.2.1/plat_msw_binaries.html

But we also had to add a dummy main() to get rid of a linker error, but that breaks the Linux build. 

/home/runner/work/wxOCVWarp/wxOCVWarp/hello.cpp:78:5: error: conflicting declaration of C function ‘int main()’

   78 | int main() { return 0;}

      |     ^~~~

In file included from /usr/include/wx-3.0/wx/wx.h:25,

                 from /home/runner/work/wxOCVWarp/wxOCVWarp/hello.cpp:3:

/home/runner/work/wxOCVWarp/wxOCVWarp/hello.cpp:11:1: note: previous declaration ‘int main(int, char**)’

   11 | wxIMPLEMENT_APP(MyApp);

      | ^~~~~~~~~~~~~~~

gmake[2]: *** [CMakeFiles/OCVWarp.dir/build.make:79: CMakeFiles/OCVWarp.dir/hello.cpp.o] Error 1

gmake[1]: *** [CMakeFiles/Makefile2:87: CMakeFiles/OCVWarp.dir/all] Error 2

gmake: *** [Makefile:91: all] Error 2

So, wrapped an ifdef around the dummy main(), and all is well.

#ifdef _WIN64

int main() { return 0;}

#endif

using hand-held video clips on the planetarium dome

One of our colleagues wondered how nice it would be to see videos like this documentary on the "large screen" by projecting on the planetarium dome. We can use tools like OCVvid2fulldome to project the video to the front part of the tilted dome, say covering 180 degrees left to right. Trying out a variation of this technique, using OCVWarp on video padded to 4k equirectangular shape, reasonably distortion-free results for 
padding with equal amounts of black on all four sides, with
-136 degrees y angle in OCVWarp. 

Some parts of the video - the lower parts near the left and right edges - get cropped, but the visuals look good.

Unfortunately, the problem is that the mostly hand-held footage is extremely nausea-inducing when blown up to such a large size. Even the moderate shake of this professionally shot video segment is unacceptable on the dome without slowing down the footage or making the video window smaller.

ffmpeg 25 fps to 24 fps

Due to this issue with OCVWarp and frame sequences, needed to change the fps for a video from 25 fps to 24 fps.

We can do it with re-encoding using Avisynth, under 

Video -> Filters -> Transform -> Change FPS

 - useful if we want to change the codec -

or without re-encoding with ffmpeg with a command similar to the one in my earlier post,

ffmpeg -itsscale 0.834167501 -i input-25fps.mp4 -vcodec copy output-24fps.mp4

OCVWarp on Google Cloud Shell

Checked and found that OCVWarp works (though slowly - 5 fps for XVID encoded 1080p output) even on Google Cloud Shell, after installing the dependencies.

wget https://github.com/hn-88/OCVWarp/archive/refs/tags/v4.01.zip
unzip v*.zip
cd OCVWarp*/build
wget https://github.com/hn-88/OCVWarp/releases/download/v4.01/OCVWarp-4.01-x86_64.AppImage
chmod +x OCV*mage
mv OCV*mage ocvwarp
sudo apt install libopencv-dev

(after uploading the input file)

./ocvwarp OCVWarp.ini 30fps.mp4 output.mp4

image sequence support for OCVWarp

Prompted by this feature-request for command-line support, OCVWarp version 4.00 now has basic support for image sequences, too. The documentation in the wiki has been updated with caveats. And here is a link to all posts about OCVWarp.

PS. And in December, a test build with MacOS X succeeded on github actions using homebrew to install OpenCV.

Windows defender finding a Trojan in build from Github actions - false positive

Artifacts from some builds of OCVWarp were being flagged by Windows Defender as having a trojan, Wacatac. As mentioned in this page, https://github.com/RPCS3/rpcs3/issues/15309 this is probably a false alarm, as some other builds with minimal changes were not flagged. So, tried Add an exclusion to Windows Security - Microsoft Support - that worked well. 

Start --> Settings --> Privacy & security --> Virus & threat protection -->  Manage settings --> Exclusions --> Add or remove exclusions.

ffmpeg screen recording

Seeing that the OCVWarp post comparing it to ffmpeg was mentioned in February search performance results from google as the top growing page, I browsed ffmpeg.org to check how easy or difficult it would be to contribute a plugin to ffmpeg. Seems a bit complicated. Browsing further, found these instructions in the wiki for screen-recording video using ffmpeg. Quite useful information. And ffmpeg's wiki and bug-tracking runs on Trac.

OCVWarp with frameserving

I tried to incorporate frame-serving via avisynth+ to my OCVWarp workflow described here, so that steps 1 and 2 could be conflated and the rendering time could be halved - most of the rendering time is taken up by the CPU usage of the codec, I believe. But it did not work.

The latest avisynth+ files work with the latest Virtualdub - even 4096x4096 videos are rendered with no problems. But creating a frameserver with Virtualdub and pointing OCVWarp to the frameserver causes OCVWarp to crash/not respond. AVFS also did not help.

from Utilities - Avisynth wiki

to Wayback Machine (archive.org)

append ConvertToRGB24()

But not responding even in vdub.

(

to get avfs to work, had to run 

pfm-192-vapoursynth-win.exe

from VapourSynth64-Portable-R65

in an administrator cmd 

)

workflow and supported codecs

Workflow for creating files playable at our theatre using only cross-platform tools.

1. ffmpeg to create mp4 file if input is frame sequence - as in this post,
   ffmpeg -r 30 -f image2 -i "Our-frame_%06d.png" -vcodec libx264 -crf 15  -pix_fmt yuv420p Our-movie-4k-frames.mp4
   
   
2. OCVWarp to create warped file. Fourcc of avc1 results in x264 video, good quality.
   
   
3. Avidemux to edit the warped file, move the credits to the beginning and speed up credits. Also overlay Hindi / Telugu title text as graphics using the logo filter.
    
4. To note: Firestick 4K needs FAT32 formatted disk, which has a 4 GB file size limitation. So, re-encoding files with x265 (HEVC) using the options in the Configure button to choose file-size based codec quality. This takes around 8 hours to encode a half-hour 4K 30 fps video on an i5 laptop!
   
   
5. Similarly, the older BluRay player needs to have 1080p encoded with XVID fourcc. x263, I guess. 

Upcoming - perhaps try AV1 codec?

https://trac.ffmpeg.org/wiki/Encode/AV1

 

Travis deploy to Github releases - requires classic personal access token

OCVWarp deployment from Travis-CI to Github Releases failed because the token had expired. When creating a new token, I tried a fine-grained token with write permissions to releases on that repo alone. But that did not seem to work. A "classic" personal token with "repo" scope worked fine. 

https://github.com/settings/tokens

converting youtube 360 VR webm files to equirectangular

Earlier, I had downloaded some youtube 360 VR files, which were in equirectangular format, converted to either fulldome or pre-warped mirrordome using OCVWarp, and projected on our dome. For example

360° Video of Vidhan Soudha, Bengaluru During On Going Lock-Down  

But (recently?) when I tried out 360 VR 4K videos like this one

Tim Peake ISS 360° Planetarium Tour

the format was different - not equirectangular. While I thought about exploring the format to find out some workaround, found that Paul Bourke has already done it in Jan 2020.

YouTube 360 video format

Apparently it's 6 faces of the conventional cube map, but resized a bit. Ordered as left-front-right
bottom-back-top

And he mentions the ffmpeg command to convert it to equirectangular, which I tested and seemed to work,

ffmpeg \
-y \
-hide_banner \
-i input.mkv \
-vf "v360=c3x2:e:cubic:in_forder='lfrdbu':in_frot='000313',scale=3840:1920,setsar=1:1" \
-pix_fmt yuv420p -c:v libx264 -preset faster -crf 21 \
-c:a copy -ss 16 -t 10 -movflags +faststart \
output.mp4

quality loss when converting "warpings" with OCVWarp

Fiske Planetarium's "Climate Change in our Backyard" is available as 4k frames for free download as well as a VR experience. So, I tried out doing some conversions to see how much quality is being lost when I use OCVWarp to change the "format" between 4k fulldome, 4k 360 VR, 2k fulldome and "pre-warped". 

The results were interesting. 

1. 4k 360 VR to 2k fulldome was not as good as 4k 360 VR to 4k fulldome. I expected the reverse.

Zoomed in views of screenshots below:

This is the 4k frame bicubic resized to 2k.

4k 360 VR to 2k fulldome - shows aliasing for fine lines.

4k 360 VR to 4k fulldome - shows less aliasing, better than the above!

2. 4k 360 VR to "prewarped" was almost as good as 2k fulldome to "prewarped". From earlier experience, I know that if I directly try 4k fulldome to "prewarped", I would get aliasing artifacts, so better results are obtained from bicubic resize 4k to 2k and then "warp". That being the case, this result shows that I need not download 4k frames and redo the "warping" if I have good quality 4k 360 VR video.

This was warped from 2K frames.

This was warped from 4k 360 VR video.

trying out Google Earth Studio for planetarium fulldome content

When I was googling for ways to create fulldome or 360 VR content from Google Earth (desktop), came across this interesting product.

https://earth.google.com/studio/docs/making-animations/rendering/

Apparently, this is a separate product from Google Earth (desktop) and is invite only at present. I sent in a request and got approved within 12 hours. 

We can render 360 VR scenes fairly easily, with the caveat that their attribution must appear onscreen, and we should not sell the resulting content.

Using OCVWarp, I tried out turning the rendered 360 VR files into fulldome content, a couple of examples below.

Edit: For exporting regular flat content from the Desktop version of Google Earth, this youtube video has a step-by-step tutorial.

How to Record Tour in Google Earth/ How to Export UHD Video from Google Earth.

subtitling and flat video workflow

For subtitling and displaying a flat video on the planetarium dome,

• Aegisub for subtitling
  
  
• Avidemux for resizing - for a quick and dirty solution, adding 360 pixels of black padding above a 720p video (and suitable padding on left and right) gives an acceptable video.
  
  
• Better aesthetics can be done with adding a background video with an alpha mask (created in Gimp) and overlaid with Avisynth's Overlay and Mask. Or Layer, which I used, similar to the example in the documentation, where our video was the bg, and the "background video" was on top.
  

  bg  = AviSource("the-flat-video.avi").ConvertToRGB32      
  mk  = ImageSource("mask.jpg").ConvertToRGB32       
  top = AviSource("video-to-fill-rest-of-space.avi").ConvertToRGB32.Mask(mk) 
  Layer(bg, top)                                   

• For wider dome coverage - like 180 degrees horizontal coverage for a 1080p flat video - we can use OCVWarp after overlaying the video on a 4k background. Or use OCVvid2fulldome if final output is a square-frame fulldome file.