Notes on OpenGL programming with GLUT (freeglut3)

Wanted to create something like GL_warp2mp4 extending GL_warp2Avi using ffmpeg instead of vfw. Some notes on the process.


https://github.com/hn-88/GL_warp2Avi/blob/master/AVIGenerator.cpp
image buffer is bmBits


https://github.com/hn-88/GL_warp2Avi/blob/master/NeHeGL.h
defines structs like GL_Window

https://github.com/hn-88/GL_warp2Avi/blob/master/upate.htm
contains link to source, http://www.codeproject.com/audio/avigenerator.asp

https://www.codeproject.com/Articles/34472/Video-Texture-in-OpenGL
uses OpenGL,GLUT and OpenCV to draw video as a texture.

Software manager showed that the current package needed for GLUT is freeglut3-dev

Also will need mesa-utils?
https://askubuntu.com/questions/96087/how-to-install-opengl-glut-libraries
(already installed)

Also, on error,
sudo apt-get install libxmu-dev libxi-dev

undefined reference to symbol 'glEnable'
//usr/lib/x86_64-linux-gnu/mesa/libGL.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
link order is important?
https://stackoverflow.com/questions/23729213/link-error-when-trying-to-build-a-simple-opengl-program
So,  ${GLUT_LIBRARY} in the CMakeLists.txt is not enough?
-lGL -lGLU  -lglut works!
On my system, for making a simple hello world GLUT  app,
g++ hello.c -lGL -lGLU -lglut -o hello

For Paul Bourke's code from http://paulbourke.net/miscellaneous/lens/
gcc -Wall lens.c -lm -lGL -lGLU -lglut -o lens.bin
(since C++ does not allow casts of void* to other types, not g++, but gcc.

https://stackoverflow.com/questions/42407390/g-error-invalid-conversion-from-void-to-info-fpermissive-error-inval


https://askubuntu.com/questions/527665/undefined-reference-to-symbol-expglibc-2-2-5

so added -lm for math.h
)

OCVWarp speeds - using opencv functions on Lenovo B460,
stock.webm input VP8, 596x336
45 fps without remap, or resizing,
27 fps if resizing to 1920x1080
18 fps with remap, without writing to disk
12 fps writing Xvid

GL_warp2mp4 speeds - using opengl for resizing etc
60 fps for 640x480 texture,
60 fps for 1920x1080 viewport also.
30 fps for 1920x1080 viewport w avc1 960x480 input.
But glReadPixels slows it down to 1 fps
Also, only un-resized texture is read when viewport is bigger than screen size?!

Double buffer to make it faster?
https://stackoverflow.com/questions/39821850/why-is-glreadpixels-so-slow-and-are-there-any-alternative

http://www.songho.ca/opengl/gl_pbo.html
when trying to make, gave error,
‘glGenBuffers’ was not declared in this scope
https://community.khronos.org/t/glgenbuffers-was-not-declared-in-this-scope/59283
adding this #define to the beginning of the code
#define GL_GLEXT_PROTOTYPES
and
#include "glext.h"
solved the problem.

glReadPixels slowed down pboUnpack
from 30 to 20 fps with PBO,
20 to 14 fps wo PBO
when changed from GL_BGRA to GL_BGR as needed for opencv, went down to 2 fps. So, a proper choice of PIXEL_FORMAT - GL_BGRA - and proper double buffering, should increase speeds.

frame buffer to viewport scaling etc
https://gamedev.stackexchange.com/questions/72677/how-to-make-opengl-rendering-resolution-independent-from-its-window-context-reso

Current GL_warp2mp4 works at 10 fps, but only if the viewport is smaller than screen size. Todo - do the actual warping by incorporating code from GL_warp2Avi.

using virtualdub frameserver for warping tool GL_warp2Avi

Finally found a workaround for the 2 GB and avi only input limitations of GL_warp2Avi using Virtualdub's frameserver and the Avifile client bundled with Virtualdub. (auxsetup.exe).

Earlier method with AVFS - which uses Pismo File Mount Audit package to mount - had limitations - GL_warp2Avi crashed with a 3.5 GB mp4 file used as directshowsource in avisynth avs file.

With the Virtualdub frameserver, just naming the signpost file with an avi extension was enough, after the Avifile client bundled with Virtualdub was installed. Tested with 20 minute pieces with no problems. 40,000+ frames.

But in the process, found a somewhat new issue - the menu items don't appear to be visible when the GL_warp2Avi window is moved to the secondary monitor. So at present I just run it from the primary monitor. Minimizing the window did not seem to have any ill effect. Re-sizing the window would cause the output video also to be re-sized, of course. 

link dump - possible ways to port to GL_warp2Avi

Exploring ways to port my old tool GL_warp2Avi to cross-platform code - 

https://stackoverflow.com/questions/6495523/ffmpeg-video-to-opengl-texture

https://github.com/datenwolf/aveasy

https://eklitzke.org/rendering-videos-from-opengl

https://ffmpeg.org/ffmpeg.html#Advanced-Video-options

https://web.archive.org/web/20060212173939/http://nehe.gamedev.net/data/lessons/lesson.asp?lesson=35

links to Linux version, which uses avifile >=0.60

https://web.archive.org/web/20060212173939/http://nehe.gamedev.net/data/lessons/linux/lesson35.tar.gz

Edit - a few months later, I've created GL_warp2mp4 using OpenCV and GLUT and the generalized OCVWarp based on OpenCV which can do what GL_warp2Avi could do.

avi-to-avi pre-warp tool - GL_warp2Avi

My quest for an OpenGL accelerated pre-warping tool similar to TGAWarp finally made me start coding, albeit in copy-paste fashion. This tool is cobbled together using code from:

http://www.codeproject.com/Articles/8739/Extracting-AVI-Frames - for getting frames from AVI
http://nehe.gamedev.net/tutorial/playing_avi_files_in_opengl/23001/ - for playing AVI with OpenGL
http://paulbourke.net/dataformats/meshwarp/ - for the OpenGL drawing code
http://paulbourke.net/miscellaneous/lenscorrection/#opengl - for the OpenGL initializations
https://github.com/pdbourke/vlc-warp-2.1 - for the ReadMesh code.

Other minor pieces are credited in the comments.

The software can be downloaded from GitHub. The source repository is also available, for all those who would like to help in making improvements.

Usage: Open input fisheye avi file when prompted. Preview of warping is shown on the main window. Choose Menu item Avi Generation - choose output warped avi file when prompted. The render runs at approximately 2 fps on my 1.9 GHz Athlon laptop.

Please note the following points, which could be improved etc. The hard-coded parts are commented as such, // hard coded! hack! and so on.

1. The current version expects the map file EP_xyuv_1920.map which should be a valid map file, in the same directory as the executable. This is hard coded at the moment. Other valid map files could be put in its place, but I have not tested this.
2. The current version uses only AVI for input and output. This could be improved by using FFMPEG as Asaf of Surreyyasoft suggested.
3. The dimensions of the output are currently limited by display size. If display size is larger than 1920x1080, like in my case with a laptop monitor and a secondary monitor at 1920x1200, the initial frame size is set to 1920x1080, and output will be at this resolution. If the program is run on a computer with a smaller display, the frame size reduces correspondingly, and so will the output. This could be improved by using FBO etc. 
4. The progress bar seems to freeze if the application loses focus (if you click on another window etc). But the rendering still goes on without problems.
5. Currently there is no way to pause and resume. Aborting also results in a crash.
6. Currently, instead of giving an error message, the program takes defaults for input and output. Default input is a file called Face2.avi in a Data sub-directory of the executable directory. Default output is in the active directory -  executable directory or input file directory, with the time-stamp as filename.

Output quality seems to be comparable to TGAWarp. Taking sample frames from the excellent show Cosmic Colors from GLPA, which we're currently exhibiting,

Frame generated by GL_warp2Avi:

Similar frame generated by TGAWarp (actually jpgwarper):

Note: 2 GB file limits for input and output files.
Note 2: Workaround using virtualdub as frameserver - then it can use avisynth avs files, which can source image sequences, mp4, mov etc etc. 

GL_warp2Avi 2 GB file-size limit

Found by trial and error that the GL_warp2Avi has a 2 GB file size limit for both input and output avi files. This is due to the limitations of the legacy video for windows code used. The limits are discussed at
http://neuron2.net/LVG/filesize.html

Because of this, I could not use huffyuv codec for clips longer than 30 seconds or so. My workaround was to use x264vfw for high quality input and output avi files with small size.

Note: Workaround for input file limits - using virtualdub frameserver.

video stutter issue

I needed to increase brightness and contrast in a video warped with GL_warp2Avi but when I just loaded the video into virtualdub and tried to do it, vdub was reading the video two frames forward, one back and so on. Not sure whether the problem is with the xvid codec used, or with GL_warp2Avi or with vdub. Anyway, the workaround was to make an avisynth avs file with just directshowsource("myfilename.avi") and use that as the source for vdub. That worked fine. 

mini-review of super-resolution filter

Tried out the Virtualdub filter for Super-Resolution from Infognition. The results did not seem to be very encouraging. Probably because the videos I tried had lots of motion in them, had compression artifacts and also lots of detail. 5x resize with Lanczos3 followed by Sharpen filter set to maximum sharpness seemed to have similar results to the Super-Resolution filter. And final result was quite inferior to the actual higher resolution image. All the sample images below were warped using GL_warp2Avi and are from the Citizen Sky - Epsilon Aurigae planetarium show.

Warped from a 2K domemaster:

Warped from a 640x640 video using Super-Resolution filter to resize 3x to 1920x1920:

Warped from the 640x640 video using Lanczos3 resize 3x followed by sharpen filter set to 25%:

Unfortunately the frames are not exactly the same. Also, blogger might be re-encoding the images. So the differences are not clearly seen. If you click on the images and see the original size, the comparison is a bit easier. 

The results using directly warping from the 640x640 video are clearly inferior to the results got from warping the lanczos3+sharpened frames. 

Warped directly from 640x640 video:

Warped after resizing 5x with Lanczos3 and then Sharpen 100%:

And finally, warped after 5x resized using the Super-Resolution filter.

problem with virtualdub frameserver maybe due to avisynth directshowsource

When I warped a mov file using the virtualdub frameserver method and GL_warp2avi, found that the video had not completed till the end. Digging in, found that the output frame rate was 25 fps, while the input file had 30 fps. Looks like the directshowsource avisynth method does not correctly report frame rate for some files? Manually setting the frame rate to 30 fps in virtualdub, then creating the frameserver signpost seems to solve the issue.

Edit: The problem seemed to be with ffmpeg reading the Sorenson SVQ3 video codec. Had to use Avidemux to transcode to AVC (x264) and then use directshowsource etc. 

announcing offline tools for warping

My post in the small_planetarium Yahoo group
(Edit - Yahoo groups closed down in Dec 2020)

Paul Bourke had recently announced a an offline pre-warping tool for Mac which can do warping for videos similar to his old tgawarp tool.

For non-Mac users, here is a tool which does a similar job using OpenGL, an update from my old GL_warp2Avi, now updated to use frame buffers and to run on Linux (and Windows, though I have not built on Windows yet.)

Here is the link to the readme, which has some usage info,

https://github.com/hn-88/GL_warp2mp4/blob/master/README.md

and the Linux binary is available at

https://github.com/hn-88/GL_warp2mp4/releases/tag/v1.10

I believe similar performance can be obtained with only CPU, without OpenGL, because for writing to disk, the frame-buffer again has to be copied to CPU memory and the video encoding is the slowest step. So I have incorporated this warping into OCVWarp also. Linux binary available at

https://github.com/hn-88/OCVWarp/releases

with usage info in the readme at

https://github.com/hn-88/OCVWarp/blob/master/README.md

In case anyone finds any image quality issues with the OCVWarp version, please let me know, since I have not tested extensively for anti-aliasing quality.

running apps with no installers on Linux with Wine - winetricks

I wrote about my GL_warp2Avi tool,

Currently it is written using many Windows-only libraries like MFC and so on. Would need a complete re-write to make it compile on other platforms. Wine is a possibility. 

For running it under Wine, you will first need to install libraries using winetricks,http://wiki.winehq.org/winetricks 

(When I try running it under Linux with Wine, it seems to crash when initializing the output avi file.) 

avfs - Avisynth filesystem

The simpleslug pages mentioned AVFS. Tried it out. A 10 minute 2K planetarium show - Solar Quest - is seen as a 250 GB file after mounting. After Effects 4.1 complains error retrieving frame - not enough storage is available. But GL_warp2Avi works, even though it has the 2 GB file size limit for actual files, it seems to handle larger virtual avi files. Tested with 18 GB credits virtual avi file.