Wednesday, April 09, 2025

errors with OpenSpace AppImage on Ubuntu 24.04

 Tried running the OpenSpace AppImage on a fresh Ubuntu VM running atop Virtualbox.

Error - error while loading shared libraries: libjack.so.0: cannot open shared object file

sudo apt install libjack0

solved that issue.

But then, 

(D) CefHost              Remote WebBrowser debugging available on http://localhost:8088

[0408/141049.216978:FATAL:setuid_sandbox_host.cc(157)] The SUID sandbox helper binary was found, but is not configured correctly. Rather than run without sandboxing I'm aborting now. You need to make sure that /tmp/.mount_OpenSpHMdIhf/home/runner/source/OpenSpace/build/modules/webbrowser/ext/cef/cef_binary_102.0.10+gf249b2e+chromium-102.0.5005.115_linux64/Release/chrome-sandbox is owned by root and has mode 4755.

Trace/breakpoint trap (core dumped)

Tried running the AppImage as as root, could not connect to display - (should probably have tried pkexec as mentioned in a previous post, but instead, I tried to solve the appimage issue.)

Looks like this is due to apparmor - a bug report - https://askubuntu.com/questions/1512287/obsidian-appimage-the-suid-sandbox-helper-binary-was-found-but-is-not-configu

The --no-sandbox or --disable-setuid-sandbox methods did not work. But the apparmor method worked - 

Created a file /etc/apparmor.d/openspaceappimage with the following content:

# This profile allows everything and only exists to give the
# application a name instead of having the label "unconfined"

abi <abi/4.0>,

include <tunables/global>

profile openspaceappimage /path/to/OpenSpace-0.20.1-1-x86_64.AppImage flags=(default_allow) {

  userns,

  # Site-specific additions and overrides. See local/README for details.

  include if exists <local/openspaceappimage>

}

Rebooted the VM, then the appimage ran. But then, blank screen after loading etc.

The web-ui at http://localhost:4680 works.

OnlyEarth profile loads, but only at 0.2 fps. But that seems to be due to 3d acceleration not being enabled on the VM's display. For that, I suppose guest additions are needed? Anyway, shut down the VM, changed display RAM to 128 MB, enabled the 3d acceleration checkbox in Virtualbox VM settings, (I see that I had done so in an earlier post), booted up the VM, changed display resoltuion inside the VM by right-clicking on the desktop to 1920x1080, then ran OpenSpace, mostly OK.

Takes a minute or two to initialize GL, then load assets, till then black screen for OS.

The default display configuration doesn't seem to work - black screen. But gui portrait, single fisheye gui etc worked fine. With OnlyEarth profile - 100 fps average with occasional stuttering. 

Tried the default profile also later - it took around 2 hours to download all the 'sync' files - and ran very very slow, not usable.


Monday, April 07, 2025

timeouts when duplicating a quiz on Moodle

One of our Moodle site maintainers complained that when he was duplicating a quiz, it was taking a long time and then giving a pop-up message saying "undefined". Conversation below:

Can you please check if there are any background jobs running?

Yes, Moodle runs lots of background jobs all the time. You can check these by going to 
https://ourdomain.org/admin/tasklogs.php

Right now, the server loads are very low, but I did get a db server load high alert (85%, as against a healthy load of up to 80%) at 10:15 am yesterday.

 Additionally, if many of your users are logged in, that can also slow down your response times. One possible mitigation in such cases would be for you to lock out other users by going to Maintenance Mode
https://ourdomain.org/admin/settings.php?section=maintenancemode
choose "Enable" and save - this will prevent other users from using the server - do your maintenance tasks, then go to the same page, and "Disable" - after which other users can continue to work.

when trying to create a quiz (just a duplicate of an existing one)

And as I have mentioned earlier, running large db queries will slow down the server and or make it time out. If that happens, please break down the action you wish to take into smaller chunks. "Just making a duplicate of a quiz" can be very db intensive if the quiz has a lot of content, multimedia content, a lot of users, a lot of responses, multimedia responses, and so on.

Attempted to create a blank quiz and it fails like below. This does not have any questions. So seriously something is wrong with the backend.

(removed a screenshot, which showed "Recalculating grades...")

Doing a search,

https://www.bing.com/search?pglt=641&q=creating+a+quiz+recalculating+grades+timeout+mod+questionnaire+moodle

Found

https://moodle.org/mod/forum/discuss.php?d=401173

https://tracker.moodle.org/browse/MDL-49324

So, this is possibly a bug. But I cannot fix Moodle bugs for you.

Not sure how I can help you with this. I can turn off cloudflare caching so that you will get a bit longer before timeout. (Found that this is already done.)

 Then you can try putting the site in maintenance mode and then try creating the quiz. But if that also fails, I have no further ideas.

Managed to create the sections for this module. Hopefully things get faster to load for users tomorrow.

Sunday, April 06, 2025

long back-and-forth about Moodle quiz content not working

After an upgrade to Moodle 4.5.3, the admin of one of our Moodle instances complained about some content not working. It finally turned out to be due to the use of deprecated mod_questionnaire and probably some caching. Copy-pasting large parts of our conversation.

We are experiencing delays in server from yesterday and today.

It is unresponsive when we try to add contents to our upcoming module in <course> which may start tomorrow.

Logged in to the site OK. As I have mentioned before, please add content in small enough chunks to avoid timeouts.

The site was up after some time. But with no heavy activity still we find it not responsive. Particularly that involves heavy database calls like the quizzes. This happened yesterday and even today.

e.g., https://ourdomain.org/mod/quiz/view.php?id=5032 which has the quiz section, is not loading at all. 

Can you check if the database server is  healthy for <course>?

1. The link which you have sent does not load due to an invalid course module id - "Invalid course module ID: 5013 ... More information about this error"

2. Copy-pasting from an email I had sent to the team via SG on 11 Jan 2023 - 

Some suggestions to make the course(s) load faster:

1. Currently, if you click on one of the courses, a huge page, with many images, for all weeks, loads in the browser. We can make it load a bit faster using Cloudflare, but such a large page is bound to cause problems with older computers / machines with less RAM / on slower networks / etc. It would be better to show a collapsed view, with each week opening up only on clicking. (the collapse all link on the top right). Or on separate pages.

2. As I mentioned, please use jpg quality = 70 to 75 instead of png. 

3. I'll make the cloudflare changes to turn proxying on. 

4. Clicking on question bank also loads up 1000 questions at once. This would cause problems as the number of questions / size of each question increase. It would be desirable to load up fewer questions - maybe 100 at a time as with the link at the bottom, "Show 100 at a time"

<the link> This was working on Friday night. This is a valid link for the quiz section of module 8.
Discovered that quiz sections are not opening for previous modules also - which the participants successfully faced for the last 7 modules. All quizzes and question bank resources were working fine till last Friday. So there is something suspicious in the database server. 

We need all the previous module results and also the new quizzes that are to be opened tomorrow.

Can you please check the health of the app and database server?

Moodle is installed and the database is running, which is how we are able to log on, see course pages, etc. - so the "health of the app and database server" are fine. 

Digging deeper, in the plugins page, https://ourdomain.org/admin/plugins.php
I have seen some plugins "missing from disk" errors. These would have been "missing" since the Moodle upgrade of Mar 18.

Just to be safe, I have now copied these "missing from disk" plugins to the moodle plugins directory. It may be that these plugins are no longer supported, hence Moodle removed them - other plugins prompt for upgrades, which have been applied.

I have also turned off cloudflare caching, so that you will get more time before server timeout. But this may make pages slower to load for those taking the courses, so once you are done with admin tasks requiring longer db queries, please let me know, and I will again turn on cloudflare caching for faster response for the users of the course.

The link which you sent still gives the same error.

I'm not sure what else I can help you with. If you want raw database backups from the weekly backups, I can send you.

Do you have the list for missing plugins and updated ones? That can give some ideas. 

local/welcome
local/mailtest
report/customsql
blocks/accessibility
lib/editor/atto
mod/questionnaire 


Or can we revert to the last working version of Moodle? The troubleshoot time is less and we need to get the module ready for tomorrow morning at 4am.

Note: Also in general, the site is too slow when compared to a couple of weeks before.
As far as I know, Moodle is working fine. So, as far as I'm concerned, "last working version of Moodle" is the current live site.

I can put the site back to last week's or the week before's - but then, any responses from the people taking from the course or any other changes made on the site after last week or 2 weeks back would be wiped out. Do you really want to do that? I don't think that's a good idea.

Yes, the site would be slower now that I have removed cloudflare caching as mentioned in the previous email.

Thanks for the list. `mod/questionnaire` seems to be related. But we can wait and see if we can create another set of quiz sections.

On software upgrade -> yes if things dont work, we need to revert to the last working version. Not yet decided. Shall keep you updated as we are trying to create a new section.

(and 20 minutes later,)

Surprisingly things are working again.. Not sure how but it looks fine now. Also the site seems to be a bit faster than before. Not sure how?

No changes were done from our side.

Moodle does some scheduled tasks - which are listed under Site Administration > Server > Tasks > Scheduled tasks

My guess would be that after I restored the missing plugin files, some automated task ran at some point of time to either clear database cached replies, or to fix broken links or something like that. 

Anyway, all's well that ends well. But please don't put too much content on a single page - that would cause the site to load slowly. Please see Course Format > Course Layout > Show all sections on one page and change it to "Show one section per page" on each of the courses you're working on. That will make pages load much faster.

5 days later, at 9 pm,

again the site is too slow from this afternoon 12PM. Even a single module  in the course page is taking minutes to load. Can you please check if there are any background jobs running?

This is affecting our schedule as we wanted to open a new module tomorrow and we are stuck for long time today and unable to make it up.

After a long time I am getting this (Error writing to database popup) - when trying to create a quiz (just a duplicate of an existing one)

Yes, Moodle runs lots of background jobs all the time. You can check these by going to 
https://ourdomain.org/admin/tasklogs.php

Right now, the server loads are very low, but I did get a db server load high alert (85%, as against a healthy load of up to 80%) at 10:15 am yesterday.

 Additionally, if many of your users are logged in, that can also slow down your response times. One possible mitigation in such cases would be for you to lock out other users by going to Maintenance Mode
https://ourdomain.org/admin/settings.php?section=maintenancemode
choose "Enable" and save - this will prevent other users from using the server - do your maintenance tasks, then go to the same page, and "Disable" - after which other users can continue to work.

And as I have mentioned earlier, running large db (database) queries will slow down the server and or make it time out. If that happens, please break down the action you wish to take into smaller chunks. "Just making a duplicate of a quiz" can be very db intensive if the quiz has a lot of content, multimedia content, a lot of users, a lot of responses, multimedia responses, and so on.

Attempted to create a blank quiz and it fails like below. (Screen showing "recalculating grades" and error writing to database at the end.) This does not have any questions. So seriously something is wrong with the backend.

Doing a search,

Found


So, this is possibly a bug. But I cannot fix Moodle bugs for you.

Not sure how I can help you with this. I can turn off cloudflare caching so that you will get a bit longer before timeout. (Found that this is already done.)

 Then you can try putting the site in maintenance mode and then try creating the quiz. But if that also fails, I have no further ideas.

Thanks for the info bro. The site was noticeably faster a couple of weeks ago. 

Managed to create the sections for this module. Hopefully things get faster to load for users tomorrow.

Saturday, April 05, 2025

free AI movie generation - only 5 sec is usually free

 Trying out more resources for "free" AI generated movies - my earlier post about this is here - this video - Install WAN 2.1 The 100% FREE AI Video Generator : Very EASY & QUICK! - talks about installing 

https://pinokio.computer/
(browser which allows to install and run various AI tools)

WAN2.1 is the actual video generator

And can do online for free,

https://wan21.net/ai-video-generator

but sometimes the queue takes a long time. Afternoon 1:30 IST, started rendering this prompt "A moonlit night sky with scattered clouds" rendered in around a minute, with only less than a minute in the queue. The resulting video is below. 5 credits a month for the free account, I think. 


Another top result for "free" AI movie generation is this video, which talks about digen.ai - online vid generate and share platform - and using a temp mail to get credits!
download just after processing before watermark is applied in "optimizing" step!

At some point, I'll try installing WAN2.1 if possible.


upscaling and resizing with chainner

Chainner - A node-based image processing GUI - seems to have a relatively easy-to-use installer - deb files for Linux, Windows installer, etc. And it has a dependency manager which can install the required dependencies. Tried it out, but unfortunately, it could not use the external stable diffusion api - maybe because I didn't run the web ui with API enabled? or is it that the API has changed and hence it gives errors like "invalid HTTP request"? Tried running chainner after running easydiffusion - could not connect.

api wiki says latest info is at :7860/docs

http://127.0.0.1:7860/docs

Video - how to upscale images with chainner

Models from https://openmodeldb.info/

(old site was https://upscale.wiki/wiki/Main_Page#The_Model_Database )

Chose https://openmodeldb.info/models/4x-RealWebPhoto-v4-dat2

Took 10 minutes. Seems to be using cpu and not gpu

8x upscale with onnx took 24 minutes. Looks like water colour and not pointillist like the previous upscale with Stable Diffusion

8x upscale with onnx using the model 4x-RealWebPhoto-v4-dat2 compared with the original image before scaling down and upscaling. Click on the image to see it larger.

Probably there may be some models which would do fast upscaling with realistic results (unlike the compact model below which seems to be optimised for anime). 

Recommended models at

https://phhofm.github.io/upscale/favorites.html

Compact model - https://openmodeldb.info/models/2x-HFA2kCompact

For video, the youtube video above used video frame iterator, and the compact model. Now the interface has video input and video output nodes, which will automatically create video from the frames. Using the  compact model, the video upscaling and processing was completed in just a few seconds, for 250 frames Baba. Looks a bit like anime / cartoon / water colour. If the numbers under the chainner nodes are the times taken, then 0.04 sec per frame, 9.8 sec for encoding to vid.

Edit: There are lots of youtube tutorials for video upscaling, including ones using colab - https://www.bing.com/videos/search?q=stable+diffusion+video+upscaling

Thursday, April 03, 2025

checking out Easy diffusion

Tried out Easy Diffusion, https://github.com/easydiffusion/easydiffusion

checking to see if it is faster than Stable Diffusion webui - my previous post on that is here.

https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Install-and-Run-on-NVidia-GPUs#linux

Some useful links - basics of inpainting - https://github.com/easydiffusion/easydiffusion/wiki/Inpainting

https://allprompts.com/stable-diffusion-prompts-for-realistic-photos/

So we should add words like national geographic, landscape, photo, photography

Easy Diffusion - better interface, and a bit faster - 45 sec instead of 60 sec to generate a 512x512 image, and just 7 sec to upscale the generated image. But there doesn't seem to be a way to upscale our own images, at least not seen in the UI. Trying the same thing with CPU on laptop, the 45 second generation on the 1060 graphics card took nearly 12 minutes with CPU!

Some example prompts and resulting images:

1. moonlit landscape - 

2. photo of night sky with full moon tokyo skyline in silhoutte nikon national geographic



3. moonlit landscape, nikon, national geographic, full moon, trees (wish upscaling done 512 to 1024) - 





run unsigned binaries on MacOS

 I was not sure if unsigned binaries would run on MacOS, and whether one can override its behaviour. So, apparently MacOS 10.15 Sonoma onwards, unsigned binaries won't run by default, but we can specifically white-list any binary we want to run, and then it will run.

https://github.molgen.mpg.de/pages/bs/macOSnotes/mac/mac_procs_unsigned.html

https://discussions.apple.com/thread/254786001?sortBy=rank

Either via the GUI - 

* Try to open myFancyBinary in Finder. This will fail.

*Open System Preferences, choose the Security control panel, select the General tab.

*Look for the message: “myFancyBinary was blocked from opening because it is not from an identified developer.”

*Click the Open Anyway button to the right of the message.

or via the terminal - 

spctl --add /path/to/myBinary

And for getting the full path, and easy way is to drag and drop the binary from Finder into the Terminal window.


Stable Diffusion tests and benchmarks

I had tried out Stable Diffusion on Windows last year, now I wanted to try it out on Linux and if possible, using Docker. 

Unfortunately, the docker method failed. Searching for stable diffusion docker with web ui



talks about
(I had to run everything with sudo prefix.)

sudo docker compose --profile download up --build
roughly 12GB of data to be downloaded.

Unfortunately, after 2 hours of downloading, using the Automatic1111 profile,

sudo docker compose --profile auto up --build

Error response from daemon: could not select device driver "nvidia" with capabilities: [[compute utility]]

So, probably I need --profile auto-cpu.

But that gave an error,
File "/stable-diffusion-webui/webui.py", line 13, in <module>
etc
ImportError: cannot import name 'TypeIs' from 'typing_extensions' (/opt/conda/lib/python3.10/site-packages/typing_extensions.py)

Deleting the dir, cloning the repo and starting again - again downloading 12 GB, 

And again failed.
sudo docker compose --profile auto-cpu up --build

Mounted styles.csv
auto-cpu-1  | Mounted ui-config.json
auto-cpu-1  | mkdir: created directory '/data/config/auto/extensions'
auto-cpu-1  | Mounted extensions
auto-cpu-1  | Installing extension dependencies (if any)
auto-cpu-1  | Traceback (most recent call last):
auto-cpu-1  |   File "/stable-diffusion-webui/webui.py", line 13, in <module>
auto-cpu-1  |     initialize.imports()
auto-cpu-1  |   File "/stable-diffusion-webui/modules/initialize.py", line 23, in imports
auto-cpu-1  |     import gradio  # noqa: F401
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/gradio/__init__.py", line 3, in <module>
auto-cpu-1  |     import gradio.components as components
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/gradio/components/__init__.py", line 3, in <module>
auto-cpu-1  |     from gradio.components.bar_plot import BarPlot
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/gradio/components/bar_plot.py", line 7, in <module>
auto-cpu-1  |     import altair as alt
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/__init__.py", line 649, in <module>
auto-cpu-1  |     from altair.vegalite import *
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/vegalite/__init__.py", line 2, in <module>
auto-cpu-1  |     from .v5 import *
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/vegalite/v5/__init__.py", line 2, in <module>
auto-cpu-1  |     from altair.expr.core import datum
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/expr/__init__.py", line 11, in <module>
auto-cpu-1  |     from altair.expr.core import ConstExpression, FunctionExpression
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/expr/core.py", line 6, in <module>
auto-cpu-1  |     from altair.utils import SchemaBase
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/utils/__init__.py", line 14, in <module>
auto-cpu-1  |     from .plugin_registry import PluginRegistry
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/utils/plugin_registry.py", line 13, in <module>
auto-cpu-1  |     from typing_extensions import TypeIs
auto-cpu-1  | ImportError: cannot import name 'TypeIs' from 'typing_extensions' (/opt/conda/lib/python3.10/site-packages/typing_extensions.py)
auto-cpu-1 exited with code 1

Then I thought I would try chainner, https://chainner.app/download
gives a deb file, for ease of installation. It has a dependency manager also. 

But the dependency manager does not install stable diffusion! So, trying

(
current python version is 3.11.5 on this Linux mint based on ubuntu 24.04 - which may run into problems,
- we're supposed to run on python 3.10-venv
)

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacty of 3.94 GiB of which 31.50 MiB is free. Including non-PyTorch memory, this process has 3.73 GiB memory in use. Of the allocated memory 3.60 GiB is allocated by PyTorch, and 77.43 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF


Stable diffusion model failed to load

Can try lowvram command, since we have the NVidia 1060 -  

export COMMANDLINE_ARGS="--lowvram"
./webui.sh

Success. Does a 512x512 generate in ~ 1 minute, 4x ESRGAN upscale resize in ~30 sec.
this has a listing of how to make prompts.

My primary use case was for upscaling. I took a picture which I had clicked with the mobile camera in its "50 megapixel" mode, which shows quite a bit of "softening" when viewed at 100% - then cropped to 4096x4096, scaled down to 512x512 (1k images were causing out of VRAM errors) and then scaled up to 4096x4096. Interesting to see the results for a jpg show lots of ringing, while the results for a png seems to make things sharper. 

A portion of jpg input on the left, with output on the right, processed as 
scale down to 512 px, save as jpg, and then upscale to 4x - this shows ringing. Click on the image to view larger size.

A portion of jpg input on the left, with output on the right, processed as 
scale down to 512 px, save as png, and then upscale to 4096 - this shows an interesting pointillistic effect, and has made edges if the image sharper - clearly seen in the foreground trees.  Click on the image to view larger size.

I'll write a separate post on Easy Diffusion, since this post has already become quite long.

Tuesday, April 01, 2025

experiments with LLMs and docker

Tried out some "easy to use locally" Large Language Models (LLMs) with the motivation of being able to use them with our own documents

Looking for easy ways to install and run them, via Run Local LLMs on Hardware from $50 to $50,000 - We Test and Compare!

ollama list
ollama pull llama3.1:latest (~4 GB - the download went at nearly 4 MBps on the Airtel 40 Mbps connection, and around 6 MBps on Airtel 5G - 12 minutes)
ollama pull llama3.2:latest (~2 GB, and runs faster on low-end hardware)

(run as ollama run llama3.1:latest --verbose to get tokens/sec.
)

With 3.1, it was going at ~4 tokens per second on the desktop. CPUs, I think. With llama3.2, it was going at 23 tokens / sec - quite an acceptable speed. On the Windows laptop, running on CPU, llama3.2 at 9 tokens / sec.


Wanted to try the docker route - installed docker via https://docs.docker.com/engine/install/ubuntu/ by setting up docker's repository. All my docker commands needed to be prefaced by sudo, since I didn't go through the post-install steps.

Trying to run open-webui with the commandline given in the youtube video, 

sudo docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama 
has the steps.

sudo docker pull ghcr.io/open-webui/open-webui:main

For Nvidia GPU support, add --gpus all to the docker run command:
Single user mode - 

sudo docker run -d -p 3000:8080 --gpus all -e WEBUI_AUTH=False -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

That also failed. "Unable to load models". Then, found that we need to start ollama separately in another terminal window, from How to Install Ollama, Docker, and Open WebUI on Windows - YouTube
and we need a different set of parameters in the command line if ollama is already installed on our system. So, got the openwebui to work with two modifications:
ran from administrator cmd (on Windows)
without -d (so that I can see the errors if any - do not detach)
and must start ollama first in another window with
ollama run llama3.2

 docker run  -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

But connecting stable diffusion to open-webui seems to be complicated - 

So, will run Stable Diffusion (for image generation and upscaling) from its own web ui instead.


creating ai agents - just links

 free plan is 1000 executions, 14 days.

via

Create Your FIRST AI Agent Today - youtube video

(uses paid n8n, paid chatgpt etc)

serpapi 100 searches per month in free plan - https://serpapi.com/pricing

n8n.io - can run locally also.

https://n8n.io/itops/ - using n8n - nodemation - to automate user creation etc. https://www.youtube.com/watch?v=ovlxledZfM4

https://blog.n8n.io/how-to-make-ai-chatbot/

Monday, March 31, 2025

set up app password for sending emails from server

One of our wordpress servers was to be used to send some emails to users. So, changed the ssmtp "from" account from our internal server's google account to another one, meant to be used for this server. 
  • Created a new google workspace account
  • Logged in to it for the first time
  • Set up recovery email, recovery phone, authenticator, enabled two-factor authentication
  • Created an app password
  • edited the /etc/ssmtp/ssmtp.conf with the new userid and app password.

Sunday, March 30, 2025

rclone - GUI and remote server-side copy possibilities

 Tried out the GUI at https://rcloneview.com/src/download.html using the Linux AppImage. The remote already configured for my user account was detected, but the directory listing didn't seem to work. Neither did the "Mount". 

Then, server-side copy possibilities - when doing remote to remote copies, the local network is used - so that's not really useful. And on the same remote, a server-side copy is done using hard links - https://forum.rclone.org/t/sftp-remote-server-side-copy/41867 - so that's a bit iffy. Server side copy might be useful for copying between two different cloud providers. The fastest option for our SFTP remotes would be to ssh into one of the machines, install rclone there, create an sftp remote for the other machine, and do the copy from the ssh shell - that would be much faster than bringing all the data to our local network and sending it back.

Saturday, March 29, 2025

gravity pdf plugin on wordpress needs php-gd

 While installing a plugin Gravity PDF on one of our wordpress sites, the users got an error message, PHP Extension GD Image Library not found.

Installed php-gd with

 sudo apt install php-gd

to solve this issue.

(Edit - Later, there was an issue raised about an add-on called Gravity Export which generates a public URL containing Gravity Form entries, with the public URL not being accessible. I guessed that was due to permission issues on WordPress - the export had been sent as email to the administrator.)

Thursday, March 27, 2025

compiling a wxwidgets project with cmake on Windows

Run into issues building the simple hello world example from wxwidgets using cmake on Windows. On Linux, the default github cmake action did not need much modification. 
Linux cmake build workflow

But for building on Windows, we needed to add some env variables in the workflow

env:
  # Customize the CMake build type here (Release, Debug, RelWithDebInfo, etc.)
  BUILD_TYPE: Release
  WXWIN: C:/wxWidgets-3.2.6
  wxWidgets_ROOT_DIR: C:/wxWidgets-3.2.6
  wxWidgets_LIBRARIES: C:/wxWidgets-3.2.6/include
  wxWidgets_INCLUDE_DIRS: C:/wxWidgets-3.2.6/lib/vc14x_x64_dll
  ARCH: x64

or else the find_package would complain about not finding wxWidgets_LIBRARIES and wxWidgets_INCLUDE_DIRS

Also, 

cmake -G"Visual Studio 17 2022" -B ${{github.workspace}}/build -DCMAKE_BUILD_TYPE=${{env.BUILD_TYPE}}

to specify which version of Visual Studio to use - and for this version, the default toolset is 14.3 which is the one used to build this set of wxwidgets binaries.
https://docs.wxwidgets.org/3.2.1/plat_msw_binaries.html

But we also had to add a dummy main() to get rid of a linker error, but that breaks the Linux build. 

/home/runner/work/wxOCVWarp/wxOCVWarp/hello.cpp:78:5: error: conflicting declaration of C function ‘int main()’
   78 | int main() { return 0;}
      |     ^~~~
In file included from /usr/include/wx-3.0/wx/wx.h:25,
                 from /home/runner/work/wxOCVWarp/wxOCVWarp/hello.cpp:3:
/home/runner/work/wxOCVWarp/wxOCVWarp/hello.cpp:11:1: note: previous declaration ‘int main(int, char**)’
   11 | wxIMPLEMENT_APP(MyApp);
      | ^~~~~~~~~~~~~~~
gmake[2]: *** [CMakeFiles/OCVWarp.dir/build.make:79: CMakeFiles/OCVWarp.dir/hello.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:87: CMakeFiles/OCVWarp.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2

So, wrapped an ifdef around the dummy main(), and all is well.

#ifdef _WIN64
int main() { return 0;}
#endif

Tuesday, March 25, 2025

rclone over ssh / sftp

 Tried installing the latest rclone - version v1.69.1 using the deb file and testing an ssh / sftp remote. Finally got it to work. Used the key_file option and passed on the path to our AWS-generated PEM file as the key_file - all other options were left as default with <ENTER>

A point to note is that if we do 

rclone ls nameofourremote:/

it will list all the files on the remote recursively! Instead, we can do

rclone lsd nameofourremote:/some/path

to list only the directories inside the specified path.

Edit: We can mount the drive remotely with sshfs also - on Windows, 

https://green.cloud/docs/how-to-mount-remote-file-systems-using-sshfs-on-windows/

On Linux,

https://www.digitalocean.com/community/tutorials/how-to-use-sshfs-to-mount-remote-file-systems-over-ssh

Apparently sshfs is deprecated, and rclone may be faster also - 

https://obviy.us/blog/comparing-sshfs-rclone-mount/

setting up a server with a dotnet api

A subset of the previous post. Steps carried out - 

- created a fresh user using adduser

- MySQL database was already running.

- created the subdomains for website and API

- created the appropriate virtual servers for the above.

- copied the uat files to prod

- created a service running on a port like 5000 or 5010 etc for the dot net api at /etc/systemd/system/ourapi.service

systemctl start ourapi

systemctl enable ourapi

(Once the team changes the files referred to by the api service , I will need to restart the service.)

- tested api and website url


Sunday, March 23, 2025

Large Language Models with our own documents - RAG for LLM

RAG = Retrieval Augmented Generation - can be done locally as well as on cloud - explanatory video - 

Feed Your OWN Documents to a Local Large Language Model!

In the video, openwebui is used to run the local LLM - 

https://docs.openwebui.com/features/

If using docker (for ease of setup), and we want to copy our documents to the LLM's docker container, Docker copy command - 

Copying Files to and from Docker Containers - GeeksforGeeks

docker container cp source_local container:/path

sudo docker cp ~/Desktop/to-be-copied.txt 135950565ad8:/to-be-copied.txt

(the container id like 135950565ad8 above, is usually seen as a root@containerid prompt in the running container)

And apparently in the latest versions of openwebui, there is integration inside the UI itself for uploading documents - https://docs.openwebui.com/features/rag#local-and-remote-rag-integration

"Local documents must first be uploaded via the Documents section of the Workspace area to access them using the # symbol before a query. Click on the formatted URL in the that appears above the chat box. Once selected, a document icon appears above Send a message, indicating successful retrieval."

See this post for some initial setup, experiments and benchmarks - https://hnsws.blogspot.com/2025/04/experiments-with-llms-and-docker.html

Saturday, March 22, 2025

dramatically different Azure pricing based on region

For our upcoming server migration (Edit - described here)

D2as v6 Central India is almost the same price as 
B2s South India!

D2as v6 is not available in South India.

Even cheaper is B2asv2.

Or D2asv4 - https://cloudprice.net/ allows a comparison.



Wednesday, March 19, 2025

message from Microsoft Azure - Convert to Azure role-based access control (RBAC)

 The email sent to us, saying "To avoid service disruptions, convert classic admin roles that still need access to your subscription to Azure RBAC roles immediately." quoted a subscription id, and checking that subscription, found that only one admin had classic admin roles, who was already added as "Owner" in RBAC roles. So, no action taken. 

Tuesday, March 18, 2025

some problems running Reaper on Linux

Trying to run Reaper on Linux, first there was an error message that libasound library could not be found,

https://askubuntu.com/questions/482478/libasound-so-2-cannot-open-shared-object-file-no-such-file-or-directory

sudo apt-get install liboss4-salsa-asound2

sudo apt install pulseaudio

But the real problem was, if you install the i686 instead of the x86-64 on an AMD64 64-bit system, you will also get this error message.

https://forum.cockos.com/showthread.php?t=255012

(that was the issue. )

Then, once Reaper opened, error opening audio device.

JACK is default, maybe I should install it?

Other options are pulse audio and alsa.

pulse audio is installed, but when I choose that option, Reaper hangs.

A reboot (after pulse audio was installed) solved the issue.

Saturday, March 15, 2025

upscaling a video (from HD to 4k, for example) using ESRGAN on Google Colab

 Tried out the first result in a web search for upscaling using google colab - 

https://colab.research.google.com/github/yuvraj108c/4k-video-upscaler-colab/blob/main/4k_Video_Upscaler_Colab_(Real_ESRGAN).ipynb

Copied to my google drive to make the changes persistent - the only changes needed were the paths to the input and output files. 

Without modifications, the upscaled video went to the google drive recycle bin instead - it looks like there is a missing 'mv' in the command = f"'{final_video_path}' '{save_directory_drive}'/'{final_video_name}'"

Anyway, input video with compression artifacts gets upscaled to output video with compression artifacts. If there's some clean video, will take a look. The upscaling inference goes at approx 2 seconds per frame on the T4 google colab GPU.

Thursday, March 13, 2025

planets "flyby" renders with OpenSpace


The full versions of these clips - more frames as well as 4096x4096 resolution - are available at https://archive.org/details/sun-2-rotate

Copy-pasting the notes posted on Github, for creating these "flybys" - 

  • for openspace to focus on a planet without making the camera rotate around it - In the settings > Navigation Handler there is a "Follow Anchor Node Rotation" and a "Follow Anchor Node Rotation Distance", smaller distance = the closer you can be to the object before following its rotation - https://openspacesupport.slack.com/archives/C055D75TJ9K/p1718704511126989?thread_ts=1718701308.415139&cid=C055D75TJ9K

  • For changing the length of the planet trails (orbit paths) - https://docs.openspaceproject.com/releases-v0.20/using-openspace/scripting/console/index.html for getting the trails length property, checking logs/ScriptLog.txt openspace.setPropertyValue("{planetTrail_solarSystem}.Renderable.Enabled", false) works. It is case sensitive. In Trail Appearance, currently the default is LineLength=1 Line Fade Amount is a small amount - something like 0.25. Increasing Line Fade Amount to 0.4 increases the length of the trail shown. openspace.setPropertyValue("{planetTrail_solarSystem}.Renderable.Appearance.LineFadeAmount", 0.4) in the console scripting window by hitting ` (backtick) key We need to again close the console window by hitting ` again, otherwise keyboard shortcuts like t to toggle all trails won't work - the keystrokes will be captured by the console.

  • The Sun size was increased to 36x and Sun glare was turned off for the Mercury + Sun clip.

escaping backticks in markdown

 While writing markdown in github, wanted to know how to escape the backtick ` character. 

https://stackoverflow.com/questions/24313204/how-does-one-escape-backticks-in-markdown

Apparently, we can do ``` ` ``` which will render as ` (note the space between the three initial backticks and the single one. We can also do \` when the backtick is not supposed to be inside a code block. 


Wednesday, March 05, 2025

google classroom overview video tutorial

 Shows both teacher view and student view - How to Use Google Classroom - Tutorial for Beginners from @TeachersTech

https://www.youtube.com/watch?v=pl-tBjAM9g4


Moodle server expansion

Copy-pasting from an email thread, responding to a query on how we can move forward with expansion plans - context was that a large number of users (presumably editing on Moodle) brought down the db server:

Various options:

1. Use CMS for content creation, and LMS for content distribution. Use only one or two concurrent users for loading content onto LMS. For training purposes, https://moodle.org/demo can be used instead of our server.

2. If the above does not work for you, consider shifting to Google Classroom - we can probably get an edu account, which would allow 500 concurrent users per class. Even in the free version, 250 concurrent teachers and students can log in. If you're predominantly going to use H5P, there are some workarounds for using H5P on google classroom, like https://h5p.org/node/137178 

Another option is Google Sites, which also allows easy creation of content pages like Wordpress, but free for us, and we don't need to add any servers - https://support.google.com/a/answer/6399191

3. If that also does not work for you, then consider repurposing some of your old developer workstations into servers, and use them instead for training + multi-user content creation purposes; at other times, when only content distribution is taking place, we can switch to the existing Azure servers within 5 minutes or so.

4. If none of the above, and you continue to want to work with Moodle, consider Moodle hosting service providers in India at ~$30 a month (compared to ~$60 a month we would spend on overages if we upgrade our Azure servers beyond the free credits) - https://cloudpap.com/blog/moodle-hosting-in-india/

Sunday, March 02, 2025

Customized Moodle app bug and solution

 Copy-pasting from a request for help in the Moodle forum - "we're facing a strange issue, and any help is welcome. The questions in the feedback module are visible when viewed on a web browser, but the questions are not visible when using a customized Moodle app. Even if we just build the Moodle app for Android without any customizations, the resulting app doesn't show the questions. "

I see now that this is a bug reported in the Bug Tracker for the app, https://tracker.moodle.org/browse/MOBILE-4698

The code in the "beta" branch has the fix, and the bug is fixed there, as I check today.

Friday, February 28, 2025

online panorama stitcher

 Via OpenCV directly in the browser (webassembly + webworker) - DEV Community

latsic/imgalign: Webapplication for image stitching and aligning

Needle - took some 30 seconds to a minute to load OpenCV on my machine, the actual stitching took less than two seconds with the sample 3 images.

using hand-held video clips on the planetarium dome

One of our colleagues wondered how nice it would be to see videos like this documentary on the "large screen" by projecting on the planetarium dome. We can use tools like OCVvid2fulldome to project the video to the front part of the tilted dome, say covering 180 degrees left to right. Trying out a variation of this technique, using OCVWarp on video padded to 4k equirectangular shape, reasonably distortion-free results for 
padding with equal amounts of black on all four sides, with
-136 degrees y angle in OCVWarp. 
Some parts of the video - the lower parts near the left and right edges - get cropped, but the visuals look good.

Unfortunately, the problem is that the mostly hand-held footage is extremely nausea-inducing when blown up to such a large size. Even the moderate shake of this professionally shot video segment is unacceptable on the dome without slowing down the footage or making the video window smaller.

Tuesday, February 25, 2025

appimage build on github actions - potential issue

linuxdeployqt will not allow building of appimages with ubuntu versions above 20.04 till 20.04 is out of support, which is end of May, 2025.

 But github actions has already started "brownout" of 20.04 - The Ubuntu-20.04 brownout takes place from 2025-02-01. For more details, see https://github.com/actions/runner-images/issues/11101

A workaround would be to use go-appimage instead, which does not have this hard limit on OS version. As was used for OpenSpace-AppImage.

fake astronomy posts on social media

I got a message about this,


Copy-pasting from my reply, 
As is usual with most email forwards, this is fake. The actual paths are described in the post below,


Then there was the "planets lining up" forward from January - for which my reply was https://earthsky.org/tonight/planetary-alignment-january-25-2025/



Thursday, February 20, 2025

adding two wordpress sites to an existing server

We had a server which had Apache and Mysql already installed (as well as a couple of Wordpress sites already). Steps followed, in brief, to add two more Wordpress sites:
  1. mysql> create database wordpresswebsitestaging;

    CREATE USER "stagingwpuser"@"localhost" IDENTIFIED BY "ThePasswordH3r3";
    GRANT ALL PRIVILEGES ON wordpresswebsitestaging.* TO "stagingwpuser"@"localhost";
    FLUSH PRIVILEGES; 

    https://developer.wordpress.org/advanced-administration/before-install/creating-database/

  2. Creating the web root directories and a user who can sftp in to upload files etc if needed
    sudo adduser thewpwebsiteadmin
    sudo su thewpwebsiteadmin
    cd /home/thewpwebsiteadmin
    mkdir directory1
    mkdir directory2


    We need to give read permissions to the entire path for the webserver to be able to read these directories, so
    sudo chmod -R +rx /home/thewpwebsiteadmin


  3. Created DNS entries in cloudflare. In this case, CNAME records.

  4. Copied existing files in /etc/apache2/sites-available as templates for the new sites. Since we were using cloudflare origin certificates, there was no need to create new certificates with certbot etc. After editing the conf files as needed, with the /home/thewpwebsiteadmin/directory1 and directory2 as home directories, enabled the sites with
    sudo a2ensite newsite1 newsite1ssl newsite2 newsite2ssl
    sudo systemctl reload apache2


  5. Path to latest wordpress from https://developer.wordpress.org/advanced-administration/before-install/howto-install/
    wget https://wordpress.org/latest.tar.gz
    tar -xzvf latest.tar.gz
    cp -R wordpress/* directory1
    etc

  6.  Visit the new sites with a browser to supply the database configuration etc, the wp-config file gets generated, create a new file and paste the generated file to the wordpress root directories as
    nano wp-config.php

  7.  Create ssh key and add to authorized_keys - 
    ssh-keygen -t rsa -m PEM -f theadminuserkey.pem
    mkdir .ssh
    mv theadminuserkey.pem.pub .ssh/authorized_keys

    and set appropriate permissions for the .ssh directory and file - 
    chmod 700 .ssh
    cd .ssh
    chmod 600 authorized_keys

Monday, February 17, 2025

cloudflared or ssh tunnel for reverse-proxying on-premise servers

Exploring what we can do with cloudflare - 


2. But BSNL, Airtel etc would block ports. So, it would be far easier to use a tunnel with an outgoing ssh connection to a VM or something like that. That way, we don't have to depend on port-forwarding and availability of unblocked ports. We can use -R in the ssh tunnel for reverse-proxying.

3. But then, we may not even need a VM - currently Cloudflare Zero Trust Web access supports private apps, which would ask clients for authentication, and then allow them to connect to our private server via cloudflared tunnels.
https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/get-started/create-remote-tunnel/
and give it a public hostname with
Starting from the page below, we can go "next, next" to finish all the steps to achieve ZT web access.


Saturday, February 15, 2025

app api call slowdown, QUIC is suspected

It was reported that one of our Android apps was experiencing delays and timeouts in api calls, which hit a dot net backend running on Linux. Apparently the errors they saw were
 net::ERR_QUIC_PROTOCOL_ERROR

I suggested that they disable QUIC in their browser, https://thegeekpage.com/err_quic_protocol_error/

But apparently the app was having the errors and not in a browser. Since I'm not a dot net or Android developer, I'm not sure how one can disable QUIC in a generalized manner. But I could configure the server's firewall to explicitly Deny QUIC requests (UDP requests on port 443) which would then immediately make the client fall back to HTTP/2 (TCP packets).

There was an update pending on the server. Completing that and rebooting, still the issue continued. "For the past week, we've been experiencing buffering and delays in the app, especially on 3G networks". Maybe there was some change in either the network or Android or something like that? Cause could be any of those. Anyway: set up a firewall rule to Deny UDP to ports 80 and 443, and also disabled QUIC for the entire domain in cloudflare, https://developers.cloudflare.com/speed/optimization/protocol/http3/

That seems to have solved the issue. 

Monday, February 10, 2025

Miscellaneous Blender VSE settings used for the recent project

Some settings and shortcuts I revised when working on a recent project - 

I thought that I should prevent the creation of "proxies" in Blender VSE (Video Sequence Editor) since the working SSD was only 256 GB, with only around 100 GB free. 

First, I set the 
Proxy setup to Manual
Proxy render size to 25%
Then I thought of disabling proxies altogether, so set the 'Use Proxies' checkbox to False (pull out menu on right of rendered view)

Image showing all these, from the page above,


How to rotate an HDRI background - https://www.youtube.com/watch?v=fqd4mc42sZQ
Shading tab - Choose World instead of Object as the subject of the shading editor, rotate using Vector Mapping node ahead of the single image HDRI as in the screenshot below,


Cutting video clips into pieces which do not show the previously existing clip sections when stretched - is by "Hold Split" - https://vse-docs.readthedocs.io/video_editing/edit/montage/transform.html#split
Shift + K to 'hold-split' at the cursor.

Shift + A is the shortcut key to add speed control strip (and many other things) - https://vse-docs.readthedocs.io/video_editing/edit/effects/speed.html - Add > Effect > Speed Control

Similarly, for a cross-fade transition, select begin clip, Shift click to select end clip, then Shift+A - Add > Transition > Cross - https://vse-docs.readthedocs.io/video_editing/edit/effects/transition.html - we can change the order of the transition later, if needed, from the side bar, by changing the Input 1 and Input 2 strips as in the screenshot below.




Sunday, February 09, 2025

listing ffmpeg commandline options


You can see available presets including lossless
with
ffmpeg -h encoder=hevc_nvenc

List of the hwaccels available in the ffmpeg executable, 
ffmpeg -hwaccels

So, added
-preset lossless
to my commandline, and it worked. I think it was something like

ffmpeg -r 30 -start_number -003875 -f image2 -preset lossless -i "OpenSpace%06d.png" -c:v hevc_nvenc ../filename.mp4

Got the message
The selected preset is deprecated. Use p1 to p7
(but it is doing the encoding.)
There is also -tune lossless option. But I didn't try that.


Thursday, February 06, 2025

simple static multi-lingual titles instead of tracked titles

Instead of going the tracked title route, I settled on a simpler static title overlay using Blender VSE, because

  • It's far less time-consuming
  • Hindi text does not render well as a Blender text object

So,the workflow was:

  • Create the title text in a document editor like Google Docs
  • Take a screenshot of the black text on white background
  • Invert to white text in black background on Gimp
  • Overlay over "warped" video directly with opacity fading in from 0 to 0.5 over a second - OR -
  • If overlaying over a fulldome clip, first "warp" the flat text to match the fulldome clip's curvature using the method in this old post.

In particular, after trial and error, the particulars seem to be:

  • 36 point font size text to take screenshot
  • stretch the text to make it square (halve the width)
  • put the stretched screenshot at the bottom of a 4096x4096 transparent image
  • make the screenshot even smaller if required, or use a larger transparent image
  • merge down the layer if the filter doesn't act on the entire image
  • use Gimp's Filter > Distorts > Polar co-ordinates with "Map backwards" checked (otherwise the text would be inverted left to right) and save as png preserving transparency (alpha channel).


 


Payzapp mini review

I tried using HDFC Bank's Payzapp app for payments - it has features like loading up the Payzapp wallet with credit card, then transfer by UPI to bank accounts, etc. But, 

  • they doubled the fee for Payzapp wallet loading from credit card
  • many merchants, including the most important one which I needed to pay > 10k per month, blocked access to Payzapp payments due to fees they're charging merchants
  • a billdesk payment of APSPDCL went through, but did not reflect in the "paid" bills. Billdesk said merchant had rejected payment, I need to pay again. Payzapp says refund has been issued, but there is no way to check except by seeing the balance - which I have now emptied, because the back-and-forth between Payzapp and Billdesk had taken more than a month.
     

Overall, a very poor experience. So I will use Payzapp only for the convenience of paying HDFC Bank credit cards with it, will block all notifications from it, and not keep any wallet balance. Even when making HDFC bank credit card payment, the app failed to load this morning, and loaded again later in the day. Also, the Payzapp app shows "apply coupons" when making a Rs. 4000 payment, but the coupon says min. transaction Rs. 5000, and the "apply coupon" button is not shown when I was actually making the Rs. 10,000 transaction! Bad HDFC! We can contrast this with Amazon.in's method of showing exactly how much "cashback" or "savings" will accrue for each payment method.

Tuesday, February 04, 2025

monitoring expiring certificates with Red Sift

 LetsEncrypt sent out an email that they would no longer be sending certificate expiry warning emails, and recommended we sign up to Red Sift's free plan. 

We can recommend Red Sift Certificates Lite, which provides free expiration emails for up to 250 active certificates: https://redsift.com/pulse-platform/certificates-lite

After signing up, there was a certificate warning, but it turned out to be a false alarm - a certificate which had been re-issued onto a new machine.