Ubuntu using userland on Android

Trying out UserLAnd - https://userland.tech/features

via https://www.geeksforgeeks.org/linux-on-android/

Rs. 190 to buy pro version.

Downloaded 80 MB, on 5G took less than 2 minutes.

Setting up file system - another 2 minutes, then came up without any browser or remote desktop.

Find ip address from Android settings > Wifi > Connection > scroll down to Details.

Port is 5951, Via

https://github.com/CypherpunkArmory/UserLAnd/issues/1133

xtightvncviewer

VNC pw is found via Filesystem > long press on "apps" > Edit 

sudo apt update

Reading package lists... Error!

W: Unable to read /etc/apt/sources.list - RealFileExists (38: Function not implemented)

sudo apt-get update

works

sudo apt-get upgrade

says 204 packages to be upgraded.

Connection failed, since vnc was being upgraded.

ssh with username=userland did not work over the network on port 22, but worked OK on localhost.

Problems when trying to install a browser - 

Preparing to unpack .../firefox_1%3a1snap1-0ubuntu2_arm64.deb ...

debconf: unable to initialize frontend: Dialog

debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)

debconf: falling back to frontend: Readline

=> Installing the firefox snap

==> Checking connectivity with the snap store

===> System doesn't have a working snapd, skipping

Unpacking firefox (1:1snap1-0ubuntu2) ...

Setting up firefox (1:1snap1-0ubuntu2) ...

update-alternatives: using /usr/bin/firefox to provide /usr/bin/gnome-www-browser (gnome-www-browser) in auto mode

update-alternatives: using /usr/bin/firefox to provide /usr/bin/x-www-browser (x-www-browser) in auto mode

Command '/usr/bin/firefox' requires the firefox snap to be installed.

Please install it with:

snap install firefox

 sudo apt-get install snapd

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

snapd is already the newest version (2.67.1+22.04).

 sudo snap install firefox 

error: cannot communicate with server: Post "http://localhost/v2/snaps/firefox": dial unix /run/snapd.socket: connect: no such file or directory

snap install firefox

error: cannot communicate with server: Post "http://localhost/v2/snaps/firefox": dial unix /run/snapd.socket: connect: no such file or directory

via

https://packages.debian.org/sid/arm64/firefox/download

 wget http://http.us.debian.org/debian/pool/main/f/firefox/firefox_137.0.2-1_arm64.deb

sudo dpkg -i firefox*.deb

Selecting previously unselected package firefox.

(Reading database ... 59497 files and directories currently installed.)

Preparing to unpack firefox_137.0.2-1_arm64.deb ...

Unpacking firefox (137.0.2-1) ...

dpkg: dependency problems prevent configuration of firefox:

 firefox depends on libasound2t64 (>= 1.0.16); however:

  Package libasound2t64 is not installed.

 firefox depends on libatk1.0-0t64 (>= 1.12.4); however:

  Package libatk1.0-0t64 is not installed.

 firefox depends on libc6 (>= 2.39); however:

  Version of libc6:arm64 on system is 2.35-0ubuntu3.9.

 firefox depends on libevent-2.1-7t64 (>= 2.1.8-stable); however:

  Package libevent-2.1-7t64 is not installed.

 firefox depends on libglib2.0-0t64 (>= 2.38.0); however:

  Package libglib2.0-0t64 is not installed.

 firefox depends on libgtk-3-0t64 (>= 3.13.7); however:

  Package libgtk-3-0t64 is not installed.

 firefox depends on libnss3 (>= 2:3.109~); however:

  Version of libnss3:arm64 on system is 2:3.98-0ubuntu0.22.04.2.

 firefox depends on libvpx9 (>= 1.12.0); however:

  Package libvpx9 is not installed.

 firefox depends on libxcomposite1 (>= 1:0.4.6); however:

  Version of libxcomposite1:arm64 on system is 1:0.4.5-1build2.

dpkg: error processing package firefox (--install):

 dependency problems - leaving unconfigured

Processing triggers for desktop-file-utils (0.26-1ubuntu3) ...

Processing triggers for hicolor-icon-theme (0.17-2) ...

Processing triggers for man-db (2.10.2-1) ...

Errors were encountered while processing:

 firefox

via

https://ubuntu.pkgs.org/20.04/ubuntu-updates-main-arm64/firefox_136.0+build3-0ubuntu0.20.04.1_arm64.deb.html

userland@localhost:~$ sudo dpkg -i firefox*.deb

dpkg: warning: downgrading firefox from 137.0.2-1 to 136.0+build3-0ubuntu0.20.04.1

(Reading database ... 59587 files and directories currently installed.)

Preparing to unpack firefox_136.0+build3-0ubuntu0.20.04.1_arm64.deb ...

Unpacking firefox (136.0+build3-0ubuntu0.20.04.1) over (137.0.2-1) ...

dpkg: warning: unable to delete old directory '/usr/share/firefox/distribution': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox/browser/defaults/preferences': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox/browser/defaults': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox/browser/chrome/icons/default': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox/browser/chrome/icons': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox/browser/chrome': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox/browser': Directory not empty

dpkg: warning: unable to delete old directory '/usr/share/firefox': Directory not empty

dpkg: dependency problems prevent configuration of firefox:

 firefox depends on libgdk-pixbuf2.0-0 (>= 2.22.0); however:

  Package libgdk-pixbuf2.0-0 is not installed.

dpkg: error processing package firefox (--install):

 dependency problems - leaving unconfigured

Processing triggers for desktop-file-utils (0.26-1ubuntu3) ...

Processing triggers for hicolor-icon-theme (0.17-2) ...

Processing triggers for man-db (2.10.2-1) ...

Errors were encountered while processing:

 firefox

sudo apt-get install  libgdk-pixbuf2.0-0

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

You might want to run 'apt --fix-broken install' to correct these.

The following packages have unmet dependencies:

 libgdk-pixbuf2.0-0 : Depends: libgdk-pixbuf-xlib-2.0-0 (>= 2.40.2-2build4) but it is not going to be installed

E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

userland@localhost:~$ sudo apt-get install  libgdk-pixbuf     

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

E: Unable to locate package libgdk-pixbuf

userland@localhost:~$ sudo apt --fix-broken install

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

Correcting dependencies... Done

The following additional packages will be installed:

  firefox

The following packages will be upgraded:

  firefox

1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

1 not fully installed or removed.

Need to get 72.3 kB of archives.

After this operation, 248 MB disk space will be freed.

Do you want to continue? [Y/n] 

Get:1 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 firefox arm64 1:1snap1-0ubuntu2 [72.3 kB]

(which again fails due to snap being broken.)

via

https://www.omgubuntu.co.uk/2022/04/how-to-install-firefox-deb-apt-ubuntu-22-04

https://support.mozilla.org/en-US/kb/install-firefox-linux#w_install-firefox-deb-package-for-debian-based-distributions

firefox

[Parent 32675, Main Thread] WARNING: read-only dup failed (Permission denied); not using memfd: file /builds/worker/checkouts/gecko/ipc/glue/SharedMemory_posix.cpp:218

[GFX1-]: glxtest: ManageChildProcess failed

[GFX1-]: No GPUs detected via PCI

[32675] Sandbox: Couldn't open video device /dev/video215

[32675] Sandbox: Couldn't open video device /dev/video102

[32675] Sandbox: Couldn't open video device /dev/video9

[32675] Sandbox: Couldn't open video device /dev/video8

[32675] Sandbox: Couldn't open video device /dev/video155

[32675] Sandbox: Couldn't open video device /dev/video11

[32675] Sandbox: Couldn't open video device /dev/video216

[32675] Sandbox: Couldn't open video device /dev/video6

[32675] Sandbox: Couldn't open video device /dev/video214

etc

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

[GFX1]: no fonts - init: 1 fonts: 14 loader: 0

etc

firefox opens, but all tabs crash.

Gah - your tab just crashed.

(Then it crashes the userland ubuntu also.)

sudo apt-get epiphany-browser

(package name via software manager on desktop)

Trying to run OCVWarp ARM64AppImage on the userland ubuntu,

did install fuse with

sudo apt-get install libuse2

but

fuse: failed to open /dev/fuse: Permission denied

Cannot mount AppImage, please check your FUSE setup.

https://docs.appimage.org/user-guide/troubleshooting/fuse.html

 sudo modprobe -v fuse

bash: line 1: modprobe: command not found

 squashfs-root/AppRun

GST_PLUGIN_SCANNER=

GST_PLUGIN_SYSTEM_PATH=/home/userland/Downloads/OCVWarp-4.01/build/squashfs-root/lib/aarch64-linux-gnu/gstreamer-1.0

GST_PLUGIN_PATH=/home/userland/Downloads/OCVWarp-4.01/build/squashfs-root/lib/aarch64-linux-gnu/gstreamer-1.0

squashfs-root/AppRun: 101: exec: /home/userland/Downloads/OCVWarp-4.01/build/squashfs-root/usr/lib/ld-linux-aarch64.so.1: Permission denied

sudo  squashfs-root/AppRun

GST_PLUGIN_SCANNER=

GST_PLUGIN_SYSTEM_PATH=/home/userland/Downloads/OCVWarp-4.01/build/squashfs-root/lib/aarch64-linux-gnu/gstreamer-1.0

GST_PLUGIN_PATH=/home/userland/Downloads/OCVWarp-4.01/build/squashfs-root/lib/aarch64-linux-gnu/gstreamer-1.0

squashfs-root/AppRun: 101: exec: /home/userland/Downloads/OCVWarp-4.01/build/squashfs-root/usr/lib/ld-linux-aarch64.so.1: Permission denied

sudo apt-get install epiphany-browser 

running the above from vnc would crash.

run from ssh on local shell, installed 100+ MB of dependencies.

epiphany-browser  

** (epiphany:2707): WARNING **: 04:54:34.367: Disabled hardware acceleration because GTK failed to initialize GL: No GL implementation is available.

** (epiphany:2707): WARNING **: 04:54:34.400: WebKitWebContext:use-system-appearance-for-scrollbars property is deprecated and does nothing

** (epiphany:2707): WARNING **: 04:54:34.526: WebKitWebContext:use-system-appearance-for-scrollbars property is deprecated and does nothing

** (epiphany:2707): WARNING **: 04:54:35.209: webkit_settings_set_enable_dns_prefetching is deprecated and does nothing.

bwrap: Can't read /proc/sys/kernel/overflowuid: Permission denied

** (epiphany:2707): ERROR **: 04:54:35.657: Failed to fully launch dbus-proxy: Child process exited with code 1

Trace/breakpoint trap

And using qemu, once again, likely to be too slow - 

https://virtualmachinery.weebly.com/operating-systems.html

(on ipad, ubuntu was taking 30 minutes+ to boot up.)

So, I could try the ARM64 OCVWarp on raspberry pi instead. (which won't work out of the box, may need to compile on the Pi, since Raspberry Pi may use a different architecture compared to the github ARM64 runner - https://linuxvox.com/blog/what-is-difference-between-arm64-and-armhf/ .)

airtel's horrendous "AI" customer service

I was trying to recharge another prepaid Airtel number with "international roaming plan 2997" for 365 day validity.

Tried with Airtel thanks app from my Airtel mobile - failed.

Tried from website - failed.

Tried calling customer care - they said try going to a shop.

Tried from shop - failed.

Tried sending them an email to 121@in.airtel.com - "We noticed that you're contacting us from an email ID that isn't linked to your Airtel account. 

You shall write back to us from your registered ID or reach out to airtel thanksapp get instant support 24x7 right from the help and support section."

The help and support section is a chatbot. There didn't seem to be any option to contact a human being. 

Finally, I thought maybe trying from the Airtel app on the actual prepaid account for which Intl. Roaming was needed - success. Edit - half an hour later, that transaction was reversed, too!  

Airtel's own customer service people did not know this, the shop guys (tried two different shops) did not know this, it is not mentioned anywhere that these intl. roaming plans need to be bought from the phone which is actually going to roam! 

UTM for OpenSpace on iOS / MacOS

Found on https://getutm.app/faq/#what-are-the-limitations

"The lack of hardware virtualization on Apple A-chips means that even for ARM code we must re-compile it with JIT. Therefore performance would never reach the levels possible with KVM. There is also no support for GPU virtualization so that means no DirectX or OpenGL. This makes most modern games non-playable." So, OpenSpace is most probably not suitable to run via UTM on iPad / iOS.

More about JIT - https://faq.altstore.io/altstore-classic/enabling-jit/altjit

Found on https://mac.getutm.app/

"UTM does not currently support GPU emulation/virtualization on Windows and therefore lacks support for 3D acceleration (e.g. OpenGL and DirectX). You may be able to run older games with software rendering options, but nothing with hardware acceleration. There is experimental support for hardware OpenGL acceleration on Linux through Virgl."

So, OpenSpace on Ubuntu might be doable - at least with performance similar to a Windows box without NVidia graphics - on a Mac with UTM.

---------------------------

Details of what didn't work

Found the limitations the hard way, trying for many hours on the (9th gen) iPad (since I don't have access to a Mac at the moment). 

UTM SE - with no JIT - took half an hour or more to boot Ubuntu 14.04 (that's not a typo, it really was the old version 14.04)

Trying to enable JIT - 

Directly on AltStore - need a Hackintosh or Mac for it? - https://github.com/altstoreio/AltStore/issues/1349

Jitstreamer-EB - https://jkcoxson.com/jitstreamer

says use stikJIT - https://stikdebug.xyz/

StikJIT user manual

https://github.com/StephenDev0/StikJIT/blob/main/user-manual.md

Apparently enabling JIT is a one-time installation - https://faq.altstore.io/altstore-classic/enabling-jit

We can't use Altstore PAL since we're not in the EU - tried and failed.

To install a downloaded ipa, go to Altstore > My apps (at the bottom), and click the + on the top. The ipa should be visible. 

(if ipa is not visible, we may need to navigate using the files app, and "open with" Altstore.

Then, it asks for apple creds.)

Transferred the pairing file using icloud drive.

Long press on UTM, then "enable jit", then try to open the vm, then

error trying to connect to spice server.

If StosVPN is turned off, then AltJIT not found, can't run VMs without JIT.

FAILED TO CONNECT TO SPICE

https://github.com/utmapp/UTM/discussions/6362

To force quit, 

swipe up and hold to open app switcher, then swipe up on the app to force-quit it.

Jitstreamer EB on Debian guide is here,

https://github.com/jkcoxson/JitStreamer-EB/blob/master/install-docs/jitstreamer-eb-debian-docker-instructions.md (not yet tried this.)

To try - minimal Ubuntu or something like that without 3d acceleration, which may not need spice.

debian 11 ARM64 lxde with spicetools installed, from gallery.

It was a zip file. Open in Downloads > unzipped. Long press and open in UTM.

But same error, QEMU exited no error message.

UTM also says Virtualization is not supported on your device.

So we can only use Emulate.

Still, same error. Then, I tried UTM SE, which took half an hour to boot the old version of Ubuntu as noted at the top of this post.

cloudflare plugin for certbot

Found this option, 

--dns-cloudflare      Obtain certificates using a DNS TXT record (if you are using Cloudflare for DNS). (default: False)

from

https://eff-certbot.readthedocs.io/en/stable/man/certbot.html

And documentation for this plugin is at

https://certbot-dns-cloudflare.readthedocs.io/en/stable/index.html

(example invocation at end)

and how to install plugin is 

sudo apt-get -y install python3-certbot-dns-cloudflare

https://installati.one/install-python3-certbot-dns-cloudflare-ubuntu-20-04/

automating backup with python - rclone and google drive

There was a python script written to

1. Compress the audio files that are created from the previous backup date to now into a folder and zip it 

2. Export the zip file to Gdrive.

2. Extract the mysql backup into a .sql file, export that also into the gdrive folder.

3. delete the past backups. 

For this, rclone and zip needed to be installed. After that, I passed on abbreviated instructions for setting up an rclone remote for GDrive - 

You will need to use the method "(N) running rclone on a machine without web browser" from the instructions onscreen, which are also shown at

https://rclone.org/drive/

As mentioned here, 

https://hnsws.blogspot.com/2022/11/rclone-and-out-of-band-flow.html

you need to download the latest rclone and install on your local system (where you have access to a web browser, where you can log on to google and give permission for rclone etc when prompted), and follow the onscreen instructions to set up rclone config for your desired google drive login.

After they set up the script, remote etc and found them working, set up a daily cron for 12:15 midnight,

15 0 * * Sun /usr/bin/python3 /home/path-to/backup_service.py 

which the user can edit from their login using

crontab -e

migrating web servers and associated scripts using rclone copy

As mentioned in the previous post, our server migration involved consolidating a web server into a DB server - we chose this method, since cloning the DB server was quick, while export followed by import of MySQL databases takes quite a bit longer as noted in the earlier 2024 migration post.

Copied the ssh private key for the destination server to the source server, and created an rclone remote using SFTP to the destination server. 

zip -r -q webserver1.zip /path/to/webserver/files

(repeated this for all the web server directories later)

rclone copy webserver1.zip rcloneremote:/home/azureuser

(repeated this for all the other zip files later)

From the destination server,

unzip webserver1.zip

cd path/to/webserver/

sudo mv files /path/to/webserver

cd /path/to/webserver

sudo chown -R azureuser:www-data .

sudo chmod -R 775 .

----------------------------------------------

Then, the apache config files - 

sudo zip -r -q etcapache.zip /etc/apache2

(rclone copy etc, and change ownership to root)

and the letsencrypt certificates etc,

sudo zip -r -q etclets.zip /etc/letsencrypt

(rclone copy etc, and change ownership to www-data for the file/directory which it needs, and the others to root)

-----------------------------------------------

With these, the files to be copied for the web servers were done. And I could start them up - 
sudo a2ensite webserver1
sudo systemctl reload apache2
etc.

Also needed were

• installing all the necessary apache and php modules - which I had done earlier using the steps at the 2024 migration post.
• installing restic for backups as per the backups section of that post.
• rclone copy, individually, for the .sh backup scripts and the notifier directory.
• Copy-pasting lines from .config/rclone/rclone.config  as noted in the restic backup script - the rclone config was modified by copy pasting the extra remotes from source to destination.
• Copy-pasting cron jobs from both crontab -e and sudo crontab -e
• Setting up postfix with gmail smtp as mentioned at this post for which 
  sudo apt remove ssmtp
  sudo apt install mailutils postfix
  and then editing the conf files was required.

-----------------------------------------------

Then, we had awstats running, so 

sudo zip -r -q etcawstats.zip /etc/awstats

for the config files, and

sudo apt install awstats

with editing out the cron which is auto installed at
sudo nano /etc/cron.d/awstats

and then copying across the awstats data - the DataDir was set to this directory -  

sudo zip -r -q varlibawstats.zip /var/lib/awstats

(then rclone copy, then unzip on the destination machine, move and set appropriate permissions etc) 

----------------------------------------------------

Tested the backup and load alert notification scripts, checked awstats the next day to verify that the stats were updating correctly.

------------------------------------------------------

Edit: one week later - some more configurations which were missed out.

sudo apt install certbot python3-certbot-apache

sudo certbot --apache

This was sufficient to enable auto-updates for the letsencrypt SSL certificates which had been copied over from the old server.

On another server, onto which we had migrated a single domain, but which already had other active letsencrypt SSL encrypted websites,

sudo certbot --cert-name subdomain.ourdomain.org

for renewing only this one, without disturbing the others.

For removing a certificate later, we can use

certbot delete --cert-name example.com

[so you will need to know the exact cert-name - not the specific FQDN(or domain name) within the cert]

[you can get the cert names with: certbot certificates]

https://community.letsencrypt.org/t/how-do-i-remove-a-certificate/121475

-----------------------------------------------------------

One of the Moodle instances gave an error, Adhoc task failed: assignfeedback_editpdf\task\convert_submission,The configured path to ghostscript is not correctly set: The ghostscript path points to a non-existent file

Solution was to install ghostscript,
sudo apt install ghostscript

--------------------------------------------------------

Then the php max_file_upload also had to be modified according to this earlier post, https://hnsws.blogspot.com/2021/03/some-post-install-changes-needed-for.html

sudo nano /etc/php/8.3/apache2/php.ini

( Ctrl+w to find and change post_max_size

Ctrl+w to find and change upload_max_filesize

Ctrl+w to find and change max_execution_time  - set to 600 for now.)

sudo systemctl restart apache2

and also did the same for /etc/php/8.3/cli/php.ini

-----------------------------------------------------------

When upgrading a plugin, a warning was shown that php extension soap was not installed, and that update will continue only after it is installed. So,

sudo apt install php-soap

-------------------------------------------------------------

Edit - a few days later - 

https://linuxize.com/post/how-to-change-hostname-on-ubuntu-22-04/

sudo hostnamectl

sudo hostnamectl set-hostname our-desired-hostname

-------------------------------------------------------------

Then, a warning while updating a plugin - max_input_vars must be at least 5000

sudo nano /etc/php/8.3/apache2/php.ini

sudo nano /etc/php/8.3/cli/php.ini

sudo systemctl restart  apache2

-------------------------------------------------------------

migrating Azure VM from one tenant to another

Azure documentation talks about migrating a VM from one region to another region - https://learn.microsoft.com/en-us/azure/resource-mover/tutorial-move-region-virtual-machines

https://argano.com/insights/articles/microsoft-azure-clone-vm-how-to-clone-a-vm-in-azure.html

(this seems to be within same tenant)

But we wanted to shift from one tenant to another. Following this post, 
https://medium.com/@morsi.masmoudi/complete-guide-to-cloning-and-moving-a-vm-from-one-azure-tenant-to-another-8fa65dd6955a

but with some changes - instead of a file share, creating a disk from snapshot seems to need a blob container for the "Create Managed Disk" applet nowadays. Following the exact steps as above results in not finding any container in that storage account.

Shifted using the following steps:

1. Enabled CLI maintenance mode on the Moodle instances with 
sudo -u www-data /usr/bin/php admin/cli/maintenance.php --enable
- verified that this does stop the cron also, which would otherwise hit the database every minute.

2. Shut down the database VM, went to its OS disk, created snapshot, went to the snapshot's page, "Export snapshot", copied the export snapshot url. 

3. I had earlier created a test VM in the destination tenant, in the Central India region which is the cheapest at present. Had also created a storage account, and a blob container. Also, had installed azcopy on that VM with the following - https://www.thomasmaurer.ch/2019/05/how-to-install-azcopy-for-azure-storage/

#Download AzCopy

wget https://aka.ms/downloadazcopy-v10-linux

 

#Expand Archive

tar -xvf downloadazcopy-v10-linux

 

#(Optional) Remove existing AzCopy version

sudo rm /usr/bin/azcopy

 

#Move AzCopy to the destination you want to store it

sudo cp ./azcopy_linux_amd64_*/azcopy /usr/bin/

4. Created SAS token (Shared Access Signature) from Azure portal > thestorageaccount - the default 3600 seconds may be too short if we need to do multiple operations. Enabled container, blob and file, all three, and copied the blob url and sas token strings. The blob url would be of the form https://storageaccountname.blob.core.windows.net/?sv=SAS-token - we need to edit this, and add nameofourcontainer/desired-name-of-file.vhd before the ?sv=SAS-token

5. SSH'd into that VM, and ran azcopy. (I could have avoided creating the VM, installing azcopy etc by using an Azure shell instead.) The command will be of the form:

azcopy copy "https://md-abcefg12nmg.z12.blob.storage.azure.net/snapshot/abcd?sv=2018-03-28&sr=b&si=this-is-the-snapshot-url-1234566b61d3&sig=zj9jh0ABCDEDFGHHJtnkD9yK0rvJ44I%3D" "https://lmsmigratestorage.blob.core.windows.net/testcontainer/LMSDBsnapshot.vhd?sv=2024-11-04&ss=bfqt&srt=sco&sp=abcdefghytfx&se=2025-04-21T11:49:12Z&st=2025-04-20T03:49:12Z&spr=https&sig=qwertyuiopasdfghjkABCDEFGuLYlgBQlWZE%3D"

# Actually ran steps 2 and 5 three times, azcopy took the following times:

# Elapsed Time (Minutes): 2.034 <-- LMS db OS disk

# Elapsed Time (Minutes): 1.2004 <-- LMS php disk

# Elapsed Time (Minutes): 22.0734 <-- LMS data disk
 

6. From Azure portal, in the destination tenant, Home > Create > Managed Disks > (Choosing Source as Blob storage) > Browse > choosing our VHD snapshot file in blob storage. Creating disk from snapshot, creating snapshot etc are very fast operations, these operations complete in less than a minute.  

7. From the created disk, if it is an OS disk, Create VM. Note that in our case, the default "Operating System type" was v1, but our original VM was v2, so I created the new VM also as v2. 

8. The data disk can be attached later to the VM, even when it is running - "Attach existing disk". Caution - attaching an OS disk as data disk does not seem to work properly - that new OS disk becomes the primary OS disk! 

9. In our case, we were cloning the db VM and copying the php files (and postgresql db) from the php server into the db VM, since we wanted to consolidate the two VMs into a single one. So, I had to copy over from the old VM to the new VM using rclone copy. What I needed to copy, and the steps taken, will be a separate post.

(What does not work:

1. "Export disk" and trying to use that using azure cli - this will not work. We need to create a snapshot first, then export the snapshot, and use the snapshot to re-create the disk in the new tenant.

2. Mounting azure blob storage on a VM and using wget to copy the snapshot VHD to blob storage will not help us to create the disk in the new tenant - azcopy is mandatory.

3. wget -O mydisksnapshot.vhd "https://snapshot-export-URL" followed by
azcopy copy mydisksnapshot.vhd "https://blob-storage-url" will work, but takes double the time - azcopy works fine directly from snapshot url to blob storage url. Caveat is that we must make a container first, and then use azcopy copy.) 

clearing up after docker

 Via https://stackoverflow.com/questions/76833923/safe-way-to-clean-var-lib-docker

/var/lib/docker was using up 16 GB. 

sudo docker system prune --all --volumes

WARNING! This will remove:

  - all stopped containers
  - all networks not used by at least one container
  - all anonymous volumes not used by at least one container
  - all images without at least one container associated to them
  - all build cache

Total reclaimed space: 7.794GB

installed an alarm applet

 On Linux Mint running Xfce, via https://forums.linuxmint.com/viewtopic.php?f=47&t=191832

Installed via Software Manager.

KAlarm needed to download 400 MB! So, instead chose Alarm-clock-applet which was only 563 kB. Shows up in the Applications menu as Alarm Clock. Can set time or timer.

postgres restore from backup - 2 minutes

A quick note on time taken - 

Install and config - 

as per

https://hnsws.blogspot.com/2024/10/moving-multiple-moodle-instances-from.html

which links to

https://documentation.ubuntu.com/server/how-to/databases/install-postgresql/index.html

rclone config

then rclone copy

rclone copy *.sql remotename2024:/home/theuser

psql -h 127.0.0.1 -U username -d databasename < dbdump-noowner.sql -W ThePassWord

This took just two minutes.

decided not to migrate Moodle from MySQL to Postgres

Though Moodle is believed to have better performance on PostgreSQL than on MySQL, 

There is a migrate tool

https://bobcares.com/blog/moodle-mysql-to-postgres/

But could have issues,

Some troubles you may find when using this tool:

If there is any object in the current database (columns, tables,...) not included in the XMLDB schema or the other way around the dbtransfer won't be executed until these objects are removed (this may happen if Moodle has been upgraded from earlier versions, if some plugin hasn't been correctly uninstalled...).

https://docs.moodle.org/405/en/Database_transfer

Also, "Depending on the size of your database, the process may take several hours."

So, decided not to migrate to Postgres at the moment.

Edit - update in Aug 2026 - ChatGPT says, (private URL for my reference - https://chatgpt.com/c/6a9267b3-5d78-83ee-9bed-f3db27750248 ) "My current assessment is that a PostgreSQL migration before the Moodle 5.3 upgrade is unlikely to deliver enough performance improvement to justify the migration effort and testing risk. I'd spend the effort on caching, PHP tuning, and perhaps a RAM upgrade first."

Change	Expected impact
MySQL → PostgreSQL	0–10%
MySQL tuning	5–20%
Redis/MUC tuning	10–50%
Faster CPU	Often noticeable
More RAM (8 → 16 GB)	Often more useful than DB migration

moving a MySQL database from one host to another

We had earlier moved a Moodle install from one server to another, and now wanted to move its database also to localhost.

Using this to set up mysql, creating database + user, and later testing a connection - 

https://www.digitalocean.com/community/tutorials/how-to-install-lamp-stack-on-ubuntu#step-2-installing-mysql

sudo mysql -p worked for logging in as root user.

create database our_master_db;

CREATE USER 'our_db_admin'@'%' IDENTIFIED BY 'OurPassword';

GRANT ALL ON our_master_db.* TO 'our_db_admin'@'%';

https://hnsws.blogspot.com/2024/10/moving-multiple-moodle-instances-from.html#dbs

to export and restore dbs.

date; mysql -h 127.0.0.1 -u our_db_admin -p our_master_db < our_master_db.OldServerdb-2024.15-04-2025.sql; date

This showed that it completed in some 20 minutes. After that, needed to change the connection settings and port in Moodle's config.php

resize OS disk for Azure VM

Wanted to resize one of our Azure VM OS disks. ChatGPT gave instructions including a manual rescan and expanding of filesystem - 

Rescan Disk: echo 1 | sudo tee /sys/class/block/sda/device/rescan

Expand Filesystem: sudo resize2fs /dev/sda1

But actually, just needed to stop the vm in azure portal, resize in azure portal, then start in azure portal - since it is the OS disk and we're rebooting, the changes are automatically noted.

Microsoft's documentation for Windows VM also notes this, lower down in this page.

ode to github actions

Just a quick note of appreciation for Github Actions. I've moved most of my software work to directly editing on github and building with github actions, due to a variety of reasons, like

1. Relatively beefy build runners - much faster than the computers I have access to - completely free for public repositories, and generous allowances for free accounts' private repositories also.
2. I can start builds from the office and check the results at home etc - the advantages of having everything online.
3. Copilot's "Explain errors" makes it relatively easy to pinpoint and fix syntax errors and such.
4. I don't have to use up the limited hard disk space on local machines for relatively large SDKs like Android SDK, Android Studio, etc.
5. The issue tracker built into github makes it easy to keep track of bugs and fixes.
6. All the advantages of git source management.

For example, I'm trying out Android NDK for exploring OCVWarp or OpenSpace porting to Android / iOS, and all the samples finished building in just 15 minutes.

asked chatgpt about running x86-64 binaries with qemu on apple silicon

First, asked about running x86_64 Debian OS on Apple Silicon, since this medium article was paywalled - 
https://medium.com/@hathibel/debian-minimal-on-apple-silicon-qemu-setup-for-x86-64-3d6d2bc15e7d - 

ChatGPT's reply - 

Running x86-64 Debian with FluxBox on Apple Silicon (M1/M2/M3) using QEMU is a great way to emulate a lightweight Linux desktop environment on ARM-based Macs. Since Apple Silicon uses the ARM64 architecture, and you want to emulate x86_64, this setup requires full system emulation (not virtualization).

(snipping the detailed setup instructions for QEMU, since UTM is easier to use.)

and the reply ended with "QEMU x86 emulation is CPU intensive. Expect modest performance."

Then asked about how to run an OpenGL windows exe on apple silicon - it gave detailed pros and cons of the following options - 

1. Crossover (not free)
2. Wine + WineSkin or WineskinServer
3. UTM (QEMU-based VM with Windows x86_64)
4. Parallels ARM64 desktop (not free) with BoxedWine (a full x86 Wine stack on ARM)

Edit: Since this was for running OpenSpace, later got OpenSpace running on UTM after compiling it for ARM64 on Github Ubuntu ARM runners.

change submodule for a git repository

 When trying to build OpenSpace for ARM64, came across issues with webbrowser ext - CEF binary being downloaded was cef_binary_102.0.10+gf249b2e+chromium-102.0.5005.115_linux64.tar.bz2 while we need ...linuxarm64.tar.bz2 for ARM64

In this case, this file seems to be responsible - https://github.com/OpenSpace/OpenSpace/blob/master/modules/webbrowser/cmake/cef_support.cmake

which we can fork and modify. In general, for other submodules not present in the main repo, the procedure seems to be: 

How do I replace a git submodule with another repo? - Stack Overflow

If the location (URL) of the submodule has changed, then you can simply:

1. Modify the .gitmodules file in the repo root to use the new URL.
2. Delete the submodule folder in the repo rm -rf .git/modules/<submodule>.
3. Delete the submodule folder in the working directory rm -rf <submodule>.
4. Run git submodule sync. git submodule sync --recursive
5. Run git submodule update.

More complete info can be found elsewhere:

Changing remote repository for a git submodule

If repo history is different then you need to checkout new branch manually:

git submodule sync --recursive
cd <submodule_dir> 
git fetch
git checkout origin/master
git branch master -f
git checkout master

And apparently with Git 2.25 (Q1 2020), you can modify it.

git submodule set-url -- <path> <newurl>

git clone error and more - trying OpenSpace compile on Raspberry Pi

An initial trial - just to see what breaks - trying to git clone OpenSpace and build it on a Raspberry Pi 4.

cloning into 'OpenSpace'...

error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408

fatal: expected 'packfile'

Apparently, this was due to an apt upgrade happening in another terminal window. Tried again later, and the git clone succeeded.

Now the error is during the generation of cmake files, regarding not finding gcc-13 - compiling gcc may take 6 hours.

https://forums.raspberrypi.com/viewtopic.php?t=362502

some post says don't cross compile.

https://gcc.gnu.org/install/
make bootstrap-lean to use less disk space.

Maybe I'll first try on a github runner to look out for ARM errors - 

https://github.blog/news-insights/product-news/arm64-on-github-actions-powering-faster-more-efficient-build-systems/

"We expect to begin offering Arm runners for open source projects by the end of the year. "

https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories

"The arm64 Linux runner is in public preview and subject to change."

libcef seems to be available for arm64, but some bugs for RPi

https://github.com/chromiumembedded/cef/issues/3803

Edit - arm64 build fails due to wrong CEF binary - will need to fork the project, make modifications ... 

prerequisites for running OpenSpace on Raspberry Pi

Initially, when I powered up the Pi, it showed a low-voltage warning on the Desktop (which I was accessing via RealVNC.) Disconnecting the powered USB hub, booting up with a 60W mobile charger - still the same warning. Then tried switching the USB-C cable to the charger's thicker cable as per
https://forums.raspberrypi.com/viewtopic.php?t=380368. That worked, no further warnings.

Check if mesa (software opengl rendering) is already the latest,

https://en.linuxadictos.com/table-23-3-arrives-with-official-support-for-rpi-5-experimental-support-for-nvk-and-more.html

On earlier pis, it used to be Fake KMS (KMS = Kernel Module Setting.)

https://forums.raspberrypi.com/viewtopic.php?t=255478

Comparison of Raspberry pi 5 with and without fan

https://www.digitec.ch/en/page/raspberry-pi-5--is-the-active-cooler-or-the-housing-with-fan-worth-it-31545

and lots of info in the comments of

https://www.raspberrypi.com/news/heating-and-cooling-raspberry-pi-5/

including the chart,

https://www.raspberrypi.com/app/uploads/2023/10/Screenshot-2023-10-04-at-23.15.30.png

About OpenGL compatibility -   

The VideoCore VII is capable of OpenGL ES 3.1 and Vulkan 1.2 according to

https://www.phoronix.com/review/raspberry-pi-5-graphics

and according to

https://docs.openspaceproject.com/releases-v0.20/contribute/development/compiling/index.html#hardware-requirements

OpenGL 3.3 is required.

So, we might need to use mesa software rendering,

https://raspberrypi.stackexchange.com/questions/75071/running-modern-opengl-on-raspberry-pi

OpenGL 3.3 is "mostly interoperable" with ES 2.0 - 

https://stackoverflow.com/questions/4519264/opengl-es-versus-opengl/11172984#11172984

Just in case this fake KMS is required on pi5

https://raspberrypi.stackexchange.com/questions/75071/running-modern-opengl-on-raspberry-pi

or just use qemu on pi?

https://raspberrypi.stackexchange.com/questions/4296/can-i-emulate-x86-cpu-to-run-teamspeak-3-server
https://marko97it.github.io/rpi5-qemu-win11/userguide/

(TPM mentioned above, is Trusted Platform Module.)

"AMD64 QEMU is currently way too broken to run a stable chroot"

in the comments of 

https://thepi.io/how-to-run-x86-programs-on-the-raspberry-pi/

and

https://github.com/ptitSeb/box86

is also 32 bit,

from https://raspberrypi.stackexchange.com/questions/108360/how-can-we-run-amd64-architecture-on-a-raspberry-pi-4

https://www.novaspirit.com/2019/04/15/run-x86-arm/

But "forget trying to run windows on it"

from

https://forums.raspberrypi.com/viewtopic.php?t=242211

Another possibility may be to cross compile.

https://earthly.dev/blog/cross-compiling-raspberry-pi/

Just try box64 first?

https://github.com/orgs/AppImage/discussions/1294

box64 with prebuilt binaries,

box64/docs/COMPILE.md at main · ptitSeb/box64

Error:

Cannot mount AppImage, please check your FUSE setup : r/debian

apt install fuse led to

The following packages will be removed:

fuse3 gvfs-fuse ntfs-3g xdg-desktop-portal xdg-desktop-portal-gtk xdg-desktop-portal-wlr

The following NEW packages will be installed:

  fuse

0 upgraded, 1 newly installed, 6 to remove and 4 not upgraded.

But even with that, the same fuse error for the x86-64 appimage running on box64. So, put back fuse3, ntfs-3g, and others. The warnings / errors seen are:

[BOX64] Binary search path: ./:bin/:/usr/local/sbin/:/usr/local/bin/:/usr/sbin/:/usr/bin/:/sbin/:/bin/:/usr/local/games/:/usr/games/

[BOX64] Looking for /media/user/SSD256GB01/rpiOpenSpace/OpenSpace-0.20.1-1-x86_64.AppImage

[BOX64] Rename process to "OpenSpace-0.20.1-1-x86_64.AppImage"

[BOX64] Warning, CALL to 0x0 at 0x401efd (0x401efc)

This doesn't look like a squashfs image.

[BOX64] Warning: Unsupported Syscall 0x54h (84)

Cannot mount AppImage, please check your FUSE setup.

You might still be able to extract the contents of this AppImage 

if you run it with the --appimage-extract option. 

See https://github.com/AppImage/AppImageKit/wiki/FUSE 

for more information

permissions needed to rename themes and plugins wordpress folders

 One of our wordpress websites was not entering recovery mode. 

... email from Wordpress regarding a fatal error while doing a setting in the menu using Astra theme. We had encountered such an error earlier also but could recover the site through the recovery link. However today on clicking on the recovery link it says recovery not initialized. ...

So, we had to disable plugins in order to be able to access the site. To follow the SSH/SFTP method described in this page - https://kinsta.com/knowledgebase/disable-wordpress-plugins/

But ... files could be opened but cannot rename since it says permission denied

So, just adding the ssh user to the www-data group was not enough - 

sudo adduser  thewordpresswebsiteadminuser www-data

also had to chmod the parent directory to 775 - was 755 earlier. 

cd theparentdirectory 
chmod -R 775 *

Then renaming worked.

Fixed by rebooting - VM on AWS

One of our VMs on AWS seemed to have some issue - it was not reachable by https or ssh. Apparently the network was down though the machine was running. Logged on to AWS portal, shut it down, and started it again, now it is working. 

errors with OpenSpace AppImage on Ubuntu 24.04

 Tried running the OpenSpace AppImage on a fresh Ubuntu VM running atop Virtualbox.

Error - error while loading shared libraries: libjack.so.0: cannot open shared object file

sudo apt install libjack0

solved that issue.

But then, 

(D) CefHost              Remote WebBrowser debugging available on http://localhost:8088

[0408/141049.216978:FATAL:setuid_sandbox_host.cc(157)] The SUID sandbox helper binary was found, but is not configured correctly. Rather than run without sandboxing I'm aborting now. You need to make sure that /tmp/.mount_OpenSpHMdIhf/home/runner/source/OpenSpace/build/modules/webbrowser/ext/cef/cef_binary_102.0.10+gf249b2e+chromium-102.0.5005.115_linux64/Release/chrome-sandbox is owned by root and has mode 4755.

Trace/breakpoint trap (core dumped)

Tried running the AppImage as as root, could not connect to display - (should probably have tried pkexec as mentioned in a previous post, but instead, I tried to solve the appimage issue.)

Looks like this is due to apparmor - a bug report - https://askubuntu.com/questions/1512287/obsidian-appimage-the-suid-sandbox-helper-binary-was-found-but-is-not-configu

The --no-sandbox or --disable-setuid-sandbox methods did not work. But the apparmor method worked - 

Created a file /etc/apparmor.d/openspaceappimage with the following content:

# This profile allows everything and only exists to give the
# application a name instead of having the label "unconfined"

abi <abi/4.0>,

include <tunables/global>

profile openspaceappimage /path/to/OpenSpace-0.20.1-1-x86_64.AppImage flags=(default_allow) {

  userns,

  # Site-specific additions and overrides. See local/README for details.

  include if exists <local/openspaceappimage>

}

Rebooted the VM, then the appimage ran. But then, blank screen after loading etc.

The web-ui at http://localhost:4680 works.

OnlyEarth profile loads, but only at 0.2 fps. But that seems to be due to 3d acceleration not being enabled on the VM's display. For that, I suppose guest additions are needed? Anyway, shut down the VM, changed display RAM to 128 MB, enabled the 3d acceleration checkbox in Virtualbox VM settings, (I see that I had done so in an earlier post), booted up the VM, changed display resoltuion inside the VM by right-clicking on the desktop to 1920x1080, then ran OpenSpace, mostly OK.

Takes a minute or two to initialize GL, then load assets, till then black screen for OS.

The default display configuration doesn't seem to work - black screen. But gui portrait, single fisheye gui etc worked fine. With OnlyEarth profile - 100 fps average with occasional stuttering. 

Tried the default profile also later - it took around 2 hours to download all the 'sync' files - and ran very very slow, not usable.

arm64 Windows 11 on Raspberry Pi 5

Using QEMU - https://marko97it.github.io/rpi5-qemu-win11/

Requires Pi5 and Micro SD card or a SSD drive with more than 64 GB of space - I have a Pi4, so didn't try it.

another tunnelling solution like cloudflare

 Another product which offers ssh tunnels - earlier post about cloudflare's offering is https://hnsws.blogspot.com/2025/02/cloudflare-for-reverse-proxying-on.html

Twingate: It's time to ditch your VPN

via

5 reasons EVERYONE needs a home server

timeouts when duplicating a quiz on Moodle

One of our Moodle site maintainers complained that when he was duplicating a quiz, it was taking a long time and then giving a pop-up message saying "undefined". Conversation below:

Can you please check if there are any background jobs running?

Yes, Moodle runs lots of background jobs all the time. You can check these by going to 

https://ourdomain.org/admin/tasklogs.php

Right now, the server loads are very low, but I did get a db server load high alert (85%, as against a healthy load of up to 80%) at 10:15 am yesterday.

 Additionally, if many of your users are logged in, that can also slow down your response times. One possible mitigation in such cases would be for you to lock out other users by going to Maintenance Mode

https://ourdomain.org/admin/settings.php?section=maintenancemode

choose "Enable" and save - this will prevent other users from using the server - do your maintenance tasks, then go to the same page, and "Disable" - after which other users can continue to work.

when trying to create a quiz (just a duplicate of an existing one)

And as I have mentioned earlier, running large db queries will slow down the server and or make it time out. If that happens, please break down the action you wish to take into smaller chunks. "Just making a duplicate of a quiz" can be very db intensive if the quiz has a lot of content, multimedia content, a lot of users, a lot of responses, multimedia responses, and so on.

Attempted to create a blank quiz and it fails like below. This does not have any questions. So seriously something is wrong with the backend.

(removed a screenshot, which showed "Recalculating grades...")

Doing a search,

https://www.bing.com/search?pglt=641&q=creating+a+quiz+recalculating+grades+timeout+mod+questionnaire+moodle

Found

https://moodle.org/mod/forum/discuss.php?d=401173

https://tracker.moodle.org/browse/MDL-49324

So, this is possibly a bug. But I cannot fix Moodle bugs for you.

Not sure how I can help you with this. I can turn off cloudflare caching so that you will get a bit longer before timeout. (Found that this is already done.)

 Then you can try putting the site in maintenance mode and then try creating the quiz. But if that also fails, I have no further ideas.

Managed to create the sections for this module. Hopefully things get faster to load for users tomorrow.

long back-and-forth about Moodle quiz content not working

After an upgrade to Moodle 4.5.3, the admin of one of our Moodle instances complained about some content not working. It finally turned out to be due to the use of deprecated mod_questionnaire and probably some caching. Copy-pasting large parts of our conversation.

We are experiencing delays in server from yesterday and today.

It is unresponsive when we try to add contents to our upcoming module in <course> which may start tomorrow.

Logged in to the site OK. As I have mentioned before, please add content in small enough chunks to avoid timeouts.

The site was up after some time. But with no heavy activity still we find it not responsive. Particularly that involves heavy database calls like the quizzes. This happened yesterday and even today.

e.g., https://ourdomain.org/mod/quiz/view.php?id=5032 which has the quiz section, is not loading at all. 

Can you check if the database server is  healthy for <course>?

1. The link which you have sent does not load due to an invalid course module id - "Invalid course module ID: 5013 ... More information about this error"

2. Copy-pasting from an email I had sent to the team via SG on 11 Jan 2023 - 

Some suggestions to make the course(s) load faster:

1. Currently, if you click on one of the courses, a huge page, with many images, for all weeks, loads in the browser. We can make it load a bit faster using Cloudflare, but such a large page is bound to cause problems with older computers / machines with less RAM / on slower networks / etc. It would be better to show a collapsed view, with each week opening up only on clicking. (the collapse all link on the top right). Or on separate pages.

2. As I mentioned, please use jpg quality = 70 to 75 instead of png. 

3. I'll make the cloudflare changes to turn proxying on. 

4. Clicking on question bank also loads up 1000 questions at once. This would cause problems as the number of questions / size of each question increase. It would be desirable to load up fewer questions - maybe 100 at a time as with the link at the bottom, "Show 100 at a time"

<the link> This was working on Friday night. This is a valid link for the quiz section of module 8.

Discovered that quiz sections are not opening for previous modules also - which the participants successfully faced for the last 7 modules. All quizzes and question bank resources were working fine till last Friday. So there is something suspicious in the database server. 

We need all the previous module results and also the new quizzes that are to be opened tomorrow.

Can you please check the health of the app and database server?

Moodle is installed and the database is running, which is how we are able to log on, see course pages, etc. - so the "health of the app and database server" are fine. 

Digging deeper, in the plugins page, https://ourdomain.org/admin/plugins.php

I have seen some plugins "missing from disk" errors. These would have been "missing" since the Moodle upgrade of Mar 18.

Just to be safe, I have now copied these "missing from disk" plugins to the moodle plugins directory. It may be that these plugins are no longer supported, hence Moodle removed them - other plugins prompt for upgrades, which have been applied.

I have also turned off cloudflare caching, so that you will get more time before server timeout. But this may make pages slower to load for those taking the courses, so once you are done with admin tasks requiring longer db queries, please let me know, and I will again turn on cloudflare caching for faster response for the users of the course.

The link which you sent still gives the same error.

I'm not sure what else I can help you with. If you want raw database backups from the weekly backups, I can send you.

Do you have the list for missing plugins and updated ones? That can give some ideas. 

local/welcome
local/mailtest
report/customsql
blocks/accessibility
lib/editor/atto
mod/questionnaire 

Or can we revert to the last working version of Moodle? The troubleshoot time is less and we need to get the module ready for tomorrow morning at 4am.

Note: Also in general, the site is too slow when compared to a couple of weeks before.

As far as I know, Moodle is working fine. So, as far as I'm concerned, "last working version of Moodle" is the current live site.

I can put the site back to last week's or the week before's - but then, any responses from the people taking from the course or any other changes made on the site after last week or 2 weeks back would be wiped out. Do you really want to do that? I don't think that's a good idea.

Yes, the site would be slower now that I have removed cloudflare caching as mentioned in the previous email.

Thanks for the list. `mod/questionnaire` seems to be related. But we can wait and see if we can create another set of quiz sections.

On software upgrade -> yes if things dont work, we need to revert to the last working version. Not yet decided. Shall keep you updated as we are trying to create a new section.

(and 20 minutes later,)

Surprisingly things are working again.. Not sure how but it looks fine now. Also the site seems to be a bit faster than before. Not sure how?

No changes were done from our side.

Moodle does some scheduled tasks - which are listed under Site Administration > Server > Tasks > Scheduled tasks

My guess would be that after I restored the missing plugin files, some automated task ran at some point of time to either clear database cached replies, or to fix broken links or something like that. 

Anyway, all's well that ends well. But please don't put too much content on a single page - that would cause the site to load slowly. Please see Course Format > Course Layout > Show all sections on one page and change it to "Show one section per page" on each of the courses you're working on. That will make pages load much faster.

5 days later, at 9 pm,

again the site is too slow from this afternoon 12PM. Even a single module  in the course page is taking minutes to load. Can you please check if there are any background jobs running?

This is affecting our schedule as we wanted to open a new module tomorrow and we are stuck for long time today and unable to make it up.

After a long time I am getting this (Error writing to database popup) - when trying to create a quiz (just a duplicate of an existing one)

Yes, Moodle runs lots of background jobs all the time. You can check these by going to 

https://ourdomain.org/admin/tasklogs.php

Right now, the server loads are very low, but I did get a db server load high alert (85%, as against a healthy load of up to 80%) at 10:15 am yesterday.

 Additionally, if many of your users are logged in, that can also slow down your response times. One possible mitigation in such cases would be for you to lock out other users by going to Maintenance Mode

https://ourdomain.org/admin/settings.php?section=maintenancemode

choose "Enable" and save - this will prevent other users from using the server - do your maintenance tasks, then go to the same page, and "Disable" - after which other users can continue to work.

And as I have mentioned earlier, running large db (database) queries will slow down the server and or make it time out. If that happens, please break down the action you wish to take into smaller chunks. "Just making a duplicate of a quiz" can be very db intensive if the quiz has a lot of content, multimedia content, a lot of users, a lot of responses, multimedia responses, and so on.

Attempted to create a blank quiz and it fails like below. (Screen showing "recalculating grades" and error writing to database at the end.) This does not have any questions. So seriously something is wrong with the backend.

Doing a search,

https://www.bing.com/search?pglt=641&q=creating+a+quiz+recalculating+grades+timeout+mod+questionnaire+moodle

Found

https://moodle.org/mod/forum/discuss.php?d=401173

https://tracker.moodle.org/browse/MDL-49324

So, this is possibly a bug. But I cannot fix Moodle bugs for you.

Not sure how I can help you with this. I can turn off cloudflare caching so that you will get a bit longer before timeout. (Found that this is already done.)

 Then you can try putting the site in maintenance mode and then try creating the quiz. But if that also fails, I have no further ideas.

Thanks for the info bro. The site was noticeably faster a couple of weeks ago. 

Managed to create the sections for this module. Hopefully things get faster to load for users tomorrow.

free AI movie generation - only 5 sec is usually free

 Trying out more resources for "free" AI generated movies - my earlier post about this is here - this video - Install WAN 2.1 The 100% FREE AI Video Generator : Very EASY & QUICK! - talks about installing 

https://pinokio.computer/
(browser which allows to install and run various AI tools)

WAN2.1 is the actual video generator

And can do online for free,

https://wan21.net/ai-video-generator

but sometimes the queue takes a long time. Afternoon 1:30 IST, started rendering this prompt "A moonlit night sky with scattered clouds" rendered in around a minute, with only less than a minute in the queue. The resulting video is below. 5 credits a month for the free account, I think. 

Another top result for "free" AI movie generation is this video, which talks about digen.ai - online vid generate and share platform - and using a temp mail to get credits!

download just after processing before watermark is applied in "optimizing" step!

At some point, I'll try installing WAN2.1 if possible.

upscaling and resizing with chainner

Chainner - A node-based image processing GUI - seems to have a relatively easy-to-use installer - deb files for Linux, Windows installer, etc. And it has a dependency manager which can install the required dependencies. Tried it out, but unfortunately, it could not use the external stable diffusion api - maybe because I didn't run the web ui with API enabled? or is it that the API has changed and hence it gives errors like "invalid HTTP request"? Tried running chainner after running easydiffusion - could not connect.

api wiki says latest info is at :7860/docs

http://127.0.0.1:7860/docs

Video - how to upscale images with chainner - 

Models from https://openmodeldb.info/

(old site was https://upscale.wiki/wiki/Main_Page#The_Model_Database )

Chose https://openmodeldb.info/models/4x-RealWebPhoto-v4-dat2

Took 10 minutes. Seems to be using cpu and not gpu

8x upscale with onnx took 24 minutes. Looks like water colour and not pointillist like the previous upscale with Stable Diffusion. 

8x upscale with onnx using the model 4x-RealWebPhoto-v4-dat2 compared with the original image before scaling down and upscaling. Click on the image to see it larger.

Probably there may be some models which would do fast upscaling with realistic results (unlike the compact model below which seems to be optimised for anime). 

Recommended models at

https://phhofm.github.io/upscale/favorites.html

Compact model - https://openmodeldb.info/models/2x-HFA2kCompact

For video, the youtube video above used video frame iterator, and the compact model. Now the interface has video input and video output nodes, which will automatically create video from the frames. Using the  compact model, the video upscaling and processing was completed in just a few seconds, for 250 frames Baba. Looks a bit like anime / cartoon / water colour. If the numbers under the chainner nodes are the times taken, then 0.04 sec per frame, 9.8 sec for encoding to vid.

Edit: There are lots of youtube tutorials for video upscaling, including ones using colab - https://www.bing.com/videos/search?q=stable+diffusion+video+upscaling

checking out Easy diffusion

Tried out Easy Diffusion, https://github.com/easydiffusion/easydiffusion

checking to see if it is faster than Stable Diffusion webui - my previous post on that is here.

https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Install-and-Run-on-NVidia-GPUs#linux

Some useful links - basics of inpainting - https://github.com/easydiffusion/easydiffusion/wiki/Inpainting

https://allprompts.com/stable-diffusion-prompts-for-realistic-photos/

So we should add words like national geographic, landscape, photo, photography

Easy Diffusion - better interface, and a bit faster - 45 sec instead of 60 sec to generate a 512x512 image, and just 7 sec to upscale the generated image. But there doesn't seem to be a way to upscale our own images, at least not seen in the UI. Trying the same thing with CPU on laptop, the 45 second generation on the 1060 graphics card took nearly 12 minutes with CPU!

Some example prompts and resulting images:

1. moonlit landscape - 

2. photo of night sky with full moon tokyo skyline in silhoutte nikon national geographic

3. moonlit landscape, nikon, national geographic, full moon, trees (wish upscaling done 512 to 1024) - 

run unsigned binaries on MacOS

 I was not sure if unsigned binaries would run on MacOS, and whether one can override its behaviour. So, apparently MacOS 10.15 Sonoma onwards, unsigned binaries won't run by default, but we can specifically white-list any binary we want to run, and then it will run.

https://github.molgen.mpg.de/pages/bs/macOSnotes/mac/mac_procs_unsigned.html

https://discussions.apple.com/thread/254786001?sortBy=rank

Either via the GUI - 

* Try to open myFancyBinary in Finder. This will fail.

*Open System Preferences, choose the Security control panel, select the General tab.

*Look for the message: “myFancyBinary was blocked from opening because it is not from an identified developer.”

*Click the Open Anyway button to the right of the message.

or via the terminal - 

spctl --add /path/to/myBinary

And for getting the full path, and easy way is to drag and drop the binary from Finder into the Terminal window.

Stable Diffusion tests and benchmarks

I had tried out Stable Diffusion on Windows last year, now I wanted to try it out on Linux and if possible, using Docker. 

Unfortunately, the docker method failed. Searching for stable diffusion docker with web ui

https://sygil-dev.github.io/sygil-webui/docs/Installation/docker-guide/

Stable Diffusion in Docker | Open Source AI Image Generation Self-Hosted

talks about

https://github.com/AbdBarho/stable-diffusion-webui-docker

https://github.com/AbdBarho/stable-diffusion-webui-docker/wiki/Setup

(I had to run everything with sudo prefix.)

sudo docker compose --profile download up --build

roughly 12GB of data to be downloaded.

Unfortunately, after 2 hours of downloading, using the Automatic1111 profile,

sudo docker compose --profile auto up --build

Error response from daemon: could not select device driver "nvidia" with capabilities: [[compute utility]]

So, probably I need --profile auto-cpu.

But that gave an error,

File "/stable-diffusion-webui/webui.py", line 13, in <module>

etc

ImportError: cannot import name 'TypeIs' from 'typing_extensions' (/opt/conda/lib/python3.10/site-packages/typing_extensions.py)

Deleting the dir, cloning the repo and starting again - again downloading 12 GB, 

And again failed.

sudo docker compose --profile auto-cpu up --build

Mounted styles.csv

auto-cpu-1  | Mounted ui-config.json

auto-cpu-1  | mkdir: created directory '/data/config/auto/extensions'

auto-cpu-1  | Mounted extensions

auto-cpu-1  | Installing extension dependencies (if any)

auto-cpu-1  | Traceback (most recent call last):

auto-cpu-1  |   File "/stable-diffusion-webui/webui.py", line 13, in <module>

auto-cpu-1  |     initialize.imports()

auto-cpu-1  |   File "/stable-diffusion-webui/modules/initialize.py", line 23, in imports

auto-cpu-1  |     import gradio  # noqa: F401

auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/gradio/__init__.py", line 3, in <module>

auto-cpu-1  |     import gradio.components as components

auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/gradio/components/__init__.py", line 3, in <module>

auto-cpu-1  |     from gradio.components.bar_plot import BarPlot

auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/gradio/components/bar_plot.py", line 7, in <module>

auto-cpu-1  |     import altair as alt

auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/__init__.py", line 649, in <module>

auto-cpu-1  |     from altair.vegalite import *

auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/vegalite/__init__.py", line 2, in <module>

auto-cpu-1  |     from .v5 import *

auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/vegalite/v5/__init__.py", line 2, in <module>

auto-cpu-1  |     from altair.expr.core import datum

auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/expr/__init__.py", line 11, in <module>

auto-cpu-1  |     from altair.expr.core import ConstExpression, FunctionExpression

auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/expr/core.py", line 6, in <module>

auto-cpu-1  |     from altair.utils import SchemaBase

auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/utils/__init__.py", line 14, in <module>

auto-cpu-1  |     from .plugin_registry import PluginRegistry

auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/utils/plugin_registry.py", line 13, in <module>

auto-cpu-1  |     from typing_extensions import TypeIs

auto-cpu-1  | ImportError: cannot import name 'TypeIs' from 'typing_extensions' (/opt/conda/lib/python3.10/site-packages/typing_extensions.py)

auto-cpu-1 exited with code 1

Then I thought I would try chainner, https://chainner.app/download

gives a deb file, for ease of installation. It has a dependency manager also. 

But the dependency manager does not install stable diffusion! So, trying

https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Install-and-Run-on-NVidia-GPUs#linux

(

current python version is 3.11.5 on this Linux mint based on ubuntu 24.04 - which may run into problems,

https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/9473

- we're supposed to run on python 3.10-venv

)

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacty of 3.94 GiB of which 31.50 MiB is free. Including non-PyTorch memory, this process has 3.73 GiB memory in use. Of the allocated memory 3.60 GiB is allocated by PyTorch, and 77.43 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Stable diffusion model failed to load

Can try lowvram command, since we have the NVidia 1060 -  

https://techtactician.com/stable-diffusion-low-vram-memory-errors-fix/

export COMMANDLINE_ARGS="--lowvram"

./webui.sh

Success. Does a 512x512 generate in ~ 1 minute, 4x ESRGAN upscale resize in ~30 sec.

https://github.com/AUTOMATIC1111/stable-diffusion-webui

this has a listing of how to make prompts.

My primary use case was for upscaling. I took a picture which I had clicked with the mobile camera in its "50 megapixel" mode, which shows quite a bit of "softening" when viewed at 100% - then cropped to 4096x4096, scaled down to 512x512 (1k images were causing out of VRAM errors) and then scaled up to 4096x4096. Interesting to see the results for a jpg show lots of ringing, while the results for a png seems to make things sharper. 

A portion of jpg input on the left, with output on the right, processed as 

scale down to 512 px, save as jpg, and then upscale to 4x - this shows ringing. Click on the image to view larger size.

A portion of jpg input on the left, with output on the right, processed as 

scale down to 512 px, save as png, and then upscale to 4096 - this shows an interesting pointillistic effect, and has made edges if the image sharper - clearly seen in the foreground trees.  Click on the image to view larger size.

I'll write a separate post on Easy Diffusion, since this post has already become quite long.