letsencrypt certificate not renewed

Short answer - it was due to ca-certificates not being updated. 

One of our web servers started showing 'certificate expired' errors. Checking, found that manually running
certbot --apache
gave errors - 
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
among others.

Googling, found several suggestions at
https://community.letsencrypt.org/t/certificate-verify-failed/69444

Reinstalled certbot as recommended at
https://stackoverflow.com/questions/53870702/why-is-certbot-renew-giving-bad-handhake-error

apt install certbot

Still the same error was seen. 

Tried
tail -n100 /var/log/letsencrypt/letsencrypt.log
- that did not show anything more than the same error as above,

Checked
more /etc/hosts
no issues there. 

Tried
curl -v https://acme-v02.api.letsencrypt.org/directory
which showed a certificate error for that server, which implied that I needed to reinstall the ca-certificates file also.

apt install ca-certificates

Then,
certbot --apache
worked - the web server was working at this point.

Then, I thought it would be better to update all the software on it to prevent such issues - 

apt update
apt upgrade

During the update, certbot could not find the http conf file for one of the domains, so I had to do a2ensite for the relevant site later. 

# a2ensite name-of-conf-file
# service apache2 reload

finding manufacture date of a laptop

Was curious about the age of the machine I'm working on - a Macbook running Linux Mint. On MacOS, one can find details of the hardware by going to the About this Mac menu item from the Apple menu, copying the serial number and then checking sites like EveryMac.com or
https://serial-number-decoder.com/apple-serial-number/apple-serial.php

On linux, found that
dmidecode -t system
gives 

System Information
Manufacturer: Apple Inc.
Product Name: MacBookPro6,2
Version: 1.0
Serial Number: WxxxxxxYAGW
UUID: xxxxDA02-xxxxx-0A56-89AF-2ABCDxxxF
Wake-up Type: Power Switch
SKU Number: System SKU#
Family: MacBook Pro

and that with the serial-number-decoder.com link above tells me 
"This 'MacBook Pro (15-inch, Mid 2010)' was made in Shanghai China in week 02 of 2011."

using Google sign in for our php portal

Following the basic idea explained at
https://phppot.com/php/php-login-script-with-session/
for Google sign-in support for our portal - in our case, it is a postgresql database and not mysql, and we already have the database in place.

We also added a button to sign out of current google account, to make it more user-friendly for people who use multiple google accounts.

<form action="https://accounts.google.com/Logout" method="get" target="logoutframe">
  <input type="submit" value="Sign out from google account" class="w3-button w3-block w3-blue" id="googSO">
</form>

In our case, it was implemented by SC with a function in our User class, $user->processGoogleLogin() so that google_login/login.php has

$google_account_info = $google_oauth->userinfo->get();
$isLoggedIn = $user->processGoogleLogin($google_account_info->email);

and processGoogleLogin() has a select statement with the condition

WHERE u.id = r.user_id AND 
  r.role_id = m.role_id AND
  u.email ilike $1

custom role for user able to start and stop vm in Azure

Apparently there is no built-in role at present in Azure which allows a user the permissions to start and stop a VM in Azure (and not much else). Even the Virtual Machine Admin role doesn't. According to this SO post,

1. Add Custom Role
2. Select "Clone a role" and role to close is "Virtual Machine User Login"
3. Click Next
4. Select add permissions
5. Scroll  down to "Microsoft.Compute.VirtualMachines" and tick
Microsoft.Compute/virtualMachines/start/action"
"Microsoft.Compute/virtualMachines/powerOff/action"
"Microsoft.Compute/virtualMachines/deallocate/action"
6. Click Next, select subscription, Next, Next then "Create".

And then after creating the role, we can assign it to a user.

download a file using powershell

One of the first things we try to do on a fresh Windows server install would be to download and install Chrome or Firefox, due to the extreme tiresomeness of "locked down Internet Explorer" which doesn't allow any downloads by default. A quick way to do that is via powershell, courtesy this page - 

powershell -command "& { (New-Object Net.WebClient).DownloadFile('https://example.com/file.zip', 'c:\somefile.zip') }"

finding files modified in the last 10 days using Linux find -mtime

I needed to find all files modified in the last 10 days in a directory using the Linux bash shell. One of the ways listed was using find -mtime,

find /var/the/relevant/dir -mtime -10

AJAX code snippet using Google Apps Script as backend

A drop-down list called Chapter gets updated by the following code - 

window.addEventListener('load', getChapterData); document.getElementById("board").addEventListener('change', updateChpDropdown); document.getElementById("standard").addEventListener('change', updateChpDropdown); document.getElementById("subject").addEventListener('change', updateChpDropdown); function updateChpDropdown() { if (document.getElementById("dataready").innerHTML == "Yes") { // clear the dropdown document.getElementById("chapter").innerHTML = '<option value="Choose">Chapter</option>'; var ddl = document.getElementById("chapter"); var chosenboard = document.getElementById("board").value; var chosensubject = document.getElementById("subject").value; var chosenstandard = document.getElementById("standard").value; for (var j=0; j < chapObj.length; j++ ) { if (chapObj[j]["boardname"]==chosenboard && chapObj[j]["subjectname"]==chosensubject && chapObj[j]["standardname"]==chosenstandard) { var option = document.createElement("OPTION"); option.innerHTML = chapObj[j]["chaptername"] ; option.value = chapObj[j]["chaptername"] ; ddl.appendChild(option); } } } } function getChapterData() { var xmlhttp = new XMLHttpRequest(); xmlhttp.onreadystatechange = function() { if (this.readyState == 4 && this.status == 200) { chapObj = JSON.parse(this.responseText); document.getElementById("dataready").innerHTML = "Yes"; } }; xmlhttp.open("GET", "https://script.google.com/macros/s/ID_OF_GAS_SCRIPTyL/exec",true); xmlhttp.send(); } </script>

find a deleted file in a git repository

https://stackoverflow.com/questions/7203515/how-to-find-a-deleted-file-in-the-project-commit-history

git log --all --full-history -- "**/thefile.*"

This outputs a list of commits which reference the deleted file.

commit d1c --snip hash -- 51ae
Author: Name <email>
Date:   Sat Mar 6 13:05:11 2021 +0000

    validate password

commit 161--snip hash --1cf04
Author: Name <email>
Date:   Wed Mar 3 03:22:55 2021 +0000

    Changes for postgresql DB

and so on. 

recording Google Meet sessions

Google Meet offered recording to drive, and later restricted the recording to these editions of Google Workspace. So, I get lots of queries on how those without those editions can record. 

One option was to stream the Google Meet on youtube live via services like streamyard. The youtube live stream is automatically recorded, and there is no hassle of saving locally and then uploading. The streamyard free plan has a limit of 20 hours a month.

If you wish to record more than that, it may be advisable to record locally either using software or a hardware screen-grabber.

One of the ways to do that would be using the free software called OBS - https://obsproject.com/

Record google meet call with OBS, https://www.youtube.com/watch?v=2ipPdxOZ27U

Some changes in settings from the above video in my notes on recording with Linux Mint on a 2010 model Macbook Pro - 30 fps was fine.

Had to "Lock X server when capturing" in the Window capture (Xcomposite) settings for the Meet video to be not black. Else, after clicking join, video becomes black. 

Around 60% cpu for 720p output rescaled from screen resolution.

Moodle MySQL collation warning

While upgrading Moodle on one of our sites, There was a warning, with a link to https://docs.moodle.org/311/en/MySQL_full_unicode_support

The admin of the site reported, 

Received an error message, "Error writing to database" on trying to add an emoji. 

Though emojis were not so important, Indian language support was required.

On checking the collations, found that the db used
Default Charset = utf8 and 
Default Collation = utf8_general_ci
just like another test site of ours, and that copy-pasting Indian language text was working - 

நமது தளத்தில் தமிழைச் சோதிக்கவே இந்த உரை.

यह पाठ हमारी साइट में हिंदी का परीक्षण करने के लिए है।

ഈ വാചകം ഞങ്ങളുടെ സൈറ്റിൽ മലയാളം പരീക്ഷിക്കുന്നതിനുള്ളതാണ്.

ಈ ಪಠ್ಯವು ನಮ್ಮ ಸೈಟ್‌ನಲ್ಲಿ ಕನ್ನಡವನ್ನು ಪರೀಕ್ಷಿಸಲು.
ఈ వచనం మన సైట్‌లో తెలుగును పరీక్షించడానికి.

(Made with translate.google.com )

So I did not change the collation to utf8mb4_unicode_ci as suggested in the Moodle documentation in the link at the beginning of this post, since this is a potentially dangerous operation, and can take a long time and can break things, so I would need to take precautions like making a backup, a rollback plan and so on. 

postgresql read-only user

There seem to be different ways of creating a read-only user in postgresql, depending on the version, and depending on whether you want the user to have access to future tables and so on. 

https://stackoverflow.com/questions/760210/how-do-you-create-a-read-only-user-in-postgresql

And when creating a user like that, we have to connect to the database first, using

\connect dbname
to prevent errors

https://dba.stackexchange.com/questions/173635/postgres-permission-denied-for-schema

adding the UserWay accessibility widget to Moodle

The way to add the accessibility widget for all users, as against only one user, seems to be :
https://docs.moodle.org/311/en/Header_and_footer

I added to the HEAD section at
https://our.server.org/admin/settings.php?section=additionalhtml

and now on the widget appears fine.

Need to check it out and see if it gives something better than the default "accessibility settings" at the bottom left corner built in to Moove theme on Moodle.

login with google

This post describes login with google using MySQL and PHP - 
https://www.w3jar.com/login-with-google-account-using-php-mysqli-source-code/

Taking that as the starting point, we could implement in our code where the db and users already exist in postgresql. 

javascript error and resolution

There was a page which had our google app script embedded, but on which the app only showed a spinner instead of loading up on Firefox. Checking the error on the browser console, found
Uncaught SyntaxError: invalid range in character class common.js:10:54

Finally found that this was due to a hyphen not being escaped in the common.js file - the hyphen should be escaped, or else it becomes a range from the previous char (.) to the next char (+), which gives an "Invalid range" error, preventing the loading of the entire js file, leading to the function w3_open() not being found.

           if (!element.value.match(/^[0-9a-zA-Z @,.-+]+$/)) {

was to be replaced by 

            if (!element.value.match(/^[0-9a-zA-Z @,.\-+]+$/)) {

   

zoom vs meet vs bbb vs teams

Came across this detailed video comparison of teaching with zoom vs meet vs bbb vs teams. But it ignores one factor which is crucial for us, which is cost.

git pull Moodle upgrade

The steps to do a command-line upgrade using git are summarized at
https://moodle.org/mod/forum/discuss.php?d=403666

While doing the upgrade, git complained that changes have been made, so it was aborting the pull. As discussed in this page - 
https://moodle.org/mod/forum/discuss.php?d=228497
I tried 
git stash
git pull

That worked. 

one-time download from GDrive - and a library to parse URLs

An interesting implementation of a temporary download link from Google Drive using Google Apps Script - https://github.com/tanaikech/One_Time_Download_for_Google_Drive

And a library to parse URLs - URI.js linked from https://googleappscripting.com/working-with-urls/ 

concurrent user limits for google apps script

User Tanaike has answered this question also on stackoverflow with testing details - the answer seems to be less than 30 concurrent calls to a google apps script, though if calls are separated by one second, more users can use the script. 

difference in parsing csv - Google Sheets and Drive vs LibreOffice Calc

One of our google apps scripts did not work properly when data was copy-pasted into its source Google sheet by opening the data in a csv file from LibreOffice Calc, but worked when data was copy-pasted from Google Drive's csv preview. 

Apparently some fields had been interpreted differently by Calc and Google, and some fields had changed places - probably due to commas within fields, or maybe due to newlines within fields. 

User rate limit exceeded in Google Apps Script

API call to drive.files.insert failed with error: User rate limit exceeded
- there are various mitigation strategies mentioned in this page, but what I usually use are calls to Logger.log - that seems to take the amount of time sufficient for staying within the 10 calls per second rate.

exporting Odia html text as Google doc

Exporting a set of Oriya (Odia) language html files stored inside a database to google docs using the usual technique which I had used for English, viz - 

var ablob = Utilities.newBlob(assethtml, MimeType.HTML, "asset.html");
var AssetGDocId = Drive.Files.insert(
{ title: data_array[i][8] + ' temp', // Column I is Asset title
mimeType: MimeType.GOOGLE_DOCS, parents: [{"id": destFolderID}] }, ablob).id;

Did not work. Reason was that the Odia files did not specify the encoding, and consisted of content like

When "good" Odia content was created using Google's phonetic Odia keyboard provided by Google Input Tools, the exported html looks like this, 

The current workaround is to just export the database contents to HTML instead of converting to GDoc. Then, by saving the database contents as a html file, installing the relevant font, opening in LibreOffice Writer (or MS Word, I guess) and then saving as pdf, correctly rendered Odia text is obtained. 

removing an app from the Android App Store (Play Store)

The video at https://www.youtube.com/watch?v=bqEOQzg_kp8 has the current steps - 

• Go to play.google.com/console
• Click on app
• On left menu, scroll down to Setup, under which click on Advanced Settings
• Choose Unpublish.

moving files and folders to shared drives in Google Workspace

For non-Admin users to move entire folders to Shared Drives in Google Workspace (formerly GSuite), the admin has to set suitable permissions - from admin.google.com, Admin Roles -> Create new role, call it something like Shared Drive Admin or whatever, and give that role the permission under Services > Drive and Docs > Settings > Move any file or folder into shared drives

After that, assign this admin role to whichever user needs it. Then, that user can drag and drop folders also into shared drives as below. (If this is not done, only files can be dragged and dropped into shared drives, folders have to be manually re-created.)

Dragging and dropping files from My Drive to a Shared Drive - walkthrough video below - 

This could be a good option if some users on Google Workspace are running out of disk space due to files in Drive. Another option would be to move the files to another google account. This video's second half shows how to transfer files from one account to another using MultCloud. The first part of the video is inaccurate, since the file owner is not changed even if moved from Shared with Me to another folder. If not using MultCloud, the method to transfer a file from one account to another is:

1. Share the file from account1 with account2 as Editor.
2. Choose to make account2 the owner by clicking the "Make Owner" option as in the screenshot below.
   
   
   
   
   
   
   
   
   
3. Account2 gets an email saying that this item appears in My Drive of account2
4. Move file to suitable folder by signing in with account2

This is quite cumbersome, so I must try out MultCloud and see if it makes things easier.

 

Edit 17 Jun 2022 - There is another tool called Drive Migrator also,

https://chrome.google.com/webstore/detail/drive-migrator/nakklajdcijlkfagghhcdofbgbhddoed?hl=en

scripts to copy publicly available files from Alfresco

There was a requirement to back up all the files in an Alfresco repository to a filesystem. A google shared drive mounted using rclone was used in this case. 

Caveat - rclone mounted gdrive is excruciatingly slow when there are a large number of files in the destination folder, even when using your own client id. It's much better to use some other remote mount like a remote local drive or something like that, and then upload from there to google drive. 

Anyway, the scripts and sql used in this case are listed at https://github.com/hn-88/bashDLalf
which uses the grep -v method to remove lines containing specific substrings.

(We did not use these bulk scripts because only a sample "site" was configured on our installation. And this Alfresco Bulk Export tool needed the Alfresco war file to be modified etc, which I was hesitant to do.)

Edit: 5 May 2022 - Got an email from Google, "OAuth out-of-band (OOB) flow will be deprecated on October 3, 2022, to protect users from phishing and app impersonation attacks." ... "Apps using OOB in testing mode will not be affected." ... etc. 

The client ID I had created for rclone was indicated in the email. So, I deleted that client ID, since I'm currently not using it. 

mysql timeouts with DBeaver

When doing custom queries on a remote MySQL server, DBeaver was timing out and I would need to "Invalidate/Reconnect" the connection every time. Found this old discussion about it, in my case the solution was to change the timeout from 20000 to 2000000 in the menu item
Database -> Driver Manager -> MySQL -> Driver properties tab

ad hoc moodle database queries

Some queries needed by the team for customized Moodle reports using the ad-hoc database queries plugin - https://github.com/hn-88/ad-hoc-moodle-database-queries

website connection issues - most probably client-side DNS misconfiguration

One of our websites was not responding when trying to connect with Chrome incognito tab from offices behind a Sophos network security appliance. Interestingly, the error message shown was "xx.yy.zz.kk refused to connect" with the relevant IP address being an Airtel address and not a github/cloudflare address as it should be, since the website was hosted on github with cloudflare proxying. 

Most probably this was due to some sort of misconfigured DNS, I guess, since V reported that it was working after the alt DNS IP 8.8.8.8 was added along with cyberoam's IP address (which was primary DNS) in the wireless router.

But the fact that only incognito tabs had this issue was interesting. Maybe due to this - in incognito mode, Chrome seems to be connecting to dns in a different manner, which seemed to have the issue - https://blog.google/products/chrome/more-intuitive-privacy-and-security-controls-chrome/

fresh moodle installation

Earlier, I had just duplicated moodle installations by backing up and restoring. Now a fresh install, for which:

1. Set up the moodle files using git as in my previous post.
   
   
2. Logged on to the database server, and created the required database and user as mentioned in the documentation - not as a single command, which doesn't seem to work, but as separate commands. Since the user is not on localhost, it is currently set up as 'username'@'%'
   mysql> CREATE DATABASE moodle DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
   mysql> CREATE USER moodleuser@'%' IDENTIFIED BY 'yourpassword';
   mysql> GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,CREATE TEMPORARY TABLES,DROP,INDEX,ALTER ON moodle.* TO moodleuser@'%';
   
   
   
3. Caveat - must log in as root with mysql -u root -p before doing the above, since other mysql users don't have all the required permissions. In this case, the root user's password was similar to the other user's password.
   
   
4. Ran the installation from the CLI. The first time I ran it, www-data user could not write to that directory. So, changed the permissions to allow group write -
   chmod g+w the_git_dir
   cd the_git_dir
   sudo -u www-data php admin/cli/install.php
   This took quite a while - around 5 minutes - for all the database table creation etc. So, probably it would have been wiser for me to do this in a screen session. 
   
   
5. Then made the site available via apache, with a letsencrypt certificate:
   cd /etc/apache2/sites-available
   sudo cp anothersite.conf thissite.conf
   sudo nano thissite.conf
   sudo a2ensite thissite.conf
   sudo systemctl reload apache2
   sudo certbot --apache
   
   

delete a user in mysql

I had created a user in mysql but was unable to run scripts as that user. Reason was that special characters were present in the password, which we should avoid. So, deleted and re-created. 

mysql -u root -h servername.tld -p mysql

SELECT User,Host FROM mysql.user;

REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'whicheveruser'@'localhost';

DROP USER 'whicheveruser'@'localhost';

cloudflare proxied self-signed web server issues

There was a problem with one of our websites set up with a different domain, where another site was being served up instead of the site configured with relevantdomain.conf virtualhost in apache. Turned out the issue was due to the way in which we had set up cloudflare proxying and SSL.

The apache server had been set up with a self-signed certificate, cloudflare was using flexible SSL mode, and a page rule had been applied in cloudflare for http to https redirection. But the web server had been configured with the self-signed cert for another domain. Maybe this was the reason for cloudflare to serve up the default.conf domain instead of the relevantdomain.conf.

Anyway, tidying up by using
certbot-auto --apache
and installing letsencrypt certificates for all the domains on that server fixed up the issue. The usual caution - when doing automatic http validation, turning cloudflare proxying off may be required. But for renewing the domains, this doesn't seem to be necessary, as tested with 
certbot-auto renew --dry-run

Later set up automatic renewal with 
45 2 * * 6 cd /etc/letsencrypt/ && ./certbot-auto renew && /etc/init.d/apache2 restart

not booting after timeshift restore

An upgrade from Linux Mint 19.3 to 20 did not end well. dpkg got stuck with some circular dependencies, probably because I had not removed all "foreign" packages before the upgrade. Then tried a fresh install of Linux Mint 20.1 and and upgrade to 20.2. Unfortunately, there seems to be a regression for Macbook support - the screen becomes corrupted after suspend and wake up. 

So, I tried to restore the timeshift backup I had made before the upgrade. That took a couple of hours, since I had backed up 40+ GB of home directory also. But then the system wouldn't boot into that restored partition.
Error message:  Alert: UUID=<whatever> does not exist, dropping to a shell. 

After looking at many places like here and here, finally I thought of checking the refind file boot/refind_linux.conf since I was booting with refind. That immediately showed up the issue. The boot/refind_linux.conf on the relevant partition, /dev/sda3 in this case, was pointing to a different UUID compared to the UUID reported by 
sudo blkid

So I just had to boot into another working Linux installation, edit this file boot/refind_linux.conf on /dev/sda3 with the correct UUIDs, and save it. The next boot worked fine. 

making a book readable on the Kindle from a set of articles

There are some pages with links to a large number of articles, like this one. Looking around for easily converting all those articles into a single book readable on an e-ink reader like the Kindle, my first trial was with Calibre and its News feature. Just adding that html page as the source did not work - the created ebook only had that page and nothing else.

Next, trying creating an RSS feed with fivefilters and giving that feed to Calibre - limited to 5 links, so not sure how it would work with the hundreds of old articles. 

Trying HTTrack to download pages - was taking a long time due to downloading CSS, images, etc. And we were not really interested in images.

So, used DownloadThemAll, choosing to download only *.htm and *.html linked from the index page, created a table of contents html page as noted in the FAQ. But Calibre complained of broken image links and stopped the conversion. 

Then tried converting all the html to txt using html2text - did not work, maybe my commandline piping was incorrect? Next, tried converting all the html to txt using HtmlAsText, running it as 

wine HtmlAsText.exe

This worked without a hitch. Then, made another table of contents html file with all the html files replaced by the txt files in the table of contents. But that conversion too ended up with only the table of contents being added to the ebook. 

Next, tried
cat *.txt > combinedfile.txt
and converted the combinedfile.txt. This created a combined ebook, but with ugly hard line-breaks every other line. Then, chose to enable heuristics processing with all options, and converted again. This time, got good results. With heuristics processing, the conversion takes longer, around 5-6 minutes for a 3 MB text file.  

So, to sum up - 

• DownloadThemAll
• HtmlAsText
• cat
• Calibre with heuristics enabled.

Edit - Later, I tried out another method using conversion to RSS and then calibre's built-in news feed reading capabilities, which might be easier. .

assigning roles to users on Moodle

On one of our Moodle installations, there was a request to change permissions of a couple of users - to assign the "manager of course category" status of one user to another. Though Moodle has an easy way to make a user a manager of the entire site, Site Administration -> Users -> Assign system roles - this particular permission assignment was a bit more tricky to navigate. One solution turned out to be:

1. Navigate to a course for which this user has manager status as an Admin or Manager
2. Choose Course "Actions Menu" (Gear icon on right corner with the Moove theme)
3. Choose "More" -> Users tab -> "Check permissions"
4. Filter for the user whose permissions you want to change, click to select and click "Show this user's permissions"
5. In this case, it showed, among other things, Manager in Category:TheRelevantCourseCategory
6. Clicking on Category:TheRelevantCourseCategory, then again the "Actions Menu" (Gear icon on right corner with the Moove theme), Assign Roles
7. That gives us a clickable table to assign roles (like Manager) in that particular category.

detailed steps for moodle upgrade using git

1. Official documentation at https://docs.moodle.org/311/en/Upgrading is a bit sparse on details like where to locate the plugins - I’m attempting to do this manually using
   find . -name “customcert”
   and so on for each of the “additional plugins” found on the plugins overview page at
   https://oursite.tld/admin/plugins.php?updatesonly=0&contribonly=1

2. This youtube video gives a good overview of how to manage a moodle installation and upgrades with git - https://www.youtube.com/watch?v=npks4TngSvo which gives a good presentation, tips for upgrading, a rapid summary and explanation for the documentation at https://docs.moodle.org/311/en/Git_for_Administrators

3. Creating the git directory and doing a test run with devel2.oursite.tld, which has the home dir /var/www/site  - commands -
   cd /var/www
   sudo mkdir sitegit
   sudo chown azureuser:www-data sitegit
   git clone git://git.moodle.org/moodle.git sitegit
   cd sitegit
   git branch --track MOODLE_311_STABLE origin/MOODLE_311_STABLE
   git branch --track MOODLE_39_STABLE origin/MOODLE_39_STABLE
   git checkout MOODLE_39_STABLE
   # first make sure 3.9 works, then upgrade 3.11
   
   

4. List of additional plugins and commands to copy them -
   cp -R ../site_learning/mod/customcert ./mod
   # have to copy the moove pix_plugins after copying moove theme first
   cp -R ../site_learning/mod/game ./mod
   cp -R ../site_learning/mod/hvp ./mod
   # have to copy the moove pix_plugins after copying moove theme first
   # not migrating admin/tool/objectfs
   cp -R ../site_learning/course/format/onetopic ./course/format
   cp -R ../site_learning/report/coursesize ./report
   cp -R ../site_learning/theme/moove ./theme
   cp -R ../site_learning/theme/moove/pix_plugins/mod/customcert ./theme/moove/pix_plugins/mod
   cp -R ../site_learning/theme/moove/pix_plugins/mod/hvp ./theme/moove/pix_plugins/mod
   # not migrating local/azure_storage
   # not migrating local/edwiserbridge
   cp ../site/config.php .

5. Change owner group from azureuser:azureuser to azureuser:www-data
   sudo chown -R azureuser:www-data *
   
   

6. Take a backup and create symlinks for easy switching between old and versions.
   cd ..
   sudo mv site site_old && sudo ln -s site_old site
   # sudo chown azureuser:www-data site < - - - this doesn’t work
   
   

7. Check that the site is working, and that the version is seen as 3.9.2+ at https://devel2.oursite.tld/admin/index.php?cache=0
   Site Administration -> notifications.
   
   

8. Change over to the new code
   sudo rm site && sudo ln -s sitegit site
   
   

9. Check if Site Administration -> notifications reflects the new code - it prompts for update for minor upgrade. Have to go plugin by plugin and upgrade as needed. This took just a couple of minutes.
   
   

10. Now removed those plugins which we did not copy over, by going to Site Administration -> Plugins -> Plugins overview -> Additional plugins and uninstall for everything except those in step 4.
    customcert
    game
    hvp
    onetopic
    coursesize
    
    

11. Now trying upgrading directly to 3.11 using git checkout -
    git checkout MOODLE_311_STABLE
    
    

12. Only warning seen was PHP setting max_input_vars is recommended to be at least 5000. Clicked through all the prompts, Notifications page shows 3.11
    
    

13. Cron not run for 5 minutes error message went away after a couple of minutes. But some sort of css issue, page formatting jumbled. Purged caches with
    sudo -u www-data php admin/cli/purge_caches.php
    But still the issue persists. Looks like the accessibility toolbar.
    
    

14. https://moodle.org/mod/forum/discuss.php?d=335856 gave the clue of ownership issues. Did Step 5. again. Stil not solved. https://moodle.org/mod/forum/discuss.php?d=325116 suggests deleting the theme. Deleted by
    cd /var/www/sitegit/theme
    ls
    rm -R adaptable
    rm -R moove
    That solved the issue. Adaptable theme is not available for 3.11, perhaps that was the issue. Again installed moove theme manually, and changed to moove theme, OK.
    
    

15. Trying to upgrade customcert,
    Validating mod_customcert ... Error
      [Warning] Target location already exists and will be removed [/var/www/sitegit/mod/customcert]  
    [Error] Write access check [/var/www/sitegit/mod/customcert] Installation aborted due to validation failure

16. https://moodle.org/mod/forum/discuss.php?d=366460 pointed to permission issues. Found that mod/customcert did not have w set for group, so set for all mod with
    cd /var/www/sitegit/mod
    chmod  -R g+w *
    Solved.
    
    

17. Found php.ini file location /etc/php/7.4/apache2/php.ini from https://devel2.oursite.tld/admin/phpinfo.php and modified to remove the warning about max_input_vars in step 12.
    
    

18. For migrating oursite.tld, additional steps of copying over data from azure blob storage to local disk. Rclone would help to copy in advance and later sync only those files which have been added later.
    
    

19. Compared speeds of downloads with oursite.tld without Cloudflare caching and devel2 with Cloudflare caching - not significantly different especially if cloudflare caching is on.
    11 sec for file.pptx (4.8 MB) from devel2 (3.5 Mbps)
    60 sec for afile.ppt (35.7 MB) from oursite (4.76 Mbps)
    48 sec for afile.ppt (35.7 MB) from devel2 (5.95 Mbps)
    So, migrating oursite to local disk from Azure blob storage will not have too much of a speed penalty. Is likely to save costs also since Azure storage charges for transfers and so on.
    
    

20. In order to migrate oursite.tld filedir from Azure Blob storage to local HDD, started copying using a SAS token and SAS URL generated from the Azure portal.
    portal.azure.com -> Resource groups -> site-rgname -> sitelearning (storage account) -> Containers -> sitelms -> Shared Access Tokens -> Generate SAS token and URL.
    
    

21. On the server, installed rclone and set it up with
    sudo apt  install rclone
    rclone config
    setting up a new remote called azurecontainer by passing the SAS URL when prompted.
    
    

22. Created directory structure by copying from /var/www/site_learning_files to /var/www/site_data_disk/site_learning_data
    
    

23. Tested rclone sync from the Azure container to the 00 subdirectory of filedir with
    rclone sync azurecontainer:sitelms/00 filedir/00 -P
    Worked, finished in less than a second.
    
    

24. Started screen, and then ran
    rclone sync azurecontainer:sitelms/ filedir/ -P
    Showed ETA of 18 minutes for 34 GB. Total is 54 GB, so would probably take longer. Actual time taken was around 45 minutes for 54 GB.
     

25. Can sync again just before doing the migration in a few seconds. And then can remove the remote using
    rclone config delete azurecontainer
    
    

26. Creating git directory for site_learning just like step 3:
    cd /var/www
    sudo mkdir site_learning_git
    sudo chown azureuser:www-data site_learning_git
    git clone git://git.moodle.org/moodle.git site_learning_git
    cd site_learning_git
    git branch --track MOODLE_311_STABLE origin/MOODLE_311_STABLE
    git branch --track MOODLE_39_STABLE origin/MOODLE_39_STABLE
    git checkout MOODLE_39_STABLE
    # first make sure 3.9 works, then upgrade 3.11
    
    

27. List of additional plugins and commands to copy them, as in step 4, except the last line for config.php which should be from site_learning -
    cp -R ../site_learning/mod/customcert ./mod
    # have to copy the moove pix_plugins after copying moove theme first
    cp -R ../site_learning/mod/game ./mod
    cp -R ../site_learning/mod/hvp ./mod
    # have to copy the moove pix_plugins after copying moove theme first
    # not migrating admin/tool/objectfs
    cp -R ../site_learning/course/format/onetopic ./course/format
    cp -R ../site_learning/report/coursesize ./report
    cp -R ../site_learning/theme/moove ./theme
    cp -R ../site_learning/theme/moove/pix_plugins/mod/customcert ./theme/moove/pix_plugins/mod
    cp -R ../site_learning/theme/moove/pix_plugins/mod/hvp ./theme/moove/pix_plugins/mod
    # not migrating local/azure_storage
    # not migrating local/edwiserbridge
    cp ../site_learning/config.php . cp ../site_learning/*.html . The last line is to copy site policy agreements if any, if they are located in the root directory.
    
    

28. Like step 5, change owner group from azureuser:azureuser to azureuser:www-data
    sudo chown -R azureuser:www-data *
    
    

29. Take a backup and create symlinks for easy switching between old and versions just like step 6.
    cd ..
    sudo mv site_learning site_learning_old && sudo ln -s site_learning_old site_learning
    
    

30. Site working, but showed an initial error message on first login, later none. This concludes the pre-maintenance mode section.
    
    
    

31. Beginning the migration - Put the site in maintenance mode from Site Administration -> Server -> Maintenance mode.
    
    

32. Took a backup of the database by logging in to the database VM,
    Commands from https://hnsws.blogspot.com/2020/12/mysql-and-postgresql-command-line-cheat.html
    ssh site_admin@40.90.169.113
    mysqldump -u site_db_admin -p site_learning_master_db -r LMSbk20210808.sql
    (took just a few seconds, sql file was 333 MB in size.)
    
    

33. Logged out of the Database VM, back on the php VM, equivalent commands to step 8 -
    sudo rm site_learning && sudo ln -s site_learning_git site_learning
    
    

34. Need to sync the local data directory with the latest changes on Azure blob storage -
    screen -x
    rclone sync azurecontainer:sitelms/ filedir/ -P
    Transferred:      149.056M / 149.056 MBytes, 100%, 22.964 MBytes/s, ETA 0s
    Errors:                 0
    Checks:             15701 / 15701, 100%
    Transferred:          148 / 148, 100%
    Elapsed time:        6.4s
    
    

35. Made sure the permissions were proper -
    cd /var/www/site_data_disk
    sudo chown -R azureuser:www-data site_learning_data
    sudo chmod -R 775 site_learning_data
    
    

36. Before starting the upgrade, we need to change config.php to point to the new data directory.
    cd /var/www/site_learning_git
    nano config.php
    Changed the line
    //$CFG->dataroot  = '/var/www/site_learning_files';
    To
    $CFG->dataroot  = '/var/www/site_data_disk/site_learning_data';
    And commented out
    //$CFG->alternative_file_system_class = '\tool_objectfs\azure_file_system';
    
    

37. Copied the files from the existing data directory, other than the filedir directory, to the new data directory -
    cd /var/www/site_data_disk/site_learning_data
    cp -Rv /var/www/site_learning_files/lang .
    cp -Rv /var/www/site_learning_files/models .
    cp -Rv /var/www/site_learning_files/muc .
    
    

38. Then started the upgrade process from the web console. Got
    Fatal error: $CFG->dataroot is not specified in config.php! Exiting. https://moodle.org/mod/forum/discuss.php?d=370441
    Indicates permission issues for the parent dir. Tried
    sudo chown -R www-data:www-data /var/www/site_data_disk
    sudo chown -R www-data:www-data /var/www/site_data_disk/site_learning_data
    Still no. Then tried
    sudo chown www-data:www-data config.php
    And tried cloudflare removing the proxying for learning - still no go.
    
    

39. Changed back the config file with
    cd /var/www
    sudo rm site_learning && sudo ln -s site_learning_old site_learning
    Then the site works.
    
    

40. Edited the file again to modify the existing line in /var/www/site_learning_git/config.php instead of creating a new line - that worked. Issue would probably be CRLF (line ending character) related, since the original config.php was probably created on Windows and I was editing it on Linux.
    
    

41. Did step 33 again -
    sudo rm site_learning && sudo ln -s site_learning_git site_learning
    
    

42. Working. Then uninstalled azure_storage, objectfs and edwiserbridge from plugins using the web interface, from Site Administration -> Plugins -> Plugins Overview -> Additional plugins
    
    

43. Changed to Boost theme to prevent errors in moove theme during upgrade from Site Administration -> Appearance -> Theme selector
    
    

44. Then the upgrade to 3.11 by running
    cd site_learning_git
    git checkout MOODLE_311_STABLE
    
    

45. Then doing the upgrade from the web interface by clicking on Site Administration and following the prompts. When trying to upgrade the plugins, got the Error that the directory is not writable. So, made sure all the git subdirectories were writable by the www-data group with
    sudo chmod -R 775 /var/www/site_learning_git
    
    

46. Removed maintenance mode.
    
    

47. Changed theme back to Moove.
    
    

48. Changed cloudflare back to proxied. Done.
    
    

49. Cleaning up -
    Changed permissions of config.php to read-only
    sudo chmod -w /var/www/site_learning_git/config.php
    sudo chmod g-w /var/www/site_learning_git/config.php
    
    

50. Try to not have 777 permissions on the moodledata directory,
    sudo chmod 755 /var/www/site_data_disk
    sudo chmod 755 /var/www/site_data_disk/site_learning_data
    (both are chown www-data:www-data)
    Site seems to be working fine.