Work Stuff: detailed steps for moodle upgrade using git

Official documentation at https://docs.moodle.org/311/en/Upgrading is a bit sparse on details like where to locate the plugins - I’m attempting to do this manually using
find . -name “customcert”
and so on for each of the “additional plugins” found on the plugins overview page at
https://oursite.tld/admin/plugins.php?updatesonly=0&contribonly=1
This youtube video gives a good overview of how to manage a moodle installation and upgrades with git - https://www.youtube.com/watch?v=npks4TngSvo which gives a good presentation, tips for upgrading, a rapid summary and explanation for the documentation at https://docs.moodle.org/311/en/Git_for_Administrators
Creating the git directory and doing a test run with devel2.oursite.tld, which has the home dir /var/www/site - commands -
cd /var/www
sudo mkdir sitegit
sudo chown azureuser:www-data sitegit
git clone git://git.moodle.org/moodle.git sitegit
cd sitegit
git branch --track MOODLE_311_STABLE origin/MOODLE_311_STABLE
git branch --track MOODLE_39_STABLE origin/MOODLE_39_STABLE
git checkout MOODLE_39_STABLE
# first make sure 3.9 works, then upgrade 3.11
List of additional plugins and commands to copy them -
cp -R ../site_learning/mod/customcert ./mod
# have to copy the moove pix_plugins after copying moove theme first
cp -R ../site_learning/mod/game ./mod
cp -R ../site_learning/mod/hvp ./mod
# have to copy the moove pix_plugins after copying moove theme first
# not migrating admin/tool/objectfs
cp -R ../site_learning/course/format/onetopic ./course/format
cp -R ../site_learning/report/coursesize ./report
cp -R ../site_learning/theme/moove ./theme
cp -R ../site_learning/theme/moove/pix_plugins/mod/customcert ./theme/moove/pix_plugins/mod
cp -R ../site_learning/theme/moove/pix_plugins/mod/hvp ./theme/moove/pix_plugins/mod
# not migrating local/azure_storage
# not migrating local/edwiserbridge
cp ../site/config.php .
Change owner group from azureuser:azureuser to azureuser:www-data
sudo chown -R azureuser:www-data *
Take a backup and create symlinks for easy switching between old and versions.
cd ..
sudo mv site site_old && sudo ln -s site_old site
# sudo chown azureuser:www-data site < - - - this doesn’t work
Check that the site is working, and that the version is seen as 3.9.2+ at https://devel2.oursite.tld/admin/index.php?cache=0
Site Administration -> notifications.
Change over to the new code
sudo rm site && sudo ln -s sitegit site
Check if Site Administration -> notifications reflects the new code - it prompts for update for minor upgrade. Have to go plugin by plugin and upgrade as needed. This took just a couple of minutes.
Now removed those plugins which we did not copy over, by going to Site Administration -> Plugins -> Plugins overview -> Additional plugins and uninstall for everything except those in step 4.
customcert
game
hvp
onetopic
coursesize
Now trying upgrading directly to 3.11 using git checkout -
git checkout MOODLE_311_STABLE
Only warning seen was PHP setting max_input_vars is recommended to be at least 5000. Clicked through all the prompts, Notifications page shows 3.11
Cron not run for 5 minutes error message went away after a couple of minutes. But some sort of css issue, page formatting jumbled. Purged caches with
sudo -u www-data php admin/cli/purge_caches.php
But still the issue persists. Looks like the accessibility toolbar.
https://moodle.org/mod/forum/discuss.php?d=335856 gave the clue of ownership issues. Did Step 5. again. Stil not solved. https://moodle.org/mod/forum/discuss.php?d=325116 suggests deleting the theme. Deleted by
cd /var/www/sitegit/theme
ls
rm -R adaptable
rm -R moove
That solved the issue. Adaptable theme is not available for 3.11, perhaps that was the issue. Again installed moove theme manually, and changed to moove theme, OK.
Trying to upgrade customcert,
Validating mod_customcert ... Error
[Warning] Target location already exists and will be removed [/var/www/sitegit/mod/customcert]
[Error] Write access check [/var/www/sitegit/mod/customcert] Installation aborted due to validation failure
https://moodle.org/mod/forum/discuss.php?d=366460 pointed to permission issues. Found that mod/customcert did not have w set for group, so set for all mod with
cd /var/www/sitegit/mod
chmod -R g+w *
Solved.
Found php.ini file location /etc/php/7.4/apache2/php.ini from https://devel2.oursite.tld/admin/phpinfo.php and modified to remove the warning about max_input_vars in step 12.
For migrating oursite.tld, additional steps of copying over data from azure blob storage to local disk. Rclone would help to copy in advance and later sync only those files which have been added later.
Compared speeds of downloads with oursite.tld without Cloudflare caching and devel2 with Cloudflare caching - not significantly different especially if cloudflare caching is on.
11 sec for file.pptx (4.8 MB) from devel2 (3.5 Mbps)
60 sec for afile.ppt (35.7 MB) from oursite (4.76 Mbps)
48 sec for afile.ppt (35.7 MB) from devel2 (5.95 Mbps)
So, migrating oursite to local disk from Azure blob storage will not have too much of a speed penalty. Is likely to save costs also since Azure storage charges for transfers and so on.
In order to migrate oursite.tld filedir from Azure Blob storage to local HDD, started copying using a SAS token and SAS URL generated from the Azure portal.
portal.azure.com -> Resource groups -> site-rgname -> sitelearning (storage account) -> Containers -> sitelms -> Shared Access Tokens -> Generate SAS token and URL.
On the server, installed rclone and set it up with
sudo apt install rclone
rclone config
setting up a new remote called azurecontainer by passing the SAS URL when prompted.
Created directory structure by copying from /var/www/site_learning_files to /var/www/site_data_disk/site_learning_data
Tested rclone sync from the Azure container to the 00 subdirectory of filedir with
rclone sync azurecontainer:sitelms/00 filedir/00 -P
Worked, finished in less than a second.
Started screen, and then ran
rclone sync azurecontainer:sitelms/ filedir/ -P
Showed ETA of 18 minutes for 34 GB. Total is 54 GB, so would probably take longer. Actual time taken was around 45 minutes for 54 GB.
Can sync again just before doing the migration in a few seconds. And then can remove the remote using
rclone config delete azurecontainer
Creating git directory for site_learning just like step 3:
cd /var/www
sudo mkdir site_learning_git
sudo chown azureuser:www-data site_learning_git
git clone git://git.moodle.org/moodle.git site_learning_git
cd site_learning_git
git branch --track MOODLE_311_STABLE origin/MOODLE_311_STABLE
git branch --track MOODLE_39_STABLE origin/MOODLE_39_STABLE
git checkout MOODLE_39_STABLE
# first make sure 3.9 works, then upgrade 3.11
List of additional plugins and commands to copy them, as in step 4, except the last line for config.php which should be from site_learning -
cp -R ../site_learning/mod/customcert ./mod
# have to copy the moove pix_plugins after copying moove theme first
cp -R ../site_learning/mod/game ./mod
cp -R ../site_learning/mod/hvp ./mod
# have to copy the moove pix_plugins after copying moove theme first
# not migrating admin/tool/objectfs
cp -R ../site_learning/course/format/onetopic ./course/format
cp -R ../site_learning/report/coursesize ./report
cp -R ../site_learning/theme/moove ./theme
cp -R ../site_learning/theme/moove/pix_plugins/mod/customcert ./theme/moove/pix_plugins/mod
cp -R ../site_learning/theme/moove/pix_plugins/mod/hvp ./theme/moove/pix_plugins/mod
# not migrating local/azure_storage
# not migrating local/edwiserbridge
cp ../site_learning/config.php . cp ../site_learning/*.html . The last line is to copy site policy agreements if any, if they are located in the root directory.
Like step 5, change owner group from azureuser:azureuser to azureuser:www-data
sudo chown -R azureuser:www-data *
Take a backup and create symlinks for easy switching between old and versions just like step 6.
cd ..
sudo mv site_learning site_learning_old && sudo ln -s site_learning_old site_learning
Site working, but showed an initial error message on first login, later none. This concludes the pre-maintenance mode section.
Beginning the migration - Put the site in maintenance mode from Site Administration -> Server -> Maintenance mode.
Took a backup of the database by logging in to the database VM,
Commands from https://hnsws.blogspot.com/2020/12/mysql-and-postgresql-command-line-cheat.html
ssh site_admin@40.90.169.113
mysqldump -u site_db_admin -p site_learning_master_db -r LMSbk20210808.sql
(took just a few seconds, sql file was 333 MB in size.)
Logged out of the Database VM, back on the php VM, equivalent commands to step 8 -
sudo rm site_learning && sudo ln -s site_learning_git site_learning
Need to sync the local data directory with the latest changes on Azure blob storage -
screen -x
rclone sync azurecontainer:sitelms/ filedir/ -P
Transferred: 149.056M / 149.056 MBytes, 100%, 22.964 MBytes/s, ETA 0s
Errors: 0
Checks: 15701 / 15701, 100%
Transferred: 148 / 148, 100%
Elapsed time: 6.4s
Made sure the permissions were proper -
cd /var/www/site_data_disk
sudo chown -R azureuser:www-data site_learning_data
sudo chmod -R 775 site_learning_data
Before starting the upgrade, we need to change config.php to point to the new data directory.
cd /var/www/site_learning_git
nano config.php
Changed the line
//$CFG->dataroot = '/var/www/site_learning_files';
To
$CFG->dataroot = '/var/www/site_data_disk/site_learning_data';
And commented out
//$CFG->alternative_file_system_class = '\tool_objectfs\azure_file_system';
Copied the files from the existing data directory, other than the filedir directory, to the new data directory -
cd /var/www/site_data_disk/site_learning_data
cp -Rv /var/www/site_learning_files/lang .
cp -Rv /var/www/site_learning_files/models .
cp -Rv /var/www/site_learning_files/muc .
Then started the upgrade process from the web console. Got
Fatal error: $CFG->dataroot is not specified in config.php! Exiting. https://moodle.org/mod/forum/discuss.php?d=370441
Indicates permission issues for the parent dir. Tried
sudo chown -R www-data:www-data /var/www/site_data_disk
sudo chown -R www-data:www-data /var/www/site_data_disk/site_learning_data
Still no. Then tried
sudo chown www-data:www-data config.php
And tried cloudflare removing the proxying for learning - still no go.
Changed back the config file with
cd /var/www
sudo rm site_learning && sudo ln -s site_learning_old site_learning
Then the site works.
Edited the file again to modify the existing line in /var/www/site_learning_git/config.php instead of creating a new line - that worked. Issue would probably be CRLF (line ending character) related, since the original config.php was probably created on Windows and I was editing it on Linux.
Did step 33 again -
sudo rm site_learning && sudo ln -s site_learning_git site_learning
Working. Then uninstalled azure_storage, objectfs and edwiserbridge from plugins using the web interface, from Site Administration -> Plugins -> Plugins Overview -> Additional plugins
Changed to Boost theme to prevent errors in moove theme during upgrade from Site Administration -> Appearance -> Theme selector
Then the upgrade to 3.11 by running
cd site_learning_git
git checkout MOODLE_311_STABLE
Then doing the upgrade from the web interface by clicking on Site Administration and following the prompts. When trying to upgrade the plugins, got the Error that the directory is not writable. So, made sure all the git subdirectories were writable by the www-data group with
sudo chmod -R 775 /var/www/site_learning_git
Removed maintenance mode.
Changed theme back to Moove.
Changed cloudflare back to proxied. Done.
Cleaning up -
Changed permissions of config.php to read-only
sudo chmod -w /var/www/site_learning_git/config.php
sudo chmod g-w /var/www/site_learning_git/config.php
Try to not have 777 permissions on the moodledata directory,
sudo chmod 755 /var/www/site_data_disk
sudo chmod 755 /var/www/site_data_disk/site_learning_data
(both are chown www-data:www-data)
Site seems to be working fine.

Work Stuff

Sunday, August 08, 2021

detailed steps for moodle upgrade using git

No comments:

Post a Comment

google analytics 4 code