caution with Moodle VMSS

The Azure VMSS (Virtual Machine Scale Set) Moodle deployment in combination with automated site backups has to be used with caution. The object storage filesystem instructs the administrator to create SAS keys with only read and write permissions, and it looks like this would cause the moodle scripts which try to do deletes to fail. So, in our case, when automatic site backups were enabled, blob storage usage began to balloon.

Also, occasionally, the backup would cause some issue due to which the object storage filesystem would stop writing to blob storage and would instead fill up the local hard disk, causing moodle to complain "session partition full". One quick check and the first place to delete would be the temp directory, especially if backup is turned on. 

Even after running some scripts (a bash script is mentioned in this forum thread) and deleting all the unnecessary backups and orphaned files, blob capacity did not come down, only container size came down. So, my brute-force solution was to avoid blob storage and use local hard disk storage instead. 

Chrome blocking http links

Google Chrome browser now blocks http links from pages served over https. Even if the http linked pages have https redirect enabled, Chrome continues to block. Only way out would be to replace all http links with https links. 

HTTP redirect was with the rule which was created by certbot etc,

RewriteEngine on
RewriteCond %{SERVER_NAME} =our.server.name
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

backing up to Google drive with rclone

Backing up from a server to google drive using rclone directly on the server, using screen so that I don't have to be logged in  for the entire multi-hour process - 1 GB takes around 2 hours. So it will take another couple of days for the process to be completed.

Commands were like

rclone copy srcDirectory gdrivearchivercloneremotename:"backupDir/destDirectory"

and so on. 

Example output:

Transferred:        7.493G / 7.493 GBytes, 100%, 396.555 kBytes/s, ETA 0s
Transferred:        46441 / 46441, 100%
Elapsed time:   5h30m20.6s

finding which files are causing disk to get full from linux command line

One of our servers ran out of disk space, so in order to diagnose the issue from the terminal, used the technique mentioned in this askubuntu post, that is,
du -cha --max-depth=1 / | grep -E "M|G"

Then, to reduce log file usage by journald, as this thread says,

edit /etc/systemd/journald.conf
Activate the SystemMaxUse= option there, e.g. as SystemMaxUse=100M to only use 100 MB.
service systemd-journald restart to activate the changed configuration.

Then, to clear up emails for root from /var/mail - they are mostly cron emails - various techniques discussed here, easiest would be to just empty it - but the safest way to do it might be using the mail command, 
sudo su -
mail
delete *
q
which immediately cleared up all the root emails.
(By default, if you press enter and read a mail, it gets saved to an mbox file in that user's home directory, so in this case, /root/mbox )

Then, redirecting the cron job emails to /dev/null to prevent this from happening again, by appending
> /dev/null 2>&1
to all those cron jobs. 

 

adding SSL (https) to http mp3 streams - icecast-kh

The earlier reverse-proxy solution had hiccups when thousands of listeners connected. So, explored directly using HTTPS on icecast. Then PB discovered icecast-kh which has improved SSL certificate handling, without needing a restart when the certificate is updated - 

an interesting fork of icecast2 called icecast-kh

https://github.com/karlheyes/icecast-kh

https://github.com/xiph/Icecast-Server/issues/20

https://github.com/AzuraCast/AzuraCast/issues/358 says the following:

8/5/2017

--autodetect SSL connections on incoming sockets. No need for in listen-socket now but is still there for compatability. (THIS IS BIG!!! We no longer need to use different ports for encrypted and unencrypted dramatically reducing complexity)

--add ssl-private-key in paths to allow for combined PEM or for separate SSL key/certificate files. (THIS IS BIG TOO! No longer need a separate process after updating let's encrypt to combine your fullchain and private cert into one pem file for icecast to read it!)

--select https/http URL in autogenerated m3u based on incoming request.

So, this was what was implemented, with a LetsEncrypt certificate. Compiled with

 $ cd <directory where you have extracted the tar.gz>
 $ ./configure --prefix=/home/ouruser/icecastkhSSL
 $ make
 $ make install

 The SSL certificate is updated through change_ssl_certificate.sh

cat /etc/ssl/private/ourserver.tld.key   /root/.acme.sh/ourserver.tld/fullchain.cer > /path/to/icecast.pem.letsencrypt
chown ouruser:ouruser /path/to/icecast.pem.letsencrypt
cp /path/to/icecast.pem   /path/to/icecast.pem.backup
#echo "Backed up icecast.pem to icecast.pem.backup..."
cp /path/to/icecast.pem.letsencrypt    /path/to/icecast.pem

Cron is run as root every day after acme script

43 0 * * * "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null

50 0 * * * /home/path/to/scripts/change_ssl_certificate.sh > /dev/null

server upgrade, certificates

We planned a streaming server upgrade. During the upgrade process, we would transfer the DNS to our web server, making it the temporary streaming server. 

According to one of the replies at

https://community.letsencrypt.org/t/how-to-set-up-lets-encrypt-https-after-moving-site-to-new-server/114776/2

the easiest way for temporary certificate transfer would be to just copy the certificates and icecast proxy server executable from our streaming server to our web server.

So,

1. copy the certificates and 2nd icecast server to the web server

2. check that the web server returns the correct certificate by locally setting /etc/hosts file DNS record of our stream url to the web server's IP address

3. on cloudflare, change the IP address from the streaming ip address to the web server's ip address for the stream url A record

After the upgrade is done, again do the steps 1, 2 and 3 for going from the web server to the streaming server. If everything goes well, the certificates and executable will remain on on the streaming server, and we will not need to copy them back. 

making GMail the default mail handler

Opening mailto links in GMail - 
https://developers.google.com/web/updates/2012/02/Getting-Gmail-to-handle-all-mailto-links-with-registerProtocolHandler

(Not sure if this 2012 technique still works for Firefox and Chrome).

Firefox has a support page for this -
https://support.mozilla.org/en-US/kb/change-program-used-open-email-links

As also Chrome - 
https://support.google.com/a/users/answer/9308783?hl=en

conditional formatting in Google sheets

Highlighting cells in Google sheets when they fulfil certain conditions - 
https://stackoverflow.com/questions/21899516/how-to-highlight-cell-if-value-duplicate-in-same-column-for-google-spreadsheet