server admin activities for an Asst. Admin

Excerpts from my reply to a colleague in a sister institution who had joined recently and wanted to get up to speed with his duties - 

These are the activities I have done to help the LMS team - 

1. Setting DNS records in Cloudflare - 

https://ns1.com/resources/dns-types-records-servers-and-queries

https://en.wikipedia.org/wiki/Cloudflare

https://support.cloudflare.com/hc/en-us/articles/360019093151-Managing-DNS-records-in-Cloudflare

The DNS records are currently handled by V----, I help him when needed.

2. Helping with load testing for servername.our.domain - R--- did most of the work - 

https://hnsws.blogspot.com/2020/12/load-testing-moodle-server-log-in-with.html

https://docs.moodle.org/310/en/Performance_recommendations#Hardware_configuration

Please see the test reports made by Prof. S----- and R-----.

3. Setting up SSL (https) using bitnami's tool - bncert-tool

https://hnsws.blogspot.com/2020/12/location-of-apache-configuration-files.html

In case the machine in question is not a bitnami VM,

https://hnsws.blogspot.com/2020/11/setting-up-letsencrypt-certificates-for.html

https://certbot.eff.org/

4. For doing the above (point 3.), you will need to know how to use the command-line, and how to use ssh - 

https://www.ucl.ac.uk/isd/what-ssh-and-how-do-i-use-it

https://en.wikipedia.org/wiki/PuTTY

https://www.chiark.greenend.org.uk/~sgtatham/putty/faq.html

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html

5. If you are not familiar with the Linux terminal (command-line), 

https://www.google.com/search?q=getting+started+with+the+linux+command+line

creating a backup of a web server

Process followed for creating a copy of one of our web servers on a backup server - 

• Copied the conf files using sftp.
• Created the user using adduser 
• mkdir downloads as the new user,
• added to www-data group and changed permissions as on our server,
  # chown nameofuser:www-data downloads
  # chmod 755 downloads
  
  
• Check if server2 has enough disk space before running rsync.
  Currently seems to have ~600 GB available.
  
  
• Ran a2ensite for all the missing sites, and 
• service apache2 reload
  

Edit: Now that the servers run https sites, must remember to set up suitable ACME clients as available for the respective OS, for managing LetsEncrypt certificates. 

For up-to-date Ubuntu-based Linux distros, 
sudo apt install certbot python3-certbot-apache

For older distros without support, acme.sh is what I have used in the past.

For Windows, win-acme is a good option.

certbot missing apache plugin

I too faced the same issue described in the post below, with the same solution - 
apt install python3-certbot-apache

https://community.letsencrypt.org/t/certbot-missing-apache-plugin/58579

Moodle directory permissions

"Invalid permissions detected when trying to create a directory. Turn debugging on for further details." 

Simple fix would be:

sudo chown -R  www-data:www-data /var/www/our_moodle_dir
sudo chown -R  www-data:www-data /var/www/our_moodle_files_dir

mount a new data disk on an Azure Linux VM

https://blog.e-zest.com/how-to-create-attach-and-mount-a-disk-to-linux-vm-microsoft-azure

Basically create the disk from the Azure portal, then

• find the name of the device using dmesg - for eg. /dev/sdc
• sudo fdisk /dev/sdc
• sudo mkfs -t ext4 /dev/sdc1
• sudo mkdir DriveFolder; sudo mount /dev/sdc1 DriveFolder
• append to /etc/fstab using sudo blkid to find the UUID of the disk. Eg. 
  UUID=33333333-3b3b-3c3c-3d3d-3e3e3e3e3e3e   /datadrive   ext4   defaults,nofail   1   2
  

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/attach-disk-portal
has a better set of instructions. 

mounting Azure blob storage

We looked at the different ways in which Azure storage could be mounted on a Linux Moodle instance, using blobfuse

https://docs.microsoft.com/en-us/azure/storage/blobs/storage-how-to-mount-container-linux

but did not find any mentions using

mount

or

crontab -l (for various users)

Edit: It turned out it was using Moodle's Object storage file system plugin, which manages all the storage from within Moodle. But see this caveat.

Fresh install of Moodle on Ubuntu 20.04 with apt

Using the command at https://www.tecmint.com/install-moodle-in-ubuntu/

sudo apt update
sudo apt install php-common php-iconv php-curl php-mbstring php-xmlrpc php-soap php-zip php-gd php-xml php-intl php-json libpcre3 libpcre3-dev graphviz aspell ghostscript clamav

Then git-based install after

sudo apt install git

Edit: For the error "PHP has not been properly configured with the MySQLi extension for it to communicate with MySQL. Please check your php.ini file or recompile PHP."
we can follow the installation instructions at
https://docs.moodle.org/310/en/Step-by-step_Installation_Guide_for_Ubuntu

Installing the lapp stack on Ubuntu 20

https://kb.amijani.net/web-server/how-to-install-lapp-apache-php-postgresql-on-ubuntu-20-04/

Basically, 
sudo apt install apache2
sudo apt install php7.4 libapache2-mod-php7.4 openssl php-imagick php7.4-common php7.4-curl php7.4-gd php7.4-imap php7.4-intl php7.4-json php7.4-ldap php7.4-mbstring php7.4-pgsql php-ssh2 php7.4-xml php7.4-zip unzip

Then install PostgreSQL with
https://www.digitalocean.com/community/tutorials/how-to-install-postgresql-on-ubuntu-20-04-quickstart

connecting to a pptp VPN

A VPN to some servers was configured on Windows 10 clients as 
Type of sign-in info: User name and password
and under "Advanced", 
VPN type: Automatic.

Other windows clients were able to log on to the VPN with no problems, but when I tried with Linux Mint and the default pptp VPN settings, the connection was failing. My first thought was that some setting was missing, but it turned out that the reason for the VPN to fail was that my internet connection was via a mobile hotspot - just connecting via a broadband connection did the trick, I did not have to do any port-forwarding or any settings on the router. And then I found that some mobile hotspots work too - 
LG Q6 with Airtel 4G on primary SIM slot - no
LG Q6 with Airtel 3G on primary SIM slot - sometimes works. 
Redmi 4 with Jio SIM - works
Realme U1 with Jio SIM in secondary SIM slot - no
JioFi router (2018 model) - works
BSNL ADSL broadband with wifi router - works

With the default routing, all connections are routed through the VPN, and normal internet browsing doesn't work. With the help of this post, changed the routing so that only connections to those machines which are on the VPN are routed through the VPN.

/var/log/syslog gave the internal gateway info, which is on the private subnet, different from the external gateway which is on a public subnet.

 

making a rectangle in Gimp

1. Select using Rectangle Select Tool
2. Edit - Stroke Selection

rsync, screen and ssh-agent

On some of our other Ubuntu servers, rsync in a cron job as root works fine with ssh and key-based authentication. But not on a CentOS server I recently logged on to, because ssh-agent was not automatically started on login.

So, I have to run the following - 

eval `ssh-agent -s` 
ssh-add /path/to/keyfile.pem

and then do the ssh -i keyfile user@remoteserver

If I want to run a long rsync job from within screen, I need to run the above commands again from within screen, or else the ssh-agent is not accessible from within screen.

The next issue was the use of rsync with directories which had spaces in them. Instead of laboriously escaping each space with a backslash, I used the --protect-args method.

rsync -azvhrt --protect-args -e ssh  "/local path/with spaces/directory to copy" "user@remotemachine.tld:/remote/path/with spaces/"

php oauth server - link dump

PHP 5

https://www.sitepoint.com/creating-a-php-oauth-server/

https://bshaffer.github.io/oauth2-server-php-docs/

PHP 7

https://github.com/thephpleague/oauth2-server

Using firebase (PHP 7)

https://github.com/jeromegamez/firebase-php-examples

Edit: We finally did not use any of these, and instead authenticated using google sign-in against our own database as mentioned at https://hnsws.blogspot.com/2021/12/using-google-sign-in-for-our-php-portal.html

creating rectangular boxes with Gimp

Just to remember - rectangular marquee to select - shortcut key is R
Then, menu item Edit - Stroke selection

Line width of 4 pixels looks quite good for onscreen viewing, like below. 

forcing chrome to reconnect to a website

For purposes like profiling site load speeds etc, making Chrome close connection to a web server and start over - 

https://stackoverflow.com/questions/37170812/force-chrome-to-close-re-open-all-tcp-tls-connections-when-profiling-with-the-ne

Go to

chrome://net-internals#sockets

and close idle sockets.

Also works for sessions opened with SSH tunnels, when we want to close the connection immediately.

HTTP to HTTPS redirection on Windows Server and IIS

Certbot can automatically add http to https redirection when used with Apache on Linux, by appending

RewriteEngine on
RewriteCond %{SERVER_NAME} =whatever.tld
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

To do the same on Windows, I followed the method given at
https://www.ssl.com/how-to/redirect-http-to-https-with-windows-iis-10/

for redirecting http to https on IIS+Windows. 

• Download and install the URL Rewrite module
• In IIS Manager, choose the relevant website, double-click URL Rewrite
• Click Add Rules on RHS, create a Blank Rule in inbound section.
• Choose Requested URL: Matches Pattern and Using: Regular Expression, with the Pattern: (.*) and Ignore Case ticked. 
• Logical Grouping: Match All, then Add...
• In the Add Condition box, Condition Input: {HTTPS}, Check if input string: Matches the Pattern, Pattern: ^OFF$, Ignore case: ticked.
• In the Action setting, Edit Inbound Rule, set Action type: Redirect, Rewrite URL: https://{HTTP_HOST}/{REQUEST_URI} ,  Append query string: uncheck, Redirect type:  Permanent (301).
• Finally, click Apply in the Actions section on Right-hand side.

This would create a web.config in the site's home directory, with contents like:

<?xml version="1.0" encoding="UTF-8"?>

<configuration>

    <system.webServer>

        <rewrite>

            <rules>

                <rule name="HTTPS Redirect" stopProcessing="true">

                    <match url="(.*)" />

                    <conditions>

                        <add input="{HTTPS}" pattern="^OFF$" />

                    </conditions>

                    <action type="Redirect" url="https://{HTTP_HOST}{REQUEST_URI}" appendQueryString="false" />

                </rule>

            </rules>

        </rewrite>

    </system.webServer>

</configuration>

scripts to alert admin when hard disk is full, and to delete older files

Proposed solution to alert admin when a server hard disk becomes close to full, and for auto deleting old files - 

1. Script to email us when hard disk becomes full -
https://www.google.com/search?q=script+to+email+me+when+hard+disk+crosses+free
gives this script,
https://www.linuxjournal.com/content/tech-tip-send-email-alert-when-your-disk-space-gets-low

by adjusting the THRESHOLD=90 line, we can adjust when the script will email us. 10 GB out of 500 GB hard disk means we need to put THRESHOLD=98

#!/bin/bash
CURRENT=$(df / | grep / | awk '{ print $5}' | sed 's/%//g')
THRESHOLD=98
if [ "$CURRENT" -gt "$THRESHOLD" ] ; then
    mail -s 'Disk Space Alert' mailid@domainname.com << EOF
Your root partition remaining free space is critically low. Used: $CURRENT%
EOF
fi

And a cron to run this daily like 
@daily ~/ourScriptName.sh

2. Script to automatically delete files older than one month - 

https://www.google.com/search?q=script+to+automatically+delete+files+older+than+one+month
gives us this,
https://tecadmin.net/delete-files-older-x-days/
which lists the steps manually. 

If we just write this as a script and put a cron job like the earlier script, this should work. Something like
find /our/path -name "*.zip" -type f -mtime +30 -delete

error on booting Linux Mint 20

https://www.google.com/search?q=Buffer+I%2FO+error+on+dev+sda7%2C+logical+block+32495328%2C+async+page+read

https://www.linuxquestions.org/questions/linux-hardware-18/buffer-i-o-error-on-dev-sdb1-async-page-read-4175600715/

Hardware error might be fixed by zeroing out the partition and then reformatting, the last posts above indicate.

upgrade Linux Mint 18.3 to 20

Tried the upgrade as per
https://community.linuxmint.com/tutorial/view/2416

Regenerating fonts cache ... failed.

(Probably this was due to Google Chrome or Google Fonts which were installed?)

So, just installing 20.1 as a fresh install on lenovoPC.

icecast reload

Tried an implementation of icecast compiled with ssl support - but needs reload whenever the certificate is changed.

(Edit - see the later implementation using icecast-kh which does not have this reload requirement. )

Tried
kill -HUP  processid

after
ps -A | grep icecast

to find process id.

But that did not seem to work - neither as non-root user nor as root. probably because the process was started as nohup.

So, did

kill processid

Verified with
ps -A | grep icecast

that the process was killed,

then did

nohup /path/bin/icecast -c /path/icecast_ssl.xml > sh.out 2> sh.err < /dev/null &

This seems to work fine even though the order of the certificate is reversed - cer first and priv second as compared to the other way for the self-signed cert.
Edit - this followed the cat strategy at
https://forum.armbian.com/topic/11436-solved-icecast2-and-ssl/

And again, seems to work fine even though it has RSA PRIVATE KEY instead of PRIVATE KEY as in the self-signed certificate.

Edit - see the later implementation using icecast-kh which does not have this reload requirement. 

too many authentication failures with ssh

The reason for the error "too many authentication failures" was the presence of too many ssh keys in my .ssh directory and the solution was to add the option IdentitiesOnly like

ssh -o "IdentitiesOnly=true" -i key.pem user@server.tld

or for password-based login, can also use

ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no user@server.tld