adding two wordpress sites to an existing server

We had a server which had Apache and Mysql already installed (as well as a couple of Wordpress sites already). Steps followed, in brief, to add two more Wordpress sites:

1. mysql> create database wordpresswebsitestaging;
   
   CREATE USER "stagingwpuser"@"localhost" IDENTIFIED BY "ThePasswordH3r3";
   GRANT ALL PRIVILEGES ON wordpresswebsitestaging.* TO "stagingwpuser"@"localhost";
   FLUSH PRIVILEGES; 
   
   https://developer.wordpress.org/advanced-administration/before-install/creating-database/
   
   
2. Creating the web root directories and a user who can sftp in to upload files etc if needed
   sudo adduser thewpwebsiteadmin
   sudo su thewpwebsiteadmin
   cd /home/thewpwebsiteadmin
   mkdir directory1
   mkdir directory2
   
   We need to give read permissions to the entire path for the webserver to be able to read these directories, so
   sudo chmod -R +rx /home/thewpwebsiteadmin
   
   
3. Created DNS entries in cloudflare. In this case, CNAME records.
   
   
4. Copied existing files in /etc/apache2/sites-available as templates for the new sites. Since we were using cloudflare origin certificates, there was no need to create new certificates with certbot etc. After editing the conf files as needed, with the /home/thewpwebsiteadmin/directory1 and directory2 as home directories, enabled the sites with
   sudo a2ensite newsite1 newsite1ssl newsite2 newsite2ssl
   sudo systemctl reload apache2
   
   
5. Path to latest wordpress from https://developer.wordpress.org/advanced-administration/before-install/howto-install/ - 
   wget https://wordpress.org/latest.tar.gz
   tar -xzvf latest.tar.gz
   cp -R wordpress/* directory1 etc
   
   
6.  Visit the new sites with a browser to supply the database configuration etc, the wp-config file gets generated, create a new file and paste the generated file to the wordpress root directories as
   nano wp-config.php
   
   
7.  Create ssh key and add to authorized_keys - 
   ssh-keygen -t rsa -m PEM -f theadminuserkey.pem
   mkdir .ssh
   mv theadminuserkey.pem.pub .ssh/authorized_keys
   and set appropriate permissions for the .ssh directory and file - 
   chmod 700 .ssh
   cd .ssh
   chmod 600 authorized_keys

cloudflared or ssh tunnel for reverse-proxying on-premise servers

Exploring what we can do with cloudflare - 

1. If we want to run a server on another port, cloudflare does support ports other than 80 and 443 - via origin rules
https://developers.cloudflare.com/rules/origin-rules/features/#destination-port
via
https://old.reddit.com/r/CloudFlare/comments/12km27p/using_non_http_https_ports/
dynamic dns with
https://developers.cloudflare.com/dns/manage-dns-records/how-to/managing-dynamic-ip-addresses/#ddclient
https://dnsomatic.com/docs/software

Also, a list of supported ports?

https://developers.cloudflare.com/fundamentals/reference/network-ports/

2. But BSNL, Airtel etc would block ports. So, it would be far easier to use a tunnel with an outgoing ssh connection to a VM or something like that. That way, we don't have to depend on port-forwarding and availability of unblocked ports. We can use -R in the ssh tunnel for reverse-proxying.

3. But then, we may not even need a VM - currently Cloudflare Zero Trust Web access supports private apps, which would ask clients for authentication, and then allow them to connect to our private server via cloudflared tunnels.
https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/get-started/create-remote-tunnel/

and give it a public hostname with

https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/configure-tunnels/remote-management/

Starting from the page below, we can go "next, next" to finish all the steps to achieve ZT web access.

https://developers.cloudflare.com/learning-paths/zero-trust-web-access/connect-private-applications/

app api call slowdown, QUIC is suspected

It was reported that one of our Android apps was experiencing delays and timeouts in api calls, which hit a dot net backend running on Linux. Apparently the errors they saw were
 net::ERR_QUIC_PROTOCOL_ERROR

I suggested that they disable QUIC in their browser, https://thegeekpage.com/err_quic_protocol_error/

But apparently the app was having the errors and not in a browser. Since I'm not a dot net or Android developer, I'm not sure how one can disable QUIC in a generalized manner. But I could configure the server's firewall to explicitly Deny QUIC requests (UDP requests on port 443) which would then immediately make the client fall back to HTTP/2 (TCP packets).

There was an update pending on the server. Completing that and rebooting, still the issue continued. "For the past week, we've been experiencing buffering and delays in the app, especially on 3G networks". Maybe there was some change in either the network or Android or something like that? Cause could be any of those. Anyway: set up a firewall rule to Deny UDP to ports 80 and 443, and also disabled QUIC for the entire domain in cloudflare, https://developers.cloudflare.com/speed/optimization/protocol/http3/

That seems to have solved the issue. 

Miscellaneous Blender VSE settings used for the recent project

Some settings and shortcuts I revised when working on a recent project - 

I thought that I should prevent the creation of "proxies" in Blender VSE (Video Sequence Editor) since the working SSD was only 256 GB, with only around 100 GB free. 

https://vse-docs.readthedocs.io/video_editing/setup/proxies.html

First, I set the 

Proxy setup to Manual

Proxy render size to 25%

Then I thought of disabling proxies altogether, so set the 'Use Proxies' checkbox to False (pull out menu on right of rendered view)

Image showing all these, from the page above,

How to rotate an HDRI background - https://www.youtube.com/watch?v=fqd4mc42sZQ

Shading tab - Choose World instead of Object as the subject of the shading editor, rotate using Vector Mapping node ahead of the single image HDRI as in the screenshot below,

Cutting video clips into pieces which do not show the previously existing clip sections when stretched - is by "Hold Split" - https://vse-docs.readthedocs.io/video_editing/edit/montage/transform.html#split
Shift + K to 'hold-split' at the cursor.

Shift + A is the shortcut key to add speed control strip (and many other things) - https://vse-docs.readthedocs.io/video_editing/edit/effects/speed.html - Add > Effect > Speed Control

Similarly, for a cross-fade transition, select begin clip, Shift click to select end clip, then Shift+A - Add > Transition > Cross - https://vse-docs.readthedocs.io/video_editing/edit/effects/transition.html - we can change the order of the transition later, if needed, from the side bar, by changing the Input 1 and Input 2 strips as in the screenshot below.

listing ffmpeg commandline options

trac.ffmpeg.org/wiki/HWAccelIntro

You can see available presets including lossless

with

ffmpeg -h encoder=hevc_nvenc

List of the hwaccels available in the ffmpeg executable, 

ffmpeg -hwaccels

So, added
-preset lossless
to my commandline, and it worked. I think it was something like

ffmpeg -r 30 -start_number -003875 -f image2 -preset lossless -i "OpenSpace%06d.png" -c:v hevc_nvenc ../filename.mp4

Got the message
The selected preset is deprecated. Use p1 to p7
(but it is doing the encoding.)
There is also -tune lossless option. But I didn't try that.

simple static multi-lingual titles instead of tracked titles

Instead of going the tracked title route, I settled on a simpler static title overlay using Blender VSE, because

• It's far less time-consuming
  
• Hindi text does not render well as a Blender text object
  

So,the workflow was:

• Create the title text in a document editor like Google Docs
• Take a screenshot of the black text on white background
• Invert to white text in black background on Gimp
• Overlay over "warped" video directly with opacity fading in from 0 to 0.5 over a second - OR -
  
• If overlaying over a fulldome clip, first "warp" the flat text to match the fulldome clip's curvature using the method in this old post.

In particular, after trial and error, the particulars seem to be:

• 36 point font size text to take screenshot
• stretch the text to make it square (halve the width)
• put the stretched screenshot at the bottom of a 4096x4096 transparent image
• make the screenshot even smaller if required, or use a larger transparent image
• merge down the layer if the filter doesn't act on the entire image
  
• use Gimp's Filter > Distorts > Polar co-ordinates with "Map backwards" checked (otherwise the text would be inverted left to right) and save as png preserving transparency (alpha channel).
  

 

Payzapp mini review

I tried using HDFC Bank's Payzapp app for payments - it has features like loading up the Payzapp wallet with credit card, then transfer by UPI to bank accounts, etc. But, 

• they doubled the fee for Payzapp wallet loading from credit card
• many merchants, including the most important one which I needed to pay > 10k per month, blocked access to Payzapp payments due to fees they're charging merchants
• a billdesk payment of APSPDCL went through, but did not reflect in the "paid" bills. Billdesk said merchant had rejected payment, I need to pay again. Payzapp says refund has been issued, but there is no way to check except by seeing the balance - which I have now emptied, because the back-and-forth between Payzapp and Billdesk had taken more than a month.
   

Overall, a very poor experience. So I will use Payzapp only for the convenience of paying HDFC Bank credit cards with it, will block all notifications from it, and not keep any wallet balance. Even when making HDFC bank credit card payment, the app failed to load this morning, and loaded again later in the day. Also, the Payzapp app shows "apply coupons" when making a Rs. 4000 payment, but the coupon says min. transaction Rs. 5000, and the "apply coupon" button is not shown when I was actually making the Rs. 10,000 transaction! Bad HDFC! We can contrast this with Amazon.in's method of showing exactly how much "cashback" or "savings" will accrue for each payment method.

monitoring expiring certificates with Red Sift

 LetsEncrypt sent out an email that they would no longer be sending certificate expiry warning emails, and recommended we sign up to Red Sift's free plan. 

We can recommend Red Sift Certificates Lite, which provides free expiration emails for up to 250 active certificates: https://redsift.com/pulse-platform/certificates-lite

After signing up, there was a certificate warning, but it turned out to be a false alarm - a certificate which had been re-issued onto a new machine. 

manual setup of awstats with static report pages

To improve responsiveness, tried to make awstats generate static pages instead of dynamic report generation with the cgi-bin perl script.

Most probably the unresponsive nature was due to the DNS settings in the /etc/awstats/*.conf files for the domains we were reporting on. As the conf file says,

# If you want/need to set DNSLookup to 1, don't forget that this will
# dramatically reduce AWStats's update process speed. Do not use on large web
# sites.

Then, there was also the dynamic DNS lookup option, which also needed to be set to 0 - 

# For very large sites, setting DNSLookup to 0 (or 2) might be the only
# reasonable choice. DynamicDNSLookup allows to resolve host names for
# items shown in html tables only, when data is output on reports instead
# of resolving once during log analysis step.
# Possible values:
# 0 - No dynamic DNS lookup

Then two more settings were changed, changed to 0

# Possible values:
#  0  - Report is not shown at all
#  1  - Report is shown in main page with an entry in menu and default columns
# XYZ - Report shows column informations defined by code X,Y,Z...
#       X,Y,Z... are code letters among the following:
#        U = Unique visitors
#        V = Visits
#        P = Number of pages
#        H = Number of hits (or mails)
#        B = Bandwidth (or total mail size for mail logs)
#        L = Last access date
#        E = Entry pages
#        X = Exit pages
#        C = Web compression (mod_gzip,mod_deflate)
#        M = Average mail size (mail logs)

# Show domains/country chart
# Context: Web, Streaming, Mail, Ftp
# Default: PHB, Possible column codes: UVPHB
ShowDomainsStats=0

# Show hosts chart
# Context: Web, Streaming, Mail, Ftp
# Default: PHBL, Possible column codes: PHBL
ShowHostsStats=0

Then, with much trial and error, the following lines were added to the root crontab for creating the static pages, and an index file was created to easily go to the relevant server/month page.

sudo su -
crontab -e

58 23 * * * /usr/share/awstats/tools/awstats_buildstaticpages.pl -update -dir=/var/www/pathtostatsfromawstats/ -config=server1.org -builddate=$(date '+\%Y\%m') > /dev/null  2>&1

 56 23 * * * /usr/share/awstats/tools/awstats_buildstaticpages.pl -update -dir=/var/www/pathtostatsfromawstats/ -config=server2.org -builddate=$(date '+\%Y\%m') > /dev/null  2>&1

and so on. 

There were lots of emails to root being generated every 10 minutes from cron, saying awstats.pl could not read the log files in /var/log/apache2/ourlogfile.log - it turned out that these were being generated by another cron which would probably have been created by the awstats installation via apt, which was located in /etc/cron.d directory, the file was /etc/cron.d/awstats which was being run as an unprivileged user, hence unable to read log files etc. So I just commented out those awstat update lines in that file. To note:

# /etc/crontab: system-wide crontab

# Unlike any other crontab you don't have to run the `crontab'

# command to install the new version when you edit this file

# and files in /etc/cron.d. These files also have username fields,

# that none of the other crontabs do.

Initial setup -

followed the post

https://hnsws.blogspot.com/2012/01/configuring-voxel-server-for-sgh-and_24.html

to get Ubuntu install documentation,

https://help.ubuntu.com/community/AWStats

and followed that for the setup, with separate access.log files for each domain set in their respective conf files.

 

To set up static pages.

https://winterdrake.com/awstats-tip-creating-static-pages/ but more details at https://www.netadmintools.com/configuring-awstats-with-static-pages/

 

Found the path to awstats executables (perl scripts) - /usr/share/awstats/tools -

via the listing at

awstats_7.8-2_all.deb Ubuntu 22.04 LTS Download

https://ubuntu.pkgs.org/22.04/ubuntu-main-amd64/awstats_7.8-2_all.deb.html

And if we need to password protect it,

https://www.digitalocean.com/community/tutorials/how-to-set-up-password-authentication-with-apache-on-ubuntu-20-04

Running the perl script without any arguments gave the listing of possible arguments -  

perl /usr/share/awstats/tools/awstats_buildstaticpages.pl

Usage:
awstats_buildstaticpages.pl (awstats_options) [awstatsbuildstaticpages_options]

  where awstats_options are any option known by AWStats
   -config=configvalue is value for -config parameter (REQUIRED)
   -update             option used to update statistics before to generate pages
   -lang=LL            to output a HTML report in language LL (en,de,es,fr,...)
   -month=MM           to output a HTML report for an old month=MM
   -year=YYYY          to output a HTML report for an old year=YYYY

  and awstatsbuildstaticpages_options can be
   -awstatsprog=pathtoawstatspl AWStats software (awstats.pl) path
   -dir=outputdir               Output directory for generated pages
   -diricons=icondir            Relative path to use as icon dir in <img> links
   -builddate=%YY%MM%DD         Used to add build date in built pages filenames
   -staticlinksext=xxx          Build pages with .xxx extension (default .html)
   -buildpdf[=pathtohtmldoc]    Build a PDF file after building HTML pages.
                                 Output directory must contains icon directory
                                 when this option is used (need 'htmldoc')

At first, I thought I would need to manually create directories and chown them to www-data

# mkdir 202502
# chown www-data:www-data 202502
 

but then I could use the builddate parameter instead. Note that in the builddate parameter, when the $(date) is called in a cron script, the % characters have to be escaped.

Reports for older year/month combinations could be created like
perl /usr/share/awstats/tools/awstats_buildstaticpages.pl -dir=/var/www/pathtostatsfromawstats/ -config=ourserver.org -year=2024 -month=12 -builddate=202412

And finally, asked Github Copilot to generate a form with dummy servernames and paths, which I manually edited to get the correct paths, as below -

<script>
        function redirectToUrl() {
            var server = document.getElementById("server").value;
            var year = document.getElementById("year").value;
            var month = document.getElementById("month").value;
            var url = "https://ourstatsurlforstatsfromawstats/awstats." + server + year + month + ".html";
            window.location.href = url;
        }
    </script>
</HEAD>
<body>
    <form onsubmit="event.preventDefault(); redirectToUrl();">
        <label for="server">Select Server:</label>
        <select id="server" name="server">
            <option value="1.org.">CMS</option>
            <option value="2.org.">devel2</option>
            
        </select>
        <br /><br />
        <label for="year">Select Year:</label>
        <select id="year" name="year">
            <option value="2024">2024</option>
            <option value="2025">2025</option>
            <option value="2026">2026</option>
        </select>
        <br /><br />
        <label for="month">Select Month:</label>
        <select id="month" name="month">
            <option value="01">01</option>
            <option value="02">02</option>
            <option value="03">03</option>
            <option value="04">04</option>
            <option value="05">05</option>
            <option value="06">06</option>
            <option value="07">07</option>
            <option value="08">08</option>
            <option value="09">09</option>
            <option value="10">10</option>
            <option value="11">11</option>
            <option value="12">12</option>
        </select>
        <br /><br />
        <input type="submit" value="Submit" />
    </form>

               
</body>

resizing a window in xfce

 Some windows like the file manager Thunar allow resizing by dragging any corner, but not the Microsoft Edge browser window. Solution - press ALT, right-click inside the window and drag in any direction.

https://unix.stackexchange.com/questions/61037/how-to-resize-application-windows-in-an-arbitrary-direction-not-vertical-and-no

show date-time instead of date in Thunar

On the Linux Mint XFCE desktop, Thunar is the default file manager, and by default, it just shows "Today" or the created date of the file. Not useful to find the latest in several versions created in a few minutes time, or to see how long it took to generate file sequences etc. But happily, it can easily be fixed, instead of having to open a terminal and do ls -l

https://forums.linuxmint.com/viewtopic.php?t=241840
Menu > Edit > Preferences - very first tab (Display), Date Format - drop down list right along the bottom is a format like yyyy-mm-dd hh:mm:ss

boot logs and more with journalctl

On machines with systemd, we see systemd logs with journalctl - How to Read and Edit Systemd Logs using Journalctl in linux

https://www.geeksforgeeks.org/how-to-read-and-edit-systemd-logs-using-journalctl-in-linux/

journalctl -b -1 for previous boot logs.

These still work - 

tail /var/log/syslog
tail /var/log/dmesg 

Other logs still seen in /var/log are auth.log and boot.log

-----------------

On newer systems, there is no /var/log/syslog - 

https://linuxblog.io/how-to-diagnose-oom-errors-on-linux-systems/

- /var/log/syslog does not exist.

https://unix.stackexchange.com/questions/607221/tail-f-log-but-with-systemd

https://linuxhandbook.com/journalctl-command/

journalctl -r for reverse chronological order.

spacebar to move forward, b to move backward.

adjusting pitch and narration speed in Reaper

Quick and dirty pitch shifting + increasing speed of narration in Reaper - using the "Playback Rate" item setting. Increasing playback rate increases the speed of the narration, of course. Keeping the "Preserve pitch" checkbox ticked allows the voice to retain the current pitch. And a negative value for "Pitch adjust (semitones)" makes the voice deeper. 

downloading youtube videos in 4k

Youtube has recently changed some of its video delivery code, so that online tools like ssyoutube.com etc don't provide videos higher than 360p. Even tools promising free 4k downloads like 4kdownload.com could only provide a 1080p version. Checking if open-source tools can do better - searched for yt dl on github, found https://github.com/ytdl-patched/ytdl-patched which had been updated recently. 

Usage - just needed to give

yt-dlp https://www.youtube.com/suitable-url

 

and it downloaded a 4k version.

Fulldome video - simulated view of moving through a gas cloud or Nebula

Trying to create a different version of this video by ESO - https://www.youtube.com/watch?v=lj3t_gjuXWk - led me to various searches, where apparently IFSRenderer was supposed to be easier to create Nebula videos than Blender. There's the disadvantage that it runs only on Windows, and also doesn't seem to be accelerated much using an NVidia graphics card? So, these renders are only 1024x1024 - also, since these are supposed to be clouds, we can probably scale them to 4k - and probably just get a softer look for the nebula. These renders went at around 4 to 5 frames per minute on the computer with the NVidia 1060 card. 

Fulldome video suitable for playback in Planetariums - 

And one more (which was actually done earlier, on a laptop with integrated Intel graphics) - 

The saved settings for these are uploaded to github,
https://github.com/hn-88/IFSRenderer-saves

Zoom out to the edge of the observable universe in a single shot without cuts

Similar to the previous post, but in a "single take" zoom motion all the way from Earth to the edge of the observable Universe. A fulldome video made with OpenSpace - details as in my previous post.

The OpenSpace "recording" is shared at https://github.com/hn-88/openspace-scripts/blob/main/user/recordings/ToEdgeofU-singletake.osrectxt

The 4K (4096x4096) full resolution videos are available from archive.org - https://archive.org/details/to-edge-of-u-single-zoom-4k-hevc-nvenc-30fps

 

 A "reference" video with text overlay is also created, so that we can note the distances from Earth as the "camera" moves outwards - 

Earth to the Moon - fulldome video

Zooming out from a view of the Earth to a close view of the Moon, with a shot similar to the iconic "Earthrise" photo, and moving beyond the Moon - fulldome video suitable for planetariums created with OpenSpace. 

 

The "recording" file used to create this image is shared at https://github.com/hn-88/openspace-scripts/tree/main/user/recordings/Moon-Earthrise-beyond

As noted in the readme there, this was created with v2.1 of OSREC-interp and not with v3.0, v3.0 had bugs - jerky motion.

The full resolution 4096x4096 video is shared on archive.org - https://archive.org/details/earthto-moon-earthrise-beyond-hevc-nvenc

More of our fulldome videos on archive.org are at  https://archive.org/search?query=creator%3A%22www.saispace.in%22

delete files found with "find"

Trying to create an AppImage for OpenSpace led to many learnings. Among them - how to delete the files found with the "find" command - 

https://stackoverflow.com/questions/43412885/fastest-way-to-delete-files-from-a-directory-tree-whose-names-contain-a-certain

find . -name '*.cpp' -print0 | xargs -0 -P2 rm

Explanation - 

"If you're on Linux or have the GNU find and xargs commands, then use -print0 with find and -0 with xargs to handle file names containing spaces and other odd-ball characters." - https://stackoverflow.com/questions/864316/how-to-pipe-list-of-files-returned-by-find-command-to-cat-to-view-all-the-files

"xargs reads items from the standard input, delimited by blanks (which can be protected with double or single quotes or a backslash) or new‐lines, and executes the command (default is echo) one or more times with any initial-arguments followed by items read from standard input.  Blank lines on the standard input are ignored." ... "Because Unix filenames can contain blanks and newlines, this de‐ fault behaviour is often problematic; filenames containing blanks and/or newlines are incorrectly processed by xargs. In these situations it is better to use the -0 option, which prevents such problems. When using this option you will need to ensure that the program which produces the input for xargs also uses a null character as a separator. If that program is GNU find for example, the -print0 option does this for you." - https://www.man7.org/linux/man-pages/man1/xargs.1.html

and the -P option is "-P max-procs, --max-procs=max-procs
              Run up to max-procs processes at a time"

patch files using patch

For creating the OpenSpace AppImage, we need to apply some changes to the cfg file, and the most straightforward way to do it in the build script might be using patch.

https://www.baeldung.com/linux/shell-edit-patch-files

https://www.howtogeek.com/415442/how-to-apply-a-patch-to-a-file-and-create-patches-in-linux/

So we create a file with the needed changes and use diff to create the patch.

diff -u old/slang.c new/slang.c > slang.patch

logging load average and memory usage on Linux servers

Just redirecting the output of top to a text file, like top >> logtop, we get lot of unprintable characters due to its refresh nature.

uptime gives load average.

https://linuxconfig.org/how-to-obtain-linux-s-system-load-average-report-since-last-reboot

top can do various things, like batch mode, -b

https://unix.stackexchange.com/questions/352594/add-timestamps-to-top-command-output

Probably more efficient to just log timestamps, free and uptime.

So, put in a cron job to run every 15 minutes,

*/15 * * * * /home/path/to/logtopscript.sh

$ more logtopscript.sh

#!/usr/bin/bash

echo $(date '+%Y-%m-%d:%H:%M:%S') uptime: $(uptime) free: $(free)  >> /home/path/to/toplog.txt

This can be imported into spreadsheets like LibreOffice Calc using <SPACE> as the delimiter, but has problems with uptime's output not being exactly the same length - sometimes

up 1 hour

sometimes

up 6 days,  8:33

etc. It has issues for 3-4 data points every day, which need to be manually finagled. Probably a regex to select only the load average can solve that.