backing up mysql databases to google shared drive

Following the unixcraft script, also seen on github, implemented a weekly backup for some of our mysql databases. 

# 05 0 * * Sun /path/to/script.sh

as root.  

Edit: The reason for the issue below was different. Explained in a separate post here. 

When I added an rclone move line after the backup line to the script, it was moving incomplete 20 kB files to the shared google drive. So, am running the rclone move as a separate script after half an hour,

 # 35 0 * * Sun /path/to/script.sh

which has lines similar to

MBD="$DEST/mysql"
rclone --config "/home/ouruser/.config/rclone/rclone.conf" move $MBD/ gshareddrive:/

The database dump takes only a couple of minutes, so the half an hour gap is more than sufficient. 

 Some things to remember about using this script:

• If we're defining only a few databases to be backed up, like
  # Store list of databases
  DBS="db1 db2"
  
  then we need to disable/comment out the DBS= line which appears later,
  #DBS="$($MYSQL -u $MyUSER -h $MyHOST -p$MyPASS -Bse 'show databases')"
  
  
• As seen above, the list of databases (or those to be ignored, in the IGGY variable) are to be entered separated by spaces.
  
  
• The script changes the directory permissions for security, so it is meant to be run as root. Otherwise, we will find that it could not write the files (if run as an unprivileged user).
  
  
• Edit: Have to provide hard-coded full path to MYSQLDUMP - see this post.
  

Bitnami mysql VM disk usage - disable binlog

Found that the disk utilization for one of our database servers was 83%. Checking with df -h and du -sh, found that the directory with most data usage was 

/bitnami/mysql/data$ sudo du -sh *
192K    #ib_16384_0.dblwr
8.2M    #ib_16384_1.dblwr
676K    #innodb_temp
4.0K    auto.cnf
1.1G    binlog.000867
1.1G    binlog.000868
1.1G    binlog.000869
1.1G    binlog.000870
1.1G    binlog.000871
1.1G    binlog.000872
1.1G    binlog.000873
1000M   binlog.000874
1.1G    binlog.000875

So, these binary logs were taking up all the space, when the actual database itself was only less than a GB. Followed this to disable binary logging - How to Disable Binary Logging in MySQL on Bitnami servers? (supportsages.com) 

sudo /opt/bitnami/ctlscript.sh stop mysql
sudo nano /opt/bitnami/mysql/conf/my.conf

and added the disable line after
[mysqld]
disable_log_bin

sudo /opt/bitnami/ctlscript.sh start mysql

Waited for some days, but the bin logs were not getting auto deleted. So, manually deleted them and did a restart of mysql with

 sudo /opt/bitnami/ctlscript.sh restart

Now the disk is only 30% used.

 

 

file not found error on Google Apps Script Drive API for files on shared drives - workaround

Failing to get a file in Google Apps Script using the Drive API - Stack Overflow

So, we must add a parameter to the Drive API call, instead of just the file id, 

Drive.Files.get('111111FileIDxxxxxxxxxxx', {supportsAllDrives: true});

 

 

Moodle course slow to load

An issue with one of our Moodle instances - one of the courses was loading very slowly in Edit Mode, and was timing out. Another issue - the question bank was not loading at all.

1. Question bank issue - The database table prefix_question_statistics showed a(n abnormally?) high number of entries -  that table alone is 49 MB,  with more being added every day. Apparently that was the issue. Statistics was disabled for the question bank, and the question bank started loading again. 

Some suggestions to make the course(s) load faster:

a. Currently, if you click on one of the courses, a huge page, with many images, for all weeks, loads in the browser. We can make it load a bit faster using Cloudflare, but such a large page is bound to cause problems with older computers / machines with less RAM / on slower networks / etc. It would be better to show a collapsed view, with each week opening up only on clicking. (the collapse all link on the top right). Or on separate pages.

b. Use jpg, of quality = 70 to 75 instead of png. 

c. Turn on cloudflare proxying. 

d. Clicking on question bank also loads up 1000 questions at once. This would cause problems as the number of questions / size of each question increase. It would be desirable to load up fewer questions - maybe 100 at a time as with the link at the bottom, "Show 100 at a time" - I have clicked it now. 

2. Course not loading / slow to load - Further troubleshooting tips -
Moodle in English: Very large course takes a minute to load unless "editing" is on

" It's almost always because somebody is trying to build the learning material on the course page itself rather than in activities/resources "behind" the page. There's nothing wrong with that, of course, but it's does have an effect.

We had a course with 120 videos embedded in labels on the course page. It took 10 minutes plus to load - even on a good connection."

So, probably not a good idea to embed all your content on the course page itself, they should be "activities/resources behind the page".


Also, another person says:
"To track down the culprit I usually switch to one section per page. With one section per page, only the content for that pages loads, so some sections will load normally, but your dragger will still drag."

How to show one topic per page:
Course format types in Moodle · Technology Help · Lafayette College

 

(Apparently the "embedding videos in labels" was one of the issues. Another suggestion which S passed on was: 

another way to improve performance overall is to move out three folders namely (cache, localcache and temp) out of moodle data folder. Here is a tutorial for the same.  https://youtu.be/qXnSlwYzDfU )

trying out Google Earth Studio for planetarium fulldome content

When I was googling for ways to create fulldome or 360 VR content from Google Earth (desktop), came across this interesting product.

https://earth.google.com/studio/docs/making-animations/rendering/

Apparently, this is a separate product from Google Earth (desktop) and is invite only at present. I sent in a request and got approved within 12 hours. 

We can render 360 VR scenes fairly easily, with the caveat that their attribution must appear onscreen, and we should not sell the resulting content.

Using OCVWarp, I tried out turning the rendered 360 VR files into fulldome content, a couple of examples below.

Edit: For exporting regular flat content from the Desktop version of Google Earth, this youtube video has a step-by-step tutorial.

How to Record Tour in Google Earth/ How to Export UHD Video from Google Earth.

Deprecated SSL/TLS version warning from NIC-CERT - action taken on cloudflare

Copy-pasting from an email exchange after one of our institutions received an email saying that some/all of their domains had deprecated versions of TLS - 

There are pros and cons for disabling these.

1. You can see which clients will be affected by looking at the last answer at

https://security.stackexchange.com/questions/227394/should-i-disable-tls-1-0-and-tls-1-1-support-on-my-web-servers

2. If you log in to your cloudflare dashboard, click on your domain, then SSL/TLS --> Overview on the left hand pane, you can find a percentage of users still using TLS 1.0 etc.

3. The security risk is that the users who use TLS 1.0 etc have a finite risk of being tricked into going to some other site masquerading as our site, since the certificate is vulnerable, and then stealing the data they send to our site. Example attacks are at
https://www.acunetix.com/blog/articles/tls-vulnerabilities-attacks-final-part/

4. If you want to disable TLS 1.0 and 1.1, you can do it from your cloudflare dashboard, click on the relevant website, go to SSL/TLS --> Edge Certificates and choose minimum TLS version to be 1.2

 

https://developers.cloudflare.com/ssl/edge-certificates/additional-options/minimum-tls/ 

 

They have now moved to a higher min TLS version.

tunneling through ssh

We wanted to protect one of our database servers by closing its open ports and only allowing tunneling via ssh or a vpn. Since this post indicates that ssh is less cpu intensive than openvpn, wanted to keep an ssh tunnel open across reboots. Autossh seems to be a ready-made solution for this - 

autossh(1): monitor/restart ssh sessions

via

linux - How to reliably keep an SSH tunnel open?

So, in our case, via

Create an ssh tunnel background service with autossh and systemd (systemctl)

and

16.04 - Start autossh on system startup - Ask Ubuntu 

 

sudo apt install autossh

sudo nano /etc/systemd/system/ourtunnel.service

[Unit]
Description=our tunnel service
After=network.target network-online.target sshd.service

[Service]
ExecStart=/usr/bin/autossh -i /home/ouruser/.ssh/id_rsa_ourkey -L 9999:localhost:9999 -NT ourserveruser@our.remoteserver.tld

[Install]
WantedBy=multi-user.target

The -NT switch is important - otherwise, the sshd on the remote server complains, "Pseudo-terminal will not be allocated because stdin is not a terminal." And the service status will show that it failed - 

service ourtunnel status

 autossh[494191]: ssh exited prematurely with status 0; autossh exiting

ssh(1): OpenSSH SSH client - Linux man page (die.net)

 

linux - Pseudo-terminal will not be allocated because stdin is not a terminal - Stack Overflow

After a few days of use, top shows the CPU usage - around 6.6% for autossh on the tunnel client which runs a web server, and the tunnel server which runs a db shows 8 to 12% CPU usage for sshd.

 

 

issues with a Moodle course

One of our institutions had some trouble with one of their courses not loading - or being very slow to load. After first suspecting server issues, some excerpts from an email exchange:

Some suggestions to make the course(s) load faster:

0. Have finished the upgrade now, and the site is working. The database table prefix_question_statistics shows a(n abnormally?) high number of entries -  that table alone is 49 MB,  with more being added every day - around 11:15 this morning a large number of records were added - this might have been one of you adding another set of questions?

 

1. Currently, if you click on one of the courses, a huge page, with many images, for all weeks, loads in the browser. We can make it load a bit faster using Cloudflare, but such a large page is bound to cause problems with older computers / machines with less RAM / on slower networks / etc. It would be better to show a collapsed view, with each week opening up only on clicking. (the collapse all link on the top right). Or on separate pages.

2. As I mentioned, please use jpg quality = 70 to 75 instead of png. 

3. I'll make the cloudflare changes to turn proxying on. 

4. Clicking on question bank also loads up 1000 questions at once. This would cause problems as the number of questions / size of each question increase. It would be desirable to load up fewer questions - maybe 100 at a time as with the link at the bottom, "Show 100 at a time" - I have clicked it now. 

I do not think anyone was adding questions. I have only been opening that page to troubleshoot. The problem still existed even after upgrade. However, I found a fix by disabling question statistics which seems to be triggering every time we access the question bank.
 

For now this should work till moodle finds a permanent solution for the statistics trigger. 

 

Checking the server, I found node js (which runs our notifier service) was running at 60% CPU. I have stopped it now and restarted the webserver with

cd notifierdirectory
sudo forever stop index.js

sudo service apache2 restart

So, now the site itself is loading.

https://our.site/course/view.php?id=4

loads fine.

But when I turn on edit mode, the page becomes very large, and hence loads slowly.

I guess the same issue is affecting https://our.site/course/view.php?id=6 in edit mode.

 

--Later--

 

Further troubleshooting tips -
Moodle in English: Very large course takes a minute to load unless "editing" is on

"It's almost always because somebody is trying to build the learning material on the course page itself rather than in activities/resources "behind" the page. There's nothing wrong with that, of course, but it's does have an effect.

We had a course with 120 videos embedded in labels on the course page. It took 10 minutes plus to load - even on a good connection."

So, probably not a good idea to embed all your content on the course page itself, they should be "activities/resources behind the page".

Now, whether you are already doing this, or how to do this if you are not doing this - maybe S can help.

Also, another person says:
"To track down the culprit I usually switch to one section per page. With one section per page, only the content for that pages loads, so some sections will load normally, but your dragger will still drag."

How to show one topic per page:
Course format types in Moodle · Technology Help · Lafayette College

 

 

delete old files - Google Apps Script

https://stackoverflow.com/questions/66060908/google-script-delete-files-within-a-folder-based-on-the-last-modified-date

gives an unattributed code snippet, but seems to be OK. (I've not yet implemented this.) 

https://gist.github.com/mbierman/f29f8e7846d2824a81c4e74c5522a34b

uses getDateCreated() instead of searchFiles() & modifiedDate. This second script also has an elegant way of emailing using Logger.getLog()

If we want to permanently delete instead of sending to trash, which we might want to do in case of rapidly filling up drives and so on, there is the Files API delete function, https://developers.google.com/drive/api/v2/reference/files/delete#examples which might be called from Apps script by enabling the Drive API on the LHS and Drive.Files.delete(fileid).

Filelink with Thunderbird

Copy-pasting from an email exchange with PB

I tried the filelink feature in Thunderbird with the "Send Server" .... It worked once and then kept failing after that. Having Google drive work with Thunderbird to send large attachments as a link would have been the ideal solution.Have you used any other method to send large attachments in a mail client?

Looking at the FileLink Send provider,

FileLink provider for Send :: Add-ons for Thunderbird

it says that Firefox Send was discontinued by Mozilla, and is now just "Send" and maintained by someone else.

The url to the send server has also changed.

Maybe if you re-install the Send plugin with the latest send provider as given on the link above (send.vis.ee), it might work.

Another option, of course, would be for the user to directly go to 

https://send.vis.ee/

to send the files, or

https://wetransfer.com/

 

Edit - thinking again of this issue, it might have been some sort of Firewall issue which caused the feature to work once and fail after that. The internet connection goes through a Sophos Firewall device (or something similar) with IPS (Intrusion Prevention System). 

 

Windows 11 login background images

Windows 11 - by default - shows some very pretty background images on the login screen. Apparently, these images are called windows spotlight images.

To get them,

How to Find Windows Spotlight Images (lifewire.com)

My path was

 

C:\Users\myusername\AppData\Local\Packages\Microsoft.Windows.ContentDeliveryManager_cw5n1h2txyewy\LocalState\Assets

From there, we can copy the files to another directory and rename them with .jpg using the method mentioned in the post above,
ren * *.jpg

caveats while using select count() in mysql

Helped by How to get multiple counts with one SQL query? - Stack Overflow I can probably improve the run time performance of https://github.com/hn-88/ad-hoc-moodle-database-queries/blob/main/Files%20by%20Asset%20type

 Another possible way to improve performance might be to change the order of the WHERE conditions. 

Apparently all queries which use the COUNT() function will lock the table until processing finishes. In this case, for some of the larger numbers like 300,000+ rows, the query takes well over a minute to run, causing the entire site to stop responding. To prevent that from happening, I have not put this query in the Moodle ad-hoc queries list. 

 To prevent timeouts from other queries, we have to limit the numbers using the parameters, like choosing to query a limited number of days/months, or limited number of users, etc.

Why we're leaving the cloud - blog post from Basecamp

An interesting blog post to which I can refer people who are obsessed with "moving to the cloud" - Why we're leaving the cloud 

 The cost and "all eggs in one basket" reliability issues are highlighted. 

Firefox "rupee symbol" overlapping problem

I reported a possible bug about Firefox showing the rupee symbol overlapped with text, like the screenshot at

https://github.com/webcompat/web-bugs/issues/116442#issuecomment-1379785455

But even I was unable to reproduce it on another machine (which had a higher resolution screen to start with). Anyway, the workaround seems to be to change the view, Ctrl- or Ctrl+ to zoom the view in or out; that fixes the rendering as mentioned in the thread above.

checking for sudo rights

Easy way to check if a user can sudo:

groups <username>

On regular Linux machines, the sudo group will turn up if the user is allowed to sudo. On bitnami VMs, the groups bitnami and bitnami_admins are used.

bitnami user and separate admin user

When setting up key-based login for one of our Bitnami servers, found that

• The bitnami user, which is the super user, has SSH logins turned off
• We can login as our_custom_user, which is also a sudo user.
• When we login as our_custom_user, the username on the prompt is shown as bitnami, but has the home directory ~ as /home/our_custom_user - so we should keep that in mind.
  

finding files modified in the last 15 days with Linux command line

Finding files modified in the last 15 days,

find ~/relevantDirectory -mtime -15 -ls > ~/last15daysfiles.txt

submitting a form without a refresh - jquery technique

This page - https://www.geeksforgeeks.org/how-to-submit-a-form-or-a-part-of-a-form-without-a-page-refresh-using-jquery/ - has a nice javascript form submitter code snippet. Another interesting thing there is that the form is submitted to https://reqres.in/api/users - which is a free API testing site. 

<script src=

"https://cdnjs.cloudflare.com/ajax/libs/jquery/3.6.0/jquery.min.js">

    </script>

    <script type="text/javascript">

    $(document).ready(function() {

        $('form').on('submit', function(event) {

            event.preventDefault();

              

            // It returns a array of object 

            let userinfo = $(this).serializeArray();

            let user = {};

            userinfo.forEach((value) => {

                  

                // Dynamically create an object

                user[value.name] = value.value;

            });

            let url = "https://reqres.in/api/users";

            $.ajax({

                method: "POST",

                url: url,

                data: user

            }).done(function(msg) {

                  

                // When the request is successful

                $('span').text('user is successfully created with Id ' + msg.id);

            }).fail(function(err, textstatus, error) {

                $('span').text(textstatus);

            });

        });

    });

    </script>

If we're using this with Google Apps Script, we need to return 'Success' using something like 

return ContentService.createTextOutput().append('Success').setMimeType(ContentService.MimeType.JAVASCRIPT);

If there is no return statement, "error" is displayed in the form response SPAN.

prevent staggered look for HTML elements with CSS

When using CSS and 'float'ing elements, if we get a staggered look with each element in subsequent rows offset to the right, that would be due to not using "clear"

CSS Layout - clear and clearfix (w3schools.com)

html - why is my css staggered? - Stack Overflow

 

Also, the clearfix overflow: auto which is mentioned in the w3schools page above would be useful in many cases. 

workaround for same-origin-policy block for AJAX query to google apps script

https://www.labnol.org/code/20197-jquery-ajax-call-google-script

This post describes the exact problem and solution - the solution seems to be to use JSONP (JSON with padding) for the data, and then client page should read the JSON and appropriately display/use the data.

• use a try catch in the Google Apps Script to prevent errors from preventing responses
• convert to JSONP with
  

  result = JSON.stringify({
      result: result,
    });
  
    return ContentService.createTextOutput(e.parameter.callback + '(' + result + ')').setMimeType(
      ContentService.MimeType.JAVASCRIPT
    );

• The AJAX query can be easily called with jQuery like
  

  var request = jQuery.ajax({
        crossDomain: true,
        url: url + encodeURIComponent(parameterIfAny),
        method: "GET",
        dataType: "jsonp"
      });

• The script with jQuery must come after jQuery is included, like for example,
  
  <head>
  <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.1/jquery.min.js"></script>
  </head>

GMailApp sendEmail - "from" needs to be an alias

Just a quick note - GMailApp SendEmail options do have a "from" address which we can choose, but the "from" needs to be an alias of the account from which the email is being sent. "replyTo" does not have that limitation, so we can use that instead. 

https://developers.google.com/apps-script/reference/gmail/gmail-app#sendemailrecipient,-subject,-body,-options

https://developers.google.com/apps-script/reference/gmail/gmail-app

 

GmailApp.sendEmail(recipient, subject, "", {

            replyTo: semail,

            name: requestorname,

            htmlBody: emailbody

        })

 

https://developers.google.com/apps-script/reference/gmail/gmail-app#sendemailrecipient,-subject,-body,-options

 

https://developers.google.com/apps-script/reference/gmail/gmail-app#advanced-parameters_1

https://developers.google.com/apps-script/reference/gmail/gmail-app#advanced-parameters_1

find all ip addresses on a network

https://www.google.com/search?q=find+all+ip+addresses+on+network

showed arp -a

It worked OK on Windows the first time. But after a day or two, arp -a still showed the old ip address, and not the refreshed ip address of another machine which had been switched on and off in the interim. So, this is not bulletproof. 

https://www.comparitech.com/net-admin/scan-for-ip-addresses-local-network/

points to Angry IP Scanner. Will try it out some time. 

creating a network share with samba on linux command line

Following this page, created a share by 
sudo nano /etc/samba/smb.conf

and adding at the end of it,

[mysharename]

path = /home/path/to/folder

valid users = myusername

read only = no

testparm

was successful, but 
sudo service smbd restart

indicated that there was no such service.

sudo apt install samba

was required.

After that, it worked, but soft links inside the shared folder were not visible. Hard links are only possible within the same physical volume, not across different disks. 

Following

https://unix.stackexchange.com/questions/5120/how-do-you-make-samba-follow-symlink-outside-the-shared-path

added the following to the smb.conf file above, in the global section,

[global]

allow insecure wide links = yes

and in the [mysharename] section,

[mysharename]

follow symlinks = yes

wide links = yes

Another smbd restart, and now it was working fine. File transfer going around 15 to 20 Mbps on our wifi.  

Moodle server issues, possible reasons and fixes

Copy-pasting from emails:

1. In November,

... had mentioned a temporary connection failure with the server when running ad-hoc queries. This could most probably have been due to some query overloading the database momentarily.

But when checking the logs, I find repeated "Web service authentication failed" messages, spaced every few minutes, from a set of regularly spaced IP addresses.

This means that our server is under distributed brute force attack. Which means that (a most-probably automated) attacker is using many different computers or devices to guess passwords (in this case web service tokens) by trying all possible combinations.

This sort of thing is common for SSH ports, but this particular attack seems to be tailored to Moodle. One way to block this particular attack would be to disable web services, but that would block the mobile app also. Will look for other solutions, may be asking in Moodle forums.

Not something which we need to worry too much about, just something we need to do to make the server a little safer.

2. Disabling "login from new device" notification emails. Disabled sending emails saying "new login from -- device",
from https://our.server.org/admin/message.php
"New login notifications"

3. Send only Digest emails from Forums -
The forum email notification setting seems to be off already. Perhaps the emails are due to someone deliberately sending a message to all forum users,
Bulk user actions - MoodleDocs
or something like that?

4. Stop send email altogether to specific users, like the bouncing users above - I have set
emailstop=1
for the user bouncinguser@somedomain.org

But I'm seeing bounced messages to noreply@someotherdomainofours.org - I'm not sure where this non-existent email id has been entered. It must be somewhere in the settings, instead of noreply@correctdomain.org entered as noreply@someotherdomainofours.org - but I could not trace out where.

 

corrupted frames repair

One of our planetarium shows had some 2-5 seconds of bad frames. Worked around it by 

• cutting the pre- and post- sections using Avisynth Trim()
• Extending a few sections of the pre- section with virtualdub - making it 15 fps and interpolating frames
• joining together the extended pre- and post- sections using Avisynth Dissolve()
  

subscribing to google group via email

Apparently we can subscribe to a google groups mailing list by just sending an email - and then replying to the confirmation email. Checking, I tried sending an email to
sai-inspires+subscribe@sssmediacentre.org
from an id which is currently not a member.

Currently, this group is set to invitation only. So, the subscribe action did not go through.

I then tried the unsubscribe email - sai-inspires+unsubscribe@sssmediacentre.org

I got a bounce message saying "Leave failed Not a member". 

So probably it would work for those who are members.

autofit in Google sheets

Adjust the column width to fit text in Google Sheets - the method seems to be 

• Highlight the column(s) you want to auto fit
• Hover the cursor over the right-border of the header of that column (or one of the columns) till it changes to the "resize by dragging" cursor
• Double-click. After a small delay depending on the size of the sheet, the autofit will happen.
  

adding Google input tools to Google Docs

If we want to type in Indian languages, Google's input tools are a good option for transliteration of Indic scripts. For enabling them inside Google Docs toolbar, the procedure is to go to

Google Account --> Personal Info --> General Preferences for the Web --> Input tools 

and enable the languages you want. There is no need to enable English in my case, since my default keyboard settings are English. But we can enable English handwriting input etc if needed, if we're using a tablet etc.

revisiting super-resolution filters and cartoon effect filters

Either super-resolution or cartoon effect could (theoretically) be used to blow up low-res video content for planetarium fulldome projection.

http://www.infognition.com/super_resolution_vdf/

 - I had a previous post about this, and also "junk" according to this - https://forum.videohelp.com/threads/398876-Super-Resolution-Plugin-For-Vdub

Lanczos resize should be better, apparently.

"Cartoon look" thread
http://forum.doom9.org/archive/index.php/t-182043.html

The Sobel etc filter near the end of the thread seems to be best for us, looks like watercolor.

Or can try any of the MSU filters
https://videoprocessing.ai/video_filters/

Also interesting - cloud based video processing, but not free,
https://www.pixop.com/pricing

auto update of LetsEncrypt certificate failed on Windows

One of our development servers had one of its domains not connect over https, most probably due to a bad certificate. At first, I wondered if it was due to the update being scheduled at night when the server was shut down. But no, the default time is 9 am, and the task runs between 9 and 10 am. The log is located at %programdata%\win-acme\$baseuri$\Log according to https://www.win-acme.com/manual/advanced-use/custom-logging and checking there, found errors like
Warning - Existing https binding "oursite.com":443"" not updated because it doesn't seem to match the new certificate!
Information - Committing 1 "https" binding changes to IIS
Information - Updating existing https binding "oursite.com":443"" (flags: 1)

Running the command-line wacs.exe and choosing to force update the certificate worked fine.

This seems to be a bug in win-acme,
https://github.com/win-acme/win-acme/issues/2076#issuecomment-1060274192

so I've now upgraded to the latest version 2.1.23 hoping the next renewal 3 months later would be fine. 

Edit - the next renewal also failed. So, shifted to cloudflare origin certificate instead, https://hnsws.blogspot.com/2023/03/setting-up-cloudflare-origin.html

bulk email solutions

Somewhat related to my mailing list trials, but larger in scale. I had a request to look at AWS Simple Email Service (SES) since the pricing is so much lower than Sendinblue et al. for their requirement of around 50k emails, but only once in 2 months or so. Since they would have difficulty with writing their own code, I mentioned Sendy to them, having come across it here. 

Another possibility was, making multiple users in their Google Workspace, and sending emails via those users. This would be easier for them to do, since it doesn't involve any coding. Unfortunately they don't put in unsubscribe links in their emails. This would probably be leading to low deliverability rates.

Putting their mailing list into a google group was another solution I had earlier suggested.

Sending mails through gmail smtp relay - 10k per user per day? - is another possibility, but more complicated -
https://www.gmass.co/blog/understanding-gmails-email-sending-limits/
https://support.google.com/a/answer/2956491

More detailed copy-paste from our email exchange about usage, pros and cons - 

SES requires you to create an application and then code the sending of emails into the application using their SDK or API

They have SDKs in a variety of languages,

Using Amazon SES with an AWS SDK - Amazon Simple Email Service

Sending through SMTP or API,

Set up email sending with Amazon SES - Amazon Simple Email Service

You will need to code your application to send using either SMTP or the API, like the example at

Sending emails programmatically through the Amazon SES SMTP interface - Amazon Simple Email Service

You cannot just use a program like Outlook or something like that, because you have limits like 1 email sent per second and so on, and it would be too painful to do that non-programmatically - 

Service quotas in Amazon SES - Amazon Simple Email Service

200 emails per 24 hours till you get upgraded on request,

1 email per second till you get upgraded on request,

and quota will increase automatically only if you send more email daily - 

Increasing your Amazon SES sending quotas - Amazon Simple Email Service

Also, please note that attachments cost extra - 

Amazon Simple Email Service Pricing | Cloud Email Service | Amazon Web Services

So, for example, if the happy new year email was sent with a 500 kB attachment to 50,000 people, 25 GB would have been sent, so 

50,000 * 0.10/1000 = $5

PLUS

25GB * 0.12 = $3 

Of course, still cheaper than Sendinblue etc.

I have tried out Sendinblue and Mailchimp APIs, example Google Apps script code is at

Google-Apps-Script-mailing-list/Test sendinblue API.gs at main · hn-88/Google-Apps-Script-mailing-list (github.com)

Google apps script intro - 

Work Stuff: Getting started with Google Apps Script (hnsws.blogspot.com)

Edit Dec 2025 - google apps script code for 'slow-sending' at this post.

kex_exchange_identification: Connection closed by remote host error with ssh

When trying to connect with ssh to our internal server, I was seeing such errors intermittently -
kex_exchange_identification: Connection closed by remote host

When I would try again, it would connect.

Of the many possible reasons, I think in our case the cause might have been a busy router or a busy server. 

Edit: A restart or something like that seems to have solved it now, a couple of days later.

private file sharing solutions

Copy-pasting from an email discussion regarding solutions for sharing large files with multiple remote teams - 

1. Shared Drive - This seems to be the easiest and simplest solution, at least to start with. Most probably you cannot increase individual users beyond 30 GB, but you can use Shared Drives -  https://www.googlecloudcommunity.com/gc/Workspace-Q-A/Trying-to-change-storage-limit-for-members-of-a-group/m-p/421866

Shared Drives currently have no limit, as far as I could see.

2. Single file size limits - With google drive, even shared drives, you have a limit of a few MB for uploading files from the web interface. If you use the Google Drive Sync desktop app, you apparently can go up to 5 TB for individual files. But limited to 750 GB per day. This page also has the explanation for the 100 GB added, "no longer available for new customers"

https://support.google.com/a/answer/172541

3. If more than 5 TB is needed - Advantage of separate server for file-sharing - would be cheaper than google drive or other cloud storage. Disadvantage - would need to choose a server in India or else uploads might be quite slow. After that, if we have https downloads, we could use cloudflare caching to speed up downloads. This apparently works even if the content is behind some login/pw authentication, for example session based auth.

getting form response in an iframe

I've used the target attribute earlier in forms in order to get a form's response in an iframe, like

<form action="rightpage.php" method="get" target="result_frame">

....

<iframe name="result_frame" width="100%" height="900">

Apparently, we can also use formtarget instead, which goes into the submit button - 

<input type="submit" formtarget="result_frame" value="Submit to frame">

how to delete EFI system partition on Windows

If we want to repartition a former system drive, Windows doesn't allow us to delete the EFI system partition from the GUI. Can be done from the terminal - How to Delete the EFI System Partition in Windows 10 or 11 | Tom's Hardware (tomshardware.com)

diskpart

list disk

sel disk 2

list partition

sel partition 2

delete partition override

Notes on building release apk using Github Actions - cordova commandline build

1. The working workflow file for building the modified version of the Moodle app version 4.1.0 is at buildSSSVV.yml
   
   
2. Though the main idea was to clone a private repo similar to this post, one difference was that I used my user's private key instead of adding a separate key-pair as deployment key - that also worked, but I should be careful to remove my private key from the destination repo as soon as I finish my builds.
   
   
3. We must clone the private repo and build the public repo as part of the same job in the github workflow, or else the file would be cleaned up and not available (unless stored as an artifact.) 
   
   
4. One stumbling block preventing builds on the first attempt was to find the correct syntax for the release build using cordova build. Turns out it needs several points which were not clearly discussed at cordova's documentation example, but was mentioned in ionic's documentation example. (Edit - all the flags required are mentioned at https://cordova.apache.org/docs/en/11.x/guide/platforms/android/index.html#using-flags - I had not seen this page earlier.) I did not create a build.json since the commandline switches worked for me. "failed to read key from store" was because I missed the --password parameter, I had put in only the --storePassword parameter. The error was not due to requiring escaping of special characters in the password.
   
   
5. We also need -- -- twice in order to get an apk instead of an aab as the output file, Cordova generate release apk instead of aab - Stack Overflow.
   
   
6. The final correct command to build (at least for this version of cordova android) was
   npx ionic cordova build android --release -- -- --packageType=apk --keystore="~/sssvv_mobile-release-key.keystore" --password=$KEY_PW --storePassword=$KEY_PW --alias=$KEY_ALIAS
   
   
7. Before trying all this, I tried building just an unsigned apk - but jarsigner signing of the unsigned release apk didn't work. I wonder why.
   jarsigner -keystore <keystore_file>  -storepass <storepass> -keypass <keypass> <unsigned_apk_file> <alias_name>
   The apk was not showing the correct icon, and "problem parsing file" when try to install. Maybe this problem is due to some version difference between the java / SDK / build tools on the build machine and the signing machine? No idea.
   
   
8. The technique of using this GUI tool to avoid typing out all the paths - Android APK Signer & Aligner - Luke Alderton - also failed, probably due to the same issue as in the above point, whatever that is. Which is why I went ahead with trying to generate the signed release apk from the workflow itself, as above.

HP 1020 printer on USB instead

Printing to the HP 1020 printer over Wifi, as noted in an earlier post, was working. But it had some issues. Multiple copies were not printing. On some days, the printer would not get detected - once it got detected after I tried something with the Android app, but on another day nothing would help. So, I got fed up and connected it via USB instead.

To get it to print over USB, 

https://forums.linuxmint.com/viewtopic.php?t=291099

led to

https://easylinuxtipsproject.blogspot.com/p/hp-printers.html

sudo apt-get install hplip-gui

(it was version 3.21.8 for current linux mint 20.3 kernel 5.15)

Deleted and installed after turning off wifi = worked.

initializing a 2D array in google apps script

Trying code like:

var resultasarray = [];  

  

  while (TableData.next()) {

    for (var j = 0; j < numberOfColumns; j++) {

       resultasarray[i][j] = TableData.getString(j+1));

  }

  i++;

       

    }

threw errors, TypeError: Cannot read properties of undefined (reading '0')

Using push() instead, worked.

var resultasarray = [];  

  

  while (TableData.next()) {

    var rowvals = [];

    for (var j = 0; j < numberOfColumns; j++) {

        rowvals.push(TableData.getString(j+1));

        //Logger.log(TableData.getString(j+1))

    }

    resultasarray.push(rowvals)

    i++;

  }

ad-hoc reports - device information

Copy-pasting from an email exchange regarding the addition of device information to files downloaded ad-hoc report on Moodle:

the device used by the user whether tablets supplied by us or smartphones

This information is not directly captured in the logs, so we cannot put this directly in the files downloaded per user report.

But we do have the

prefix_user_devices table,

which logs which mobile device a user has logged in from, including the time-stamp of when they first logged in from that device.

So, we could write a separate queries for things like:

1. How many users are using the tablets provided by us

2. How many users are using mobile phones or other tablets

3. Which devices a particular user has used, including the time periods

and so on. 

Also, we must note:

This table would not have entries if the user has logged in using a web browser - even if it was a tablet or mobile. Only those devices which use the Moodle app or our mobile app would be logged in this table.

mailchimp-like free services without the banner, sendinblue and curl

Mailchimp's free plan nowadays has quite an intrusive banner. "Grow your business with Mailchimp". This makes it quite inappropriate for non-commercial emails. Older methods like this one, to make the banner less intrusive, don't seem to work. Looking around for alternatives, one of the top search results was sendinblue, with 300 free emails per day, including transactional emails also. Added bonus - the transactional emails don't even have an unsubscribe link by default, we can add it though. My test implementation is at https://github.com/hn-88/Google-Apps-Script-mailing-list/blob/main/Mailing%20List/Test%20sendinblue%20API.gs

While testing the sendinblue api and the mailchimp api, found some points by trial and error. Just related to curl, syntax of multi-line strings and so on.

1. curl's --user parameter can be translated to the following snippet -
   'headers': {
           "Authorization": "Basic " + Utilities.base64Encode("anystring" + ":" + mailchimpApiKey)
       }
   for use in Google Apps scripts.
   
   
2. curl's --header parameters should be used as such - just take care to escape double-quotes and single quotes inside the strings like \' or \\\' as needed.
   'headers': {
           "accept": "application/json",
           "content-type": "application/json",
           "api-key" :  sibApiKey
       }
3. Sendinblue's transactional emails don't include unsubscribe links by default, but we can add them by including the placeholder {{ unsubscribe }}
   <a href={{ unsubscribe }}>To unsubscribe, click here</a>
   
   
4. Multi-line data needs to be handled with care. In Bash, for curl, we put \ at the end of lines for line breaks between parameters, but for a single parameter (like --data) which has a multi-line parameter, we can use single quotes in Bash+curl and single back-ticks in GAS for json objects as can be seen at the following lines of code here. But for JSON to be valid, there should be no line-breaks within the key-value pairs - so, the htmlContent has to be entered in a single line with no breaks. Even adding \n inside the string causes the API to complain about invalid JSON.
   

 

batch conversion of canon raw files

There were some photos in Canon Raw .CR2 format which were to be uploaded to shutterfly.com which accepts only JPG, JPEG, BMP, PNG, HEIC and HEIF - not CR2 or RAW. So I looked at batch conversions. Most probably it would have been possible to do it in Irfanview, but for Linux, I found this - https://askubuntu.com/questions/483379/how-to-convert-cr2-to-jpg-or-png

In my case, I used

sudo apt install darktable 

for pic in *.CR2
do
     darktable-cli "$pic" "$(basename ${pic%.CR2}.jpg)";
done 

It took nearly 10 seconds per file, but it did the job.

PPF information

My PPF (Public Provident Fund) account was with SBI's main branch in Calicut, while my savings bank account with SBI is in Puttaparthi. So, when I got an email from SBI Trivandrum asking if I wanted to enable standing instructions, I did not find my PPF account directly linked in my SBI netbanking, nor did I know my account number off-hand. Asking a bank employee in Calicut, got the account number. Then, found that the account number and balance are available in two places:

1. SBI's Netbanking --> Other links --> My SBI World - shows the "net worth", which includes PPF. Also supposed to show transactions if any. The account number is shown in full here.
   
   
2. The monthly statement from cbssbi.cas@sbi.co.in has the PPF info also in the statement. The account number is partially blanked out here. 

In my case, trying UPI payments to the account did not work. A colleague who works in finance said that this was probably due to the account being inactive for the last few years - we need to call them, make the account active, and then close it or deposit more etc. The toll-free lines did not work, of course. And nobody answers the landline number of the branch. Probably will have to contact some bank employee and get this done. There is time - one more year till maturity as per the statement.

log out all users from php sessions based website - and superuser shell for shell wildcard expansions

There was a request to log out all users from the website based on this code. At first I thought just restarting the server after applying some updates would do it. But no. 

https://stackoverflow.com/questions/25740705/how-to-log-out-all-active-logged-in-users-in-php-from-admin-side

Since our site was based on php sessions and nothing was being saved to the database for user sessions, here is what worked:

First found the path using a small php page which had the line 

php echo session_save_path()

(or phpinfo would have worked?)

then just 

sudo sh -c 'rm -v /var/lib/php/sessions/sess*'

About why just 

sudo rm sess* 

won't work, that's because the wildcard expansion is done by the shell and not by rm. 

Notes on a simple mailing list written in Google Apps Script

There are lots of examples about mail-merge with Google Apps Script. 

Using Mailchimp for blog updates by email had the problem that only the default feed would work, and not a custom feed as mentioned in my earlier post. 

So, I've attempted to create a mailing list manager in Google Apps Script. Subscribe and unsubscribe forms are to be done (TBD). Some notes, in addition to the references in the inline comments - 

• Add unsubscribe link - https://ravgeetdhillon.medium.com/add-unsubscribe-link-in-emails-using-google-apps-script-475c938b3e9f
  
  
• For parsing RSS, the XmlService - https://developers.google.com/apps-script/reference/xml-service/xml-service
  and https://www.labnol.org/code/19733-parse-xml-rss-feeds-google-scripts
  
  
• Example script which gets the most recent XKCD comic from the RSS feed and emails it to yourself - https://gist.github.com/roblaplaca/11025c3e0cae33f3fc0b
  
  
• Verifying emails - https://stackoverflow.com/questions/26577293/verify-email-entered-in-a-google-form-using-google-apps-script
  
  
• I might probably go for a verification email instead, like https://www.youtube.com/watch?v=gi_xlZ5K9rs
  
  
• Simple GAS RSS reader - https://www.youtube.com/watch?v=vFOH1AQaSl8 - but I think the Atom RSS used by blogger has different fields.
  
  
• Shortcut to get json data - https://gist.github.com/mhawksey/1442370 -
  
  var response = UrlFetchApp.fetch('https://YOURSITEHERE.com');
  var json = response.getContentText();
  var rawdata = JSON.parse(json);
  var rawdata = JSON.parse(response.getContentText());
  Logger.log(rawdata);
  var data = [];
  data.push(rawdata);
  SpreadsheetApp.getActiveSheet().appendRow(data)
  
  

concatenating mp4 files without re-encoding with ffmpeg

Following https://ottverse.com/3-easy-ways-to-concatenate-mp4-files-using-ffmpeg/ and https://finance.uw.edu/recmgt/resources/list-your-folder-structure-windows

• dir *.mp4 /B > filelist.txt
• In notepad, find/replace VIDEO_ with file VIDEO_
• ffmpeg -f concat -safe 0 -i filelist.txt -c copy mergedVideo.mp4

After that, speeding up the video 10x using Avisynth to open the file in VirtualDub - the two videos below. The second video's last part seems to have some missing segments, probably because the dashcam's power adapter came out of the 12V socket. I've now replaced it with one which I had purchased earlier, which has a better grip. 

The camera-generated timelapse and other info is at this post.

preventing related videos on embedded youtube videos

If a youtube video is embedded with the default embed code, after the video plays, youtube shows a selection of "related content" which is usually not from the same channel. According to this thread, we can now only prevent other channel content from being displayed - by adding ?rel=0 to the embed code. Like youtube.com/embed/idOfVideo?rel=0

some Reaper audio cleanup ideas

Use the Ripple Delete tool to enable Ripple Delete either for a single track or for all tracks -

https://www.youtube.com/watch?v=AW5G-60jH34
describes a useful click pop removal using trim tool - we can change the fade-in-out lengths for the trim tool at
Preferences -> Media Item defaults for new items from 10 ms to 50 ms.

And in using this for click-pop removal, we can pinpoint the correct location to use the trim tool by using the Spectrograph view - Spectral Edit -> View Spectrograph. In the track image below, the orange vertical line is the click - 

Some cleaning up tips
https://www.youtube.com/watch?v=Ar4hIbqTzAY
NR etc.

Screenshots from my current settings for audio cleanup and sweetening for voice-overs (VO) - 

1. Noise reduction with ReaFir to remove the background hiss,

2. Gating out background noises

3. Some compression

4. Followed by a little bit of reverb for the vocals.

This earlier post gives the method I have used to do RMS normalization for audio clips, using the limiter and manual control of the limiter threshold by watching the RMS levels on the Master track. 

adding copyright statement to Moodle footer - Moove theme

There was a request to add a custom content copyright statement to the footer of one of our Moodle sites which used the Moove theme.

The easiest way to add the copyright footer seems to be via the custom CSS box at

https://moodlesite.tld/admin/settings.php?section=themesettingmoove#theme_moove_advanced

Using the technique at

https://www.foxinfotech.in/2020/03/adding-text-content-to-an-element-using-css.html

So, in this case, the css added was:

footer#page-footer .supportemail::before {content: "© 2010 - 2022 Our custom copyright statement."} 

 

All the raw SCSS added currently:

footer#page-footer .supportemail::before {content: "© 2010 - 2022 Our custom copyright statement. All rights reserved."}

.card-body {
min-height: 100%;
}
footer#page-footer .copyright  {display:none;}

nav.navbar .navbar-brand .logo img{

max-width:100%;

height:auto;

}

nav.navbar .navbar-brand .logo{

display:inline!important;}