kex_exchange_identification: Connection closed by remote host error with ssh

When trying to connect with ssh to our internal server, I was seeing such errors intermittently -
kex_exchange_identification: Connection closed by remote host

When I would try again, it would connect.

Of the many possible reasons, I think in our case the cause might have been a busy router or a busy server. 

Edit: A restart or something like that seems to have solved it now, a couple of days later.

private file sharing solutions

Copy-pasting from an email discussion regarding solutions for sharing large files with multiple remote teams - 

1. Shared Drive - This seems to be the easiest and simplest solution, at least to start with. Most probably you cannot increase individual users beyond 30 GB, but you can use Shared Drives -  https://www.googlecloudcommunity.com/gc/Workspace-Q-A/Trying-to-change-storage-limit-for-members-of-a-group/m-p/421866

Shared Drives currently have no limit, as far as I could see.

2. Single file size limits - With google drive, even shared drives, you have a limit of a few MB for uploading files from the web interface. If you use the Google Drive Sync desktop app, you apparently can go up to 5 TB for individual files. But limited to 750 GB per day. This page also has the explanation for the 100 GB added, "no longer available for new customers"

https://support.google.com/a/answer/172541

3. If more than 5 TB is needed - Advantage of separate server for file-sharing - would be cheaper than google drive or other cloud storage. Disadvantage - would need to choose a server in India or else uploads might be quite slow. After that, if we have https downloads, we could use cloudflare caching to speed up downloads. This apparently works even if the content is behind some login/pw authentication, for example session based auth.

getting form response in an iframe

I've used the target attribute earlier in forms in order to get a form's response in an iframe, like

<form action="rightpage.php" method="get" target="result_frame">

....

<iframe name="result_frame" width="100%" height="900">

Apparently, we can also use formtarget instead, which goes into the submit button - 

<input type="submit" formtarget="result_frame" value="Submit to frame">

how to delete EFI system partition on Windows

If we want to repartition a former system drive, Windows doesn't allow us to delete the EFI system partition from the GUI. Can be done from the terminal - How to Delete the EFI System Partition in Windows 10 or 11 | Tom's Hardware (tomshardware.com)

diskpart

list disk

sel disk 2

list partition

sel partition 2

delete partition override

Notes on building release apk using Github Actions - cordova commandline build

1. The working workflow file for building the modified version of the Moodle app version 4.1.0 is at buildSSSVV.yml
   
   
2. Though the main idea was to clone a private repo similar to this post, one difference was that I used my user's private key instead of adding a separate key-pair as deployment key - that also worked, but I should be careful to remove my private key from the destination repo as soon as I finish my builds.
   
   
3. We must clone the private repo and build the public repo as part of the same job in the github workflow, or else the file would be cleaned up and not available (unless stored as an artifact.) 
   
   
4. One stumbling block preventing builds on the first attempt was to find the correct syntax for the release build using cordova build. Turns out it needs several points which were not clearly discussed at cordova's documentation example, but was mentioned in ionic's documentation example. (Edit - all the flags required are mentioned at https://cordova.apache.org/docs/en/11.x/guide/platforms/android/index.html#using-flags - I had not seen this page earlier.) I did not create a build.json since the commandline switches worked for me. "failed to read key from store" was because I missed the --password parameter, I had put in only the --storePassword parameter. The error was not due to requiring escaping of special characters in the password.
   
   
5. We also need -- -- twice in order to get an apk instead of an aab as the output file, Cordova generate release apk instead of aab - Stack Overflow.
   
   
6. The final correct command to build (at least for this version of cordova android) was
   npx ionic cordova build android --release -- -- --packageType=apk --keystore="~/sssvv_mobile-release-key.keystore" --password=$KEY_PW --storePassword=$KEY_PW --alias=$KEY_ALIAS
   
   
7. Before trying all this, I tried building just an unsigned apk - but jarsigner signing of the unsigned release apk didn't work. I wonder why.
   jarsigner -keystore <keystore_file>  -storepass <storepass> -keypass <keypass> <unsigned_apk_file> <alias_name>
   The apk was not showing the correct icon, and "problem parsing file" when try to install. Maybe this problem is due to some version difference between the java / SDK / build tools on the build machine and the signing machine? No idea.
   
   
8. The technique of using this GUI tool to avoid typing out all the paths - Android APK Signer & Aligner - Luke Alderton - also failed, probably due to the same issue as in the above point, whatever that is. Which is why I went ahead with trying to generate the signed release apk from the workflow itself, as above.

HP 1020 printer on USB instead

Printing to the HP 1020 printer over Wifi, as noted in an earlier post, was working. But it had some issues. Multiple copies were not printing. On some days, the printer would not get detected - once it got detected after I tried something with the Android app, but on another day nothing would help. So, I got fed up and connected it via USB instead.

To get it to print over USB, 

https://forums.linuxmint.com/viewtopic.php?t=291099

led to

https://easylinuxtipsproject.blogspot.com/p/hp-printers.html

sudo apt-get install hplip-gui

(it was version 3.21.8 for current linux mint 20.3 kernel 5.15)

Deleted and installed after turning off wifi = worked.

initializing a 2D array in google apps script

Trying code like:

var resultasarray = [];  

  

  while (TableData.next()) {

    for (var j = 0; j < numberOfColumns; j++) {

       resultasarray[i][j] = TableData.getString(j+1));

  }

  i++;

       

    }

threw errors, TypeError: Cannot read properties of undefined (reading '0')

Using push() instead, worked.

var resultasarray = [];  

  

  while (TableData.next()) {

    var rowvals = [];

    for (var j = 0; j < numberOfColumns; j++) {

        rowvals.push(TableData.getString(j+1));

        //Logger.log(TableData.getString(j+1))

    }

    resultasarray.push(rowvals)

    i++;

  }

ad-hoc reports - device information

Copy-pasting from an email exchange regarding the addition of device information to files downloaded ad-hoc report on Moodle:

the device used by the user whether tablets supplied by us or smartphones

This information is not directly captured in the logs, so we cannot put this directly in the files downloaded per user report.

But we do have the

prefix_user_devices table,

which logs which mobile device a user has logged in from, including the time-stamp of when they first logged in from that device.

So, we could write a separate queries for things like:

1. How many users are using the tablets provided by us

2. How many users are using mobile phones or other tablets

3. Which devices a particular user has used, including the time periods

and so on. 

Also, we must note:

This table would not have entries if the user has logged in using a web browser - even if it was a tablet or mobile. Only those devices which use the Moodle app or our mobile app would be logged in this table.

mailchimp-like free services without the banner, sendinblue and curl

Mailchimp's free plan nowadays has quite an intrusive banner. "Grow your business with Mailchimp". This makes it quite inappropriate for non-commercial emails. Older methods like this one, to make the banner less intrusive, don't seem to work. Looking around for alternatives, one of the top search results was sendinblue, with 300 free emails per day, including transactional emails also. Added bonus - the transactional emails don't even have an unsubscribe link by default, we can add it though. My test implementation is at https://github.com/hn-88/Google-Apps-Script-mailing-list/blob/main/Mailing%20List/Test%20sendinblue%20API.gs

While testing the sendinblue api and the mailchimp api, found some points by trial and error. Just related to curl, syntax of multi-line strings and so on.

1. curl's --user parameter can be translated to the following snippet -
   'headers': {
           "Authorization": "Basic " + Utilities.base64Encode("anystring" + ":" + mailchimpApiKey)
       }
   for use in Google Apps scripts.
   
   
2. curl's --header parameters should be used as such - just take care to escape double-quotes and single quotes inside the strings like \' or \\\' as needed.
   'headers': {
           "accept": "application/json",
           "content-type": "application/json",
           "api-key" :  sibApiKey
       }
3. Sendinblue's transactional emails don't include unsubscribe links by default, but we can add them by including the placeholder {{ unsubscribe }}
   <a href={{ unsubscribe }}>To unsubscribe, click here</a>
   
   
4. Multi-line data needs to be handled with care. In Bash, for curl, we put \ at the end of lines for line breaks between parameters, but for a single parameter (like --data) which has a multi-line parameter, we can use single quotes in Bash+curl and single back-ticks in GAS for json objects as can be seen at the following lines of code here. But for JSON to be valid, there should be no line-breaks within the key-value pairs - so, the htmlContent has to be entered in a single line with no breaks. Even adding \n inside the string causes the API to complain about invalid JSON.
   

 

batch conversion of canon raw files

There were some photos in Canon Raw .CR2 format which were to be uploaded to shutterfly.com which accepts only JPG, JPEG, BMP, PNG, HEIC and HEIF - not CR2 or RAW. So I looked at batch conversions. Most probably it would have been possible to do it in Irfanview, but for Linux, I found this - https://askubuntu.com/questions/483379/how-to-convert-cr2-to-jpg-or-png

In my case, I used

sudo apt install darktable 

for pic in *.CR2
do
     darktable-cli "$pic" "$(basename ${pic%.CR2}.jpg)";
done 

It took nearly 10 seconds per file, but it did the job.

PPF information

My PPF (Public Provident Fund) account was with SBI's main branch in Calicut, while my savings bank account with SBI is in Puttaparthi. So, when I got an email from SBI Trivandrum asking if I wanted to enable standing instructions, I did not find my PPF account directly linked in my SBI netbanking, nor did I know my account number off-hand. Asking a bank employee in Calicut, got the account number. Then, found that the account number and balance are available in two places:

1. SBI's Netbanking --> Other links --> My SBI World - shows the "net worth", which includes PPF. Also supposed to show transactions if any. The account number is shown in full here.
   
   
2. The monthly statement from cbssbi.cas@sbi.co.in has the PPF info also in the statement. The account number is partially blanked out here. 

In my case, trying UPI payments to the account did not work. A colleague who works in finance said that this was probably due to the account being inactive for the last few years - we need to call them, make the account active, and then close it or deposit more etc. The toll-free lines did not work, of course. And nobody answers the landline number of the branch. Probably will have to contact some bank employee and get this done. There is time - one more year till maturity as per the statement.

log out all users from php sessions based website - and superuser shell for shell wildcard expansions

There was a request to log out all users from the website based on this code. At first I thought just restarting the server after applying some updates would do it. But no. 

https://stackoverflow.com/questions/25740705/how-to-log-out-all-active-logged-in-users-in-php-from-admin-side

Since our site was based on php sessions and nothing was being saved to the database for user sessions, here is what worked:

First found the path using a small php page which had the line 

php echo session_save_path()

(or phpinfo would have worked?)

then just 

sudo sh -c 'rm -v /var/lib/php/sessions/sess*'

About why just 

sudo rm sess* 

won't work, that's because the wildcard expansion is done by the shell and not by rm. 

Notes on a simple mailing list written in Google Apps Script

There are lots of examples about mail-merge with Google Apps Script. 

Using Mailchimp for blog updates by email had the problem that only the default feed would work, and not a custom feed as mentioned in my earlier post. 

So, I've attempted to create a mailing list manager in Google Apps Script. Subscribe and unsubscribe forms are to be done (TBD). Some notes, in addition to the references in the inline comments - 

• Add unsubscribe link - https://ravgeetdhillon.medium.com/add-unsubscribe-link-in-emails-using-google-apps-script-475c938b3e9f
  
  
• For parsing RSS, the XmlService - https://developers.google.com/apps-script/reference/xml-service/xml-service
  and https://www.labnol.org/code/19733-parse-xml-rss-feeds-google-scripts
  
  
• Example script which gets the most recent XKCD comic from the RSS feed and emails it to yourself - https://gist.github.com/roblaplaca/11025c3e0cae33f3fc0b
  
  
• Verifying emails - https://stackoverflow.com/questions/26577293/verify-email-entered-in-a-google-form-using-google-apps-script
  
  
• I might probably go for a verification email instead, like https://www.youtube.com/watch?v=gi_xlZ5K9rs
  
  
• Simple GAS RSS reader - https://www.youtube.com/watch?v=vFOH1AQaSl8 - but I think the Atom RSS used by blogger has different fields.
  
  
• Shortcut to get json data - https://gist.github.com/mhawksey/1442370 -
  
  var response = UrlFetchApp.fetch('https://YOURSITEHERE.com');
  var json = response.getContentText();
  var rawdata = JSON.parse(json);
  var rawdata = JSON.parse(response.getContentText());
  Logger.log(rawdata);
  var data = [];
  data.push(rawdata);
  SpreadsheetApp.getActiveSheet().appendRow(data)
  
  

concatenating mp4 files without re-encoding with ffmpeg

Following https://ottverse.com/3-easy-ways-to-concatenate-mp4-files-using-ffmpeg/ and https://finance.uw.edu/recmgt/resources/list-your-folder-structure-windows

• dir *.mp4 /B > filelist.txt
• In notepad, find/replace VIDEO_ with file VIDEO_
• ffmpeg -f concat -safe 0 -i filelist.txt -c copy mergedVideo.mp4

After that, speeding up the video 10x using Avisynth to open the file in VirtualDub - the two videos below. The second video's last part seems to have some missing segments, probably because the dashcam's power adapter came out of the 12V socket. I've now replaced it with one which I had purchased earlier, which has a better grip. 

The camera-generated timelapse and other info is at this post.

preventing related videos on embedded youtube videos

If a youtube video is embedded with the default embed code, after the video plays, youtube shows a selection of "related content" which is usually not from the same channel. According to this thread, we can now only prevent other channel content from being displayed - by adding ?rel=0 to the embed code. Like youtube.com/embed/idOfVideo?rel=0

some Reaper audio cleanup ideas

Use the Ripple Delete tool to enable Ripple Delete either for a single track or for all tracks -

https://www.youtube.com/watch?v=AW5G-60jH34
describes a useful click pop removal using trim tool - we can change the fade-in-out lengths for the trim tool at
Preferences -> Media Item defaults for new items from 10 ms to 50 ms.

And in using this for click-pop removal, we can pinpoint the correct location to use the trim tool by using the Spectrograph view - Spectral Edit -> View Spectrograph. In the track image below, the orange vertical line is the click - 

Some cleaning up tips
https://www.youtube.com/watch?v=Ar4hIbqTzAY
NR etc.

Screenshots from my current settings for audio cleanup and sweetening for voice-overs (VO) - 

1. Noise reduction with ReaFir to remove the background hiss,

2. Gating out background noises

3. Some compression

4. Followed by a little bit of reverb for the vocals.

This earlier post gives the method I have used to do RMS normalization for audio clips, using the limiter and manual control of the limiter threshold by watching the RMS levels on the Master track. 

adding copyright statement to Moodle footer - Moove theme

There was a request to add a custom content copyright statement to the footer of one of our Moodle sites which used the Moove theme.

The easiest way to add the copyright footer seems to be via the custom CSS box at

https://moodlesite.tld/admin/settings.php?section=themesettingmoove#theme_moove_advanced

Using the technique at

https://www.foxinfotech.in/2020/03/adding-text-content-to-an-element-using-css.html

So, in this case, the css added was:

footer#page-footer .supportemail::before {content: "© 2010 - 2022 Our custom copyright statement."} 

 

All the raw SCSS added currently:

footer#page-footer .supportemail::before {content: "© 2010 - 2022 Our custom copyright statement. All rights reserved."}

.card-body {
min-height: 100%;
}
footer#page-footer .copyright  {display:none;}

nav.navbar .navbar-brand .logo img{

max-width:100%;

height:auto;

}

nav.navbar .navbar-brand .logo{

display:inline!important;}

Mini review of Qubo Pro dashcam

There are various reviews on youtube, like this one, which give more details about this dashcam. I purchased it since it was one of the lowest priced dashcams with good reviews on Amazon. 

• The "audio notification" feature quickly turns annoying on our rough roads, it keeps chiming. So I turned it off via the app. 
  
  
• I used it initially with my existing 2 GB mini-SD card, but they recommend faster newer cards - maybe with a faster card, I won't see the frame skipping as seen in the "manual mode" recording below.
  
  
• If we download a recording, it is approx 1 minute long, as the reviews suggest. Around 30 MB. So, a 64 GB card can have around 2000 minutes or 30 hours of video.
  
  
• General recordings are FIFO auto-deleted, "Emergency" recordings, based on the accelerometer jerk detection, are not auto-deleted as quickly. We can set the number of "Emergency" recordings to keep, and it will auto-delete older ones only after that limit is reached.  
  
  
• The time-lapse feature captures 640 x 480 frames - an example embedded below. The audio track for the time-lapse is also created by the app. Apparently it is something which is licensed for Youtube uploads.

• I will add a video recorded in "manual mode" on the app below. During manual mode recording, I believe nothing is saved on the mini-SD card, and the app needs to be in the foreground with the mobile switched on. If we turn off the mobile's screen during recording, the recording is lost. We need to manually press the record icon and stop recording, only then the recording is seen in the app gallery. It is saved as mp4, but with a H265 or HEVC codec or something like that. Probably the dropped frames are due to my mobile phone's internal memory or CPU not being fast enough? Samsung Galaxy M12. 

• Edit: Found that the one minute clips recorded on the miniSD card have a small overlap of a few frames - 10 frames or so - each. So no data is supposed to be lost between clips. 1800 frames recorded in each segment. So, with the overlap, slightly less than a minute of unique content each. Filenames are like  
  VIDEO_30112022085501_1669778701951.mp4
  VIDEO_30112022085600_1669778760947.mp4
  So assuming a time-stamp in milliseconds, via Epoch Converter, 
  08:55:01.951 to 08:56:00.947 - so probably one second of overlap.
  
  
• Edit: Here are a couple of videos - one from the running car, one from the timelapse generated for an entire day of driving to Bangalore and back.
  Approx. six hours of driving = 6*3600 = 21600 seconds.
  The timelapse = approx. 3 minutes at 5 fps = 900 frames. 
  So, the timelapse was shot at approx. one frame every 30 seconds? Or some other scene-based method? Will try looking at the raw files and see.
  
  Looking at the files saved on the miniSD card, there is a folder called Timelapse created, with the saved files used to make timelapse videos. There are 5 timestamp files per minute. According to the timestamp of the file, once every 12 seconds. According to the filename, which is like
  timelapse_1669964034331.jpeg
  timelapse_1669964022031.jpeg
  subtracting the unix timestamps, 12.3 seconds between the frames.
  
  

Edit: More videos of the complete drive are available at this post.

Google Play store blues - trying in vain to update our app - shortcut to update version code

Updates to one of our apps was being blocked by Google Play console.

Initially, in the developer console, there was a Policy Declaration we had to make, stating the reason for adding a particular sensitive permission - in this case, REQUEST_INSTALL_PACKAGES permission.

There was a form to appeal the rejection, but submitting that just gave a generic reply saying the appeal was rejected.

The Moodle app's repository meanwhile mentioned this issue, and gave a fix - https://tracker.moodle.org/projects/MOBILE/issues/MOBILE-4134?filter=doneissues

But even after uploading a version of the app with these fixes, the same generic automated rejection of the update. Then I tried the "reaching out to our policy support team" link at the bottom of the email. 

It was not very clear where exactly I should reach out to the policy support team in this screen, but I guessed it was via the arrow to the right of the "Appeals" section. I specifically mentioned that this permission had been removed, but the "policy declaration" could not be removed - there is no provision on the console to do that.

Again I got a template email as a reply saying "we are unable to approve your app's use of REQUEST_INSTALL_PACKAGES", but at the bottom of that email, there was some additional information -

Step 2: Submit a compliant update or remove the permission from your app

Once you’re ready to submit a compliant version of your app:

1. Make the necessary updates to address the issue(s) identified above OR please remove the permission from your manifest AND in-app functionality.
2. Double check that your app is compliant with all other Developer Program Policies.
3. Sign in to your Play Console, upload the modified, policy compliant APK across all tracks, and deactivate the non-compliant APK(s). 
   1. To deactivate a non-compliant APK, please create a new release and upload a compliant APK to the same track.
   2. Be sure to increment the APK version number and set the release to 100% rollout, in order to successfully override and deactivate the non-compliant APK. 
4. To save any changes you make to your release, select Save.
5. When you've finished preparing your release, select Review release.

Kindly note that your changes aren't sent for review automatically. You must go to the Publishing overview page and click Send for review to submit your changes.

Aha - so I have to deactivate non-compliant APKs on all tracks - not just the Production track. And how should I deactivate?

https://stackoverflow.com/questions/48624580/how-to-deactivate-my-previous-version-retained-apk-in-play-store

So I have to go to each track and add a new version with "Create New Release". For that, I had to create one more version of the "fixed" APK, since the current "proper" release had been auto-rejected - I had to build another apk with an incremented version code. One of the rejected apks had a higher version code than the current "fixed" version. So, I added one to that to get my next version code - though retaining the same version name and version number displayed inside the app, which have to be changed via config.xml and moodle.config.json. 

For changing only the version code, I just did Edit --> Find --> "Replace in files" inside Android Studio for the old version code with the new version code, with "Scope" as the "search in" parameter. It found 7 matches in 6 files or something like that. 7 clicks of "Replace" later, clicking on "Generate signed APK" got me the fresh apk which I could upload to the Play store. Much easier than going through the entire Cordova build process. 

Now, after submitting all these tracks, it looks like the update has not been auto-rejected. I also edited the "Policy declaration" mentioning that this permission is no longer needed. Perhaps after the update is reviewed, the policy declaration change - where this permission is not listed - would also happen. We have to wait and see. Maybe a week.

Edit: The update went through in 2 days.

Avisynth Resize and not Blur

I wanted an effect similar to news videos which have blurred bands on both sides of the frame. First, I tried Avisynth Blur filter, but even with a 3x chained filter with the max settings, the blurring was not sufficient. Then I tried Avisynth Resize, something like 

BilinearResize(1920, 1080, src_left=10, src_top=10, src_width=200, src_height=300)

That did the trick. Not perfect, since only a part of the source video is being sampled, but sufficient for my purpose. 

cloudflare origin server certificate for a bitnami moodle server

An earlier post had noted various workarounds for renewing certificates of cloudflare proxied bitnami servers. When a new server was required now, I tried out the cloudflare-provided origin server certificate.

We get the certificate and private key from Cloudflare dashboard --> Domain --> SSL/TLS --> Origin server. We just have to ssh into the bitnami server and copy-paste the content of the certificate to server.crt and contents of the private key to server.key respectively, at /opt/bitnami/apache/conf/bitnami/certs/

This is so because the Bitnami Moodle installation already had the ssl configuration set up in the ssl conf file at at /opt/bitnami/apache/conf/

Edit: 10 Feb 2023 - on a fresh bitnami moodle install, just doing the above was causing bitnami services to not start up after a restart. An additional step - I needed to save the root certificate as

/opt/bitnami/apache/conf/server-ca.crt

as mentioned at serial number (4.) at

https://developers.cloudflare.com/ssl/origin-configuration/origin-ca

and linked from that page.

increase swap size on Linux

From Linux Mint forums,

[Solved] How to increase swap size.... - Linux Mint Forums

inxi -Sp

swapon

# to see if a swapfile is being used. If yes, then

sudo swapoff -a

sudo dd if=/dev/zero of=/swapfile bs=1M count=8192 status=progress

# took around 10-15 seconds

sudo chmod 0600 /swapfile

sudo mkswap /swapfile

sudo swapon -a

and then reboot for the changes to take effect. 

mailchimp as an alternative to feedburner emails

When google turned off feedburner emails, I just removed the "subscribe to emails" link. The other day, I happened to come across this link, 
https://cascadevalleydesigns.com/moving-subscribers-away-from-feedburner-and-into-mailchimp-updated-guide/

So, I'm trying out this on another blog. 

Some points to note:

• After logging in to feedburner, we can download the list of email subscribers as csv from the "three vertical dots" menu.
• We can import this with or without editing into mailchimp contacts.
• Mailchimp free account has a footer with mailchimp logo etc.
• I tried to customize my feed by choosing the rss feed to be
  myblogname.blogspot.com/feeds/posts/default?updated-min=2022-11-14T00:00:00
  because my latest posts there are dated 1992. If we choose the default feed, the 1992 posts are not shown, only 2022, 2021 etc are shown. 
• We can preview RSS feeds using online tools like RSS Feed Viewer
• As mentioned in the howto post at the top of this email, mailchimp has made it difficult to find RSS integration. Google search for mailchimp rss by email finds the relevant page with a link to the rss setup, to which we can directly click through if logged in to mailchimp. 
• After choosing the campaign email theme, we are redirected to a graphical design page. There, we need to drag and drop the RSS title and RSS content blocks into our email. Otherwise the RSS feed won't be part of the email!

Edit: Unfortunately, the RSS feed which I have mentioned above does not seem to work with Mailchimp. Since I don't want to use the blog/feeds/posts/default feed, I'm stuck with either manual campaigns on Mailchimp or some other solution. Perhaps using Google Apps Scripts, since the number of emails to be sent is very small. 

exploring jekyll for our websites

Since Github Pages support jekyll, I took a look at some jekyll templates which could be customized to fit the desired "menu links on the left, full-page content on the right" theme. 

https://www.smashingmagazine.com/2014/08/build-blog-jekyll-github-pages/

Among the supported themes, Dinky seemed to be the best bet, after customizing its colours. But the amount of work needed to customize the themes, followed by re-writing the existing HTML pages in markup, seems to be unnecessary for our needs. I can probably just tweak the css files and/or edit html tags and get the desired results. Jekyll might be more useful for someone who wants to create fresh pages every day, or blog, or something like that. 

Moodle manual backup

Followed the broad outline at https://www.handybackup.net/moodle-backup.shtml, so

1. SQL db data dump - I used DBeaver on my local machine. Using the command-line on the server would have been faster, but I was not sure if some options would have caused problems.
   
   
2. Moodle code - just for safety, zipped the entire directory - but I could have discarded the .git sub-directory.
   zip -r moodledir.zip /var/www/moodledir
   
   
3. Moodle data - Zipped only filedir directory, since other things like cache are not required for the backup. 

After using rclone to copy the 72 GB data dir zip file to a Shared drive in Google Drive, I tried using Google Drive Desktop on Windows to make a local copy. First attempt at directly copying from drive to a USB external hard disk - failed with semaphore timeout has expired error. Then tried with internal SSD, again the same error. Then I tried with rclone, success.

rclone copyto gdrive:/our_data-filedir20221112.zip ../our_data-filedir20221112.zip --progress

Transferred:       73.856 GiB / 73.856 GiB, 100%, 0 B/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:    2h40m9.5s

Blogger does not show correct preview image for youtube playlists - workaround

If we embed a youtube playlist in a blog post on Blogger, the preview image for the post shown on index pages is the youtube broken link preview image. My workaround was to embed the videos in the playlist BEFORE the playlist on the same post, in reduced size - only 81 pixels wide. Then the preview shows the preview image of the first video embedded, and all is well.

using Google Drive for desktop

Some points for using Google Drive for desktop - may be easier to use than rclone for some users, but the following can be potential stumbling blocks:

• By default, C:\Users\yourusername\Appdata\Local\Google\DriveFS is used as a temp file location - this can cause problems when trying to "stream" large files with C: drive running out of disk space.
  
  
• To change this setting - and to change from "streaming" to "mirroring" and so on, we need to right-click on the taskbar icon (which is usually hidden by default, so we have to click on the up-arrow to show this icon first) --> the Gear icon --> Preferences then again click on the Gear icon in the window which comes up. A bit clunky.
  
  
• This is the interface now (November 2022), but may change in the future.
  
  
• Edit: Google Drive for Desktop does not work very well with very large files. rclone is better in such cases. For example, see my next post.
  

rclone and out of band flow authentication error

I needed to copy some files from our server to Google Drive. Tried to download directly to my local machine where google drive desktop had been installed. Failed twice, then realized that the error shown by Filezilla was "unable to write to local file" and not connection error - the temp folder was running out of disk space, I guess. So, tried rclone to directly transfer from the server to Google Drive. 

Rclone initially gave the out of band flow authentication error. Then I installed version 1.6 by downloading the deb file,

wget https://github.com/rclone/rclone/releases/download/v1.60.0/rclone-v1.60.0-linux-amd64.deb

sudo dpkg -i rclone-v1.60.0-linux-amd64.deb

rclone config

With that, when I choose n for whether to use auto config, it asks for a local machine with a browser which has rclone installed. So I downloaded rclone to my local machine, ran rclone as requested with the required parameters as prompted, pasted the auth key, and could create the GDrive remote. 

Since I needed to copy only a single file, instead of mounting the remote, I just did

screen 

rclone copyto src/file/path remotename:/path --progress 

The file copy was going at around 2 or 3 MBytes per second when I was transferring via my local machine on BSNL fiber with Google drive desktop in "streaming" mode. 

When doing direct server to server transfer from Azure to GDrive using this rclone method, the transfer was going approx 10x faster. 200+ Mbps.

adding google analytics 4 code to Blogger

There seem to be many requests for adding google analytics 4 to Blogger, but as of now, it looks as if support is only through adding the javascript code as a custom HTML/JS block in Blogger's Layout settings - screenshot below.

Building the cordova-based Moodle app v4.0.2 on Github Actions

It turns out that building the latest version (4.0.2) of the Moodle app on github actions is much easier than on travis-ci, because

• The Travis build environment seems to have Java 8 by default, and the latest version of the Android SDK (for SDK level 31 and above) need Java 11 
• Github actions seem to have all the latest build tools etc pre-installed.

The github actions yml file, which has all the commands necessary for a debug build, is at 
https://github.com/hn-88/LMSapp/blob/main/.github/workflows/buildSSSVV.yml 

Can refer to Notes on building the cordova-based Moodle app for Android for more info on building release version, etc.

Edit: In my implementation, I used manually triggered actions - https://levelup.gitconnected.com/how-to-manually-trigger-a-github-actions-workflow-4712542f1960 - by having the line
on: workflow_dispatch

exploring cross-platform GUI coding

Checking out wxwidgets - seems to be best with code::blocks ? 

wxCrafter - The wxWidgets GUI RAD Tool (codelite.org)

Demo of wxWidgets wxCrafter CodeLite C++ on Ubuntu Linux - YouTube

subtitling and flat video workflow

For subtitling and displaying a flat video on the planetarium dome,

• Aegisub for subtitling
  
  
• Avidemux for resizing - for a quick and dirty solution, adding 360 pixels of black padding above a 720p video (and suitable padding on left and right) gives an acceptable video.
  
  
• Better aesthetics can be done with adding a background video with an alpha mask (created in Gimp) and overlaid with Avisynth's Overlay and Mask. Or Layer, which I used, similar to the example in the documentation, where our video was the bg, and the "background video" was on top.
  

  bg  = AviSource("the-flat-video.avi").ConvertToRGB32      
  mk  = ImageSource("mask.jpg").ConvertToRGB32       
  top = AviSource("video-to-fill-rest-of-space.avi").ConvertToRGB32.Mask(mk) 
  Layer(bg, top)                                   

• For wider dome coverage - like 180 degrees horizontal coverage for a 1080p flat video - we can use OCVWarp after overlaying the video on a 4k background. Or use OCVvid2fulldome if final output is a square-frame fulldome file.
  

project language shown as Mercury in Github

One of my github repos was being displayed as having Language: Mercury instead of C or C++. The problem and resolution is at https://github.com/hn-88/OCVWarp/issues/16 - following https://dev.to/katkelly/changing-your-repo-s-language-in-github-5gjo

Basically I just needed to create a .gitattributes file, with the line

*.m linguist-detectable=false

Notes on making an ad-hoc query for Moodle

Expanding on the problem and resolution at
https://github.com/hn-88/ad-hoc-moodle-database-queries/issues/1

This seems to be a classic illustration of what can go wrong when we copy code written by someone else without understanding it. Apparently Github's Copilot also has such issues.

I had blindly copied the code from https://stackoverflow.com/questions/15938185/selecting-all-the-files-along-with-their-paths-of-a-course-in-moodle without verifying if the resource join is correct. 

select count(*) as "All ppt(x) and mp3/mp4 files",

count(distinct f.contenthash) as "Only unique ppt(x) and mp3/mp4 files"

from {files} f

INNER JOIN {context} c ON f.contextid = c.id

INNER JOIN {resource} r ON c.instanceid = r.id

INNER JOIN {course} co ON r.course = co.id

where (f.filename like '%ppt%' or f.filename like '%mp%')

and co.fullname like :coursename

Apparently, in our case, where the files are inside folders, this is absolutely wrong. The contextid depends on what is mentioned in contextlevel. If contextlevel is 70 (= modules), contextid points to the entry in course_modules table. Not resource table!

So, the corrected query would be

select count(*) as "All ppt(x) and mp3/mp4 files",

count(distinct f.contenthash) as "Only unique ppt(x) and mp3/mp4 files"

from {files} f

left join {context} cont on f.contextid = cont.id

left join {course_modules} vcm on cont.instanceid = vcm.id

where

vcm.course=co.id

and vcm.module=8

and cont.contextlevel = 70

and f.component ='mod_folder'

and (f.filename like '%.ppt%' or f.filename like '%.mp%')

and co.fullname like :coursename

Unfortunately, the documentation for this was not very easily available. For example, where do we see that contextlevel=70 corresponds to modules? Only in this deprecated page, https://docs.moodle.org/dev/Roles_and_modules - archived link.

So, my notes for arriving at the above conclusion:

Symptom - when relying on contextid to determine course, a lower number of files is reported in the queries

https://moodle.org/mod/forum/discuss.php?d=214037
did not seem to be correct, since it was returning all sorts of files.

So, decided to try and understand the meaning of contextid first!

via https://moodle.org/mod/forum/discuss.php?d=200625
"In files tables, component col is the where (course, backup, mod_folder)" - so, to find list of files, we can search files table with
contextid =962 and component ='mod_folder'

To get the relevant contextid, I had used the path in LMS, which was https://server.tld/pluginfile.php/962/mod_folder/content/0/Teaching%20Resources/DD_Living%20on%20the%20Streets.pptx?forcedownload=1
Todo: get it from the database

In context table, id=962 has instanceid=667, which is what is seen in the html,
for the div
  | id="module-667"
  | data-for="cmitem"
  | data-id="667"

 Finally! Explanation of context table - https://moodle.org/mod/forum/discuss.php?d=202588 

 From https://docs.moodle.org/400/en/course_display
"contextlevel 70 is modules, then instanceid points to the mdl_course_modules table" so in the relevant context table entry, the instanceid points to course_modules table....

Checking the entry in course_modules table, id=667, course=34 (correct), module=8 (=folder, from modules table). With this info, I could put together the "correct" query above.

credit card transaction declined - Paypal India

The Canara bank credit card used as Paypal payment method for one of our units declined payment. Probably due to their tokenization deadline - 

https://canarabank.com/UPLOAD/NewsImage/Annexure-1-FAQ-Circular_Tokenization_For_Cards.pdf

My HDFC card worked fine - probably because I had enabled tokenization and online payments much earlier.

zoom effect with javascript

One possible implementation of eye-candy - a zoom effect for images on hover using javascript - https://www.youtube.com/watch?v=cJgOfuzYpM0 - (I've not tried it out yet).

Notes on building the cordova-based Moodle app for Android

A collection of the various issues faced and workarounds - building older and newer versions of the Moodle Android app. Many of these might be applicable to other cordova apps too. This post encapsulates work done, trial and error, over a one-month period.

Edit: Building v4.0.2 is described in a newer post.
Later Edit: Building v4.1.0 - release apk in Github Actions - is described in a newer post.

• Build notes on successfully building v4.0.1 - https://hnsws.blogspot.com/2022/09/modify-and-build-moodle-app-version-4.html
  
  
• Unfortunately, v400 branch of the moodle app has some really nasty bugs like https://tracker.moodle.org/browse/MOBILE-4135
  
  
• Incomplete Travis yml for building v4.0.1 - https://github.com/hn-88/moodleapptravistest/blob/937a2842c351c1cf13fe8270d30ed4521d481376/.travis.yml - there needs to be
  npm run build:prod
  before
  npm run prod:androidnorun
  
  
• As seen in the above yml, though the travis build image only has SDK 29 pre-installed, we can install SDK 30 and the relevant build tools using sdkmanager,
  yes | sdkmanager "build-tools;30.0.3"
  
  
• Travis builds were relatively easier to set up since the repo already had the appropriate build image set up in the included travis yml. Appveyor build did not work, probably because of not setting up the environment properly. Probably can be done with commands like sdkmanager above for SDK 31 and above, since they use openjdk and don't depend on Oracle's java.
  
  
• Travis yml for building v3.9.5 - https://github.com/hn-88/moodleappAppveyorTrial/blob/44a5c27212559c18b78f70d12fff952c6ed0c8fd/.travis.yml - but the resulting app doesn't work, I think, maybe due to ionic/android sdk incompatibilities.
  
  
• For the above travis yml, commands like
  npm run build:prod
  use scripts which are in package.json scripts section. We can instead use commands like
  npx ionic build --prod
  directly in the yml, so that we don't have to edit package.json. Another caution is that the options needed for the build command may vary for different versions of cordova. So we need to check the documentation or rely on the existing commands in the moodleapp repo. Some links on command-line build:
  
  https://stackoverflow.com/questions/10930331/how-to-sign-an-already-compiled-apk
  or
  https://stackoverflow.com/questions/26449512/how-to-create-a-signed-apk-file-using-cordova-command-line-interface
  make a build conf and then ...
  
  
  (or import android project to windows and build - that does not work out of the box - there are some paths which have / instead of \ or vice-versa, and some amount of path mangling has to be done.
  https://stackoverflow.com/questions/44959885/cordova-to-android-studio-error-malformed-uxxxx-encoding)
  
  

  Examples at

  Building an Android Project - Travis CI (travis-ci.com)
  

  Calc/.travis.yml at master · samlsso/Calc (github.com)
  
  
  
• Build notes on successfully building v3.9.4 - https://hnsws.blogspot.com/2021/04/customizing-and-building-moodle-app.html
  
  
• The zip path traversal vulnerability in v3.9.4 can probably be fixed easily using https://tracker.moodle.org/browse/MOBILE-3949
  which points to the diff at
  https://github.com/moodlehq/moodleapp/pull/3024/files
  
  
• For older versions of the app, we need to specify an older version of cordova-res, since the latest version of cordova-res has dependencies which need npm>12
  
  npm i -g cordova-res
  /home/homedir/.nvm/versions/node/v11.15.0/bin/cordova-res -> /home/homedir/.nvm/versions/node/v11.15.0/lib/node_modules/cordova-res/bin/cordova-res
  
  > sharp@0.29.3 install /home/homedir/.nvm/versions/node/v11.15.0/lib/node_modules/cordova-res/node_modules/sharp
  > (node install/libvips && node install/dll-copy && prebuild-install) || (node install/can-compile && node-gyp rebuild && node install/dll-copy)
  
  sharp: Installation error: Expected Node.js version >=12.13.0 but found 11.15.0
  sharp: Please see https://sharp.pixelplumbing.com/install for required dependencies
  
  
  
  from older versions at
  https://www.npmjs.com/package/cordova-res
  
  
  
  npm i -g cordova-res@0.15.1
  
  worked.
  
  
  
• Probably the travis yml above (for 3.9.5) will work for building 3.9.4, but we would need to use SDK 29, because
  
  
• Though we can change the SDK version in Android Studio 4.1 and build v3.9.4 using targetSdkversion=30, the resulting app has the problem, "Could not read database" error when trying to log in to Moodle. If we use SDK 29, this issue is not seen. Probably due to this - https://github.com/andpor/react-native-sqlite-storage/issues/462
  
  
• In any case, Google play store mandates target SDK 31 from November, and Moodle app builds targeting SDK 31 just will not work. As mentioned in https://stackoverflow.com/questions/68344424/unrecognized-attribute-name-module-class-com-sun-tools-javac-util-sharednametab,
   need to change jdk to 11 for sdk 31 and above. Then the cordova build fails for cordova<11. And older versions of the Moodle app are incompatible with higher versions of npm which are needed for cordova 11. How to target SDK 30 is mentioned at https://github.com/apache/cordova-android/issues/1178#issuecomment-814116954 - but those changes were not enough in my case. I first tried
  grep -rn . -e "targetSdk"
  via https://stackoverflow.com/questions/16956810/how-to-find-all-files-containing-specific-text-string-on-linux
  and then used Android Studio's find / replace in files to change targetSdkversion in a couple of instances of files not mentioned above, so files to modify were
  /platforms/android/project.properties
  /platforms/android/CordovaLib/project.properties
  /platforms/android/app/src/main/res/xml/config.xml
  as well as
  /platforms/android/build.gradle
  /platforms/android/gradle.properties
  (which had the cdvTargetSdkVersion=29 line)
  
  
• For changing version code of the app - Google Play store insists on incrementing version code - we need to change it in src/config.json for v3.x and config.xml for v4.x and then doing the whole npx ionic cordova remove android, then add, etc. Other methods, like changing flavors in Android Studio etc, don't work properly, with the version inside the app showing an older version if we don't do it this way. That makes sense, since the ionic build is what creates the webpages which are seen as screens in the app.
  
  
• If we try build tools 31 or later, older versions of Android Studio will complain that the build tools are corrupted. https://stackoverflow.com/questions/68387270/android-studio-error-installed-build-tools-revision-31-0-0-is-corrupted
  If we just try copying d8 to dx as mentioned by some of the people in the above post, that doesn't help with fixing all the incompatibilities. Easier to just build with compatible versions of cordova / gradle / android studio. That is, old code --> old cordova --> old build tools + old android studio.
• https://developer.android.com/studio/releases/build-tools
  mentions how we should target a particular build tools version.
  
  
  "To use a different version of the build tools, specify it using buildToolsVersion in your module's build.gradle, as follows:
   
  
  android {
      buildToolsVersion "30.0.2"
      ...
  }"
  
  
  But instead, what I was doing was, if it complained that build tools 31 is corrupted, I would just unselect it from the SDK manager window - uninstall it. Then, by default, Android Studio would use the next highest available build tools version to build. 
  
  
• We cannot build v3.9.4 with later versions of Android Studio - version 4.2, 2021.3 and so on. Probably because they use later versions of gradle which are incompatible with cordova 9. I believe gradle version 6.x is required for compiling the app without errors, while later Android Studio versions need versions 7 and above.
  
  
• https://stackoverflow.com/questions/32258183/how-to-install-an-older-version-of-android-studio
  https://developer.android.com/studio/archive
  The links to 4.2.x and 4.1.x tar.gz files were working when this was written.
  
  
• Changing the gradle versions in a project in Android Studio can be done via various methods as described at https://www.geeksforgeeks.org/how-to-update-gradle-in-android-studio/
  
  
• Changing the desired gradle and android gradle plugin versions in Android Studio is via File --> Project Structure --> Project
  
  
• While testing builds, it takes around 90 seconds for the moodle app to be available in the Chromium browser after issuing npm start - does ionic compile etc.
  
  
• If we want to build entirely from the command-line, without using Android Studio, that also should be possible if the correct versions of gradle and SDK tools are installed. Simply trying
  sudo apt install gradle
  npm run dev:android
  said "Run successful" for v400 app (cordova 10) on Linux Mint 21 (based on Ubuntu 22.04). But there may be incompatibilities with gradle versions. sdkman can help.
  
  error:
  Unrecognized option: --illegal-access=permit
  Error: Could not create the Java Virtual Machine.
  Error: A fatal exception has occurred. Program will exit.
  
  gradle -version
  
  ------------------------------------------------------------
  Gradle 4.4.1
  
  https://stackoverflow.com/questions/68565856/unrecognized-option-illegal-access-permit-when-running-cordova-build-android
  
  
  
  https://gradle.org/install/
  
  
  curl -s "https://get.sdkman.io" | bash
  
  source "/home/myhomedir/.sdkman/bin/sdkman-init.sh"
  
  sdk install gradle 7.5.1
  
  
• After gradle errors due to incompatible versions, just uninstalling gradle with
  sudo apt remove gradle
  was not sufficient. Android Studio still would not build the project. Found that we have to remove both hidden folders
  ~/.gradle and projectfolder/.gradle
  for all the crud to be removed so that Android Studio would commence to build again properly.
  
  
• Documentation about signing the apk built using cordova - 
  
  https://cordova.apache.org/docs/en/latest/guide/platforms/android/index.html#extending-buildgradle
  

  
  

  https://stackoverflow.com/questions/26449512/how-to-create-a-signed-apk-file-using-cordova-command-line-interface
  

  
  

  maybe just give the --keystore & pw? as variables?
  
  
• If signing key is not provided and a release apk is built, we can sign using apksigner or jarsigner -
  
  android - How to Sign an Already Compiled Apk - Stack Overflow
  or use this GUI tool to avoid typing out all the paths -
  
  Android APK Signer & Aligner - Luke Alderton
  
  
• https://docs.travis-ci.com/user/languages/android/#How-to-install-Android-SDK-components
  is outdated.
  
  
  platform tools 29.0.3 is what is being used, setting target api 30 seems to work.
  
  
  unfortunately, still not signed. (Probably due to incompatible command-line options in the relevant cordova version.)
  
  
  
  Also tried uber-apk-signer
  https://stackoverflow.com/questions/10930331/how-to-sign-an-already-compiled-apk - but that did not work for some reason.