Thursday, October 27, 2022

Notes on making an ad-hoc query for Moodle

Expanding on the problem and resolution at
https://github.com/hn-88/ad-hoc-moodle-database-queries/issues/1

This seems to be a classic illustration of what can go wrong when we copy code written by someone else without understanding it. Apparently Github's Copilot also has such issues.

I had blindly copied the code from https://stackoverflow.com/questions/15938185/selecting-all-the-files-along-with-their-paths-of-a-course-in-moodle without verifying if the resource join is correct. 

select count(*) as "All ppt(x) and mp3/mp4 files",
count(distinct f.contenthash) as "Only unique ppt(x) and mp3/mp4 files"
from {files} f
INNER JOIN {context} c ON f.contextid = c.id
INNER JOIN {resource} r ON c.instanceid = r.id
INNER JOIN {course} co ON r.course = co.id
where (f.filename like '%ppt%' or f.filename like '%mp%')
and co.fullname like :coursename

Apparently, in our case, where the files are inside folders, this is absolutely wrong. The contextid depends on what is mentioned in contextlevel. If contextlevel is 70 (= modules), contextid points to the entry in course_modules table. Not resource table!

So, the corrected query would be

select count(*) as "All ppt(x) and mp3/mp4 files",
count(distinct f.contenthash) as "Only unique ppt(x) and mp3/mp4 files"
from {files} f
left join {context} cont on f.contextid = cont.id
left join {course_modules} vcm on cont.instanceid = vcm.id
where
vcm.course=co.id
and vcm.module=8
and cont.contextlevel = 70
and f.component ='mod_folder'
and (f.filename like '%.ppt%' or f.filename like '%.mp%')
and co.fullname like :coursename

Unfortunately, the documentation for this was not very easily available. For example, where do we see that contextlevel=70 corresponds to modules? Only in this deprecated page, https://docs.moodle.org/dev/Roles_and_modules - archived link.

So, my notes for arriving at the above conclusion:

Symptom - when relying on contextid to determine course, a lower number of files is reported in the queries

https://moodle.org/mod/forum/discuss.php?d=214037
did not seem to be correct, since it was returning all sorts of files.

So, decided to try and understand the meaning of contextid first!

via https://moodle.org/mod/forum/discuss.php?d=200625
"In files tables, component col is the where (course, backup, mod_folder)" - so, to find list of files, we can search files table with
contextid =962 and component ='mod_folder'

To get the relevant contextid, I had used the path in LMS, which was https://server.tld/pluginfile.php/962/mod_folder/content/0/Teaching%20Resources/DD_Living%20on%20the%20Streets.pptx?forcedownload=1
Todo: get it from the database

In context table, id=962 has instanceid=667, which is what is seen in the html,
for the div
  | id="module-667"
  | data-for="cmitem"
  | data-id="667"

 Finally! Explanation of context table - https://moodle.org/mod/forum/discuss.php?d=202588 

 From https://docs.moodle.org/400/en/course_display
"contextlevel 70 is modules, then instanceid points to the mdl_course_modules table" so in the relevant context table entry, the instanceid points to course_modules table....

Checking the entry in course_modules table, id=667, course=34 (correct), module=8 (=folder, from modules table). With this info, I could put together the "correct" query above.


credit card transaction declined - Paypal India

The Canara bank credit card used as Paypal payment method for one of our units declined payment. Probably due to their tokenization deadline - 

https://canarabank.com/UPLOAD/NewsImage/Annexure-1-FAQ-Circular_Tokenization_For_Cards.pdf

My HDFC card worked fine - probably because I had enabled tokenization and online payments much earlier.

Wednesday, October 26, 2022

zoom effect with javascript

One possible implementation of eye-candy - a zoom effect for images on hover using javascript - https://www.youtube.com/watch?v=cJgOfuzYpM0 - (I've not tried it out yet).

Tuesday, October 11, 2022

Notes on building the cordova-based Moodle app for Android

A collection of the various issues faced and workarounds - building older and newer versions of the Moodle Android app. Many of these might be applicable to other cordova apps too. This post encapsulates work done, trial and error, over a one-month period.

Edit: Building v4.0.2 is described in a newer post.
Later Edit: Building v4.1.0 - release apk in Github Actions - is described in a newer post.


Thursday, October 06, 2022

finding files from linux command line

This is something which I keep needing. For a filename based search,


find /path/to/search -name "*partoffilename*"

For a substring inside the file,

grep -rn /path/to/search -e "substring to look for"
https://stackoverflow.com/questions/16956810/how-to-find-all-files-containing-specific-text-string-on-linux

first arg is path to search
-r recursive
-n show line numbers
-e pattern

Wednesday, October 05, 2022

removing large files from git history

If large files have been committed by mistake, they add to extra file-transfers for every new git clone. So, followed this - 

https://www.baeldung.com/git-remove-file-commit-history#using-git-filter-repo

After that, needed
git remote add origin git@github.com:usrname/reponame.git
and
git push -f origin hn # push -f to force push
 

setting up git

In addition to my earlier git post, the following are useful resources for setting up git with global settings locally and so on. 

For enabling push to remote repos, we can set up the username and email globally - https://kbroman.org/github/tutorial/pages/first_time.html

git config --global user.name "My name"

git config --global user.email "email@example.com"

I had already set up ssh key etc. so 

ssh -T git@github.com

said successfully authenticated. 

More info:

https://docs.github.com/en/get-started/getting-started-with-git/managing-remote-repositories

https://docs.github.com/en/get-started/importing-your-projects-to-github/importing-source-code-to-github/adding-locally-hosted-code-to-github

What I did was:

git clone url_of_repo directory_name
cd directory_name
git branch hn # to create this new branch called hn
git checkout hn # if I wanted to edit the new branch

# after edits
git add .
git commit -m "commit message"
git push origin hn 

In travis-ci, I can choose to build my branch by choosing the custom build button, and choosing my branch from the drop-down.


Monday, October 03, 2022

dynamic dns solutions with cloudflare

An update to my previous post about dynamic dns - found a powershell script and a bash script - tested both, working fine - 

https://github.com/fire1ce/DDNS-Cloudflare-Bash

and

https://github.com/fire1ce/DDNS-Cloudflare-PowerShell

These might be the better option - directly setting the A record in cloudflare instead of using a CNAME to NoIP etc, because then even if the router is replaced - as happened in my case - there would be no leakage of info. 

(This medium post was higher ranked in google,
https://adamtheautomator.com/cloudflare-dynamic-dns/
but uses some Powershell v7 features, so can't be used with Win10 without upgrading Powershell etc. 

https://4sysops.com/wiki/differences-between-powershell-versions/
)

Friday, September 30, 2022

Zoho mail issues

One of our units had problems with their Zoho email either not being delivered at all, or going to spam. The error message indicated DKIM and SPF issues. I pointed them to https://www.zoho.com/mail/help/adminconsole/dkim-configuration.html and they fixed their issue.

Thursday, September 29, 2022

site traffic report from cloudflare

Cloudflare claims to give reports without having to install server-side code, and claims to not track individual users. Our sites' report says, data from 14 of our Cloudflare sites during the month of August  

  • 14 sites
  • Cloudflare served 2.58 TB of data
  • 70.9% Data transfer saved
  • 19.38k firewall mitigated events

Wednesday, September 21, 2022

setting up IIS with dot NET core on Windows 10 - but Linux is better!

Setting up IIS and dot NET core for Windows 10 - this may be slightly different for Windows 11 and or Windows Server OSes.

1. Installing IIS - Control Panel - Programs and Features - Turn Windows Features on or off - choose IIS, IIS manager, (optionally open them up and select all the features like FTP and so on if needed).

2. Installing the dot NET Hosting bundle and deployment - First download and install dot NET core hosting bundle - googling found the updated link here - https://dotnet.microsoft.com/en-us/download/dotnet/thank-you/runtime-aspnetcore-6.0.8-windows-hosting-bundle-installer. Then "publish to folder" in Visual Studio. Then copy to the relevant folder inside the relevant IIS website.

3. Create application in IIS - (a) create a new IIS Application Pool under the .NET CLR version of “No Managed Code”.  
(b) choose Add Application for the relevant website, use the Application Pool listed above, and the folder to which the deployed app was copied
(c) and now if the website is enabled, the app should be working.

Apparently, we can try and run dot NET web apps on Linux also, 

https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#1804-

We can run console app with 
dotnet myApp.dll
 or we can publish a standalone version with
dotnet publish -c release -r ubuntu.16.04-x64 --self-contained
and run it after copying it to the Ubuntu server with
chmod +x ./appname
./appname

as in the post below.

https://stackoverflow.com/questions/46843863/how-to-run-a-net-core-console-application-on-linux

Complete howto on deploying and running on an Ubuntu server - 
https://www.c-sharpcorner.com/article/how-to-deploy-net-core-application-on-linux/

Apparently, there is a nearly 50% performance boost by running on Linux!
https://stackoverflow.com/questions/44334125/net-core-on-windows-vs-linux
https://robertoprevato.github.io/More-about-Linux-vs-Windows-hosted-ASP-NET-Core-applications-in-Azure-App-Service/


Tuesday, September 20, 2022

deleting an Azure free trial and deleting a hotmail account

Earlier, I had posted about stopping the Azure free trial from an unmanaged domain account. This post is about stopping an Azure free trial subscription and deleting a free hotmail account. 

https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/cancel-azure-subscription

portal.azure.com  --> Cost Management + Billing --> Overview --> your plan --> select Cancel --> verify that you want to cancel and select Yes, cancel.

After that, to delete the hotmail account which had that free trial,
https://www.makeuseof.com/tag/how-to-delete-outlook-hotmail-account/

account.microsoft.com --> login --> Your Info tab  --> Scroll down to the Help with Microsoft account  -->  How to close your account --> Choose whether you want Microsoft to retain your data for 30 days or 60 days --> Next etc (various security confirmations)  

This info is probably subject to change as these portals constantly undergo slight revisions in UI.

Monday, September 19, 2022

steps taken for a suspended Google Workspace domain

Unfortunately, one of our domains missed out migration to non-profit edition. Copy-pasting from some email exchanges which happened, to document one possible way of dealing with such a situation.

The following were carried out:

1. For emergency continuation of requireduser@our-lapsed-domain.tld, our-lapsed-domain.tld was upgraded to Business starter. Would cost approx Rs. 250 per month, so instead of continuing this, we would migrate the users.

2. For doing this, the easiest way was to find out the admin account of our-lapsed-domain.tld in the legacy Google Apps for Domains.

Initially, many of our domains had been administered by Mr. MM's team, so I tried searching for their hand-over email which listed all assets and all logins, in which I had also been cc'd - found our-lapsed-domain.tld also in that list. Tried with its administrative account - apparently not logged in for many years - and enrolled in Business Starter.

There were 4 accounts seen - of which only the administrative account and requireduser@our-lapsed-domain.tld had any data.

3. Have started the migration of archived emails from supersededemail@our-lapsed-domain.tld (just for safety's sake, probably these have already been saved earlier) to currentemail@anotherdomain.org

4. Will delete notused@our-lapsed-domain.tld - since it seems to be not in use.

5. Since the current email id used by Dr. is in ourdomain.org, P has been requested to ask S to create requireduser@ourdomain.org. Once this is done, requireduser@our-lapsed-domain.tld will be migrated there - perhaps an outage of a day, which can be decided by requireduser at their convenience. Outage will be only in terms of accessing old emails. Fresh emails will go to requireduser@our-lapsed-domain.tld without any outage.

6. Once the new email id is ready, we can start forwarding new email which goes to requireduser@our-lapsed-domain.tld to requireduser@ourdomain.org - for this, I have used https://improvmx.com/
in the past, and seems to be a good service.

I can set up forwarding for supersededemail@our-lapsed-domain.tld to currentemail@anotherdomain.org also if desired. And adminaccount to orgadminaccount@gmail if desired.

7. Future - checking if there are any other domains which have been orphaned like this - I will check
(a) the hand-over email from MM's team
(b) the MX records in all domains which are under our control
and make a list of all domains which have emails enabled, and who administers them.

----
Later, to S   ...
---

So, the steps would be:
1. I would reset the pw of requireduser@our-lapsed-domain.tld
2. I would log in to requireduser@our-lapsed-domain.tld and ensure that IMAP is set to ON in gmail settings.
3. I would ensure that "Allow insecure apps" is set to ON in Google account settings
4. I would then give you the password, which you can use in Google Workspace Admin panel to do the migration.
5. The screenshots below explain the process.
6. I can also call you at 2.30 pm tomorrow if you want. Google's documentation for this is at https://support.google.com/a/answer/9476255

Scroll down to Data Migration



Inside Data Migration, choose the Add user link - 



In the next screen, you may see another option,

Choose source. There, you can choose "Google Workspace" as the source. Then, it will use IMAP with imap.gmail.com as the source. If it gives an option to choose the date, give a date very long back, like 2001 or something, so that all the emails will be selected. Then if it asks whether
Migrate Junk
Migrate Deleted
etc,
just choose the default, which is not to migrate Junk/Deleted. Etc. Then, you would see the following.




There, the source email would be
requireduser@our-lapsed-domain.tld
the Google Workspace Email would be
requireduser@ourdomain.org
and the password would be the password for requireduser@our-lapsed-domain.tld, which I will send you tomorrow.

----
Later, after S had created the new account ...
---


In case you use any other services - google keep, youtube, blogger, calendar, drive, anything else - please take a backup by logging on to requireduser@our-lapsed-domain.tld and then navigating to takeout.google.com

This is important, so please let me know once you have taken the backup (or let me know if you don't use any services except email.)

So, from 2 pm tomorrow or so, you would not have access to requireduser@our-lapsed-domain.tld

The emails will start getting migrated hopefully by 2:30 pm.

Depending on the total disk size as well as the number of emails, this process can take between 1-7 days.
(The migration happens via IMAP,
https://support.google.com/a/answer/9476255?hl=en
In our experience, if total emails size is less than 2 GB, it would take only one day.)

You will start seeing the newest emails - from the last few weeks - within an hour, and the older emails will continue to be added to requireduser@ourdomain.org over the course of the migration.


direct url of play console and google apps script editor url

For uploading or editing apps in the Google Play Store, the play console url, if logged in to multiple accounts at the same time, would be like

https://play.google.com/console/u/6/

(if using the account number 6, etc.)

similar to 

https://drive.google.com/drive/u/6 etc.

The google apps script editor url is of the form

https://script.google.com/u/6/home/projects/project-id/edit

This usually works if we just double-click on the script in google drive if the editor is set to "new editor", but if the editor is set to the "classic" version of GA script editor, double-clicking on a script may not work, we need to use the above url. 

Thursday, September 15, 2022

xfce icons still not showing

Running xfce over VNC on a cloud server, the icons were not showing up, even after

and
installing xubuntu-desktop
Need to check if the  last command in the thread above works,
# gtk-update-icon-cache --force /usr/share/icons/hicolor

Tuesday, September 13, 2022

modify and build the Moodle app version 4+

Edit - There is a newer post on building the release version entirely from Github Actions - for the moodle app v4.1.0.

There are a few changes from my earlier post about building the Moodle app. I'll try to write this post as a bash script which could be run on an Ubuntu-based machine which does not have the build environment set up, so that it would be possible to use something like Travis CI to build it. The current script below will not work without a gui on headless servers, since there are some steps like previewing in Chromium, installing android studio etc require a GUI. 

# the following are for a headless server, to get a gui over VNC

https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-18-04

sudo apt update

sudo apt install xfce4 xfce4-goodies -y

sudo apt install tightvncserver -y

vncserver

# This starts up VNC server and asks to set up a password.

# The following will be after logging on to VNC - my preferred method is 

# to tunnel port 5901 over ssh.

# following the documentation at https://moodledev.io/general/app/development/setup

sudo apt install git chromium-browser -y

# from https://github.com/nvm-sh/nvm#installing-and-updating

wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash

source ~/.bashrc


git clone https://github.com/moodlehq/moodleapp.git
cd moodleapp
git checkout v4.0.1

# Here we make all the customizations, described at the bottom of this post

cd ..

git clone https://github.com/ourPrivate/Repo-with-customization-files.git

cd  customizedMoodleApp4changes

cp -Rvf * ../moodleapp

cd ../moodleapp

 

# node version should be < v15, 

# https://github.com/moodlehq/moodleapp/blob/master/package.json#L178

nvm install 14.15.0

npm install

npm start 

# npm start is to see if it works in chromium browser 

# Initially it will open the browser and complain connection refused.

# It will take around 5 minutes to compile, only then a browser reload will work.

# Even after the first compile, it takes around 90 seconds on a 16 GB machine for

# each npm start command - till the chokidar error comes up.

# Fix the chokidar error - too many watch files - with

# https://stackoverflow.com/questions/55763428/react-native-error-enospc-system-limit-for-number-of-file-watchers-reached

# echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p 

# Check if this is OK with

# cat /proc/sys/fs/inotify/max_user_watches 

 

# For building Android or iOS app, need more setup,

# https://moodledev.io/general/app/development/setup

# via my previous post, need to install cordova - 

npm install cordova

npm i -g cordova-res

sudo apt install libsecret-1-dev -y

sudo apt install openjdk-11-jdk -y

sudo add-apt-repository ppa:maarten-fonville/android-studio

sudo apt update

sudo apt install android-studio -y

# This ppa install does not add the required environment variables, so

# to find java home, via https://www.baeldung.com/find-java-home

dirname $(dirname $(readlink -f $(which javac)))

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

export PATH=$JAVA_HOME/bin:$PATH

export ANDROID_SDK_ROOT=/home/azureuser/Android/Sdk

export PATH=$ANDROID_SDK_ROOT/tools:$ANDROID_SDK_ROOT/tools/bin:$ANDROID_SDK_ROOT/platform-tools:$PATH 

nano .bashrc
# and added all these export commands to the end of .bashrc also

# We may need to install the android-30 SDK separately,

# https://developer.android.com/studio/command-line/sdkmanager

# but it is far easier to do it from within the gui SDK manager

npm run prod:android 

# This will probably fail due to not finding gradle, 

# and at this point, we can just open android studio, 

# import the platforms/android folder, and build there.

# The android studio build may complain that build tools ver 30 are not found.

# The way to install them using SDK manager is to choose

# the SDK Tools tab, and check the "Show package details"

# in order to make the Build tools 30.0.3 show up, which we can mark and install. 


 

# Unfortunately, building this way with the above steps seems to result in

# an app with the default cordova or ionic icons. To change to our icons, 

# we could use Android Studio's Image Asset Studio

https://github.com/ionic-team/capacitor-assets/issues/108 

# But the standard way to generate icons as per

# https://capacitorjs.com/docs/guides/splash-screens-and-icons

# was creating files in android directory instead of platforms/android, so

# since mv cannot do it, 

#https://unix.stackexchange.com/questions/9899/how-to-move-and-overwrite-subdirectories-and-files-to-parent-directory 

cordova-res android --skip-config --copy 

cp -Rvf android/* platforms/android

# Then build with Android Studio. 

# https://stackoverflow.com/questions/37300811/android-studio-dev-kvm-device-permission-denied

# if you want to install an Android emulator

# sudo apt install qemu-kvm

#sudo adduser $USER kvm

# sudo reboot

# To run the emulator outside Android Studio in a separate window, 

# start the emulator before opening the project

# https://stackoverflow.com/questions/70986530/android-studio-emulator-in-a-separate-window 

To sign the app with our existing keystore, followed a procedure similar to my previous post, with a change that the location of the Build types tab has changed in the latest 2021.2.1 chipmunk version of Android Studio. In short, following the official documentation (which shows screenshots of the older version)

  1. Project window --> right-click on app --> Open Module settings


  2. Modules on LHS pane --> app --> Signing configs tab --> create a new signing config by giving it some name like releaseconfig


  3. Modules --> app --> Build types tab --> choose the newly created signing config under "Signing" for the release build. (Extreme scrolling to the right was needed before the drop-down was visible, in one case)


  4. Build variants --> choose the release build



The app build so far was fine. But unfortunately, the 4.00 branch of the Moodle app has show-stopping bugs - 

https://tracker.moodle.org/browse/MOBILE-4135

https://tracker.moodle.org/browse/MOBILE-4084

So, I'll continue trying fixes for these in separate posts. One option might be to use the 3.9.4 or 3.9.5 version and fix only the zip path traversal vulnerability in them - 

https://tracker.moodle.org/browse/MOBILE-3949





Virus and threat protection not opening in Windows 10

After restoring Windows 10 on a refurbished mini-desktop purchased on Amazon, found that the Start button was not displaying any content when left-clicked, and the Virus / threat protection / Windows Defender settings page was also not opening. Tried these steps, https://www.thewindowsclub.com/virus-and-threat-protection-not-working-on-windows

  • cmd as administrator using right-click of start menu
  • ran System File checker with
    sfc /scannow
    - this showed some errors, and said they were fixed.
  • ran Deployment Image Servicing and Management with
    DISM.exe /Online /Cleanup-image /Restorehealth

It also did not report any issues, but the problem continued. Then tried anti-rootkit scan from Malwarebytes - it found and fixed some issues. Rebooted, ran another malwarebytes full scan, did windows update. Then Windows Defender and Virus threat protection page started opening. Start menu (left click) still doesn't work, but other things seem ok.

Make Windows boot by Default in Windows + Linux Mint Dual Boot System

Via https://www.techmesto.com/set-windows-as-default-in-linux-dual-boot/

sudo nano /etc/default/grub

and edit the line which says

GRUB_DEFAULT=2

(the entries start from 0, up to down.)

Monday, September 12, 2022

hotstar scaling

Happened to come across this video - Scaling hotstar.com for 25 million concurrent viewers - interesting to note the types of challenges when scaling up with record-breaking numbers of users. And there's a whole channel with such tech talks, HasGeekTV

Friday, September 09, 2022

Personal finance apps - UI - SIP delete in Groww and Valueresearchonline

In the Groww app,
--> Mutual funds tab (at the bottom)
--> Dashboard tab (at the top)
--> SIP is shown on top - click on the arrow to the right
click on the relevant sip's arrow to the right
Cancel SIP button on top.

In valueresearchonline.com's portfolio manager,
--> My investments
--> Overview
--> Action drop-down - last column on the right
--> Edit/view SIP / SWP etc