Azure AD graph retirement

 There was an email from Microsoft Azure, "Migrate Service Principals from the retiring Azure AD Graph APIs to Microsoft Graph" - clicking through, the recommendation only showed the id of the function or resource using the deprecated API, and did not provide any other information.

After an internet search, this faq showed how to identify it - 
https://learn.microsoft.com/en-us/graph/migrate-azure-ad-graph-faq#as-an-it-admin-how-do-i-identify-apps-in-my-tenant-that-use-azure-ad-graph

(It was not an application which I had created, it says the owner is "Power Virtual Agents Service" - so no action was taken. More info - https://techcommunity.microsoft.com/blog/azure-ai-services-blog/use-your-own-data-to-create-a-power-virtual-agent-with-azure-openai-service/3891860 )

automating user preferences modifications on Moodle - SikuliX

We had to change a large number of users' forum preferences in Moodle as noted in the previous post. Checking the database for changes when such changes were made, it appeared that new entries would be added to prefix_user_preferences table, with 

name = message_provider_mod_forum_digests_enabled

value = popup
and so on.

Since this was not as straightforward as just changing a flag in a user preferences table, I thought it would be safer to do it via the Moodle UI instead of messing around with database changes. 

To automate the task of changing 200+ users' preferences, I once again took the help of SikuliX. Once again, I chose the simplistic method of making SikuliX scripts with hard-coded Location(x,y) values. Using the 'Run in slow motion' mode, which shows the Location with a red cross-hair, I used moveMouse commands to find the x and y co-ordinates of the points where I wanted the clicks to happen. Unfortunately, the points were not directly 1:1 corresponding to x and y co-ordinates of my 1920x1080 fullscreen capture - Locations based on those co-ordinates threw up errors that the co-ordinates did not correspond to any location (being out of range). 

With the Edge browser screen set to approx. 80% zoom in order to show all the elements we needed on the single screen, the two scripts to update the Moodle preferences pages were as follows.

Documents > notifpref.sikuli > notifpref.py

from time import sleep

sleeptime=0.5

sleeptillpageload=2.0

urlbar=Location(925,64)

test1=Location(1300,746)

test2=Location(1405,564)

webpref=Location(1180,746)

emailpref=Location(1300,746)

id=126

while (id<345): 

  click(urlbar)

  sleep(sleeptime)

  sleep(sleeptime)

  click(urlbar)

  #type(BACKSPACE 3 times)

  # https://answers.launchpad.net/sikuli/+faq/1444

  type("\b")

  type("\b")

  type("\b")

  type(str(id))

  #type ENTER

  type("\n")

  #mouseMove(test1)

  popup("waiting for no error")

  #sleep(sleeptime)

  #mouseMove(test2)

  mouseMove(webpref)

  #sleep(sleeptime)

  click(webpref)

  #mouseMove(emailpref)

  #sleep(sleeptime)

  click(emailpref)

  id=id+1

  #popup("waiting for no error")

Documents > forumprefs.sikuli > forumprefs.py

from time import sleep

sleeptime=0.5

sleeptillpageload=2.0

urlbar=Location(925,64)

subjectonly=Location(700,525)

test2=Location(505,704)

savebutton=Location(505,704)

emailtype=Location(700,446)

id=4362

url="https://ourserver.org/user/forum.php?id="

while (id>4135): 

  click(urlbar)

  #select all

  type("a",KeyModifier.CTRL)

  #type(paste the url already copied)

  type("v",KeyModifier.CTRL)

  type(str(id))

  #type ENTER

  type("\n")

  popup("waiting for no error")

  click(emailtype)

  sleep(sleeptime)

  click(subjectonly)

  id=id-1

  mouseMove(test2)

  click(savebutton)

  popup("waiting for no error")

  

  

Moodle database server overloaded - fixes

Since I'd set up CPU and disk space monitoring emails on some of our servers, I started getting CPU usage alert emails regularly from the mysql database server VM which served some of our Moodle instances. CPU usage kept going up, not declining even during night hours. 

Troubleshooting - 

1. Checked 'currently running tasks' in all Moodle instances - Site administration > Server > Tasks > Tasks running now. Deleted the 3 tasks seen to be running for over an hour (delete modules) in the database table - since these were adhoc tasks, from the {task_adhoc} table based on adhoc taskid.
   
   
2. Checked task logs, filtered for tasks taking longer than 3 seconds, and changed the scheduled tasks settings inside those Moodle instances to reduce the frequency of those tasks. Did this for all five Moodle instances. On the newest instance, changed the following in
   Site Administration > Server > Tasks > Scheduled tasks
   
   • \enrol_category\task\enrol_category_sync -  */5 instead of * (once every five minutes instead of every minute)
     
     
   • \local_edwiserreports\task\send_scheduled_emails 05 instead of */5 (once an hour instead of once a minute)
     
     
   • \local_edwiserreports\task\update_course_progress_data 39 instead of */5
     
     
   •  \mod_customcert\task\issue_certificates_task (once a day instead of every minute)
     
     
   •  \mod_forum\task\cron_task (once an hour instead of every minute)
   • \core\task\search_index_task */47 instead of */30 
     
     
3. The newest instance had thousands of errors being generated by mod_forum - failed to send notifications - and also another instance had thousands of errors from 'failed to send login notifications' at the time of migration in May. Deleted all these failed adhoc tasks from prefix_task_adhoc for both these instances. (For deleting the rows, since the number of rows were so large, it was much faster to create a query like 
   delete from our_db.prefix_task_adhoc where classname like '%send_user_notif%' (24k rows)
   instead of deleting via the DBeaver GUI.
   
   
4.  I had tried various options to prevent mod_forum from throwing errors for 'failed to send notifications'. Finally, the options which worked seemed to be:
   + Enable mobile notifications in Site Administration > General > Notification settings
   
   + Enable Subscribed forum posts notifications for Web and Mobile, also in Site Administration > General > Notification settings, default notification preferences
   
   + Enable Subscribed forum digests notifications for Web, Mobile and Email
   
   + Go to each user and change their forum notification preferences to Digest instead of No digest (each post separate email) - I'll write a separate post on how I automated this - Site Administration > Users , find the user, click on the user profile, Administration > Preferences and there, change forum preferences and notification preferences as above. 
   
   
   

GMail rate limiting and Moodle

Some of our Moodle instances which used XOAUTH to send emails via smtp.gmail.com had issues with temporary authentication failures. Apparently GMail's smtp servers have started rate limiting after approx 100 emails were sent in 3 minutes by Moodle. Mostly mod_forum email digests.

Since Moodle itself doesn't seem to have any provision for rate limiting, we need to set up some mail relay which can retry mails which get throttled.  

SSMTP which is currently installed on the server, doesn't seem to support any advanced features like rate limiting. 

Since I'm familiar with Postfix, looked up ways to send emails from Postfix through google's smtp servers - https://computingforgeeks.com/configure-postfix-to-relay-emails-using-gmail-smtp/

After setting up postfix as above, and changing the outgoing mail configuration on our Moodle instances to the default values (which would use the server default, ie postfix), emails seem to be going out fine. 

For checking the postfix logs for errors, 

journalctl -t postfix/smtp | more
journalctl -t postfix/smtp -f # for tailing the logs

With the postfix config with a single relayhost, there are a few errors once in a while "Network unreachable" but a second later, the mail gets sent. So, use of postfwd or extra config was not needed. If needed in future, multiple postfix instances or the use of multiple relay hosts based on authentication might be options.

Currently sending two large instances' emails through these, 250 emails have gone out last night with no problems as seen via the gmail interface in the sent folder.

 

limiting download speeds with trickle

Running Sheepit render farm in the background on one of our machines, I wanted to limit the download speed of the Sheepit client, since office bandwidth was limited. 

First tried wondershaper and tc - did not work - finally ended up with trickle. Modified the desktop shortcut launcher I use to the command

trickle -s -d 1024 -u 1024 java -jar sheepit-client-ver.jar

Earlier tries and misses, and links for reference - 

https://securitynetworkinglinux.com/how-to-shape-traffic-using-wondershaper-on-ubuntu-20-04-cli/

https://askubuntu.com/questions/1523362/server-24-04-only-starts-a-single-interface-on-boot

https://superuser.com/questions/1053003/what-is-the-difference-between-eth1-and-eno1
We have eno1 instead of eth0.

Used git clone instead of the apt version as per
https://github.com/magnific0/wondershaper

Then, syntax has changed. But errors,
sudo ./wondershaper -a eno1 -d 1024
Error: Exclusivity flag on, cannot modify.
Error: Exclusivity flag on, cannot modify.
Error: Exclusivity flag on, cannot modify.
RTNETLINK answers: File exists

Trying tc with this syntax from
https://superuser.com/questions/1598721/adding-both-delay-bandwidth-restrictions-via-tc

sudo tc qdisc add dev eno1 root netem rate 10mbit
no errors shown
sudo tc qdisc add dev eno1 root netem rate 1mbit
Error: Exclusivity flag on, cannot modify.

The 2nd command above was with protonvpn on.

sudo systemctl restart systemd-networkd

sudo tc qdisc add dev eno1 root netem rate 1mbit

Error: Exclusivity flag on, cannot modify.

 

https://www.cyberciti.biz/faq/linux-restart-network-interface/
sudo ifdown eno1
ifdown: command not found.

sudo systemctl restart systemd-networkd

 

sudo tc qdisc add dev eno1 root netem rate 5mbit
no errors,
but when running protonvpn, this has no effect - download is running at 2.4MB/s

 

trickle -s -d 1024 firefox
Command 'trickle' not found, but can be installed with:
sudo apt install trickle

sudo apt install trickle
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package trickle is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'trickle' has no installation candidate

https://ubuntu.pkgs.org/22.04/ubuntu-universe-amd64/trickle_1.07-11_amd64.deb.html

 
wget http://archive.ubuntu.com/ubuntu/pool/universe/t/trickle/trickle_1.07-11_amd64.deb
Installed OK.

But man trickle says,

Furthermore, trickle will not work with statically linked executables, nor with setuid(2) executables.

 

Also available from source, https://github.com/mariusae/trickle

 

 

some blender ideas I have not yet tried

Shortcut to creating rocket scene - from https://sketchfab.com/3d-models/gslv-mk3-0426922358b4444f9887bcd551d3a5cb - zoom in the view, take screenshots, then composite it with Earth background.

Editing Blender Image Texture with Gimp - How to Link Blender with any External Image Editor! (youtube.com) - as of now, only directly edited the png texture file(s) by loading them into Gimp outside Blender.

How to make millions of bodies - https://www.youtube.com/watch?v=CQ9VmCN2EsE - How to Render Millions of Objects in Blender.

Rigid bodies physics - falling etc https://www.youtube.com/watch?v=nHVYYMG3QVY

DNA in Blender in one minute - nicely explained - https://www.youtube.com/watch?v=xgPlgiOQWPA

Biochemistry L12: Building Bacteria (E. coli) in Blender - https://www.youtube.com/watch?v=XoDDNCWyziI - that channel has lots of Molecules, Chemistry, proteins, petri-dish etc Blender tutorials. @LuminousLab, "Blender for Scientists".

Smart UV project - for easier uv unwrapping of simple objects - https://www.youtube.com/watch?v=qa_1LjeWsJg - UV > Smart UV project - https://youtu.be/qa_1LjeWsJg?feature=shared&t=820 - This did not seem to work for the cubes - would always turn out sideways for some cube faces. Did rotation and rescaling of the uv mesh manually.

Idea for Swami video composition - or a sequence of stills - 5 stills left to right, come at 1 sec intervals, 4 stills above that (found in testing that one layer of stills looks better than 2 layers.) Each frame changes every second? May not be required - could have new frames appearing every second, then after 5 seconds, the first frame changes, etc. Change would be like a page turn? - or not - quick fade to black and change also works. Another option: A large photo album, with pages turning. Each page has 2 + 2 photos? This might need animation. So, earlier option might be easier. Page turn tutorial - https://www.youtube.com/watch?v=K3lfNXAZblA

Keyframe the multiply factor, to change the speed of clips - to slowly speed up or slow down clips - https://docs.blender.org/manual/en/latest/video_editing/edit/montage/strips/effects/speed_control.html - as of now, used only the "stretch" method, which is the default for the Speed Control effect, which is easy to use. We can cut clips with Shift-K and apply different speeds etc.

Privacy blur mask in Blender VSE - https://www.youtube.com/watch?v=v0qoIRKNtnE - 

Set active camera - animate camera smoothly - https://www.youtube.com/watch?v=a7qyW1G350g

How to use google colab to render blender files - Speed Up Your Blender Renders with Free Google Colab- using Sheepit Render Farm instead as of now.

3d text in Blender - Add a text object, choose the font in the object's properties. 3D Text in Blender: Everything You Need to Know! - but has problems with Hindi rendering etc

human body anatomy - using the free z-anatomy blender model

Led on by this video, though the model was not found in the link in the description of the video, found it with a github search at Z-Anatomy/The-blend: Z-Anatomy blender template

As mentioned in the video, importing single collections like skeletal system takes a minute or so, importing multiple collections or opening the entire blend file takes a bit longer.

To import the whole thing and then save-as, deleting those features which we don't want - 

• to delete text labels, we can select all text labels by going to Object menu, Select all by type and choosing Text
  
  
• Rendering in eevee or cycles causes a cut-away view, while workbench rendering engine gives a whole face. This is probably due to some "hide" setting somewhere, but I just went with workbench rendering instead of trying to troubleshoot.
  
  
• to turn on visibility for all objects in a collection, we have to Alt-left-click on the render / viewport icon as required. ALT clicking is the trick.
  
  
• to move camera to current view, the shortcut is Ctrl+Alt+Numpad 0. Or can change the shortcut on machines which don't have the numpad, in File > User Preferences > Input, search for camera, the pref is 'align camera to view'. For Blender 4.2, the menu is Edit > Preferences > Keymap
  
  
• to prevent camera being deleted with lasso select, just hide it - select the object in object mode, Object menu > Show/Hide > Hide selected (or H is the shortcut key, Shift H to unhide after the delete operations are done.)
  
  
• Working with objects inside a collection - simply selecting the collection and deleting it does not delete the objects inside the collection. To select all objects in a collection - right-click the collection and choose Select objects. Then, Del key or x or object menu > delete
 

ffmpeg commands - image sequence to mp4

Listing some of the ffmpeg commands used in my previous post - 

ffmpeg -r 30 -start_number 1326 -f image2 -i "OpenSpace_%06d.png" -vcodec libx264 -crf 15  -pix_fmt yuv420p mars-valley-2.mp4

#and for reverse,

ffmpeg -r 60 -start_number -3037 -f image2 -i "OpenSpace%06d.png" -c:v libx265 -x265-params lossless=1 zoom-in-to-psn-lossless.mp4

and today,

ffmpeg -r 30 -i "%04d.png" -c:v libx265 -x265-params lossless=1 bloodcells.mp4

and 

ffmpeg -r 30 -i "%04d.png" -c:v libx264  ../bloodcellsx264.mp4

why it is a bad idea to have other people's product name in the name of your product/service

 Copy-pasting from an email exchange, when a well known name was proposed to be used for one of our servers - 

Why it is a bad idea to have other people's product name in the name of your product/service - though legally permissible, it is generally regarded as being due to incompetence or due to malfeasance (trying to scam people to get more traffic.)

https://www.nolo.com/legal-encyclopedia/question-trademark-infringement-use-company-name-28198.html

and so, we should choose another name.

upgrade Moodle 4.4 to 4.5

Since the theme for some of our Moodle instances was "Moove" and this theme had some updates which would work on 4.5 only after upgrading the theme, first changed the theme to the built-in theme "Boost" from the UI, completed the upgrade, upgraded the plugins including the Moove theme, changed back to Moove theme, OK.

The upgrade itself was 

screen
cd /var/www/theLMSgit
sudo -u www-data /usr/bin/php admin/cli/maintenance.php --enable
git config core.filemode false
#git branch --track MOODLE_405_STABLE origin/MOODLE_405_STABLE - this was done earlier.
git checkout MOODLE_405_STABLE
git pull
sudo chown -R azureuser:www-data .
sudo chmod -R 775 .
sudo -u www-data /usr/bin/php admin/cli/upgrade.php

Activity dashboard in google docs

 There was a request from a user that he wanted activity to be turned on - 

he is not able to see who all opened the document through the shared link.

Copy-pasting from my reply - 

There is a misunderstanding about what the 'Activity Dashboard' does do and does not do.

Even though the "Activity Dashboard Settings" is set to ON, the activity will be visible only during the time the viewer is viewing the document, and not later. 

The (relevant) document pdf seems to be shared as "Anyone on the internet with the link". So, even at the time when someone is viewing it, if not logged in, that person will be shown as some anonymous name. For example, "Anonymous Squirrel" in this case - 

Also, the activity dashboard will not show anything until some user edits the document, it will show only the following info:

linux desktop shortcut to open a directory in terminal

Following https://unix.stackexchange.com/questions/62197/how-to-make-a-desktop-shortcut-that-opens-the-terminal-but-under-a-different-d

in our case, xfce4-terminal --working-directory=/the/relevant/directory

Water vapour map

 Some possible data sources for displaying storms etc in OpenSpace - 

http://tropic.ssec.wisc.edu/real-time/mtpw2/product.php?color_type=tpw_nrl_colors&prod=global2&timespan=24hrs&anim=html5

https://bin.ssec.wisc.edu/pub/mtpw2/images/tpw_nrl_colors/global2/202409/comp20240926.000000_tpw.png

https://bin.ssec.wisc.edu/pub/mtpw2/images/tpw_nrl_colors/global2/202409

Hurricane Helene.

blender rendering settings

 By trial and error, for neuronDuplicated scene, fast rendering with acceptable quality - Blender Cycles render:

Noise threshold Off

Max time 1 seconds, 

1024 max samples, 

denoise on - seems to be fastest with good results.

For 3 cubes with video textures, 

0.5 noise threshold

4096 max samples

Max time 1 sec

denoise off

seems to be fastest with good results.

show the rules for creating username on Moodle - email based signup

There was a request for a method to show the rules for creating usernames (shouldn't have spaces, should be only lowercase) for new users registering on Moodle. 

While a help message is provided for the password, similar information is not provided for username...

My reply was:

https://our.moodle.server/admin/settings.php?section=manageauths
There is an "instructions" field available at the above link, which you can use to provide instructions to the people logging in. Maybe that might help you.

(Another option is to remove the option for users to enter Username, and instead only use Email address as username. But I'm not sure how to do that.)  

With this "instructions" field the problem seems to be mitigated - these instructions are shown on the login screen itself.

animating visibility for neurons scene in Blender

Creating this neurons scene. 

Shift selected to select multiple neurons, then Alt D to duplicate, move mouse to move, R to rotate etc.

In Blender, for keyframing visibility of objects - https://blender.stackexchange.com/questions/115526/trying-to-hide-object-during-animation

has screenshots. "Show in" > Viewports or Renders

recaptcha and Moodle

Moodle has a feature of enabling Recaptcha for email-based registration. But when I tried it with the v3 version of the google captcha, it did not work. V2 works. So, go to recaptcha site, click through the options, choose v2 and not v3

(Moodle seems to have issues with v3? or was it that I did not wait a sufficient amount of time for the changes to propagate?)

modifications to activity completion report

We've had multiple requests for modifications to Moodle's activity completion report - 31 Aug 2023, the request was to add a cohort id, for which I replied (copy-pasting from email exchange)

Need help. In the activity completion report, we do not get cohort id /name. Could you please modify this system report to include cohort id?

Within my current limited understanding of moodle, I don't think this is possible. 

We can write some sort of ad-hoc report to have the contents of the activity completion report with cohort being one of the fields, but that would end up being

very complicated, because of the different numbers and names of activities over different courses.

A simpler option might be to filter inside google sheets or Excel based on users in the cohort you desire, after importing the csv of activities as one sheet, 

and the list of users + cohort in another sheet. An example is shown here,

 progress-trainer-coordi... (link to google sheet removed)

In the Method1 sheet, the last field EZ has been filtered to include only Our_Cohort_2_2023

There is one complication. Each user can appear in more than one cohort. I've used the LOOKUP function to map user to cohort,

=LOOKUP(B27,Cohortusers!D27:Cohortusers!D3199,Cohortusers!A27:Cohortusers!A3199)

but I think the LOOKUP function returns only the last cohort to which the user belongs. 

A better option might be, if you know which cohort you want, to use a function like

=IF(COUNTIF(OnlyOurCohortusers!$D$2:OnlyOurCohortusers!$D$87, B2)>0,"Yes","No")

This has been implemented in the Method2 sheet of the same spreadsheet file.

For some reason, both these methods seem to return 137 rows, even though the OnlyOurCohort sheet has only 85 rows of data.

I'm not sure why.

I have obtained the cohort users using a "custom report", Site Administration > Reports > (Report builder) > Custom reports

User id by cohort (link removed)

-----------------

Then, in Nov 2024, the users wanted a volunteer to modify the code to add a column 'group' to the activity completion report. Again copy-pasting from an email exchange,

Some additional info to what [the volunteer] has written -

1. Activity completion report plugin is part of moodle core, hence will be updated every time Moodle has an update - which would be approx 3-4 times a year. So, any modified version in, say,

reportmodified/completion/index.php

may need to be edited / checked / corrected after each Moodle update. If it works, no problems. But if it doesn't work, code modifications would be needed.

2. Also, running queries on very large number of users has the possibility to make the database time out. Hence, filtering by smaller groups is advisable.

3. Due to 1 & 2, I would strongly recommend filtering by group, exporting to csv, and then if necessary, aggregating via Google sheets / Microsoft excel etc. 

4. The next best option would be to use ad hoc reports. Again, (2.) is a consideration, ad hoc reports can easily cause system lockups if not executed with care.

5. If (2.) is not an issue - that is, if the activity completion report of all users does not slow down the system, and if you can live with (1.) - that is, have the modified code work for now, but potentially break after a month or so, following which [the volunteer] would again need to check and re-modify the code, only then would I suggest asking him to modify the code and create a modified version of the activity completion report.

And the response was,

As of now the team is convinced that they can generate individual group wise reports. Presently there are about 10 groups. If the number of groups increases, we may need to think of a solution.

- meaning that as of now, no modified version is being used.

brain and neuron modeling in blender

How To Make a Brain in Blender

https://www.youtube.com/watch?v=7koiCppOm0o

Brain Material Blender 2.8 Easy Tutorial

https://www.youtube.com/watch?v=27WhXoA1LcQ

Download 3d model

https://free3d.com/3d-model/brain-18357.html

Creating plastic texture - 

https://www.youtube.com/watch?v=zxvjZwg7JeI

defaults, only change colour and roughness.

Final results, using the downloaded model, are at
https://github.com/hn-88/Blender-files/tree/main/brainmodel

referer in apache logs not seen in awstats

Trying to set up awstats for one of our servers which needed referer information to be captured - the referer info was not being displayed initially.

https://simonecarletti.com/blog/2009/01/logging-external-referers-with-apache/

Documentation said

https://www.awstats.org/docs/awstats_config.html#SiteDomain

better to have separate logs for each virtual server. So, set that up.

Purge old files (which contain irrelevant data during testing) by deleting from DirData

AWStats Documentation - Configuration directives and parameters

https://www.awstats.org/docs/awstats_config.html#DirData

It is in /var/lib/awstats on our server.

And the main reason for referer not being displayed was the wrong logformat - it needed to be 

LogFormat=1

and not 

LogFormat=4 # default