data transfer numbers

One of our moodle websites wanted their data transfer numbers for the last two years. I tried analog, but that gave only 14 days of data, since probably the old logs were being deleted by logrotate. Then, got 30 days of data from cloudflare, which they show for free plans, for all their sites put together - 

This shows 60 GB per month across all their websites. Analog's data of the uncached calls to one of their servers showed "Average data transferred per day: 1.98 gigabytes, over the last 14 days." Which again works out to around 60 GB per month. 

Another interesting data point is that (another of our sites) currently pays around $35 a month for their approx 1 TB data stored on AWS S3, so their data transfer is only that much. Via calculator.aws, that means that their usage is probably around 100 GB per month from S3. Or less, since they are likely to have more than the 1M GET requests in the estimate below.

Tiered price for: 1000 GB

1000 GB x 0.0250000000 USD = 25.00 USD

Total tier cost = 25.0000 USD (S3 Standard storage cost)

3,000 PUT requests for S3 Standard Storage x 0.000005 USD per request = 0.015 USD (S3 Standard PUT requests cost)

1,000,000 GET requests in a month x 0.0000004 USD per request = 0.40 USD (S3 Standard GET requests cost)

25 USD + 0.40 USD + 0.015 USD = 25.41 USD (Total S3 Standard Storage, data requests, S3 select cost)

S3 Standard cost (monthly): 25.41 USD

Inbound:

Internet: 10 GB x 0 USD per GB = 0.00 USD

Outbound:

Internet: 100 GB x 0.1093 USD per GB = 10.93 USD

Data Transfer cost (monthly): 10.93 USD

Total Monthly cost: 36.34 USD

Gimp gradients to fade

The goal was to fade out the sharp edges of high-res images used in a planetarium show. 

If we use feathered selection, radius of feathering is limited to 100, so this method is only suitable for smaller images. What I used was Layer Masks.

Creating gradients on transparency is as simple as right-click on layer, add layer mask (then layer mask is automatically selected for editing) and then drag a gradient FG to BG - ensure that FG is white and BG is black.

https://www.flickr.com/groups/48637250@N00/discuss/72157626788469288/

cumulative gradients - I created new layer "from visible", added a block of black or background below that layer, and added the next gradient. Found that we can just Apply layer mask, then add layer mask again, and so on. Need not create new layers as in the screenshot below!

creating a planetarium show with Gimp, Reaper, Blender, Openshot and other free tools

This might be a long post, with separate sections on each tool.

The Gimp - for working with still images, and also can do "convert to polar" which results in something useful for fulldome. 180 degree pans can be stretched to 270 degrees or so without too much degradation. Here is a separate post on Gimp alpha channel gradients to fade out edges of high-res images.

Reaper - not free, but reasonably priced, and very capable DAW - Digital Audio Workstation - for multitrack editing of audio files. I've been using Reaper for a looong time - since 2010 or so.

Openshot - Just for quick and dirty creation of titles. Actually this was not required, Blender's Video Sequence Editor (VSE) can do all these and more.
Openshot - display frame number -
https://www.youtube.com/watch?v=5HyYj1eJH_o

Blender - I used Blender almost exclusively in its Video Sequence Editor (VSE) mode, only using the general purpose mode for creating the long "movie-style" scrolling credits as per this video. 

Blender's VSE had most of the features from Adobe Premiere and Adobe After Effects which I needed, though the final rendering is slow. Of course, I had worked extensively with Premiere and AE only around 20 years ago! Some short notes on how to do particular tasks:

Blender 3.4: Video Editing Tutorial For Beginners. - YouTube 

Transitions

https://www.youtube.com/watch?v=5WhlRQky90w
and a better way to do simple transitions - Click on first clip to select it, Shift click 2nd clip, Add -> transition -> gamma cross

https://www.youtube.com/watch?v=0wuU-7BmtVE&list=PLxdnSsBqCrrG9vYaQMDE0A16c4M52b7vJ&index=4

Movie style end credits with long list of text file

https://www.youtube.com/watch?v=GACSUfqDl28

Very detailed movie style end credits, 75 minute video, if you want more control:

https://www.youtube.com/watch?v=608q54NbPVw

 

Some changes in Blender 3.4,

Delete default object
Delete default light
Move camera to z=5, x=0, y=0
Camera rotation to 0,0,0

Add --> Text from top menu of object mode
The Text Menu appears on top in Edit mode, for "Paste" or "Paste File"

For converting to Mesh, I used
Object --> Convert --> Mesh
with the default settings.

Materials - there is ready made Emit material
We need to make emit value 18 or so for max, not 1

Similarly, for editing the World,
World --> Surface --> Emission
can be used for a green screen background or black background etc.

For editing the keyframes to make them linear,
(Extreme left) -> Graph Editor (Shift F6)
select both keyframes, then
Key --> Interpolation mode --> Linear.

For rotation, a good slow linear rotation speed has a slope of
400 frames === 10 (degrees?) on y axis.

Unfortunately, Indic scripts and other complex scripts were not rendering properly in Blender - https://github.com/oormicreations/VSEIndic
My workaround was to create the the title in Google Docs or something, export as pdf, export as image, etc.

Blurring a face - animated masking with keyframes - https://youtu.be/NF_Q282V_uo

Free video stock footage is a thing - but very few useful clips. Youtube's CC filter and free videos and hi res images listed at
http://www.lss-planetariums.info/index.php?lang=en&menu=videos&page=free_shows
https://www.eso.org/
https://esahubble.org/
https://webb.nasa.gov/content/multimedia/images.html

Green screen chroma keying with Blender -
https://www.youtube.com/watch?v=dI1kMT3O6uk
preferable easier way, using the VSE

https://www.youtube.com/watch?v=_pPGR3d9WWc
uses the object view and nodes, has more control

Colour correcting a specific section in Blender VSE can be done with an adjustment layer strip - https://www.youtube.com/watch?v=yjON3i-890o

Lots of Blender resources and interesting reading:
https://github.com/agmmnn/awesome-blender
http://www.shadedrelief.com/natural3/pages/textures.html
https://github.com/search?q=blender+fulldome&type=repositories
https://blog.exppad.com/article/importing-actual-3d-models-from-google-maps

Edit: Also, lots of detailed info about tracking and animation in this video about how to create tracked callout titles in blender, https://www.youtube.com/watch?v=uT6qNVAKHzU

Wordpress installation on Ubuntu 22.04 with S3 bucket/Azure Storage for content files

https://linux.how2shout.com/how-to-install-wordpress-on-ubuntu-22-04-lts-server/

Created user etc as per

https://hnsws.blogspot.com/2023/05/dot-net-core-ubuntu-linux-server.html

Apparently there were some unmet dependencies,
https://ubuntu.com/tutorials/install-and-configure-wordpress#2-install-dependencies

Still got the error "Please check that the mysqli PHP extension is installed and enabled." For fixing this, uncommented the mysqli extension line in
/etc/php/8.0/apache2/php.ini 

We need sudo rights to chown a resource to other users (who have different permissions than the user we are logged on).

So, instead chmod 777 for the public html folder for WP to be able to write to the configuration file. 

Connecting to an existing S3 bucket for auto-moving content files there, and all the advantages of using an S3 bucket -
https://www.codeinwp.com/blog/wordpress-s3-guide/

Using WP CLI to make changes like installing plugins, templates etc -
https://mediatemple.zendesk.com/hc/en-us/articles/360056149791-how-to-install-wp-cli
because otherwise, WP needs FTP or FTPS access - not SFTP by default. Apparently we can add SFTP support by
sudo apt install php-ssh2
https://stackoverflow.com/questions/53203050/how-to-use-sftp-instead-of-ftp-when-updating-wordpress-plugins

I may add more to this as the configuration progresses.

Edit: 27 June - 
Instead of WP CLI, in order to facilitate plugin installation, updates etc from the WP console, we just need to chown the wp files and the directory to www-data. Not make its permissions 777.

If using Azure storage for content files, even though some how-tos indicate that using our own custom domain requires the use of Azure CDN, we can actually use Cloudflare to proxy traffic with our custom domain and https - we need to first disable proxying and verify the custom domain after setting the CNAME, at the Azure portal -> Storage Account -> Security + networking -> Networking, Custom domain tab. After verification, we can turn on proxying.

For getting the Primary Access key, we need to go to the Azure portal -> Storage Account -> Security + networking -> Access keys.

 

secrets found in repository email

There was an email reporting google services json as having the API key for a repository of ours. According to this,
https://github.com/firebase/firebase-android-sdk/issues/4425

the API key can be public. So no action taken.

Visual Studio's vs folder in git repositories

Thought of removing the .vs directory created by Visual Studio in every git repository opened by it - seems to be a pain,
https://stackoverflow.com/questions/30868544/gitignore-wont-ignore-vs-folder

So I'm just ignoring it for now.

moodle app build blues and fix

After my previous post about building the moodle app using Github actions, Moodle app version 4.1.1 would not build with the same commands. I guess there was some compatibility issues with the versions of cordova or gradle or SDK or .... 

Anyway, the newly released Moodle app version 4.2.0 builds without any issues with this basic workflow. I had seen an announcement that cordova-android is going to default to the latest version in future, so the workflow did not have to specify cordova-android@12.0.0.

Edit: 13 Jan - This commit forces a change to the moodle.config.json format. The Moodle tracker does not have any documentation about the expected format. By trial and error, found that this format works.

"sites": [

    {

      "name": "This seems to be ignored",

      "url": "https://our.moodle.instance.org"

    }

  ],

setting up an opencv C++ project in Visual Studio 2022

This video has a good overview. https://www.youtube.com/watch?v=unSce_GPwto

To summarize the points in the video:

• Install OpenCV, add dll to path if required, add OPENCV_DIR env var if required. If doing these, we would need to close and open the command prompt or Visual Studio for these to take effect.
• Empty project (or Console project)
• If right-clicking through Solution Explorer to "create new c++ file" inside Visual Studio, the file would be automatically added to the project.
• Project -> (name of project) Properties, 
  Configuration Release in dropdown
  Platform x64 in dropdown
• C/C++ -> General -> Additional Include directories -> %OPENCV_DIR%\build\include
• Linker -> General -> Additional Library directories -> %OPENCV_DIR%\build\x64\vc15\lib
• Linker -> Input -> Additional dependencies -> (add the lib file of opencv, like opencv_world480.lib for version 4.80)
   

My older post about Visual Studio 2019 with opencv and spinnaker might probably still work, with the appropriate build tools installed. Creation of props (properties) file which contains the opencv dependencies is another possibility, which I have used in my projects. For example, this github actions build works fine with the latest windows runner, where Visual Studio 2022 is installed, and where the older build tools are also installed.  

Notes on failed builds on Visual Studio 2022:

Cannot find float.h

https://stackoverflow.com/questions/24186647/c1083-cannot-open-include-file-math-h-no-such-file-or-directory

This is probably due to some path issue. For eg,

https://github.com/hn-88/OCVvid2fulldome/commit/c3499db504997c9c308f70c6b368608fb3858307

path needed an extra opencv

Similarly, for VS19 upwards, there are some differences due to a different way of precompiled header handling - no need to add an include for windows.h as in earlier versions, as seen in the code used for the youtube example linked at the top of this post. 

Another issue with Visual Studio builds would be the SDK version(s) which are installed. An answer at this post points to the location in the configuration as
Project properties -> Configuration properties -> General -> Windows SDK version

If it just says latest, then we must install the latest SDK - now it is for Windows 11, but the version number is still 10.<something>. And after the install, we may need to restart Windows too. And then in VS, may need to do "Retarget solution" and then Rebuild.

 

MIT's Scratch for teaching how to program

From a discussion about embedding a custom player in moodle, https://moodle.org/mod/forum/discuss.php?d=435456

https://scratch.mit.edu/about

Useful for teaching kids (or adults) programming, I guess. Drag and drop code elements like loops and so on. Getting started - https://www.youtube.com/watch?v=ssoRNCtmhVM

Creating an animation with scratch - https://www.youtube.com/watch?v=k4zMuBf-7Vs  - basically drawing everything and then looping - much easier for our needs to do it in Blender VSE with keyframes etc.

throw distance and brightness numbers

To evaluate the suitability of some projectors for our planetarium theatre, calculated some numbers based on projectorcentral's calculator.

Our existing projector, 

https://www.projectorcentral.com/Mitsubishi-UD8400U-projection-calculator-pro.htm

at 

0.15 m, output = 0 cm vertical offset, 8 cm x 5 cm image.

20.0 m, output = 0 cm vertical offset, 1049x656 cm image = 1229 cm diagonal

28 nits brightness @ 20 m.

15.0 m, output = 0 cm vertical offset, 1037x648 cm image = 1229 cm diagonal

50 nits brightness @ 15 m. for wide angle

15.0 m, output = 0 offset, 658x411 = 776 cm diagonal

65 nits @ 1.0x telephoto zoom.

Highlights page shows 2.0 to 10.8 m Throw Range.

With that throw range, only NEC projector with zoom lens can fit @ 10k lumens.

https://www.projectorcentral.com/NEC-PX1005QL-W-projection-calculator-pro.htm

with NP19ZL lens

15.0 m, output = 115 offset, 409x230 = 469 cm diagonal

243 nits @ 1.0x telephoto zoom.

If non-4K

https://www.projectorcentral.com/Sony-VPL-FHZ65B-projection-calculator-pro.htm?td_=49%2E2125984252&id_=483%2E858267717&l_=0

Sony VPL-FHZ65B - 6000 lumens

@zoom = 1.0x

15.0 m, output = 0 offset, 650x406 = 766 cm diagonal

57 nits @ 1.0x telephoto zoom.

https://www.projectorcentral.com/Sony-VPL-FHZ85B-projection-calculator-pro.htm

7300 lumens

66 nits.

https://www.bhphotovideo.com/c/product/1669635-REG/sony_vplfhz85_w_vpl_fhz85_7300_lumen_wuxga_laser.html

$7000

Barco F80 4K12

via paulbourke

$26,000

w GLD Zoom R9801721 lens

@15 m

141 cm offset, 556 cm diag im, 176 nits.

https://www.projectorcentral.com/Optoma-ZK507-W.htm

https://www.projectorcentral.com/Optoma-ZK507-W-projection-calculator-pro.htm#calc

0.15 m, output = 0 cm vertical offset, 9 cm x 5 cm image.

20.0 m, output = 0 cm vertical offset, 950x535 cm image = 1083 cm diagonal

26 nits brightness @ 20 m. @zoom=1.05

20.0 m, output = 0 cm vertical offset, 901x507 cm image = 1033 cm diagonal

28 nits brightness @ 20 m. @zoom=1.0

15.0 m, output = 0 cm vertical offset, 1037x583 cm image = 1190 cm diagonal

26 nits brightness @ 15 m. for wide angle @zoom=1.53x

15.0 m, output = 0 offset, 676x380 = 775 cm diagonal

49 nits @ 1.0x telephoto zoom.

TODO: check zoom on current position, distance to mirror and size on mirror. (noted below)

Current setup:

Distance to mirror centre ~ 80 cm

Width of image ~41 cm

Corresponding values on

https://www.projectorcentral.com/Mitsubishi-UD8400U-projection-calculator-pro.htm

= 1.16x zoom

@1.16x zoom, distance of 15.0 metres

== 760 x 427 cm, 872 diag, for 16x9, 52 nits.

Corresponding values on

https://www.projectorcentral.com/Optoma-ZK507-W-projection-calculator-pro.htm#calc

= 1.14x zoom

@1.14x zoom, distance of 15.0 metres

== 768 x 432 cm, 881 diag, for 16x9, 40 nits.

If we adjust for a slightly smaller 871 diag as for the mitsubishi,

@1.12x zoom, distance of 15.0 metres

== 759 x 427 cm, 871 diag, for 16x9, still 40 nits.

 

(52-40)/52 = 23% dimmer. 

 

But we can get 23% dimmer/brighter just by changing modes from theatre to custom etc, and also when the lamp age increases. So, probably this projector may be OK. https://www.projectorcentral.com/Optoma-ZK507-W.htm

How to train a Large Language Model (LLM) with your own data

There's an interesting blog post about how much effort (and money for infrastructure) is required to train a Large Language Model (LLM) with custom data - https://blog.replit.com/llm-training

Huggingface does have free rate limited APIs for some of the smaller pre-trained models, like
https://huggingface.co/google/flan-t5-base

And they do have free autotrain for custom data,

https://huggingface.co/autotrain

More links:

https://huggingface.co/google/flan-t5-xl

how to use a LLM

Hugging Face – Pricing

https://huggingface.co/google/flan-t5-small

About FLAN-T5

https://exemplary.ai/blog/flan-t5

Hardware and time costs - 

https://www.philschmid.de/fine-tune-flan-t5-deepspeed

Edit: Oct 2023 - Tutorial on how to use an LLM with Google Colab's Free tier - 
https://betterprogramming.pub/set-up-an-llm-project-using-a-free-gpu-in-google-colab-e55453bfc760

 

google groups email delivery issues

There have been delivery issues with a google group with > 40k subscribers, and over the last 3-4 days, there were issues with automated sending of emails to the group. Earlier, we used a Google Apps Script to send email, taking content from a web page. But overnight, the team changed to sending via smtp, and the group refused the incoming messages.

My first suspicion was that the From: or some other header was wrong, due to which the messages were getting rejected as not having permission to post. But on checking the sent messages to another group - which had been sent successfully - found that the headers were right, only the sending method was SMTP instead of Google Apps Script, and the content template was different.

After a couple of days of manual sending by the team with the same new template, the SMTP sending started working automatically. So probably this was some sort of anti-spam strategy by google groups. 

dot net core Ubuntu Linux server - software setup

Listing all the tasks which needed to be done:

1. Change default port of SSH server
by editing
/etc/ssh/sshd_config
#Port = 22 <-- change this to something else.

2. install and set up mysql

How To Install MySQL on Ubuntu 22.04 | DigitalOcean


3. Set up DNS in cloudflare.

4. Set up Apache virtual servers

sudo apt install apache2

needed to see the documentation, in /var/www/html/index.html, which says that
"By default, Ubuntu does not allow access through the web browser to any file outside of those located in /var/www, public_html directories (when enabled) and /usr/share (for web applications). If your site is using a web document root located elsewhere (such as in /srv) you may need to whitelist your document root directory in /etc/apache2/apache2.conf."

So, added like this,
<Directory our/custom/www/home>
        AllowOverride None
        Require all granted
</Directory>

and also, needed read permission for the entire directory tree in /our/custom/www/home,
Why apache throwing forbidden when directory is in home? - Stack Overflow


5. Set up ssl with cloudflare server origin cert - this did not work till I first set up SSL using certbot, then replaced the private key and certificate paths to point to the cloudflare origin cert. Most probably because certbot automatically set up the required configuration with
Include /etc/letsencrypt/options-ssl-apache.conf
in the virtual host file.

How To Secure Apache with Let's Encrypt on Ubuntu 22.04 | DigitalOcean

sudo apt install certbot python3-certbot-apache
sudo certbot

 Link dump of other stuff I tried:

Origin server · Cloudflare SSL/TLS docs

Ubuntu with Apache2: CSR & SSL Installation (OpenSSL) (digicert.com)

(We saved it in /etc/ssl/certs/ )

How To Troubleshoot Common Apache Errors | DigitalOcean

needed to enable modssl, so
/etc/apache2/mods-available

sudo a2enmod ssl
sudo a2enmod headers

from SSL: How to enable HTTPS with Apache 2 on Ubuntu 20.04 | ArubaCloud.com
Syntax error on line 33 of /etc/apache2/sites-enabled/002-our-site-ssl.conf:
May 06 10:28:11 ip-10-0-0-73 apachectl[2467]: SSLCertificateKeyFile: file '/etc/ssl/private/cloudflare-oursite.org.privatekey.pem'

sudo apachectl configtest
SSLCertificateKeyFile: file '/etc/ssl/private/cloudflare-oursite.org.privatekey.pem' does not exist or is empty

(This was because of saving the private key elsewhere instead of in this path).

sudo apt install certbot
Certbot doesn't know how to automatically configure the web server on this system.

(This was because python3-certbot-apache also needed to be installed.)

After changing the cert to cloudflare origin cert, I wanted to disable the cron job to get updated letsencrypt certs - but
no crontab for root. The cron job for certbot was at
/etc/cron.d/certbot
 - commented out everything.

Then just copied the same thing for uat virtual server.

6. Set up dot net

https://www.syncfusion.com/blogs/post/hosting-multiple-asp-net-core-apps-in-ubuntu-linux-server-using-apache.aspx

find which dot net core version is used by your application in IIS - Google Search

dotnet --version
3.1.410

So we need to use "Microsoft feed"
.NET and Ubuntu overview - .NET | Supported distributions

https://learn.microsoft.com/en-us/dotnet/core/install/linux-package-mixup?pivots=os-linux-ubuntu#i-need-a-version-of-net-that-isnt-provided-by-my-linux-distribution

https://manpages.ubuntu.com/manpages/xenial/man5/sources.list.5.html

https://learn.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#register-the-microsoft-package-repository

https://tecadmin.net/how-to-install-dotnet-core-on-ubuntu-22-04/
Does not support 3.1

Would need to migrate
https://learn.microsoft.com/en-us/aspnet/core/migration/31-to-60?view=aspnetcore-7.0&tabs=visual-studio

provisioning a dot net core Ubuntu Linux server on AWS

Some notes about the provisioning and setup:

• AWS EC2 is where the VMs are seen. Choosing the region is important - the dashboard will not show the VMs in your account if the region is not set correctly!
  
  
• AWS offers a very large selection of machine types. We need to research costs before choosing one. For example, when I checked the pricing calculator for Mumbai region, a12xlarge with 16 MB RAM was $0.217 hourly, c5a2xlarge was $0.207, c6a2xlarge was $0.2057, but c52xlarge was $0.362. We can't assume that any family will be lower based on name prefixes, though some series like d are much higher due to higher specs. 
  
  
• In order to provision users who can access the VM via amazon's web console, the method chosen was to create a user group, create policies to give permissions to users in that user group, and create IAM users in that user group. 
  
  
• Creating custom policies - since we wanted the users to be able to do all EC2-related tasks like starting and stopping the VM, the method used was to make a copy of the supplied "All EC2 permissions" policy, edit the JSON and instead of having * as the Resources to which it applied, filling in the Amazon Resource Name instead.
  https://docs.aws.amazon.com/managedservices/latest/userguide/find-arn.html
  
  
• Amazon gives custom login urls, like
  https://our-name.signin.aws.amazon.com/console
  https://docs.aws.amazon.com/signin/latest/userguide/introduction-to-iam-user-sign-in-tutorial.html
  After logging in, we need to go to EC2, choose Instances on the left-hand side, ensure that we have chosen the correct region in the drop-down list on top of the page, and select our instance. For restarting, we can then choose the Actions menu on top of the page, Manage Instance State --> Start / Stop / Reboot. We have set "Terminate protection" on, so that the Terminate option is greyed out - which should be used only to completely remove the instance.
• If we don't opt for a dedicated IP - called Elastic IP in AWS parlance - we can use the hostname like ec2-9-999-999-99.ap-south-1.compute.amazonaws.com where the nines seem to indicate the ip address. But I guess this name would change along with the dynamically assigned IP address if the machine were stopped and restarted after a while, so we need to assign an Elastic IP. One Elastic IP per instance are free if used on a running instance.
  

Since this post has already become long, I will put the software installation part on a separate post. 

mpg123 works well for icecast or shoutcast streams - curl to help it play https

mpg123 http://stream.sssmediacentre.org:8000

works well on raspberry pi, even doing retries so that playout resumes after short network outages, unlike browsers like Chrome or Firefox. The buffer is also a minute long, so that short outages of a few seconds are not noticed as gaps in audio.

But mpg321, which is the package which supports mpg123 now, does not seem to have support for https. With the help of curl as mentioned here,

~$ mkfifo pipe
~$ curl -L https://stream.sssmediacentre.org:8443/bhajan -o pipe & mpg123 pipe

Works well.

documentation on domain transfer from Godaddy to Nettigritty

https://domain.nettigritty.com/kb/servlet/KBServlet/faq1188.html

https://support.hostinger.com/en/articles/1583437-how-to-get-epp-code-from-godaddy

 

github pages, cloudflare and SSL

Many people have issues with Github Pages not issuing TLS certificates after following the official procedure, which is to first go to settings -> pages on the repo and set up the custom domain, and then set the CNAME for the domain to point to username.github.io.

The reason for this order of actions is to prevent domain hijacking, as github explains.

One of our domains had the same issue even after 24 hours, even though Cloudflare proxying was turned off, and the "Only DNS" mode was set. The solution was to remove the custom domain and immediately add it back in github repo settings. Then the TLS certificate request went through and the certificate was issued within a minute or two.

new whatsapp voice call spam

Whatsapp has had a feature for "Block and Report" for spam chats. Now there's a new game in town - missed whatsapp voice-calls from international numbers like
+62 857 6605 4116
+55 66 8473 9018
+7 931 703 90 02
etc.

What we can do is to block them. Tap on the number to open the Call Info page, then the three dots menu on top right (on Android), Block.

php echo does not reflect on screen till end of processing

With the code at https://github.com/hn-88/mktxt/ found that sometimes the messages would not be displayed on screen till the end of execution.

https://stackoverflow.com/questions/4669822/echo-messages-while-php-script-still-executes

- so it may be client dependent.

And yes - found that in Edge / Chrome / Chromium, the output is shown as soon as the page opens, with dynamic updating. With my version of Firefox, the output flush does not work due to browser caching.

adding multiple files in a single commit using github api

Currently we push each file separately like in this code snippet, but when I explored how to add multiple files, the implementation seems a bit too complex for our needs.

https://siddharthav.medium.com/push-multiple-files-under-a-single-commit-through-github-api-f1a5b0b283ae

https://github.blog/changelog/2021-09-13-a-simpler-api-for-authoring-commits/

"simpler" just means fewer number of calls.

 

Google Sign-in - old methods deprecated

I was attempting to get the Google username + email id for users using Google Sign-in to log on to an embedded google apps script, but the username was being returned blank.

Our earlier php code, modified from something like this was returning only email and not username when trying with the default scope or even with addScope(profile). Even the REST API didn't help.

Finally found the issue - this page states that 

"Warning: The support of Google Sign-In JavaScript platform library for Web is set to be deprecated after March 31, 2023. The solutions in this guide are based on this library and therefore also deprecated.

Use instead the new Google Identity Services for Web solution to quickly and easily sign users into your app using their Google accounts.

By default, new client IDs are now blocked from using the older platform library; existing client IDs are unaffected. New client IDs created before July 29th, 2022 may set the plugin_name to enable use of the legacy Google platform library."

So I modified the code to use the new Google Identity Services using this guide, and all was well. In order to not interfere with the earlier (working) authentication setup on the same server, which has a different set of authentication rules, the vendor, autoload etc are on a different path.

Code snippet:

$client = new Google_Client(['client_id' => $client_id]);

$payload = $client->verifyIdToken($_POST['credential']);
if ($payload) {
  //print_r($payload);
  //echo $payload['email'];
  //echo $payload['name'];
  // set the session variables needed by mediabank
  $_SESSION["userEmail"] = $payload['email'];
  $_SESSION["userName"] = $payload['name'];
  header('Location: /the-embedded-page.php');

get POST to work on PHP

Inputs from

https://stackoverflow.com/questions/9332718/how-do-i-print-all-post-results-when-a-form-is-submitted

One way would be something like

foreach ($_POST as $key => $value) {
        echo $key;
        echo $value; etc.

embedding a google apps script - too many redirects

There was a google apps script to be embedded in an iframe on one of our websites. I've done this before with no issues, as long as the XFrameOptionsMode is set properly in the script to ALLOWALL. But in this case, I was getting a too many redirects error. "accounts.google.com redirected you too many times."

Checking the network traffic (right-click -> Inspect -> Network tab), found that the app was redirecting to google's login authentication page, back to the app's original script.google.com address, and again, infinitely till the browser stops the loop after 25 times or so.

Checking the app's deployment, found that this was due to the deployment availability being set to "Anyone with a Google Account" instead of "Anyone". Once the deployment was changed to "Anyone", the embed worked fine. 

(This link has one approach in which validation can be done, but I didn't go through it in detail to check if it is embed-friendly.)

issues with a Windows server running a dot net app and mitigation

Copy-pasting my response to a message,  

server seems to be down again. It is happening quite frequently since yesterday....

I see two types of attacks on the server. Both are likely to be by automated bots, since such attacks are very common.

1. Remote Desktop

Attack

The remote desktop service (RDP, termservice) is being attacked. If a sufficiently large number of connections take place, the server will run out of memory, and other services will also start failing.

Mitigation

We can prevent these by simply stopping the remote desktop service when not in use. For this, please run the stoprdp.bat via SSH, logging in with putty etc, whenever you start RDP with restartrdp.bat, after you finish your work on the server. I have run stoprdp.bat now.

2. Attacks on the database via dotNET api

There are attacks on the database / dotNET app using the API exposed by the dotNET application. In the screenshots below (redacted in this blog post), you can see how multiple API controller endpoints are being targeted in a short time, with the database quickly reaching maximum retries.

Edit - adding another screenshot from another day - 

 

Mitigation

Assuming that these are not some tests being done by your team, we will need to

(a) check if the dot net app code-base is secured against all the usual attack vectors mentioned at

https://www.c-sharpcorner.com/article/securing-your-Asp-Net-web-applications/

In particular, exception handling via try-catch should be checked.

(b) add some sort of Denial of Service attack prevention, similar to
https://www.madskristensen.net/blog/block-dos-attacks-easily-in-aspnet/

Please note - directly using the code in the link above would not be safe, since our web app is much more complex.

Another DOS attack prevention tool is HackerSpray
https://www.codeproject.com/Articles/1112532/HackerSpray-Block-Brute-Force-and-DOS-Attacks

https://github.com/oazabir/HackerSpray

Another one is

https://stackoverflow.com/questions/19087512/how-to-restrict-dos-attack-with-web-api/56075128#56075128

(c) Prevent direct access to the API on our server, and allow ONLY access through Cloudflare.

This is supposed to mitigate many types of attacks.

 

I've now implemented blocking of all external requests on HTTP and HTTPS on the Sainin production server except via cloudflare's ip addresses

https://www.cloudflare.com/en-gb/ips/

 

For this, just had to copy-paste the ipv4 addresses from the list above, make them comma-separated, and paste them into a newly created inbound security rule in Azure portal for the nsg (network security group) of the server in question. And, of course, make sure that the relevant services' DNS records are proxied through cloudflare and not "DNS only".

 

 

web fonts for properly rendering multilingual website

When creating a website on github pages with text in English, Hindi and Telugu, the first attempt was to just copy-paste the content from a Google Doc exported as HTML. But that had two problems.

1. The text was represented as characters like &#2309;&#2306;&#2327;&#2381;&#2352;&#2375;&#2332;&#2368;
   not human-readable for easy editing.
   
   
2. Firefox was not rendering some of the characters properly, like
   
   instead of

   

   
   
   So, tried incorporating web-fonts - reading up
   
   https://teck.in/free-google-web-fonts-for-indian-languages.html
   https://fonts.google.com/knowledge/using_type/using_web_fonts
   https://www.zachleat.com/web/css-tricks-web-fonts/
   https://www.w3schools.com/css/css3_fonts.asp
    
   Google's free multilingual font Noto Sans was what I used. 
   
   
   CSS like
   @font-face {
               font-family: EnglishBold;
               src: url(images/notosans-black.ttf);
           }
   
           .eng {
               font-family: EnglishRegular;
           }
   

SSH - host key has changed

Github has changed their host key - 

https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/

So, when trying to clone a repo of mine via the SSH url, I got the message

Offending RSA key in /home/myusername/.ssh/known_hosts:3
  remove with:
  ssh-keygen -f "/home/myusername/.ssh/known_hosts" -R "github.com"
Host key for github.com has changed and you have requested strict checking.

So I did

ssh-keygen -f "/home/myusername/.ssh/known_hosts" -R "github.com"

and answered "yes" when asked to save the new key.

Expanding the sky in panoramic photos

Useful for changing the aspect ratio of panoramic photos, which are usually much more wide than they are tall - How to expand sky in any photo with GIMP - YouTube

Scale up a copy on a new layer.
Create layer mask on original layer.
Select the layer mask in the layers window to select it for editing.
Gradient black to white Foreground to transparent to blend in the orig with the scaled up sky.

What I did for the Tokyo pan was, since the sky was just blue,
did a rectangular selection of only sky,
copied the selection, pasted to new layer
and for that layer, Layer -> resize layer, expand upwards.

Scripting Stellarium 2023

There are some changes after my earlier post about creating views of the night sky using stellarium. And I've forgotten the locations of some controls. So, a set of reference notes.

• Initial setup - to set the place and time to Puttaparthi, 14.17N, 77.81E, 23 Nov 1926 4:30 am  - (F6) Location window and (F5) Date Time window. (F2) Configuration window --> Time tab to start at that time every time, or to stop time at that point. 
  
  
• Switch off captions with (F4) Sky and Viewing Options --> Sky tab, Labels and Markers.
  
  
• Switch off detailed information for selected objects in (F2) Configuration window --> Main tab, if we leave only Name checked, the name of the selected object is shown on top left corner, which would not be visible in a fulldome frame. 
  
  
• To set field of view to 180 degrees, scroll-wheel on mouse - slightly inaccurate. More accurate to use scripting, below. We can change mapping to Fisheye from (F4) Sky and Viewing Options --> Sky tab.
  
  
• Use Ctrl+T to remove/toggle the controls from view. Ctrl+S will save a screenshot. The screenshot directory is set on (F2) Configuration window --> Tools tab. Or use scripting, as below.
  
  
• Scripting example for screenshots - https://renenyffenegger.ch/notes/Wissenschaft/Astronomie/tools/Stellarium/script/API/core/screenshot
  In our case, instead of StelMovementMgr.zoomTo(25, 0.01) we wanted to change / set the FOV from 195 to 30 degrees - progressively zooming in. Scripting console (F12) or run saved scripts with (F2) Configuration window --> Scripts tab.
  for (i=195; i>30; i--) {
  StelMovementMgr.zoomTo(i, 0.01);
  core.wait(0.01);
  // Take screenshot
  core.screenshot('sirius-zoom') ;
  }
  
  
•  Unfortunately, the values mentioned in a previous post for "good-looking" night sky, which was for version 0.9 or something like that, don't seem to be valid now for version 1.2 or 2023. (F4) Sky and Viewing Options --> -> Sky tab -> Absolute scale = 1 and Relative scale = 0.5 with Limit magnitude = 5.5 or 6.0 makes the stars look good for 1080p fulldome screenshots in the current version 1.2. 

Scripting Software Bisque Seeker Theatre Edition on Windows 11

Adding to my earlier tests with Seeker Theatre Edition, on Windows 11, there are some more things to keep in mind. 

• Offline rendering with 'create movie' - only .png and .tga export works. mov or jpg don't work. This is probably due to not installing Quicktime.
  
  
• After .png export, we can create a movie with ffmpeg, for example
  ffmpeg -r 30 -f image2 -i "Our-frame_%06d.png" -vcodec libx264 -crf 15  -pix_fmt yuv420p Our-movie-4k-frames.mp4
  
  
• Copy-pasting from script examples, zooming in to our location with the following, (but this may work only after setting the time to "Now" at mid-day here :)
  
  <Software Bisque> 1 <HIDE> 1
  !Waypoint start
  <HIDE> 1
  <WAYPOINT2> 2460041.90072305361 -0.957039617708 -0.292439496712 7.64553844115e-06 -0.51761781304 -0.847675565812 0.116266653641 -0.0507515739738 0.166066153524 0.984807753012
  <HIDE> 0
  !Waypoint end
  <SSGOTO> 3 0 0 40 1.16
  <SSROLL> 0
  !Waypoint start
  <HIDE> 1
  <WAYPOINT2> 2460041.90013993671 -0.957075431798 -0.292483987095 1.50676317142e-05 -0.51761781304 -0.847675565812 0.116266653641 -0.0507515739738 0.166066153524 0.984807753012
  <HIDE> 0
  !Waypoint end
  

raspberry pi setup redux

Setting up a Raspberry Pi 4 as an Icecast player with a local video playing on the screen, notes below:

• Interesting idea for chromium tabs for display, but we have not used it, https://blog.gordonturner.com/2019/07/23/raspberry-pi-switch-chromium-browser-tabs/
  
  
• Remmina refuses to connect to realvnc server on pi - simplest solution was to install realvnc client.
  
  
• Unfortunately, high CPU usage with VLC playing the local video file. Tried 64 bit Raspberry Pi OS, 32 bit Raspberry Pi OS, finally got good results with 32 bit "Legacy" Raspbian Buster and omxplayer. With omxplayer, CPU usage is only 5% or less. With VLC, either VLC (in 64 bit) or Xorg (in 32 bit) uses 100%. Unfortunately, omxplayer has been removed from newer versions of Raspberry Pi OS. Maybe we can use vlc later after exploring various output device options, but as of now,
  runStreamAndVideo.sh has
  
  mpg123 http://stream.radiosai.org:8000 &
  #cvlc "sssmc-prconn.mp4" --fullscreen
  omxplayer -o local --loop /home/pi/sssmc-prconn_360p.mp4
  
  
• For setting up wifi after fresh installs, the easiest option was to ssh into the device using an ethernet cable nearby, and use
  sudo raspi-config
  System -> Wireless LAN
  The method of using a wpa_supplicant.conf file created in /boot seems to work only if
  (a) the country is already set
  (b) the conf file has Unix style line endings.
  The following sudo nano /boot/wpa_supplicant.conf worked once for me, didn't work for a fresh install -
  

  country=IN  #omit if US
  ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
  update_config=1
  network={
      ssid="Our-SSID"
      psk="Our-Password"
  }

  
  
• Used our old technique of creating a .desktop file in .config/autostart directory + enabling auto login to GUI + wait for network to boot (wait for network is needed, because otherwise mpg123 doesn't play.)
  
  
• sudo apt install realvnc-vnc-server
  to install VNC server, and
  sudo raspi-config
  Interface options -> VNC to enable the service at boot.
  
  
• We can control the volume easily with ssh, with the alsamixer command, up arrow and down arrow.
  
  
• Default audio device is analog out. To change to HDMI, a HDMI monitor needs to be connected, then we can use
  sudo raspi-config
  System -> Audio
  (or using VNC, right-clicking on the volume applet on top right corner)
  
  
• Instead of using our earlier method of an IR remote control to shut down the machine, we're using RaspController android app instead.
  (The device drivers have changed yet again from our earlier method.
  sudo modprobe lirc_rpi
  was replaced by
  sudo modprobe gpio_ir
  and now, with Raspbian Buster, it is
  sudo modprobe gpio_ir_recv )
  
  
• We can set the hostname of the Raspberry pi using
  sudo raspi-config
  System -> Hostname
  and we're supposed to be able to reach it using hostname.local on the local network for "devices which support mDNS" or Avahi. On the pi,
  sudo apt-get install avahi-daemon
  sudo apt-get install avahi-utils
  
  But avahi-browse -a -d local didn't seem to work. Probably some syntax issue.
  
  Also, the hostname.local didn't seem to work out of the box for some Android devices. So, set up a static ip by moving up the DHCP pool to a more restricted set on the router, and setting up the static ip by right-clicking on the wifi icon using VNC,
  Wireless and Wired networking settings ->
  Configure -> SSID (choose our SSID)
  or Configure -> interface (choose wlan0)
  and set the required IP address etc. by unchecking the "Automatic" box.
  
  
  
  

cognitive impairment detection via speech

Unfortunately not available for the general public - https://accexible.com/

edit id for google play - notes about setting update priority

When uploading an app to google play store, there was a request from the team whether we can make some updates mandatory and some non-mandatory.

Do we have any release priority setting in play store console for mandatory or non-mandatory?

https://developer.android.com/guide/playcore/in-app-updates/native

"To determine priority, Google Play uses an integer value between 0 and 5, with 0 being the default and 5 being the highest priority. To set the priority for an update, use the inAppUpdatePriority field under Edits.tracks.releases in the Google Play Developer API. All newly-added versions in the release are considered to be the same priority as the release. Priority can only be set when rolling out a new release and cannot be changed later."

To get the edit identifier:

https://stackoverflow.com/questions/62611679/google-play-developer-api-what-is-edit-identifier-for-an-apk-and-how-can-i-fi

http://frontendcollisionblog.com/javascript/2015/12/26/using-nodejs-to-upload-app-to-google-play.html

https://developers.google.com/android-publisher/getting_started

Using the Google Play API seems to be a bit complicated, for setting the update priority.

This discussion

google play developer api - How to set update priority for a new android release - Stack Overflow

points to a Firebase approach, in case you use Firebase in the project.

Sign language resources links

Possibly useful for creating Indian Sign Language app.

Existing app (for older version of android, so it's not discoverable by newer Android versions) using video for Indian Sign Language - ISL Dictionary - https://indiansignlanguage.org/android-app-for-indian-sign-language/

Existing app using Avatars for American Sign Language - Hand Talk Translator

https://simax.media/?lang=en - commercial company making Sign Language apps. 

Offline (and online) speech to text - Basic Tutorial on how to make your Android app with speech to text - https://www.geeksforgeeks.org/offline-speech-to-text-without-any-popup-dialog-in-android/

Automatic Translation of English Text to Indian Sign Language Synthetic Animations - conference article - https://aclanthology.org/W16-6319.pdf

That references
HamNoSys - Hamburg Notation System
which links to
https://vh.cmp.uea.ac.uk/index.php/SiGML_Tools

Demo of their Avatar - CoffeeScript WebGL ARP Signing Avatar

Terms of use - CC-BY-SA - https://vh.cmp.uea.ac.uk/index.php/CWASA_Conditions_of_Use

Edit 4 Apr: A summer project to convert text to Indian Sign Language - https://github.com/shoebham/text_to_isl 

How to build a WebGL-based native Android app - https://prototechsolutions.com/cad-notes/webgl-native-android-app/

Edit 5 Apr: Search engines for open data sets - 
https://www.kaggle.com/search?q=sign+language
https://datasetsearch.research.google.com/search?src=0&query=indian%20sign%20language

Collection of datasets blog post - https://redis.com/blog/datasets-for-test-databases/


Open-source AI models - build, train, deploy - Hugging Face – The AI community building the future.

reverse engineering an api

For making queries to the sssmediacentre api like for creating the schedule page, I had used the browsers' "inspect" functionality by right clicking on the page, and noting the traffic in the "network" tab. It turns out there's a nice how-to for exploring undocumented APIs at inspectelement.org

interpreting the Linux top command CPU usage stats

https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/

for interpreting the top command's %CPU usage stats.

•     us: Amount of time the CPU spends executing processes for people in “user space.”
•     sy: Amount of time spent running system “kernel space” processes.
•     ni: Amount of time spent executing processes with a manually set nice value.
•     id: Amount of CPU idle time.
•     wa: Amount of time the CPU spends waiting for I/O to complete.
•     hi: Amount of time spent servicing hardware interrupts.
•     si: Amount of time spent servicing software interrupts.
•     st: Amount of time lost due to running virtual machines (“steal time”).

Interestingly,

https://www.redhat.com/sysadmin/interpret-top-output

has a mistake about idle time - it says, "id is the percent of time idle (if high, CPU may be overworked)." It should be, if low, CPU may be overworked, right?

ssh-copy-id

Instead of manually copying over the public key to known-hosts on the SSH server, we can use ssh-copy-id to make things simpler, to implement key-based SSH authentication.

https://www.ssh.com/academy/ssh/copy-id

ssh-copy-id -i ~/.ssh/mykey user@host 

new schedule page based on json data instead of database back-end - ChatGPT for Javascript to PHP translation

Initially created a Google apps script / Google Sheet version to replicate the schedule page and audio search page - 

https://github.com/hn-88/mktxt/tree/main/GAS

Then created a php-based version to run on our local server, since remote server audio search functionality was being dropped in favour of SSSMC search page - 

https://github.com/hn-88/mktxt

Tried using ChatGPT to convert code snippets from Javascript to PHP. It was surprisingly good. Saves a lot of time.

pip install etc

https://developers.google.com/admin-sdk/directory/v1/quickstart/python

 https://stackoverflow.com/questions/47600597/import-error-cannot-import-name-opentype solved the problem.

parsing a csv with pipe symbol as separator in PHP

Following this discussion, where it was noted that "PHP's array_map() supports a third parameter which is an array representing the parameters to pass to the callback function" the method I used was:

$file = "../path/to/".$playlistDatenodash.".csv";
$filearray = file($file);
$data = array_map('str_getcsv', $filearray, array_fill(0,count($filearray),'|'));

unable to ssh into old server after client upgrade to Linux Mint 21

After upgrading my client machine to Linux Mint 21, I was unable to ssh into an old internal server of ours. Interestingly, Filezilla on the client machine was able to sftp into the old server. At first, I thought the keys in .ssh might have changed, but it was an upgrade and not a reinstall, so the old keys were still there. Then found this - https://stackoverflow.com/questions/73795935/sign-and-send-pubkey-no-mutual-signature-supported - and ssh worked when I added the parameter

ssh -o PubkeyAcceptedKeyTypes=ssh-rsa

So, probably the new client has RSA disabled, while the old internal server still uses RSA.

possible google play developer console bug - testing for wearos even when we do not target it

When a new app was uploaded to Google Play console for internal testing, the pre-launch report showed

Errors found. Fix before releasing to production.

Error

Issue 1 of 1 Detected on 1 device during testing

Stack trace etc etc.

Detected on 1 device during testing

Google Wear (virtual) 454x454 Android 11 (SDK 30) - x86 en_I

 

This happened even though we had de-selected Google Wear in the targeted devices list.

 

This seems to be a google bug - testing on WearOS even when we have not targeted WearOS -

https://support.google.com/googleplay/android-developer/thread/207447385/how-can-i-tell-pre-launch-not-to-test-with-google-wear?hl=en

So I think we can safely ignore the Google Wear crash report, at least in this case. (And the app was released later into production with no problems - I deselected pre-launch reports just in case.)