How to train a Large Language Model (LLM) with your own data

There's an interesting blog post about how much effort (and money for infrastructure) is required to train a Large Language Model (LLM) with custom data - https://blog.replit.com/llm-training

Huggingface does have free rate limited APIs for some of the smaller pre-trained models, like
https://huggingface.co/google/flan-t5-base

And they do have free autotrain for custom data,

https://huggingface.co/autotrain

More links:

https://huggingface.co/google/flan-t5-xl

how to use a LLM

Hugging Face – Pricing

https://huggingface.co/google/flan-t5-small

About FLAN-T5

https://exemplary.ai/blog/flan-t5

Hardware and time costs - 

https://www.philschmid.de/fine-tune-flan-t5-deepspeed

Edit: Oct 2023 - Tutorial on how to use an LLM with Google Colab's Free tier - 
https://betterprogramming.pub/set-up-an-llm-project-using-a-free-gpu-in-google-colab-e55453bfc760

 

google groups email delivery issues

There have been delivery issues with a google group with > 40k subscribers, and over the last 3-4 days, there were issues with automated sending of emails to the group. Earlier, we used a Google Apps Script to send email, taking content from a web page. But overnight, the team changed to sending via smtp, and the group refused the incoming messages.

My first suspicion was that the From: or some other header was wrong, due to which the messages were getting rejected as not having permission to post. But on checking the sent messages to another group - which had been sent successfully - found that the headers were right, only the sending method was SMTP instead of Google Apps Script, and the content template was different.

After a couple of days of manual sending by the team with the same new template, the SMTP sending started working automatically. So probably this was some sort of anti-spam strategy by google groups. 

dot net core Ubuntu Linux server - software setup

Listing all the tasks which needed to be done:

1. Change default port of SSH server
by editing
/etc/ssh/sshd_config
#Port = 22 <-- change this to something else.

2. install and set up mysql

How To Install MySQL on Ubuntu 22.04 | DigitalOcean


3. Set up DNS in cloudflare.

4. Set up Apache virtual servers

sudo apt install apache2

needed to see the documentation, in /var/www/html/index.html, which says that
"By default, Ubuntu does not allow access through the web browser to any file outside of those located in /var/www, public_html directories (when enabled) and /usr/share (for web applications). If your site is using a web document root located elsewhere (such as in /srv) you may need to whitelist your document root directory in /etc/apache2/apache2.conf."

So, added like this,
<Directory our/custom/www/home>
        AllowOverride None
        Require all granted
</Directory>

and also, needed read permission for the entire directory tree in /our/custom/www/home,
Why apache throwing forbidden when directory is in home? - Stack Overflow


5. Set up ssl with cloudflare server origin cert - this did not work till I first set up SSL using certbot, then replaced the private key and certificate paths to point to the cloudflare origin cert. Most probably because certbot automatically set up the required configuration with
Include /etc/letsencrypt/options-ssl-apache.conf
in the virtual host file.

How To Secure Apache with Let's Encrypt on Ubuntu 22.04 | DigitalOcean

sudo apt install certbot python3-certbot-apache
sudo certbot

 Link dump of other stuff I tried:

Origin server · Cloudflare SSL/TLS docs

Ubuntu with Apache2: CSR & SSL Installation (OpenSSL) (digicert.com)

(We saved it in /etc/ssl/certs/ )

How To Troubleshoot Common Apache Errors | DigitalOcean

needed to enable modssl, so
/etc/apache2/mods-available

sudo a2enmod ssl
sudo a2enmod headers

from SSL: How to enable HTTPS with Apache 2 on Ubuntu 20.04 | ArubaCloud.com
Syntax error on line 33 of /etc/apache2/sites-enabled/002-our-site-ssl.conf:
May 06 10:28:11 ip-10-0-0-73 apachectl[2467]: SSLCertificateKeyFile: file '/etc/ssl/private/cloudflare-oursite.org.privatekey.pem'

sudo apachectl configtest
SSLCertificateKeyFile: file '/etc/ssl/private/cloudflare-oursite.org.privatekey.pem' does not exist or is empty

(This was because of saving the private key elsewhere instead of in this path).

sudo apt install certbot
Certbot doesn't know how to automatically configure the web server on this system.

(This was because python3-certbot-apache also needed to be installed.)

After changing the cert to cloudflare origin cert, I wanted to disable the cron job to get updated letsencrypt certs - but
no crontab for root. The cron job for certbot was at
/etc/cron.d/certbot
 - commented out everything.

Then just copied the same thing for uat virtual server.

6. Set up dot net

https://www.syncfusion.com/blogs/post/hosting-multiple-asp-net-core-apps-in-ubuntu-linux-server-using-apache.aspx

find which dot net core version is used by your application in IIS - Google Search

dotnet --version
3.1.410

So we need to use "Microsoft feed"
.NET and Ubuntu overview - .NET | Supported distributions

https://learn.microsoft.com/en-us/dotnet/core/install/linux-package-mixup?pivots=os-linux-ubuntu#i-need-a-version-of-net-that-isnt-provided-by-my-linux-distribution

https://manpages.ubuntu.com/manpages/xenial/man5/sources.list.5.html

https://learn.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#register-the-microsoft-package-repository

https://tecadmin.net/how-to-install-dotnet-core-on-ubuntu-22-04/
Does not support 3.1

Would need to migrate
https://learn.microsoft.com/en-us/aspnet/core/migration/31-to-60?view=aspnetcore-7.0&tabs=visual-studio

provisioning a dot net core Ubuntu Linux server on AWS

Some notes about the provisioning and setup:

• AWS EC2 is where the VMs are seen. Choosing the region is important - the dashboard will not show the VMs in your account if the region is not set correctly!
  
  
• AWS offers a very large selection of machine types. We need to research costs before choosing one. For example, when I checked the pricing calculator for Mumbai region, a12xlarge with 16 MB RAM was $0.217 hourly, c5a2xlarge was $0.207, c6a2xlarge was $0.2057, but c52xlarge was $0.362. We can't assume that any family will be lower based on name prefixes, though some series like d are much higher due to higher specs. 
  
  
• In order to provision users who can access the VM via amazon's web console, the method chosen was to create a user group, create policies to give permissions to users in that user group, and create IAM users in that user group. 
  
  
• Creating custom policies - since we wanted the users to be able to do all EC2-related tasks like starting and stopping the VM, the method used was to make a copy of the supplied "All EC2 permissions" policy, edit the JSON and instead of having * as the Resources to which it applied, filling in the Amazon Resource Name instead.
  https://docs.aws.amazon.com/managedservices/latest/userguide/find-arn.html
  
  
• Amazon gives custom login urls, like
  https://our-name.signin.aws.amazon.com/console
  https://docs.aws.amazon.com/signin/latest/userguide/introduction-to-iam-user-sign-in-tutorial.html
  After logging in, we need to go to EC2, choose Instances on the left-hand side, ensure that we have chosen the correct region in the drop-down list on top of the page, and select our instance. For restarting, we can then choose the Actions menu on top of the page, Manage Instance State --> Start / Stop / Reboot. We have set "Terminate protection" on, so that the Terminate option is greyed out - which should be used only to completely remove the instance.
• If we don't opt for a dedicated IP - called Elastic IP in AWS parlance - we can use the hostname like ec2-9-999-999-99.ap-south-1.compute.amazonaws.com where the nines seem to indicate the ip address. But I guess this name would change along with the dynamically assigned IP address if the machine were stopped and restarted after a while, so we need to assign an Elastic IP. One Elastic IP per instance are free if used on a running instance.
  

Since this post has already become long, I will put the software installation part on a separate post. 

mpg123 works well for icecast or shoutcast streams - curl to help it play https

mpg123 http://stream.sssmediacentre.org:8000

works well on raspberry pi, even doing retries so that playout resumes after short network outages, unlike browsers like Chrome or Firefox. The buffer is also a minute long, so that short outages of a few seconds are not noticed as gaps in audio.

But mpg321, which is the package which supports mpg123 now, does not seem to have support for https. With the help of curl as mentioned here,

~$ mkfifo pipe
~$ curl -L https://stream.sssmediacentre.org:8443/bhajan -o pipe & mpg123 pipe

Works well.

documentation on domain transfer from Godaddy to Nettigritty

https://domain.nettigritty.com/kb/servlet/KBServlet/faq1188.html

https://support.hostinger.com/en/articles/1583437-how-to-get-epp-code-from-godaddy

 

github pages, cloudflare and SSL

Many people have issues with Github Pages not issuing TLS certificates after following the official procedure, which is to first go to settings -> pages on the repo and set up the custom domain, and then set the CNAME for the domain to point to username.github.io.

The reason for this order of actions is to prevent domain hijacking, as github explains.

One of our domains had the same issue even after 24 hours, even though Cloudflare proxying was turned off, and the "Only DNS" mode was set. The solution was to remove the custom domain and immediately add it back in github repo settings. Then the TLS certificate request went through and the certificate was issued within a minute or two.

new whatsapp voice call spam

Whatsapp has had a feature for "Block and Report" for spam chats. Now there's a new game in town - missed whatsapp voice-calls from international numbers like
+62 857 6605 4116
+55 66 8473 9018
+7 931 703 90 02
etc.

What we can do is to block them. Tap on the number to open the Call Info page, then the three dots menu on top right (on Android), Block.

php echo does not reflect on screen till end of processing

With the code at https://github.com/hn-88/mktxt/ found that sometimes the messages would not be displayed on screen till the end of execution.

https://stackoverflow.com/questions/4669822/echo-messages-while-php-script-still-executes

- so it may be client dependent.

And yes - found that in Edge / Chrome / Chromium, the output is shown as soon as the page opens, with dynamic updating. With my version of Firefox, the output flush does not work due to browser caching.

adding multiple files in a single commit using github api

Currently we push each file separately like in this code snippet, but when I explored how to add multiple files, the implementation seems a bit too complex for our needs.

https://siddharthav.medium.com/push-multiple-files-under-a-single-commit-through-github-api-f1a5b0b283ae

https://github.blog/changelog/2021-09-13-a-simpler-api-for-authoring-commits/

"simpler" just means fewer number of calls.

 

Google Sign-in - old methods deprecated

I was attempting to get the Google username + email id for users using Google Sign-in to log on to an embedded google apps script, but the username was being returned blank.

Our earlier php code, modified from something like this was returning only email and not username when trying with the default scope or even with addScope(profile). Even the REST API didn't help.

Finally found the issue - this page states that 

"Warning: The support of Google Sign-In JavaScript platform library for Web is set to be deprecated after March 31, 2023. The solutions in this guide are based on this library and therefore also deprecated.

Use instead the new Google Identity Services for Web solution to quickly and easily sign users into your app using their Google accounts.

By default, new client IDs are now blocked from using the older platform library; existing client IDs are unaffected. New client IDs created before July 29th, 2022 may set the plugin_name to enable use of the legacy Google platform library."

So I modified the code to use the new Google Identity Services using this guide, and all was well. In order to not interfere with the earlier (working) authentication setup on the same server, which has a different set of authentication rules, the vendor, autoload etc are on a different path.

Code snippet:

$client = new Google_Client(['client_id' => $client_id]);

$payload = $client->verifyIdToken($_POST['credential']);
if ($payload) {
  //print_r($payload);
  //echo $payload['email'];
  //echo $payload['name'];
  // set the session variables needed by mediabank
  $_SESSION["userEmail"] = $payload['email'];
  $_SESSION["userName"] = $payload['name'];
  header('Location: /the-embedded-page.php');

get POST to work on PHP

Inputs from

https://stackoverflow.com/questions/9332718/how-do-i-print-all-post-results-when-a-form-is-submitted

One way would be something like

foreach ($_POST as $key => $value) {
        echo $key;
        echo $value; etc.

embedding a google apps script - too many redirects

There was a google apps script to be embedded in an iframe on one of our websites. I've done this before with no issues, as long as the XFrameOptionsMode is set properly in the script to ALLOWALL. But in this case, I was getting a too many redirects error. "accounts.google.com redirected you too many times."

Checking the network traffic (right-click -> Inspect -> Network tab), found that the app was redirecting to google's login authentication page, back to the app's original script.google.com address, and again, infinitely till the browser stops the loop after 25 times or so.

Checking the app's deployment, found that this was due to the deployment availability being set to "Anyone with a Google Account" instead of "Anyone". Once the deployment was changed to "Anyone", the embed worked fine. 

(This link has one approach in which validation can be done, but I didn't go through it in detail to check if it is embed-friendly.)

issues with a Windows server running a dot net app and mitigation

Copy-pasting my response to a message,  

server seems to be down again. It is happening quite frequently since yesterday....

I see two types of attacks on the server. Both are likely to be by automated bots, since such attacks are very common.

1. Remote Desktop

Attack

The remote desktop service (RDP, termservice) is being attacked. If a sufficiently large number of connections take place, the server will run out of memory, and other services will also start failing.

Mitigation

We can prevent these by simply stopping the remote desktop service when not in use. For this, please run the stoprdp.bat via SSH, logging in with putty etc, whenever you start RDP with restartrdp.bat, after you finish your work on the server. I have run stoprdp.bat now.

2. Attacks on the database via dotNET api

There are attacks on the database / dotNET app using the API exposed by the dotNET application. In the screenshots below (redacted in this blog post), you can see how multiple API controller endpoints are being targeted in a short time, with the database quickly reaching maximum retries.

Edit - adding another screenshot from another day - 

 

Mitigation

Assuming that these are not some tests being done by your team, we will need to

(a) check if the dot net app code-base is secured against all the usual attack vectors mentioned at

https://www.c-sharpcorner.com/article/securing-your-Asp-Net-web-applications/

In particular, exception handling via try-catch should be checked.

(b) add some sort of Denial of Service attack prevention, similar to
https://www.madskristensen.net/blog/block-dos-attacks-easily-in-aspnet/

Please note - directly using the code in the link above would not be safe, since our web app is much more complex.

Another DOS attack prevention tool is HackerSpray
https://www.codeproject.com/Articles/1112532/HackerSpray-Block-Brute-Force-and-DOS-Attacks

https://github.com/oazabir/HackerSpray

Another one is

https://stackoverflow.com/questions/19087512/how-to-restrict-dos-attack-with-web-api/56075128#56075128

(c) Prevent direct access to the API on our server, and allow ONLY access through Cloudflare.

This is supposed to mitigate many types of attacks.

 

I've now implemented blocking of all external requests on HTTP and HTTPS on the Sainin production server except via cloudflare's ip addresses

https://www.cloudflare.com/en-gb/ips/

 

For this, just had to copy-paste the ipv4 addresses from the list above, make them comma-separated, and paste them into a newly created inbound security rule in Azure portal for the nsg (network security group) of the server in question. And, of course, make sure that the relevant services' DNS records are proxied through cloudflare and not "DNS only".

 

 

web fonts for properly rendering multilingual website

When creating a website on github pages with text in English, Hindi and Telugu, the first attempt was to just copy-paste the content from a Google Doc exported as HTML. But that had two problems.

1. The text was represented as characters like &#2309;&#2306;&#2327;&#2381;&#2352;&#2375;&#2332;&#2368;
   not human-readable for easy editing.
   
   
2. Firefox was not rendering some of the characters properly, like
   
   instead of

   

   
   
   So, tried incorporating web-fonts - reading up
   
   https://teck.in/free-google-web-fonts-for-indian-languages.html
   https://fonts.google.com/knowledge/using_type/using_web_fonts
   https://www.zachleat.com/web/css-tricks-web-fonts/
   https://www.w3schools.com/css/css3_fonts.asp
    
   Google's free multilingual font Noto Sans was what I used. 
   
   
   CSS like
   @font-face {
               font-family: EnglishBold;
               src: url(images/notosans-black.ttf);
           }
   
           .eng {
               font-family: EnglishRegular;
           }
   

SSH - host key has changed

Github has changed their host key - 

https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/

So, when trying to clone a repo of mine via the SSH url, I got the message

Offending RSA key in /home/myusername/.ssh/known_hosts:3
  remove with:
  ssh-keygen -f "/home/myusername/.ssh/known_hosts" -R "github.com"
Host key for github.com has changed and you have requested strict checking.

So I did

ssh-keygen -f "/home/myusername/.ssh/known_hosts" -R "github.com"

and answered "yes" when asked to save the new key.

Expanding the sky in panoramic photos

Useful for changing the aspect ratio of panoramic photos, which are usually much more wide than they are tall - How to expand sky in any photo with GIMP - YouTube

Scale up a copy on a new layer.
Create layer mask on original layer.
Select the layer mask in the layers window to select it for editing.
Gradient black to white Foreground to transparent to blend in the orig with the scaled up sky.

What I did for the Tokyo pan was, since the sky was just blue,
did a rectangular selection of only sky,
copied the selection, pasted to new layer
and for that layer, Layer -> resize layer, expand upwards.

Scripting Stellarium 2023

There are some changes after my earlier post about creating views of the night sky using stellarium. And I've forgotten the locations of some controls. So, a set of reference notes.

• Initial setup - to set the place and time to Puttaparthi, 14.17N, 77.81E, 23 Nov 1926 4:30 am  - (F6) Location window and (F5) Date Time window. (F2) Configuration window --> Time tab to start at that time every time, or to stop time at that point. 
  
  
• Switch off captions with (F4) Sky and Viewing Options --> Sky tab, Labels and Markers.
  
  
• Switch off detailed information for selected objects in (F2) Configuration window --> Main tab, if we leave only Name checked, the name of the selected object is shown on top left corner, which would not be visible in a fulldome frame. 
  
  
• To set field of view to 180 degrees, scroll-wheel on mouse - slightly inaccurate. More accurate to use scripting, below. We can change mapping to Fisheye from (F4) Sky and Viewing Options --> Sky tab.
  
  
• Use Ctrl+T to remove/toggle the controls from view. Ctrl+S will save a screenshot. The screenshot directory is set on (F2) Configuration window --> Tools tab. Or use scripting, as below.
  
  
• Scripting example for screenshots - https://renenyffenegger.ch/notes/Wissenschaft/Astronomie/tools/Stellarium/script/API/core/screenshot
  In our case, instead of StelMovementMgr.zoomTo(25, 0.01) we wanted to change / set the FOV from 195 to 30 degrees - progressively zooming in. Scripting console (F12) or run saved scripts with (F2) Configuration window --> Scripts tab.
  for (i=195; i>30; i--) {
  StelMovementMgr.zoomTo(i, 0.01);
  core.wait(0.01);
  // Take screenshot
  core.screenshot('sirius-zoom') ;
  }
  
  
•  Unfortunately, the values mentioned in a previous post for "good-looking" night sky, which was for version 0.9 or something like that, don't seem to be valid now for version 1.2 or 2023. (F4) Sky and Viewing Options --> -> Sky tab -> Absolute scale = 1 and Relative scale = 0.5 with Limit magnitude = 5.5 or 6.0 makes the stars look good for 1080p fulldome screenshots in the current version 1.2. 

Scripting Software Bisque Seeker Theatre Edition on Windows 11

Adding to my earlier tests with Seeker Theatre Edition, on Windows 11, there are some more things to keep in mind. 

• Offline rendering with 'create movie' - only .png and .tga export works. mov or jpg don't work. This is probably due to not installing Quicktime.
  
  
• After .png export, we can create a movie with ffmpeg, for example
  ffmpeg -r 30 -f image2 -i "Our-frame_%06d.png" -vcodec libx264 -crf 15  -pix_fmt yuv420p Our-movie-4k-frames.mp4
  
  
• Copy-pasting from script examples, zooming in to our location with the following, (but this may work only after setting the time to "Now" at mid-day here :)
  
  <Software Bisque> 1 <HIDE> 1
  !Waypoint start
  <HIDE> 1
  <WAYPOINT2> 2460041.90072305361 -0.957039617708 -0.292439496712 7.64553844115e-06 -0.51761781304 -0.847675565812 0.116266653641 -0.0507515739738 0.166066153524 0.984807753012
  <HIDE> 0
  !Waypoint end
  <SSGOTO> 3 0 0 40 1.16
  <SSROLL> 0
  !Waypoint start
  <HIDE> 1
  <WAYPOINT2> 2460041.90013993671 -0.957075431798 -0.292483987095 1.50676317142e-05 -0.51761781304 -0.847675565812 0.116266653641 -0.0507515739738 0.166066153524 0.984807753012
  <HIDE> 0
  !Waypoint end
  

raspberry pi setup redux

Setting up a Raspberry Pi 4 as an Icecast player with a local video playing on the screen, notes below:

• Interesting idea for chromium tabs for display, but we have not used it, https://blog.gordonturner.com/2019/07/23/raspberry-pi-switch-chromium-browser-tabs/
  
  
• Remmina refuses to connect to realvnc server on pi - simplest solution was to install realvnc client.
  
  
• Unfortunately, high CPU usage with VLC playing the local video file. Tried 64 bit Raspberry Pi OS, 32 bit Raspberry Pi OS, finally got good results with 32 bit "Legacy" Raspbian Buster and omxplayer. With omxplayer, CPU usage is only 5% or less. With VLC, either VLC (in 64 bit) or Xorg (in 32 bit) uses 100%. Unfortunately, omxplayer has been removed from newer versions of Raspberry Pi OS. Maybe we can use vlc later after exploring various output device options, but as of now,
  runStreamAndVideo.sh has
  
  mpg123 http://stream.radiosai.org:8000 &
  #cvlc "sssmc-prconn.mp4" --fullscreen
  omxplayer -o local --loop /home/pi/sssmc-prconn_360p.mp4
  
  
• For setting up wifi after fresh installs, the easiest option was to ssh into the device using an ethernet cable nearby, and use
  sudo raspi-config
  System -> Wireless LAN
  The method of using a wpa_supplicant.conf file created in /boot seems to work only if
  (a) the country is already set
  (b) the conf file has Unix style line endings.
  The following sudo nano /boot/wpa_supplicant.conf worked once for me, didn't work for a fresh install -
  

  country=IN  #omit if US
  ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
  update_config=1
  network={
      ssid="Our-SSID"
      psk="Our-Password"
  }

  
  
• Used our old technique of creating a .desktop file in .config/autostart directory + enabling auto login to GUI + wait for network to boot (wait for network is needed, because otherwise mpg123 doesn't play.)
  
  
• sudo apt install realvnc-vnc-server
  to install VNC server, and
  sudo raspi-config
  Interface options -> VNC to enable the service at boot.
  
  
• We can control the volume easily with ssh, with the alsamixer command, up arrow and down arrow.
  
  
• Default audio device is analog out. To change to HDMI, a HDMI monitor needs to be connected, then we can use
  sudo raspi-config
  System -> Audio
  (or using VNC, right-clicking on the volume applet on top right corner)
  
  
• Instead of using our earlier method of an IR remote control to shut down the machine, we're using RaspController android app instead.
  (The device drivers have changed yet again from our earlier method.
  sudo modprobe lirc_rpi
  was replaced by
  sudo modprobe gpio_ir
  and now, with Raspbian Buster, it is
  sudo modprobe gpio_ir_recv )
  
  
• We can set the hostname of the Raspberry pi using
  sudo raspi-config
  System -> Hostname
  and we're supposed to be able to reach it using hostname.local on the local network for "devices which support mDNS" or Avahi. On the pi,
  sudo apt-get install avahi-daemon
  sudo apt-get install avahi-utils
  
  But avahi-browse -a -d local didn't seem to work. Probably some syntax issue.
  
  Also, the hostname.local didn't seem to work out of the box for some Android devices. So, set up a static ip by moving up the DHCP pool to a more restricted set on the router, and setting up the static ip by right-clicking on the wifi icon using VNC,
  Wireless and Wired networking settings ->
  Configure -> SSID (choose our SSID)
  or Configure -> interface (choose wlan0)
  and set the required IP address etc. by unchecking the "Automatic" box.