importing emails into google groups

Sending invites to a large number of email ids from google groups, found the following. 

• The "500 per day" limit seems to include those email ids which were selected for sending, and then were not sent since they were already members of the group.
• So, sending invites one by one - only one email at a time - would result in the maximum invites sent per day. But - this is very time consuming. Even with an automated script made with SikuliX, approximately 15 seconds per email id if processing one email at a time. 
• Filtering out existing users makes the process much smoother. Script for that shared below. But the list still needs manual filtering of typos like gamil.com or gmailcom or gmail com etc. 

For comparing this list with the list of subscribers made earlier, the technique discussed here was modified to make the formula
=IF(ISERROR(MATCH(A143,Sheet2!B142:B,0)),"",A143)

That is, the existing users were copied into a sheet called Sheet2 on the same Google spreadsheet. This was further improved using the technique of ARRAYFORMULA discussed here, so that the entire column of thousands of cells were populated with one click. That is, enter the formula in the first cell, and instead of specifying just A2, mention the entire column in the form A2:A, and hit Ctrl+Shift+Enter after entering the formula.

Then a modified SikuliX script was used to do the import 10 emails at a time, which I've copy-pasted below. Captcha is not required for 10 emails or less. Alt+Shift+C is the key combination to abort the SikuliX script.

from time import sleep

sleeptime=0.5

lastlineofsheet=Location(412,656)

#Addbutton=Location(1068,286) changed to 75 percent below

Addbutton=Location(982,202)

#DirectAddToggle=Location(840,670) changed to 75 percent below

DirectAddToggle=Location(895,580)

#MembersToAddBox=Location(880,320) changed to 75 percent below

MembersToAddBox=Location(922,346)

#WelcomeMesgBox=Location(915,415)

#sometimes the gmail profile comes in the above spot

#WelcomeMesgBox=Location(1180,446) changed to 75 percent below

WelcomeMesgBox=Location(1138,438)

#TextEditOnTaskbar=Location(751,751)

#Taskbar location is often non-reproducable, avoid.

TextEditOnTaskbar=Location(600,31)

InsideTextEdit=Location(650,400)

#MemberListOnTaskbar=Location(435,751)

#SendButton=Location(1164,555) changed to 75 percent below

#SendButton=Location(1146,520) changed for 10 at a time to

SendButton=Location(1128,543)

while(True):  

  click(lastlineofsheet)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  sleep(sleeptime)

  type("c",KeyModifier.CTRL)

  sleep(sleeptime)

  click(Addbutton)

  sleep(sleeptime)

  sleep(sleeptime)

  sleep(sleeptime)

  click(DirectAddToggle)

  sleep(sleeptime)

  click(MembersToAddBox)

  sleep(sleeptime)

  type("v",KeyModifier.CTRL)

  click(TextEditOnTaskbar)

  click(InsideTextEdit)

  sleep(sleeptime)

  type("a",KeyModifier.CTRL)

  type("c",KeyModifier.CTRL)

  #click(MemberListOnTaskbar)

  sleep(sleeptime)

  click(WelcomeMesgBox)

  sleep(sleeptime)

  type("v",KeyModifier.CTRL)

  

  popup("waiting for paste")

  click(SendButton)

  popup("waiting for no error")

  #sleep(7.0)

  

  click(lastlineofsheet)

  sleep(sleeptime)

  type(Key.DOWN)

  sleep(sleeptime)

  #popup("waiting for next")

GCP costs for 2 16 GB VMs

Google Cloud Platform's Billing -> Overview shows the number of free credits remaining as well as the number of days remaining. 

Running 2 instances of e2-highmem-2 (2 vCPUs, 16 GB memory) cost approximately Rs. 600 or US$8 per day, or $250 per month, or $3000 per year.  

URL addressability API removed from Alfresco

URL Addressability API (superseded by WebScripts)

https://hub.alfresco.com/t5/alfresco-content-services-hub/alfresco-community-5-0-b-release-notes/ba-p/289468

Interestingly, this is in version 5-0-b. But ticket-based addressing of image URLs is working locally (I believe?) in 5-0-d. Does Alfresco's versioning go from d to c to b? Or is my understanding of the URL Addressability flawed?

The URL addressability documentation says that it has been obsoleted - 5

https://hub.alfresco.com/t5/alfresco-content-services-hub/url-addressability/ba-p/291489

mounting remote volumes and cloud drives - sshfs and rclone

Rclone can mount a wide variety - over 40! - of cloud storage options as drives on Linux, Windows and Mac. If we're ssh-ing into a remote machine and want to authenticate, we can tunnel the temporary web server which Rclone opens on port 53682.

Another option for remote mounting is sshfs. 

When I tried it with key-based authentication, 
sudo sshfs -o allow_other,default_permissions,IdentityFile=~/.ssh/id_rsa bitnami@xxx.xxx.xxx.xxx:/ /mnt/mymount
I got errors - connection reset by peer. 

Apparently, the full path to the key file is needed - 
sudo sshfs -o allow_other,default_permissions,IdentityFile=/home/myhomedir/.ssh/id_rsa bitnami@xxx.xxx.xxx.xxx:/ /mnt/mymount

Then it worked. 

deploying war files on wildfly

There seem to be multiple ways to deploy WAR files in Wildfly. 
https://www.baeldung.com/jboss-war-deploy

Just copying the WAR file to the wildfly/standalone/deployments directory seems the most easy for us to do. And if the WAR file is already exploded, we also need to create a appname.war.dodeploy marker file as mentioned in RedHat's documentation.

Change configuration files only after stopping wildfly -
/opt/bitnami/ctlscript.sh stop wildfly
on Bitnami Wildfly VM.

Takes ~15 sec to stop Wildfly

Takes ~5 minutes to start Wildfly and deploy wars if there are errors, but only a few seconds (less than a minute) if there are no errors.
Monitoring the log is a good way to judge if Wildfly is running or stopped - 
sudo tail -f /opt/bitnami/wildfly/standalone/log/server.log

Examples of configuration files which might need to be set up - 
wildfly/standalone/deployments/CAS.war/WEB-INF/deployerConfigContext.xml
wildfly/standalone/deployments/CAS.war/WEB-INF/cas.properties

wildfly/standalone/configuration/standalone.xml
wildfly/modules/org/CustomAppName/properties/configuration/main/Environment.properties

If a WAR file deployment fails with Null Pointer Exception (seen in the server.log file mentioned above), there is a good chance that this is due to some non-existent file or path mentioned in one of the properties files above. The usual culprit would be Windows-style paths for log or conf files. 

If a large number of WAR files are to be deployed, deployment may fail with Out of Memory errors. We can increase heap size and MetaSpace size by editing /opt/bitnami/wildfly/bin/standalone.conf line for JAVA_OPTS like

JAVA_OPTS="-Xms2G -Xmx10G -XX:MetaspaceSize=2G -XX:MaxMetaspaceSize=6G -Djava.net.preferIPv4Stack=true"

Edit - Adding data-sources - if needed, we can deploy the db driver as a module, and configure via the data source via the web console. For the postgres driver, I directly downloaded the jar from

http://localhost:9990/console/index.html#deployment;deployment=postgresql-42.2.14.jar

so gave the 

 <resource-root path="postgresql-42.2.14.jar"/>

Using the jboss-cli method for creating the db driver as a module, 

wildfly/bin/jboss-cli.sh

ran as sudo, needed to put the entire command in one line

/subsystem=datasources/jdbc-driver=postgresql:add(driver-name=postgresql, driver-module-name=org.postgresql, driver-class-name=org.postgresql.Driver)

Outcome success.

location of Apache configuration files on Bitnami Wildfly VM and easy SSL configuration with LetsEncrypt

As of this writing in Dec 2020, this discussion on configuring redirects is not directly applicable to Bitnami's Wildfly VM. The Proxy Pass commands etc are located in /opt/bitnami/apache2/conf/vhosts/wildfly-http-vhost.conf
and
wildfly-https-vhost.conf

Bitnami makes it easy for us to use https by providing an interactive script which will set up SSL certificates with LetsEncrypt as well as a cron job for renewal - 
sudo /opt/bitnami/bncert-tool

check if a service is running

One way to easily check for a service listening on a port for a Linux server running systemd is using ss - need not be run as root.

ss -tulpn

shows all listening ports 

ss -tulpn | grep 5432

would indicate if PostgreSQL is running or not, and so on.

quick temporary phpPgAdmin installation using Bitnami

On Bitnami's Alfresco VM, installing phppgadmin using 
sudo apt-get install phppgadmin
causes problems since apt-get on the platform (Debian 10) tries to install apache2 which is already installed as httpd on the Bitnami stack. 

Bitnami's LAPP stack has phpPgAdmin, so one can install it as a user, disable or stop postgres from that stack, change phppgadmin/htdocs/ conf file for accessing any other db which we may have, change apache's port if another server is running at 8080, to 18080 for example, by modifying
apache2/conf/httpd.conf
and
apache2/conf/bitnami/bitnami.conf

Virtual hosts, host alias equivalents and LetsEncrypt SSL for IIS on Windows Server

The equivalent of adding virtual hosts in Apache is adding Sites in IIS. And similar to adding a host alias, what is done is to add a site binding. IIS Manager -> Servername -> Sitename -> Edit site 

Then, getting SSL certificates from LetsEncrypt and applying them - 
https://weblog.west-wind.com/posts/2016/feb/22/using-lets-encrypt-with-iis-on-windows
suggests using LetsEncrypt-Win-Simple which is now WinAcme. Very simple interactive script as described on their website.

Also, various possible reasons for IIS errors are listed at https://superuser.com/questions/741486/page-not-found-in-iis-but-its-there

Bitnami Wildfly VM default user is called user and not manager

Bitnami's documentation mentions that the username to access the wildfly admin console is manager. But with the credentials supplied, I was unable to log on to the VM deployed using GCP marketplace (Version: 21.0.2-0-r02). Then, looking at how to change the credentials, looking at the file /opt/bitnami/wildfly/standalone/configuration/mgmt-users.properties found that the username is actually user and not manager. Then I was able to log on.

Monitoring RAM usage of a VM on GCP

CPU usage is shown by default on the monitoring console, but in order to see how much RAM is used by a VM on Google Cloud Platform, one has to first install the agent, then in the Monitoring dashboard, open Metrics explorer, choose the GCE VM Instance Resource Type and choose the agent called Memory Usage (agent.googlapis.com/memory/bytes_used). This last part is a bit non-intuitive, because there are many entries labelled Memory Usage, and we have to scroll down till agent.googlapis.com/memory/bytes_used. The resulting chart can be saved to an existing dashboard or we can create a new dashboard.

tried and failed to change Alfresco port in Bitnami VM

In some of my initial attempts, working with Bitnami's Alfresco VM, I tried a few ways to change the port and failed. Just documenting what does not work - 

https://docs.alfresco.com/4.2/tasks/share-change-port.html

mentions changing the <web-extension>/share-config-custom.xml file, not just in alfresco-global.properties -

https://docs.alfresco.com/4.2/concepts/share-configuration-files.html

The path above is tomcat/shared/classes/alfresco/web-extension/share-config-custom.xml

But probably these docs assume that Tomcat is already configured to run on the fresh port, and that is probably why just doing the above failed for me.

And finding the location of the Apache configuration files on this Bitnami VM which were doing the proxypass from port 8080 to port 80 - 

/opt/bitnami/apache2/conf/bitnami/bitnami.conf
had a line 

Include /opt/bitnami/apache2/conf/bitnami/bitnami-apps-prefix.conf

bitnami/bitnami-apps-prefix.conf has

Include /opt/bitnami/apps/alfresco/conf/httpd-prefix.conf

httpd-prefix.conf has

Include "/opt/bitnami/apps/alfresco/conf/banner.conf"

Include /opt/bitnami/apps/alfresco/conf/httpd-app.conf

and finally 

/opt/bitnami/apps/alfresco/conf/httpd-app.conf

has the proxypass lines for alfresco and share, like

<Location /alfresco>
ProxyPass ajp://localhost:8009/alfresco etc.

MySQL and PostgreSQL command line cheat sheet

 MySQL

From https://www.a2hosting.in/kb/developer-corner/mysql/managing-mysql-databases-and-users-from-the-command-line

mysql -u root -p
(or bitnami user for bitnami VMs)

show databases;
use dbname1; 
show tables; 

create database dbname;
use dbname;
GRANT ALL PRIVILEGES ON *.* TO 'username'@'localhost' IDENTIFIED BY 'password';

Example SQL script import usage

mysql -u username -p < example.sql

And if required,

drop table tablename;

or

drop database dbname;

For cloning a database, this page has instructions, which also needed the permissions as detailed here. as root user, opening mysql and

GRANT PROCESS, SELECT, LOCK TABLES ON *.* TO 'bn_alfresco'@'localhost'; 

mysqldump -u bn_alfresco -p bitnami_alfresco -r bnalf.sql

mysql -u bn_alfresco -p

CREATE DATABASE my_project_copy;

USE my_project_copy;

SOURCE bnalf.sql;

PostgreSQL

For Debian-based distributions, 
sudo apt-get install postgresql-client
psql -h <REMOTE HOST> -p <REMOTE PORT> -U <DB_USER> <DB_NAME>

\l

to show databases, or

select datname from pg_database;

And to show tables,

\c databasename
\dt

Edit: For exporting the database, various options for pg_dump, most common would be
sudo su postgres
pg_dump dbname > dumpfile.sql
or if local authentication is not supported in the hba.conf, and username/pw auth is supported,
pg_dump --no-owner --dbname=postgresql://dbuser:dbuserpasswd@host:port/dbname > dump.sql

From bash shell, as postgres user, can import like
psql dbname < dbdump.sql

From https://medium.com/coding-blocks/creating-user-database-and-adding-access-on-postgresql-8bfcd2f4a91e

create database dbname;
create user username;
(To change, alter user username with encrypted password 'MyPassWord';)
grant all privileges on database dbname to username;

If the database was imported as postgres user, and another user is given all privileges as above, a possible issue is "permission denied when accessing schema postgres". My rough workaround was to create another super user (since I did not know the postgres user's password and didn't want to change it and potentially break things) by logging on as root, su postgres, and 
CREATE USER newadmin WITH SUPERUSER PASSWORD 'newadminpassword';

changing bitnami alfresco db to postgresql and resetting the password

After following the steps below, was unable to log on as alfresco admin user - the credentials were not being accepted. The solution is at the end of the post.

https://docs.alfresco.com/4.0/tasks/postgresql-config.html
https://computingforgeeks.com/install-postgresql-11-on-debian-10-buster/

To set the postgres alfresco user password MyPassWord

sudo su -
sudo postgres

psql
alter user alfresco with password 'MyPassWord';

Downloaded driver from

https://jdbc.postgresql.org/download.html

https://docs.bitnami.com/bch/infrastructure/tomcat/get-started/get-started/

says TOMCAT_HOME is /opt/bitnami/tomcat

but for the Bitnami Alfresco Community VM, found it was

/opt/bitnami/apache-tomcat

https://docs.alfresco.com/6.1/reuse/conv-syspaths.html

indicates location of <classpathRoot> as tomcat/shared/classes, so on this bitnami VM,

/opt/bitnami/apache-tomcat/shared/classes

Ask the database the location of pg_hba.conf as mentioned at

https://askubuntu.com/questions/256534/how-do-i-find-the-path-to-pg-hba-conf-from-the-shell

In our case doing
psql -t -P format=unaligned -c 'show hba_file';
as postgres user, 

sudo nano /etc/postgresql/11/main/pg_hba.conf to add the line

host all all 127.0.0.1/32 password

and

systemctl restart postgresql

Then, restarting alfresco with

/opt/bitnami/ctlscript.sh restart

brings up alfresco by bootstrapping the required tables in the new database, but can't log in - the credentials are not accepted. 

Tried various methods as given at https://docs.alfresco.com/5.1/concepts/admin-password.html and http://www.giuseppeurso.eu/en/alfresco-tips-and-tricks-1-reset-the-admin-password/ using hashes as given at https://blog.atucom.net/2012/10/generate-ntlm-hashes-via-command-line.html and so on. But did not work.

The solution was:

Delete or comment out the line with admin credentials at /opt/bitnami/apache-tomcat/shared/classes/alfresco-global.properties like

#alfresco_user_store.adminusername=admin
#alfresco_user_store.adminpassword=209c6174da490caeb422f3fa5a7ae634

so that when alfresco is restarted with

/opt/bitnami/ctlscript.sh restart

with a blank database, the admin username and password is reset to admin admin. After logging in, the password can be set as desired from the web console.

Installing activemq on Debian 10

Following
https://www.howtoforge.com/tutorial/debian-activemq-message-broker/

to install activemq on Debian 10 for a Bitnami Alfresco VM starting from step 2, found that activemq failed to start with 
ERROR: Configuration variable JAVA_HOME or JAVACMD is not defined correctly. (JAVA_HOME='', JAVACMD='java')

Set the environment variables as mentioned at 
https://stackoverflow.com/questions/31400148/apache-activemq-5-11-1-doesnt-start-in-ubuntu
in this case, for the bitnami stack, I defined  JAVA_HOME and JAVACMD in /etc/default/activemq as

JAVA_HOME="/opt/bitnami/java" 

JAVACMD="/opt/bitnami/java/bin/java"

Now 

systemctl start activemq

systemctl enable activemq

systemctl status activemq

shows

● activemq.service - Apache ActiveMQ

   Loaded: loaded (/etc/systemd/system/activemq.service; disabled; vendor preset

   Active: active (running) 

etc. 

The default username and password for admin user are admin admin, and for non-admin user, user password. The default port is 61616, the web console configuration panel can be accessed from localhost:8161.

public key authentication fails with agent refused operation

I generated a private+public key pair on a Windows 10 machine, copied the key over to a Linux Mint 18.03 (based on Ubuntu Xenial 16.04) machine and used it to ssh into that machine with no issues. Then I generated another key pair on another Windows 10 machine, which had been upgraded to 10 from a lower version of Windows, and tried to use that key pair to log on to that machine from the Linux machine, but it failed with "agent refused operation". 

Looked at various possible reasons for this failure, finally found that the private key was of a different size, and also had the string OpenSSH PRIVATE KEY instead of RSA PRIVATE KEY at the beginning and end. This might have been due to a different OpenSSH server from MLS software I had tried installing on the 2nd Windows machine, or it could have been due to something else. Anyway, I just created one more key from the 1st Windows machine and used that key pair instead for authenticating into the 2nd machine. That worked fine.

Edit - This discussion seems to be related, and has an explanation.

Load testing a moodle server - log in with token parsed with Beautiful Soup

While trying automated logins with a simplistic python script like this -
import requests

url = 'https://theserver.tld/login/index.php'
myobj = {'username': 'theUserName', 'password': 'ThisIsThePassWord!', "logintoken": "hCBVa2gzAWqSa2u7iNzdzlE9kwpVehFf"}
x = requests.post(url, data = myobj)
print(x.text)

Moodle would return authentication failure - it was checking the logintoken, which was hard-coded above. R used a modified script to get the logintoken correctly first, which then works - 

import sys
import requests
from bs4 import BeautifulSoup 

#driver = webdriver.Chrome()
# using this technique, the testing machines would run out of RAM
# after a dozen or so Chrome instances were loaded!

#def login(url,usernameId, username, passwordId, password, submit_buttonId):

#   driver.get(url)
#   driver.find_element_by_id(usernameId).send_keys(username)
#   driver.find_element_by_id(passwordId).send_keys(password)
#   driver.find_element_by_id(submit_buttonId).click()

#myMoodleEmail = sys.argv[1]

#login("https://server.tld/login/index.php", "username", myMoodleEmail, "password", "ThePW!", "loginbtn")

#driver.close()

login_data = {

    #'logintoken': 'XPc8nw0a2gLXpgsn6njpehwEoW43mzYQ',
'username': sys.argv[1], 'password': 'ThePW!'

}

headers = {'user-agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Mobile Safari/537.36'}

with requests.Session() as s:

    url = "https://theserver.tld/login/index.php"
    r = s.get(url)
    soup = BeautifulSoup(r.content, 'html5lib')
    login_data['logintoken'] = soup.find('input', attrs = {'name': 'logintoken'})['value'] 
    r = s.post(url, data = login_data, headers = headers)
    print(r.text)

So I didn't have to try JMeter as the moodle documentation suggests, 
https://moodle.org/mod/forum/discuss.php?d=313489

which points to the moodle documentation on load testing. 
https://docs.moodle.org/dev/JMeter#Make_JMeter_test_plan

Edit: Example usage of JMeter - 
jmeter -n -t [jmx file] -l [results file] -e -o [Path to web report folder]

Edit: The final result was, on AWS, "R5A Quadruple Extra Large - 128 GB RAM, 16 Virtual CPUs, 10 GB Network" - for 2000 users to log in concurrently. 

some ssh tunnel notes

While setting up an ssh server on Windows, found that PermitTunnel is not supported by the Microsoft port of OpenSSH-Server. But then, PermitTunnel is not required to do local port forwarding at the client side, like 
ssh -L 5990:192.168.12.34:5990 user@server.name

And interestingly, this sort of "poor man's VPN" using such an ssh tunnel seems to be much faster than OpenVPN etc as mentioned here, due to having less overhead.

Installing OpenSSH on Windows 10 which had been upgraded from Windows 7

When trying to install OpenSSH on a machine running Windows 10 which had been upgraded from Windows 7, the Optional Features under Apps -> Apps and Features -> Manage Optional Features as mentioned in the documentation was completely blank. But the install with Powershell (run as administrator) method as mentioned in the same documentation page worked.
Add-WindowsCapability -Online -Name OpenSSH*

For troubleshooting permission issues, this post or this one may be useful - it talks about using PowerShell to change the permissions - ACL - on the authorized_keys file. Or, just make sure the permissions are appropriate using windows explorer -> file properties -> security -> advanced, 

For the private key (and for the authorized_keys file?), 

change the owner to the login user (if it's not already).

Disable inheritance (if it's set).

Remove all permissions for every one but this user.

Give this user “Full Control”.

avoiding multiple redirects in cloudflare

Changing the SSL settings from Flexible to Strict fixed the too many redirects error for one of the domains - https://support.cloudflare.com/hc/en-us/articles/115000219871-Troubleshooting-redirect-loop-errors

adding share links

Used this medium post which links to this codepen as a starting point for implementing share over email (a mailto link), copy to clipboard, and sharing via facebook, twitter and linkedin. Our code is on my (private) github repo.

firebase authentication tutorial without the need to set up billing - but ...

The official firebase auth tutorial from Google needs you to set up billing. This tutorial made by the community doesn't need you to set up billing. 

But since it's based on old versions of libs, with the current (newer) version of falcon, gives error at

from falcon.version import __version__  # NOQA

exporting users from a large google group

We wanted to export all the current subscribers to a large google group (>30k), and the export members functionality doesn't seem to work even for owners of the group*. So, brute-forcing to the rescue. Used SikuliX to automate the process of going to each page of subscribers, select all, copy, paste into a text file, save, and move to the next page of 100 users after sorting the users by join date earliest first. Since I didn't have opencv installed using apt on this machine and didn't want to break any opencv3.x configuration I might have done, used hard-coded locations on screen instead of Sikuli's feature detection. So, I had to manually click popups to ensure proper wait times after each operation. The script looked like this - 

 

i=18 #the page from which we're starting

while(True):  

  dragDrop(Location(721,265), Location(821,365)) #drag at the top end

  type("a",KeyModifier.CTRL)        #Ctrl a - select all

  type("c",KeyModifier.CTRL)        #Ctrl c - copy

  click(Location(1301,565))         #click inside text editor

  type("v",KeyModifier.CTRL)        #Ctrl v - paste

  popup("waiting for paste")

  click(Location(1336,450))         #click save toolbar icon

  popup("waiting for save dialog")

  click(Location(726,102))          #click on save dialog

  type(str(i))                      #save filename = pagenumber

  popup("waiting for save")

  click(Location(1311,720))

  popup("waiting for creating new file")

  click(Location(1276,450))

  i=i+1

  click(Location(1207,232))

  popup("waiting for pageload")

Then the following python script helped parse it into a csv - 

#!/usr/bin/env python3

import sys

count = 0

for line in sys.stdin:

    #print(line)

    if line.startswith("Profile image"):

        Profileim = line

        username = next(sys.stdin).strip()

        ExtOrRadiosaiId = next(sys.stdin).strip()

        if (ExtOrRadiosaiId == "External"):

            emailid = next(sys.stdin).strip()

        else:

            emailid = ExtOrRadiosaiId

        print('{},{}'.format(username, emailid))

        count = count + 1

        

print(count)

by calling it as 

cat * | parsedumpinfile.py > mycsvfilename.csv

Earlier, I had thought about importing the txt files into a spreadsheet and processing the fields there, creating a formula to copy every sixth (email id) to column next to username, taking care to select six + six + six before dragging down, using a modified method from

https://www.wikihow.com/Select-Alternate-Rows-on-a-Spreadsheet

to select every sixth row. Or using autoformat in LibreOffice Calc, which looks like this.

But the python script above is much easier and cleaner, of course. 

*Edit - For a group inside a GSuite account, this method using Directory API might work. But GSuite api and googlegroups.com api are different. Only brute-forcing works for googlegroups.com :)

low cost video capture and streaming

How times have changed since the days of the $1000 Matrox RT2000 which we purchased in 1999-2001. This HDMI frame-grabber costs only $35. But it's just a simple HDMI to USB conversion, raw frames with no encoding. So the PC needs to be capable enough to handle the high-bitrate stream and do encoding on the fly. Since these are generally used by gamers for streaming, beefy machines are not an issue. And OBS Studio is the software of choice. Then, OBS Studio has "BrowserSource" which can be used to capture - no hardware required, except OBS Studio's own hardware requirements. 

All these could be possibilities if Google goes ahead with its plan to remove access to Google Meet recording ability for Education customers.

creating and distributing private keys as pem files

I wanted to create and distribute some keys as filename.pem for people to ssh into a server like

ssh -i filename.pem username@machine.name

as is done for qwiklabs. With a bit of digging, found that the pem file is actually the private key and not the public key, and the whole process would be as follows.

ssh-keygen -t rsa -m PEM

cd .ssh

cp id_rsa filename.pem

cp id_rsa.pub authorized_keys

chmod 600 authorized_keys

References:

https://docs.oracle.com/en/cloud/paas/integration-cloud/ftp-adapter/generate-keys-pem-format.html

https://hnsws.blogspot.com/2006/01/adding-rsa-auth.html

https://serverfault.com/questions/706336/how-to-get-a-pem-file-from-ssh-key-pair

https://inkel.github.io/posts/pem-to-authorized-keys/

How to log on using the pem file with PuTTY on windows is via PuTTYgen as explained at 

https://aws.amazon.com/premiumsupport/knowledge-center/convert-pem-file-into-ppk/

for example.)

setting up letsencrypt certificates for multiple virtual hosts with apache

The preferred method is to just run certbot multiple times with each required domain, as mentioned at https://www.digitalocean.com/community/tutorials/how-to-set-up-let-s-encrypt-certificates-for-multiple-apache-virtual-hosts-on-ubuntu-14-04 

So, sudo certbot --apache -d example.com -d www.example.com

and later

sudo certbot --apache -d example2.com -d www.example2.com

or whatever. And then add to root's cron,

15 3 * * * /usr/bin/certbot renew --quiet

which checks every morning at 3:15 AM and renews any certs with validity less than 30 days more.

troubleshooting a ruby on rails web server

To troubleshoot a web server running ruby on rails, checked the history of the ssh login I had been supplied (up arrow, up arrow, etc) and found that the admin had started rails using

rails s -p 5001 -b <ipaddress> -d
rails s -p 3001 -b <ipaddress> -d

for the two sites, and that they were running apache to proxy these ports to the two websites. The sites-available had virtual hosts configured like

DocumentRoot /full/path/to/rubyonrails/project/production
...
ProxyPass / http://xx.yy.zz.ww:5001/
ProxyPassReverse / http://xx.yy.zz.ww:5001/

and so on.

Found that the apache sites-enabled directory had some wrong entries + duplicate entries, which were causing apache to go back to the default configuration with no virtual servers, hence the various errors. There may be some issues with apache, serveralias and rails as mentioned here,

https://serverfault.com/questions/300226/serving-rails-through-apache-using-proxypass

that it was serving up the test page when the main domain was a server alias. So, I created a separate virtual server conf file in sites-available, with main-domain-name.conf which solved that issue. Then ran certbot for both domains for getting letsencrypt certificates.

debugging the radiosai google assistant action's audio search issue

Going step by step, found that dialogflow was returning the value "search" instead of the words the user was using. After going through the codelabs example at https://codelabs.developers.google.com/codelabs/actions-1/ ,  saw that after the dialogflow intent was edited with new training phrases without template mode, I needed to delete the old phrases, and then retrain before the changes would take effect, as per

https://github.com/googleapis/nodejs-dialogflow/issues/219

https://cloud.google.com/dialogflow/es/docs/training

Now, instead of the word "search", dialogflow returns the correct user phrase, as in the screenshot below.

After submitting this version as a release, the corrected version is live now. 

test of creating a free tier instance, using it for multiple users like students

Though GCP's "recommended way" of adding SSH keys is quite convoluted, found that I could use my usual method to add my key, and use PasswordAuthentication yes in /etc/sshd-config to allow password-based logins for users.

As can be seen in this techrepublic post, the way to prevent newly created users from seeing each others' directories is to edit the /etc/adduser.conf file, changing the default home directory permissions from 755 to 750. We can of course do this manually with sudo chmod 750 /home/user1 and so on.

Then, using a "Free Tier" f1 micro instance, someone can do text-based teaching like conducting a C lab after installing the required build tools like apt install gcc or apt-get install build-essential. Currently the specs of a free tier compute instance are -

1 F1-micro instance per month
Scalable, high-performance virtual machines.

1 non-preemptible f1-micro VM instance per month in one of the following US regions:
Oregon: us-west1
Iowa: us-central1
South Carolina: us-east1

30 GB-months HDD
5 GB-month snapshot storage in the following regions:
Oregon: us-west1
Iowa: us-central1
South Carolina: us-east1
Taiwan: asia-east1
Belgium: europe-west1

1 GB network egress from North America to all region destinations (excluding China and Australia) per month.

Azure AD guest account and how to close an azure account

There are conflicting posts all around, saying that one can only remove subscriptions and not close an Azure account once it is opened. Since I had used this from a test domain, it was an "unmanaged organization" 

https://docs.microsoft.com/en-us/azure/active-directory/enterprise-users/users-close-account

1. Sign in to close your account, using the account that you want to close.

2. On My data requests, select Close account.

   

I used this to remove the test account I had created to delegate permissions for an app being developed by a third-party. The delegation of permissions via Azure Active Directory implies that the guest user has to switch directory to the directory of the current resource in the Azure portal, and then the guest user would have view access to the current directory name (and probably more). 

diffuse not opening - needs python2

 The graphical diff tool Diffuse was not opening - running it from the terminal gave the error message 

diffuse
  File "/usr/bin/diffuse", line 74
    print codecs.encode(unicode(s, 'utf_8'), sys.getfilesystemencoding())
               ^
SyntaxError: invalid syntax

Apparently this is because diffuse is written for python2.

So, edited the hashbang for /usr/bin/diffuse to use /usr/bin/env python2 instead of /usr/bin/env/ python, works now.

#!/usr/bin/env python2

embedding math tools in blog

geogebra.org is the Math App used in HS's blog, which is now https://jee-math.in/

font issue with GMail on Chrome after installing google fonts on Linux Mint

So I was seeing the GMail inbox in Chrome with a bold narrow font after installing google fonts. Workaround to get back to the normal view was to remove Archivo* fonts - I moved them from /usr/share/fonts/truetype/google-fonts to a local directory, and on checking Chrome, found the issue was solved. 

signing pdfs on Linux with Xournal

 I have used Adobe Acrobat to sign pdfs on Windows, now I needed a solution for Linux or Android. Adobe Fill and Sign is available for mobiles, and its workflow lets you sign on the touchscreen with a finger or stylus. Since I had a scanned signature available, I preferred the Linux solution using Xournal. I had cleaned up the image, deleting the background in Gimp and leaving only transparency and the signature saved as a png. 

1. File -> Annotate PDF,
2. Choose the Image tool, click and drag inside the document to make it give the file open dialog
3. After positioning the image, Export to PDF.

There seems to be some bugginess in the Export to PDF dialog in my version of Xournal (0.4.8), so that if I try to edit the filename in the dialog box, it fails to save. But I can just export and then edit the filename later. Howtogeek lists solutions for Mac, Windows and Linux, using Preview, Adobe Reader and Xournal respectively, and also for iOS, Chromebook and Android.

removing old kernel headers

An update to my previous post about clearing up disk space by clearing old kernel images - on this machine running Linux Mint 18.3, it appears purge-old-kernels was not removing the kernel headers. Following the post at https://www.pontikis.net/blog/remove-old-kernels-debian-ubuntu for the manual process, 

dpkg --list | grep linux-image

- old kernel images were being purged by purge-old-kernels

dpkg --list | grep linux-headers

- were not being purged.

So, did with

sudo apt-get --purge remove linux-headers-4.15.0-7* linux-headers-4.15.0-6* linux-headers-4.15.0-5* linux-headers-4.15.0-4* linux-headers-4.15.0-3* linux-headers-4.15.0-2*

and so on, since current kernel and previous one were linux-headers-4.15.0-118 and 117. This cleared up 3 GB or so.

xfce startup on vnc - Ubuntu 18.04

I followed the digitalocean instructions on setting up xfce properly over vnc for the virtual machine I was running on GCP. Initially, the ~/.vnc/xstartup  file was either blank or absent, so only a bare-bones X was coming up. After adding this in the xstartup file,

#!/bin/bash
xrdb $HOME/.Xresources
startxfce4 &

xfce came up nicely. My earlier manual startup trials were missing the xrdb line, hence xfce was not starting cleanly.

updating a static version of a website

Emergency resuscitation for a website which had been hacked, by setting up temporary hosting on github pages - the method I used was:

1. using webhttrack - follow only 2 levels down, -*.pdf, -*.zip, 0 external levels 

2. find and replace in files using regexxer

(a) temporary-domain.com with actualdomain.name
(b) replacing js injected code when found using inspect
(c) replacing http with https (This was done later, after https was made to work.)

For html, css and js files.
*html In regexxer, Find files, open a file, Find, then choose the All Files button at the bottom.

3. Copy paste into local github repo, ignoring / overwriting existing files.

4. git add .
git commit -m "git message"
git push

github pages in directories prefixed by _ - no jekyll

Stackoverflow had an answer when I wondered by some files hosted on github pages were returning 404 errors - the solution was to place a file called .nojekyll in the root directory to enable hosting of files within directories starting with underscore, like _css etc. 

ad links in youtube embed code

Poking around our website and inspecting it (using Ctrl+Shift+I on Chrome, or right-click and inspect), found  that ad_status.js and googleads.g.doubleclick.net are being called. This seems to be from the youtube embed code -

Probably there is no real workaround except drastic ones like not using youtube embed code and using a link instead etc.

Edit: 19 Nov 2020 - Today, I get an email from youtube with their updated terms of service, which includes the following line which may prick quite a few people:

YouTube's right to monetize: YouTube has the right to monetize all content on the platform and ads may appear on videos from channels not in the YouTube Partner Program.

GCP console issues with Firefox

While doing the timed qwiklabs google cloud platform assessments on coursera, I found that google cloud shell was taking a lot of time to come up on Firefox. In fact, perhaps it was not coming up at all. Initially I tried the workaround of using the google cloud sdk locally and running gcloud commands from a terminal. Later, found that the cloud shell opens fine when Chrome is used instead of Firefox. So, this adds to the list of google products which don't work properly on Firefox. Can anyone else see some parallels to the bad old days of browser wars with Microsoft?

Chrome and Google Meet issues on Windows

In a previous post, I wrote that Google Meet worked for me on Chrome and not on Firefox. Now, I work on Linux (Mint 18.3) and most of the world works on Windows. Apparently Google Meet had issues with Chrome, video not appearing, even screen sharing not working, on a Windows machine. Nz let me know the solution, which was based on this thread,  https://support.google.com/meet/thread/41815190?hl=en

That thread has so many descriptions of so many different things which can go wrong! Privacy settings, antivirus, etc etc. If you expand and view the entire thread with nearly 300 messages, it makes interesting reading for people who want to mock Windows.

 

So, enable-media-foundation-video-capture must be Disabled, otherwise all sorts of weird video issues with Google Meet. This seems to be a windows-only issue.

add javascript to Blogger

There are many methods proposed at 
https://stackoverflow.com/questions/6449733/how-can-i-add-javascript-inside-blogger

The CDATA method above seemed to work for me, from HTML view. 

And preventdefault for stopping the scroll to the top of the page - used 

return false; 

instead, working for me on Chrome also.

https://stackoverflow.com/questions/13003044/href-going-to-top-of-page-prevent

shifting a website out of google sites and into github pages

Google sites recently announced a shift to "New Sites" with a mandatory upgrade for "Old Sites". Since New Sites had some disadvantages like inability to choose our own font size for particular lines of text (as we would like to do for important announcements, making them large), I looked around for options - whether to move to Blogger or New Sites or do something else. New Sites also had the disadvantage of breaking our photo gallery page when using the migration tool. Blogger would break existing page urls. So, since there was only static content, thought about hosting it on github pages. 

The move was relatively straightforward, though I took a long time since I did a drag-and-drop commit after each change instead of using the commandline. The changes I needed to make were:

1. removed all scripts
2. jpg and css links made relative, uploaded to suitable directories
3. created a template with the above, to which the changes needed for each page were:
   
   1. the title and other header info (at 4 places near each other)
      
   2. the div with id sites-chrome-sidebar-left
   3. the div with id  sites-canvas-main-content
   4. the h3 tags with page heading 

After setting the CNAME records on the domains DNS servers, even 4 days later, the "Enforce HTTPS" was not available, with the message "Not yet available for your site because the certificate has not finished being issued." One of the suggested workarounds was to remove and re-add the custom domain. I did that, and immediately Enforce HTTPS was enabled.

Coursera - a good resource

This post on ClassCentral has a good run-down on some 1400+ completely free Coursera courses. I've been doing some Coursera courses lately, and found some of them very nicely done. Courses I'm taking or have taken are:

Fundamentals of Graphic design

Create your first web app with Python and Flask

Visual elements of User Interface design

Google IT automation with Python

In general, you can speed up some of the courses with the available video speed changer, while for some of the courses, you can learn better by pausing the video, doing some of the exercises and then resuming. 

free cloud hosting how-tos

A link dump for resources on how to host small low-traffic websites for free on Google Cloud Platform (GCP) or Microsoft Azure - the link names are self explanatory.

https://redstapler.co/cost-of-hosting-wordpress-website-on-google-cloud/

https://www.c-sharpcorner.com/article/free-web-app-hosting-on-azure/

https://build5nines.com/free-website-hosting-in-microsoft-azure/

https://www.hanselman.com/blog/PennyPinchingInTheCloudWhenDoAzureWebsitesMakeSense.aspx

Google's cloud shell and other (free) resources for teaching and learning

Excerpts from an email I sent to a colleague - 

Currently, I have not applied for any Google Cloud grant for the institution, because

1. Google's Cloud credit grants for education are granted on a per professor and per student basis, not on an Institution basis. More info on how to apply is given at

https://edu.google.com/programs/faqs/?modal_active=none#teaching-and-learning-credits

2. Google's Cloud shell (free) may not allow for allows for c programming etc, but there are many  other  c, c++ , python and other language free online compilers/interpreters for teaching and learning -

https://www.google.com/search?q=online+c%2B%2B+compiler

https://www.google.com/search?q=online+c+compiler

issue with imagemagick convert

I had an issue with converting to pdf with convert - error message, convert not authorized. Found that it was due to a Ghostscript vulnerability. 

https://askubuntu.com/questions/1081895/trouble-with-batch-conversion-of-png-to-pdf-using-convert

So I temporarily used the renaming method,
sudo mv /etc/ImageMagick-6/policy.xml /etc/ImageMagick-6/policy.xmlout

And later I commented out the Ghostscript policy restrictions in the policy.xml.

moving a user from one tenant to another on Microsoft 365

Googling, found that there are different scenarios for tenant* to tenant migration on Microsoft 365 -

• with the help of Microsoft tech support for model driven apps in dynamics 365,
• For Microsoft Teams, by manually by downloading and uploading all assets, manually recreating all users, an involved process
  
• or with the help of third party tools like Sharegate to automate the above manual process. Other 3rd party tools are mentioned in this thread. One of the solutions offers $50 per team pricing.

*Tenant means (loosely speaking) subscription account. "A tenant is a representation of an organization."

moving a domain from one Microsoft 365 tenant to another

As the title says - to move a domain from one Microsoft 365 tenant to another, since I did not have any data to be backed up, I just deleted all the users, then deleted the domain, and recreated the domain in the other tenant. In order to delete the domain in the admin centre at admin.microsoft.com I had to first create an admin user myusername@mydomain.onmicrosoft.com , log in as that user, and then delete the other existing admin user adminuser@mydomain.tld. Only then is the domain deletion permitted.  

appropriate technology - right-sizing tech solutions

Yesterday, I heard about a clever implementation of distance learning by some school here. The teachers would daily send assignments via Whatsapp with links to some online knowledge resource, along with some questions. The children are encouraged to go through the resource, answer the questions and get back to the teacher in case of questions. This is quite useful in a bandwidth-starved nation like India, where asking pupils to participate in video calls would tax their parents' resources.

Today, I had to answer some queries regarding cost calculations for hosting of a medical data collection application. I'll copy-paste it below with some modifications.

As you may know, pricing of Azure services and AWS services are given at
https://azure.microsoft.com/en-in/pricing/
and
https://aws.amazon.com/pricing/

In general, such cloud services are very much more expensive than other hosting services, if the loads are small and predictable. By small load, I would classify even radiosai.org, with 1.5 lakh visitors a month, thousands of concurrent listeners, as "small".

But in the case of wildly fluctuating loads, when you have a load balancing system and multiple servers which need to be brought online and taken offline dynamically, cloud servers then become cost effective. They are also useful for outsourcing many of the management headaches of servers, to have the peace of mind of managed hosting without the possible security issues of shared hosting.

In the case of the medical data collection tool, they recommend Azure/AWS/your own server for security and licensing reasons. As mentioned at their website, only a very basic set of specs is needed for it, which can easily be supplied by even a shared hosting provider offering Rs. 40 per month hosting. The only caveat is that the organisation who has the license should take sole responsibility for the hosting and support of the application as given in the requirements page.

If the organisation hosting the site is a non-profit, it would be possible to get non-profit credits from Microsoft for Azure, and from Amazon for AWS.
https://www.microsoft.com/en-us/nonprofits/azure
https://aws.amazon.com/government-education/nonprofits/

Azure seems to be offering a larger number of credits - $5000 vs $2000. Also, on the Azure pricing page, they mention that they will price match AWS for cloud infrastructure. So, it might seem that hosting on Azure may be the better option, if you do get non-profit credits.

Since the loads expected on this server would be small - by my definition of small as given above - I would recommend using a VM with a couple of cores and low end specs
https://azure.microsoft.com/en-in/pricing/details/virtual-machines/linux/

If you don't get non-profit credits, the lowest cost option would be to hire two shared servers from two different reputed providers, ensuring that their services are located in different physical locations (since your services are in India, you could look for Bangalore and Mumbai, for example) and have one of them as hot-spare. If you use cloudflare.com to host your dns, operating a hot spare becomes very easy. Such a hot-spare setup allows you to have the peace of mind that Azure/AWS offers, at 10x lower cost. But your provider(s) need to be reputable in the sense that they should not steal your data, and they need to ensure good security practices.

Even if for security reasons you do not want to go for shared hosting, you could get reputed dedicated servers for half the price of Azure/AWS VMs of similar specs, or host it on your premises if you have redundant internet links to your facility. Only drawback, of course, would be that you would need to manually set up as against the automated Azure setup option offered by your software. And you need to know how to do some server management and maintenance.

In the case of VMs or shared servers or dedicated servers, you can directly calculate the costs by just multiplying the per hour or per month cost by the number of hours/months in a year.

In case you wish to estimate traffic, you will need to know what sort of data is going to be uploaded/ downloaded - whether it would be predominantly just numbers, or whether images/video/audio would be included, and what are the estimated numbers of users. Again, a simple multiplication.

When looking to host something for non-profits, where there is no charge for the services being offered, we can get the best RoI by minimising cost while ensuring that performance doesn't suffer. So, for small operations such as ours, I wouldn't recommend cloud solutions except when free credits are available.

Getting USB tethering to work with Linux Mint

Our home BSNL broadband has been on the fritz since yesterday. The land-line phone is dead. Airtel 4G on my LG Q6 can do wifi hotspot also, but I preferred to get USB tethering working, since the wifi card on this Lenovo laptop sometimes has issues.

Just connecting the USB cable to the phone and computer, enabling USB tethering on the phone, I get a "Connected" notification on the computer. But "could not find server" errors. Checked and saw that the default setting assigning my phone's IP address as the DNS server. Changed that to Google DNS 8.8.8.8, and the connection works fine. Network connections - Edit the relevant connection - IPv4 tab.

adding Next and Previous post links in new blogger

Tried http://copypastesave.blogspot.com/2017/06/add-pervious-page-home-and-next-page-in.html

using revert to legacy blogger.

Step 3 occurs in 9 places!

Unfortunately, this made the entire post not display! So, reverted to adding the Previous and Next links manually. 

Google Meet issues with Firefox

When I try to join a Google Meet meeting on Firefox 78, running on Linux Mint, I get a video network error and get disconnected. On the same machine, when I connect with Chrome 84, it works fine.

So, we have to use Chrome for Google Meet.

interesting mass emailing solution

GMass uses GMail's sending interface - but now the free plan is limited to 50 emails per day. 

https://www.gmass.co/blog/gmass-pricing-frequently-asked-questions-faq/

https://www.gmass.co/blog/you-can-now-send-10000-emails-with-gmass-and-gmail/

This may be useful for non-time-sensitive emails.

find and replace http with https with sed

htm pages under the SI folder used to give a POST request to http://www.radiosai.org/program/PlayNow.php

which we needed to change to https, since otherwise the POST requests do not get redirected - 

PB did this with

SI# ls

2011  2012  2013  2014  2015  2016  (etc)

SI# find ./ -type f -exec sed -i 's/action=\"http/action=\"https/g' {} \;

More examples of sed with find are given at https://linuxize.com/post/how-to-use-sed-to-find-and-replace-string-in-files/

qt5 on Precise Pangolin

Link dump for setting up qt5 on Ubuntu 12.04 - 

https://askubuntu.com/questions/279421/how-can-i-install-qt-5-x-on-12-04-lts

https://stackoverflow.com/questions/22925717/adding-qmake-cxxflags-std-c11-to-pro-file-in-qt-not-workingon-linux-12-0

https://stackoverflow.com/questions/19836858/upgrade-gcc-4-6-3-on-ubuntu-12-04-to-4-8-2

https://www.google.com/search?q=which+version+of+gcc+adds+c%2B%2B11+support&oq=which+version+of+gcc+adds+c%2B%2B11+support

Need 4.8.1

ERROR libvlc.so.5 not found

did ldd, found the paths, copied to build dir.

https://github.com/probonopd/linuxdeployqt/issues/105

need to copy them to a dir in LD_LIBRARY_PATH

icecast ideas

Some ideas for showing "Now playing" on our icecast server, https://stackoverflow.com/questions/48327310/how-to-display-now-playing-in-my-website-with-icecast2-and-liquidsoap
but since we don't use metadata on our server, this method won't work directly for us. But perhaps I could try tinkering with ices0 to make it send the filenames alone, and then try out the worker.js as mentioned in the stackoverflow post above.

adding a domain to Microsoft 365

The procedure to add a domain is given at this support page - basically, at admin.microsoft.com, Setup -> Get your custom domain set up -> Manage

There was another domain which they wanted me to add, but that seemed to be already connected to another tenant anotherdomain.onmicrosoft.com - migration would need all these steps, while a simple delete and add with no data being retained is relatively simple as I have posted elsewhere. There are also third-party tools to do this, as I have posted.