importing emails into google groups

Sending invites to a large number of email ids from google groups, found the following. 

• The "500 per day" limit seems to include those email ids which were selected for sending, and then were not sent since they were already members of the group.
• So, sending invites one by one - only one email at a time - would result in the maximum invites sent per day. But - this is very time consuming. Even with an automated script made with SikuliX, approximately 15 seconds per email id if processing one email at a time. 
• Filtering out existing users makes the process much smoother. Script for that shared below. But the list still needs manual filtering of typos like gamil.com or gmailcom or gmail com etc. 

For comparing this list with the list of subscribers made earlier, the technique discussed here was modified to make the formula
=IF(ISERROR(MATCH(A143,Sheet2!B142:B,0)),"",A143)

That is, the existing users were copied into a sheet called Sheet2 on the same Google spreadsheet. This was further improved using the technique of ARRAYFORMULA discussed here, so that the entire column of thousands of cells were populated with one click. That is, enter the formula in the first cell, and instead of specifying just A2, mention the entire column in the form A2:A, and hit Ctrl+Shift+Enter after entering the formula.

Then a modified SikuliX script was used to do the import 10 emails at a time, which I've copy-pasted below. Captcha is not required for 10 emails or less. Alt+Shift+C is the key combination to abort the SikuliX script.

from time import sleep

sleeptime=0.5

lastlineofsheet=Location(412,656)

#Addbutton=Location(1068,286) changed to 75 percent below

Addbutton=Location(982,202)

#DirectAddToggle=Location(840,670) changed to 75 percent below

DirectAddToggle=Location(895,580)

#MembersToAddBox=Location(880,320) changed to 75 percent below

MembersToAddBox=Location(922,346)

#WelcomeMesgBox=Location(915,415)

#sometimes the gmail profile comes in the above spot

#WelcomeMesgBox=Location(1180,446) changed to 75 percent below

WelcomeMesgBox=Location(1138,438)

#TextEditOnTaskbar=Location(751,751)

#Taskbar location is often non-reproducable, avoid.

TextEditOnTaskbar=Location(600,31)

InsideTextEdit=Location(650,400)

#MemberListOnTaskbar=Location(435,751)

#SendButton=Location(1164,555) changed to 75 percent below

#SendButton=Location(1146,520) changed for 10 at a time to

SendButton=Location(1128,543)

while(True):  

  click(lastlineofsheet)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  type(Key.DOWN, KeyModifier.SHIFT)

  sleep(sleeptime)

  type("c",KeyModifier.CTRL)

  sleep(sleeptime)

  click(Addbutton)

  sleep(sleeptime)

  sleep(sleeptime)

  sleep(sleeptime)

  click(DirectAddToggle)

  sleep(sleeptime)

  click(MembersToAddBox)

  sleep(sleeptime)

  type("v",KeyModifier.CTRL)

  click(TextEditOnTaskbar)

  click(InsideTextEdit)

  sleep(sleeptime)

  type("a",KeyModifier.CTRL)

  type("c",KeyModifier.CTRL)

  #click(MemberListOnTaskbar)

  sleep(sleeptime)

  click(WelcomeMesgBox)

  sleep(sleeptime)

  type("v",KeyModifier.CTRL)

  

  popup("waiting for paste")

  click(SendButton)

  popup("waiting for no error")

  #sleep(7.0)

  

  click(lastlineofsheet)

  sleep(sleeptime)

  type(Key.DOWN)

  sleep(sleeptime)

  #popup("waiting for next")

GCP costs for 2 16 GB VMs

Google Cloud Platform's Billing -> Overview shows the number of free credits remaining as well as the number of days remaining. 

Running 2 instances of e2-highmem-2 (2 vCPUs, 16 GB memory) cost approximately Rs. 600 or US$8 per day, or $250 per month, or $3000 per year.  

URL addressability API removed from Alfresco

URL Addressability API (superseded by WebScripts)

https://hub.alfresco.com/t5/alfresco-content-services-hub/alfresco-community-5-0-b-release-notes/ba-p/289468

Interestingly, this is in version 5-0-b. But ticket-based addressing of image URLs is working locally (I believe?) in 5-0-d. Does Alfresco's versioning go from d to c to b? Or is my understanding of the URL Addressability flawed?

The URL addressability documentation says that it has been obsoleted - 5

https://hub.alfresco.com/t5/alfresco-content-services-hub/url-addressability/ba-p/291489

mounting remote volumes and cloud drives - sshfs and rclone

Rclone can mount a wide variety - over 40! - of cloud storage options as drives on Linux, Windows and Mac. If we're ssh-ing into a remote machine and want to authenticate, we can tunnel the temporary web server which Rclone opens on port 53682.

Another option for remote mounting is sshfs. 

When I tried it with key-based authentication, 
sudo sshfs -o allow_other,default_permissions,IdentityFile=~/.ssh/id_rsa bitnami@xxx.xxx.xxx.xxx:/ /mnt/mymount
I got errors - connection reset by peer. 

Apparently, the full path to the key file is needed - 
sudo sshfs -o allow_other,default_permissions,IdentityFile=/home/myhomedir/.ssh/id_rsa bitnami@xxx.xxx.xxx.xxx:/ /mnt/mymount

Then it worked. 

deploying war files on wildfly

There seem to be multiple ways to deploy WAR files in Wildfly. 
https://www.baeldung.com/jboss-war-deploy

Just copying the WAR file to the wildfly/standalone/deployments directory seems the most easy for us to do. And if the WAR file is already exploded, we also need to create a appname.war.dodeploy marker file as mentioned in RedHat's documentation.

Change configuration files only after stopping wildfly -
/opt/bitnami/ctlscript.sh stop wildfly
on Bitnami Wildfly VM.

Takes ~15 sec to stop Wildfly

Takes ~5 minutes to start Wildfly and deploy wars if there are errors, but only a few seconds (less than a minute) if there are no errors.
Monitoring the log is a good way to judge if Wildfly is running or stopped - 
sudo tail -f /opt/bitnami/wildfly/standalone/log/server.log

Examples of configuration files which might need to be set up - 
wildfly/standalone/deployments/CAS.war/WEB-INF/deployerConfigContext.xml
wildfly/standalone/deployments/CAS.war/WEB-INF/cas.properties

wildfly/standalone/configuration/standalone.xml
wildfly/modules/org/CustomAppName/properties/configuration/main/Environment.properties

If a WAR file deployment fails with Null Pointer Exception (seen in the server.log file mentioned above), there is a good chance that this is due to some non-existent file or path mentioned in one of the properties files above. The usual culprit would be Windows-style paths for log or conf files. 

If a large number of WAR files are to be deployed, deployment may fail with Out of Memory errors. We can increase heap size and MetaSpace size by editing /opt/bitnami/wildfly/bin/standalone.conf line for JAVA_OPTS like

JAVA_OPTS="-Xms2G -Xmx10G -XX:MetaspaceSize=2G -XX:MaxMetaspaceSize=6G -Djava.net.preferIPv4Stack=true"

Edit - Adding data-sources - if needed, we can deploy the db driver as a module, and configure via the data source via the web console. For the postgres driver, I directly downloaded the jar from

http://localhost:9990/console/index.html#deployment;deployment=postgresql-42.2.14.jar

so gave the 

 <resource-root path="postgresql-42.2.14.jar"/>

Using the jboss-cli method for creating the db driver as a module, 

wildfly/bin/jboss-cli.sh

ran as sudo, needed to put the entire command in one line

/subsystem=datasources/jdbc-driver=postgresql:add(driver-name=postgresql, driver-module-name=org.postgresql, driver-class-name=org.postgresql.Driver)

Outcome success.

location of Apache configuration files on Bitnami Wildfly VM and easy SSL configuration with LetsEncrypt

As of this writing in Dec 2020, this discussion on configuring redirects is not directly applicable to Bitnami's Wildfly VM. The Proxy Pass commands etc are located in /opt/bitnami/apache2/conf/vhosts/wildfly-http-vhost.conf
and
wildfly-https-vhost.conf

Bitnami makes it easy for us to use https by providing an interactive script which will set up SSL certificates with LetsEncrypt as well as a cron job for renewal - 
sudo /opt/bitnami/bncert-tool

check if a service is running

One way to easily check for a service listening on a port for a Linux server running systemd is using ss - need not be run as root.

ss -tulpn

shows all listening ports 

ss -tulpn | grep 5432

would indicate if PostgreSQL is running or not, and so on.

quick temporary phpPgAdmin installation using Bitnami

On Bitnami's Alfresco VM, installing phppgadmin using 
sudo apt-get install phppgadmin
causes problems since apt-get on the platform (Debian 10) tries to install apache2 which is already installed as httpd on the Bitnami stack. 

Bitnami's LAPP stack has phpPgAdmin, so one can install it as a user, disable or stop postgres from that stack, change phppgadmin/htdocs/ conf file for accessing any other db which we may have, change apache's port if another server is running at 8080, to 18080 for example, by modifying
apache2/conf/httpd.conf
and
apache2/conf/bitnami/bitnami.conf

Virtual hosts, host alias equivalents and LetsEncrypt SSL for IIS on Windows Server

The equivalent of adding virtual hosts in Apache is adding Sites in IIS. And similar to adding a host alias, what is done is to add a site binding. IIS Manager -> Servername -> Sitename -> Edit site 

Then, getting SSL certificates from LetsEncrypt and applying them - 
https://weblog.west-wind.com/posts/2016/feb/22/using-lets-encrypt-with-iis-on-windows
suggests using LetsEncrypt-Win-Simple which is now WinAcme. Very simple interactive script as described on their website.

Also, various possible reasons for IIS errors are listed at https://superuser.com/questions/741486/page-not-found-in-iis-but-its-there

Bitnami Wildfly VM default user is called user and not manager

Bitnami's documentation mentions that the username to access the wildfly admin console is manager. But with the credentials supplied, I was unable to log on to the VM deployed using GCP marketplace (Version: 21.0.2-0-r02). Then, looking at how to change the credentials, looking at the file /opt/bitnami/wildfly/standalone/configuration/mgmt-users.properties found that the username is actually user and not manager. Then I was able to log on.

Monitoring RAM usage of a VM on GCP

CPU usage is shown by default on the monitoring console, but in order to see how much RAM is used by a VM on Google Cloud Platform, one has to first install the agent, then in the Monitoring dashboard, open Metrics explorer, choose the GCE VM Instance Resource Type and choose the agent called Memory Usage (agent.googlapis.com/memory/bytes_used). This last part is a bit non-intuitive, because there are many entries labelled Memory Usage, and we have to scroll down till agent.googlapis.com/memory/bytes_used. The resulting chart can be saved to an existing dashboard or we can create a new dashboard.

tried and failed to change Alfresco port in Bitnami VM

In some of my initial attempts, working with Bitnami's Alfresco VM, I tried a few ways to change the port and failed. Just documenting what does not work - 

https://docs.alfresco.com/4.2/tasks/share-change-port.html

mentions changing the <web-extension>/share-config-custom.xml file, not just in alfresco-global.properties -

https://docs.alfresco.com/4.2/concepts/share-configuration-files.html

The path above is tomcat/shared/classes/alfresco/web-extension/share-config-custom.xml

But probably these docs assume that Tomcat is already configured to run on the fresh port, and that is probably why just doing the above failed for me.

And finding the location of the Apache configuration files on this Bitnami VM which were doing the proxypass from port 8080 to port 80 - 

/opt/bitnami/apache2/conf/bitnami/bitnami.conf
had a line 

Include /opt/bitnami/apache2/conf/bitnami/bitnami-apps-prefix.conf

bitnami/bitnami-apps-prefix.conf has

Include /opt/bitnami/apps/alfresco/conf/httpd-prefix.conf

httpd-prefix.conf has

Include "/opt/bitnami/apps/alfresco/conf/banner.conf"

Include /opt/bitnami/apps/alfresco/conf/httpd-app.conf

and finally 

/opt/bitnami/apps/alfresco/conf/httpd-app.conf

has the proxypass lines for alfresco and share, like

<Location /alfresco>
ProxyPass ajp://localhost:8009/alfresco etc.

MySQL and PostgreSQL command line cheat sheet

 MySQL

From https://www.a2hosting.in/kb/developer-corner/mysql/managing-mysql-databases-and-users-from-the-command-line

mysql -u root -p
(or bitnami user for bitnami VMs)

show databases;
use dbname1; 
show tables; 

create database dbname;
use dbname;
GRANT ALL PRIVILEGES ON *.* TO 'username'@'localhost' IDENTIFIED BY 'password';

Example SQL script import usage

mysql -u username -p < example.sql

And if required,

drop table tablename;

or

drop database dbname;

For cloning a database, this page has instructions, which also needed the permissions as detailed here. as root user, opening mysql and

GRANT PROCESS, SELECT, LOCK TABLES ON *.* TO 'bn_alfresco'@'localhost'; 

mysqldump -u bn_alfresco -p bitnami_alfresco -r bnalf.sql

mysql -u bn_alfresco -p

CREATE DATABASE my_project_copy;

USE my_project_copy;

SOURCE bnalf.sql;

PostgreSQL

For Debian-based distributions, 
sudo apt-get install postgresql-client
psql -h <REMOTE HOST> -p <REMOTE PORT> -U <DB_USER> <DB_NAME>

\l

to show databases, or

select datname from pg_database;

And to show tables,

\c databasename
\dt

Edit: For exporting the database, various options for pg_dump, most common would be
sudo su postgres
pg_dump dbname > dumpfile.sql
or if local authentication is not supported in the hba.conf, and username/pw auth is supported,
pg_dump --no-owner --dbname=postgresql://dbuser:dbuserpasswd@host:port/dbname > dump.sql

From bash shell, as postgres user, can import like
psql dbname < dbdump.sql

From https://medium.com/coding-blocks/creating-user-database-and-adding-access-on-postgresql-8bfcd2f4a91e

create database dbname;
create user username;
(To change, alter user username with encrypted password 'MyPassWord';)
grant all privileges on database dbname to username;

If the database was imported as postgres user, and another user is given all privileges as above, a possible issue is "permission denied when accessing schema postgres". My rough workaround was to create another super user (since I did not know the postgres user's password and didn't want to change it and potentially break things) by logging on as root, su postgres, and 
CREATE USER newadmin WITH SUPERUSER PASSWORD 'newadminpassword';

changing bitnami alfresco db to postgresql and resetting the password

After following the steps below, was unable to log on as alfresco admin user - the credentials were not being accepted. The solution is at the end of the post.

https://docs.alfresco.com/4.0/tasks/postgresql-config.html
https://computingforgeeks.com/install-postgresql-11-on-debian-10-buster/

To set the postgres alfresco user password MyPassWord

sudo su -
sudo postgres

psql
alter user alfresco with password 'MyPassWord';

Downloaded driver from

https://jdbc.postgresql.org/download.html

https://docs.bitnami.com/bch/infrastructure/tomcat/get-started/get-started/

says TOMCAT_HOME is /opt/bitnami/tomcat

but for the Bitnami Alfresco Community VM, found it was

/opt/bitnami/apache-tomcat

https://docs.alfresco.com/6.1/reuse/conv-syspaths.html

indicates location of <classpathRoot> as tomcat/shared/classes, so on this bitnami VM,

/opt/bitnami/apache-tomcat/shared/classes

Ask the database the location of pg_hba.conf as mentioned at

https://askubuntu.com/questions/256534/how-do-i-find-the-path-to-pg-hba-conf-from-the-shell

In our case doing
psql -t -P format=unaligned -c 'show hba_file';
as postgres user, 

sudo nano /etc/postgresql/11/main/pg_hba.conf to add the line

host all all 127.0.0.1/32 password

and

systemctl restart postgresql

Then, restarting alfresco with

/opt/bitnami/ctlscript.sh restart

brings up alfresco by bootstrapping the required tables in the new database, but can't log in - the credentials are not accepted. 

Tried various methods as given at https://docs.alfresco.com/5.1/concepts/admin-password.html and http://www.giuseppeurso.eu/en/alfresco-tips-and-tricks-1-reset-the-admin-password/ using hashes as given at https://blog.atucom.net/2012/10/generate-ntlm-hashes-via-command-line.html and so on. But did not work.

The solution was:

Delete or comment out the line with admin credentials at /opt/bitnami/apache-tomcat/shared/classes/alfresco-global.properties like

#alfresco_user_store.adminusername=admin
#alfresco_user_store.adminpassword=209c6174da490caeb422f3fa5a7ae634

so that when alfresco is restarted with

/opt/bitnami/ctlscript.sh restart

with a blank database, the admin username and password is reset to admin admin. After logging in, the password can be set as desired from the web console.

Installing activemq on Debian 10

Following
https://www.howtoforge.com/tutorial/debian-activemq-message-broker/

to install activemq on Debian 10 for a Bitnami Alfresco VM starting from step 2, found that activemq failed to start with 
ERROR: Configuration variable JAVA_HOME or JAVACMD is not defined correctly. (JAVA_HOME='', JAVACMD='java')

Set the environment variables as mentioned at 
https://stackoverflow.com/questions/31400148/apache-activemq-5-11-1-doesnt-start-in-ubuntu
in this case, for the bitnami stack, I defined  JAVA_HOME and JAVACMD in /etc/default/activemq as

JAVA_HOME="/opt/bitnami/java" 

JAVACMD="/opt/bitnami/java/bin/java"

Now 

systemctl start activemq

systemctl enable activemq

systemctl status activemq

shows

● activemq.service - Apache ActiveMQ

   Loaded: loaded (/etc/systemd/system/activemq.service; disabled; vendor preset

   Active: active (running) 

etc. 

The default username and password for admin user are admin admin, and for non-admin user, user password. The default port is 61616, the web console configuration panel can be accessed from localhost:8161.

public key authentication fails with agent refused operation

I generated a private+public key pair on a Windows 10 machine, copied the key over to a Linux Mint 18.03 (based on Ubuntu Xenial 16.04) machine and used it to ssh into that machine with no issues. Then I generated another key pair on another Windows 10 machine, which had been upgraded to 10 from a lower version of Windows, and tried to use that key pair to log on to that machine from the Linux machine, but it failed with "agent refused operation". 

Looked at various possible reasons for this failure, finally found that the private key was of a different size, and also had the string OpenSSH PRIVATE KEY instead of RSA PRIVATE KEY at the beginning and end. This might have been due to a different OpenSSH server from MLS software I had tried installing on the 2nd Windows machine, or it could have been due to something else. Anyway, I just created one more key from the 1st Windows machine and used that key pair instead for authenticating into the 2nd machine. That worked fine.

Edit - This discussion seems to be related, and has an explanation.

Load testing a moodle server - log in with token parsed with Beautiful Soup

While trying automated logins with a simplistic python script like this -
import requests

url = 'https://theserver.tld/login/index.php'
myobj = {'username': 'theUserName', 'password': 'ThisIsThePassWord!', "logintoken": "hCBVa2gzAWqSa2u7iNzdzlE9kwpVehFf"}
x = requests.post(url, data = myobj)
print(x.text)

Moodle would return authentication failure - it was checking the logintoken, which was hard-coded above. R used a modified script to get the logintoken correctly first, which then works - 

import sys
import requests
from bs4 import BeautifulSoup 

#driver = webdriver.Chrome()
# using this technique, the testing machines would run out of RAM
# after a dozen or so Chrome instances were loaded!

#def login(url,usernameId, username, passwordId, password, submit_buttonId):

#   driver.get(url)
#   driver.find_element_by_id(usernameId).send_keys(username)
#   driver.find_element_by_id(passwordId).send_keys(password)
#   driver.find_element_by_id(submit_buttonId).click()

#myMoodleEmail = sys.argv[1]

#login("https://server.tld/login/index.php", "username", myMoodleEmail, "password", "ThePW!", "loginbtn")

#driver.close()

login_data = {

    #'logintoken': 'XPc8nw0a2gLXpgsn6njpehwEoW43mzYQ',
'username': sys.argv[1], 'password': 'ThePW!'

}

headers = {'user-agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Mobile Safari/537.36'}

with requests.Session() as s:

    url = "https://theserver.tld/login/index.php"
    r = s.get(url)
    soup = BeautifulSoup(r.content, 'html5lib')
    login_data['logintoken'] = soup.find('input', attrs = {'name': 'logintoken'})['value'] 
    r = s.post(url, data = login_data, headers = headers)
    print(r.text)

So I didn't have to try JMeter as the moodle documentation suggests, 
https://moodle.org/mod/forum/discuss.php?d=313489

which points to the moodle documentation on load testing. 
https://docs.moodle.org/dev/JMeter#Make_JMeter_test_plan

Edit: Example usage of JMeter - 
jmeter -n -t [jmx file] -l [results file] -e -o [Path to web report folder]

Edit: The final result was, on AWS, "R5A Quadruple Extra Large - 128 GB RAM, 16 Virtual CPUs, 10 GB Network" - for 2000 users to log in concurrently. 

some ssh tunnel notes

While setting up an ssh server on Windows, found that PermitTunnel is not supported by the Microsoft port of OpenSSH-Server. But then, PermitTunnel is not required to do local port forwarding at the client side, like 
ssh -L 5990:192.168.12.34:5990 user@server.name

And interestingly, this sort of "poor man's VPN" using such an ssh tunnel seems to be much faster than OpenVPN etc as mentioned here, due to having less overhead.

Installing OpenSSH on Windows 10 which had been upgraded from Windows 7

When trying to install OpenSSH on a machine running Windows 10 which had been upgraded from Windows 7, the Optional Features under Apps -> Apps and Features -> Manage Optional Features as mentioned in the documentation was completely blank. But the install with Powershell (run as administrator) method as mentioned in the same documentation page worked.
Add-WindowsCapability -Online -Name OpenSSH*

For troubleshooting permission issues, this post or this one may be useful - it talks about using PowerShell to change the permissions - ACL - on the authorized_keys file. Or, just make sure the permissions are appropriate using windows explorer -> file properties -> security -> advanced, 

For the private key (and for the authorized_keys file?), 

change the owner to the login user (if it's not already).

Disable inheritance (if it's set).

Remove all permissions for every one but this user.

Give this user “Full Control”.

avoiding multiple redirects in cloudflare

Changing the SSL settings from Flexible to Strict fixed the too many redirects error for one of the domains - https://support.cloudflare.com/hc/en-us/articles/115000219871-Troubleshooting-redirect-loop-errors