demo of displaying content from google drive using a service account

Following
https://www.labnol.org/code/20375-service-accounts-google-apps-script

creating a service account and delegating authority domain-wide via the method at
https://developers.google.com/admin-sdk/directory/v1/guides/delegation

The google apps script's Code.gs has the following, and will work even when the doc is not shared with anyone except theuser@ourdomain.tld:

var JSON = {
    "private_key": "-----BEGIN PRIVATE KEY-----\nMI -- snip -- pJKdnI=\n-----END PRIVATE KEY-----\n",
    "client_email": "what-name-we-give@proj-name.iam.gserviceaccount.com",
    "client_id": "1020123456872",
    "user_email": "theuser@ourdomain.tld"
};

function getOAuthService() {
    return OAuth2.createService("Service Account")
        .setTokenUrl('https://accounts.google.com/o/oauth2/token')
        .setPrivateKey(JSON.private_key)
        .setIssuer(JSON.client_email)
        .setSubject(JSON.user_email)
        .setPropertyStore(PropertiesService.getScriptProperties())
        .setParam('access_type', 'offline')
        .setScope('https://www.googleapis.com/auth/drive');
}


function getUserFiles() {
    var service = getOAuthService();
    service.reset();
    //Logger.log("Getting the Access Token:");
    var atoken = service.getAccessToken()
    //Logger.log(atoken);
    if (service.hasAccess()) {      
        var url = 'https://drive.google.com/open?id=1qM_DOC_ID_5QiIJ0Ls';
        var response = UrlFetchApp.fetch(url, {
            headers: {
                Authorization: 'Bearer ' + atoken
            }
        });
        return response.getContentText()        
    }
}

function reset() {
    var service = getOAuthService();
    service.reset();
}

function doGet(e) {    
  var htmlout = HtmlService.createHtmlOutput(getUserFiles());
  return htmlout
      .setTitle('Our Team')
      .setSandboxMode(HtmlService.SandboxMode.IFRAME);
}

Google sign in on Moodle

The user experience for google sign-in on Moodle is shown at 
https://www.youtube.com/watch?v=1_HDZoir3eQ
and how to set up, for admins, is shown at 
https://www.youtube.com/watch?v=cwUWvTiGSAk
- accurate as of Nov 2020, as google's api and set up screens keeps changing.

No matching client found - error for android app

As mentioned on this page - 
https://stackoverflow.com/questions/34990479/no-matching-client-found-for-package-name-google-analytics-multiple-productf

the reason was that the package name in the google-services json and the local package name of the app had a discrepancy. 

monitoring ram usage over time on windows server

One of our servers running a .NET app was undergoing a stress test to check whether it could scale to the required number of users. RAM was the biggest question-mark. Looking for ways to monitor RAM usage, found this page which talks about creating a counter log in perfmon. And how to create a counter log is explained here. 

Basically - 

1. Start - Run - perfmon.exe
2. User defined folder -> New -> Data Collector Set and give it a name.
3. Choose Create Manually, and Next.
4. Create Data Logs -> Performance Counter and then click Next.
5. Choose the desired object(s) to log
6. (Every 5 seconds may be a good option for short collection runs)
7. Choose location to save in.
8. "Start this data collector set now" if desired.
9. To stop data logging, right-click User Defined -> Data Collector Set -> our set and click Stop.

And later to view the data, choose the View Log Data button, Data Source as Log Files, and add the relevant log file.

And caveat when viewing and using Processor usage, the Scale factor has to be taken into account. 

search for a string or a number in an entire database

With just sql queries, this would need some amount of scripting as mentioned on stackoverflow. But with visual database tools like DBeaver, it's just a simple operation of doing a full-text search, with the option of including numeric fields or not - "Search in numbers" and the option to search Large Objects or not - "Search in LOBs".

show an image from binary data using php

This SO link gives two techniques - using data URIs and using a helper function to fetch the image from some storage location. 

Edit: My initial implementation was using the simpler data URI method. Later, also implemented using the other method. 

Intially,

    // to prevent images from being dropped due to timeouts, we'll encode them inline
    // https://ads-developers.googleblog.com/2013/11/how-to-send-pdf-reports-with-adwords.html

    // to parse the img src, we'll use regex, since the html is of limited scope
    // https://stackoverflow.com/questions/14939296/extract-image-src-from-a-string/15013465
    // https://support.google.com/a/answer/1346938

    var m;
    imageurls = [];
    regexstring = '(http\\S*' + currentTicket +')'; // the \ is escaped, so it becomes doubled.
    //Logger.log(regexstring);
    //myregex = new RegExp(/(http\S*0d0a226a47229313863)/,"gi");    // the ticket is the string after S*
    myregex = new RegExp(regexstring, "gi");
    
    var imageindex=1;

    while ( m = myregex.exec( contentdata ) ) {
      imageurls.push( m[1] );
      //Logger.log(m[1])
      var imagemimetypeandenc = 'data:image/png;base64,'; // default
      if ( m[1].includes('.jpg?ticket') || m[1].includes('.jpeg?ticket') ) {
        imagemimetypeandenc = 'data:image/jpeg;base64,';
      }
      else if ( m[1].includes('.gif?ticket') || m[1].includes('.GIF?ticket') ) {
        imagemimetypeandenc = 'data:image/gif;base64,';
      }
      else if ( m[1].includes('.tif?ticket') || m[1].includes('.tiff?ticket') ) {
        imagemimetypeandenc = 'data:image/tiff;base64,';
      }

      Logger.log('%s of assetid %s',imageindex.toString(),data_array[i][0].toString());
      imageindex = imageindex + 1;
      var imageBlob = UrlFetchApp.fetch(m[1]).getBlob();
      var base64EncodedBytes = Utilities.base64Encode(imageBlob.getBytes());
      var imageencoded = imagemimetypeandenc + base64EncodedBytes;
      contentdata = contentdata.replace(m[1], imageencoded);

    }

and later, when images were available on Google Drive,

  var placeholder = 'data:image/png;base64,iVBORs_a_few_kb_of_chars_QAAAABJRU5ErkJggg==';
  // made with https://www.w3docs.com/tools/image-base64

.....

var m;
    imageurls = [];
    regexstring = 'img\\s*src\\s*=\\s*"(http\\S*)"'; // the \ is escaped, so it becomes doubled.
    // Only the http part is captured, within parenthesis
    // currently it matches only if the img src is within double quotes. "https://whatever"
    //Logger.log(regexstring);
    
    myregex = new RegExp(regexstring, "gi");
    var imageindex=1;

    while ( m = myregex.exec( contentdata ) ) {
      imageurls.push( m[1] );
      
      Logger.log('%s of assetid %s',imageindex.toString(),data_array[i][0].toString());
      imageindex = imageindex + 1;
      
      // here we use a helper if the url contains the alfresco url
      var response;
      if (m[1].includes('ticket=') ) {
        //use the helper function after extracting the filename
        Logger.log('Downloading %s using Alfresco helper', m[1])
        var fname = m[1].split('/')[9]; // from the url similar to 
        // https://www.ourdomain.tld/alfresco/d/direct/workspace/NameOfStore/f8b56b7c-long-id-a35c30be756a/filename-15092020100930PM?ticket=
        fname = fname.split('?')[0];
        // handle cases where fname does not have extension
        var urlFromHelper = UrlFetchApp.fetch(returnUrl+encodeURIComponent(fname));
        response = UrlFetchApp.fetch(urlFromHelper, { muteHttpExceptions: true });
      } 
      
      
      if (response.getResponseCode()==200) {
        var imageBlob = response.getBlob();
        var base64EncodedBytes = Utilities.base64Encode(imageBlob.getBytes());
        //var ctype = response.getHeaders().Content-Type
        var headermap = new Map(Object.entries(response.getHeaders()));
        var mimetypeofim = headermap.get('Content-Type');
        var imagemimetypeandenc = 'data:'+mimetypeofim+';base64,';

        var imageencoded = imagemimetypeandenc + base64EncodedBytes;
        contentdata = contentdata.replace(m[1], placeholder);
      }
      

And in cases where we did not want the images to be base64 encoded - for example in the case of large images, which would cause the function to run out of memory - the whole code-block above would be not needed, since the img src link in the original html would work OK.

embedding a pdf in a web page

Several ways to embed a pdf in an html page - 

1. Using the object or embed tag (deprecated)
2. Using a tags with href links (not actually embedding, linking)
3. Using iframe tags where the toolbar can be made hidden with src="/path/my.pdf#toolbar=0"
4. Using Adobe's SDK - somewhat complicated initial setup
5. If hosted on Google Drive, using an iframe with src="https://drive.google.com/file/d/FILE_ID/preview">

showing memory usage by process and user

Using ps on the linux terminal, we can check out memory usage - 
https://www.networkworld.com/article/3516319/showing-memory-usage-in-linux-by-process-and-user.html

"sort is being used with the -r (reverse), the -n (numeric) and the -k (key) options which are telling the command to sort the output in reverse numeric order based on the fourth column (memory usage) in the output from ps. If we first display the heading for the ps output, this is a little easier to see."

ps aux | head -1; ps aux | sort -rnk 4 | head -5

resubscribing to letsencrypt emails

According to this thread,
https://community.letsencrypt.org/t/accidentally-unsubscribed/14682/14
the way to resubscribe after accidentally unsubscribing would be to change the registration to email+1@mydomain.com with

certbot update_account --email yourname+1@example.com

azcopy with sas tokens and more

I struggled a bit with using azcopy to copy from one Azure storage container to another. The issue finally turned out to be that when a new SAS token was created, the default permissions were not sufficient for what I wanted to do - needed to set the expiry time and read / write permissions correctly. After doing that, the command was:

azcopy cp "https://ourstorage.blob.core.windows.net/ourname/*" "https://ourdestination.blob.core.windows.net/destname?sp=racw&st=-snip-&se=-snip-&spr=https&sv=-snip-&sr=c&sig=-snip-%3D" --recursive

51.3 %, 4919 Done, 0 Failed, 5647 Pending, 0 Skipped, 10566 Total, 2-sec Throughput (Mb/s): 3685.9743

 

- finished in around 5 min.

updating an app on Google's Android Play App store

Updating an already existing app with an already existing developer account - the process was fairly intuitive. Older tutorials point to https://market.android.com/publish/Home - that url now redirects to https://play.google.com/console. 

Basically we need to click on the relevant app, choose View Releases Overview - Release Dashboard - Create new Release.

To edit the play store listing - the text shown on the play store - we need to scroll all the way down on the left-hand side, Grow - Store presence - Main store listing.

For creating the release, we need to sign with the same keystore and signature used to sign the earlier version, etc. as in this post.

comparing Azure pricing in different locations

An interesting site - https://azureprice.net/ Makes it easy to find the region with lowest pricing for a particular type of VM. 

As the site says, pricing is also different in different currencies. 

customizing and building the moodle app

After learning how to use an older node version, the moodle app could finally be customized and built - a follow-up to my failed trials here. 

The customization steps and the relevant modified files are in this location, with the changes being mentioned in the readme as follows:

1. Changes as per the moodle doc on compiling with AOT
Edit Oct 2022 - the link content seems to have changed, so here is the archived version.

2. Change to the build.gradle to force minSdkversion 22

ext.cdvMinSdkVersion = 22

to build with android studio. This can also be done in config.xml

3. Changing all ~ and ^ package dependencies in package.json (except cordova-android itself) to the exact package. Done by replacing

"~ with "

and

"^ with "

in package.json

Or else, all sorts of dependency issues while building the 2019 release in 2021. (The nvm use 11 may also suffice for solving the dependency issues.)

The build directory takes up more than 1 GB, lots of dependencies are downloaded. 

4. Customization - Changes as per the Configuration heading at this post,

config.xml

reduced SplashScreenDelay

and SplashShowOnlyFirstTime true

also, very important,

content src="https://sssvidyavahini.org" would make the app like a simple webview.

We don't want that, so leave the content src as it is, 

but change the siteurl in src/config.json

and onlyallowlistedsites true.

We also set android-minSdkVersion" value="22" in config.xml

google-services.json - changed the package name

and changed the version in src/config.json , ensured it is different, but same number of digits.

5. Changing logo and splash screen as per above url and also resources/android/icon-background.png and icon-foreground.png

and in src dir assets/img

Build steps - 

Set up environment variables as per

https://cordova.apache.org/docs/en/10.x/guide/platforms/android/

JDK tar.gz needed oracle login, created as my official email

Set JAVA_HOME

https://docs.oracle.com/cd/E19182-01/821-0917/inst_jdk_javahome_t/index.html

in .bashrc, added the following.

export JAVA_HOME=/home/mac/Downloads/jdk1.8.0_281
export PATH=$JAVA_HOME/bin:$PATH
export ANDROID_SDK_ROOT=/home/mac/Android/Sdk
export PATH=$ANDROID_SDK_ROOT/tools:$ANDROID_SDK_ROOT/tools/bin:$ANDROID_SDK_ROOT/platform-tools:$PATH

As mentioned in the cordova documentation, we can open a cordova project inside android studio for the final build - choose the import gradle project option, and choose the platforms/android directory.

But we need to edit www folder outside android studio, then copy over changes by doing cordova build - in our case, npx etc as below.

The build process is detailed in the following steps to build document:

# https://docs.moodle.org/dev/Setting_up_your_development_environment_for_Moodle_Mobile_2

nvm install node 11.15.0

# https://www.sitepoint.com/quick-tip-multiple-versions-node-nvm/

nvm use 11

# this is very important - if starting again in another terminal window, must do nvm use 11

# or else all sorts of hard to diagnose errors occur during the build.

sudo apt-get install libsecret-1-dev

npm install 

# cordova.plugins.diagnostic: Diagnostic plugin - ERROR: ENOENT: no such file or directory, open '/home/mac/StudioProjects/LMSappBuildTrial/config.xml'

# added 1829 packages from 1022 contributors and audited 1958 packages in 98.592s

npx cordova prepare 

# Current working directory is not a Cordova-based project.

# https://stackoverflow.com/questions/21276294/cordova-current-working-directory-is-not-a-cordova-based-project

#mkdir www

npm install cordova

# this is also important. Without this, the www directory is not populated for the final prod build.

npm i -g cordova-res

# This is also important - without this, the image resources won't get processed

# config.xml was missing, so copied over with git

# No platforms added to this project. Please use `cordova platform add <platform>`.

npx ionic cordova platform add android --verbose

#Source path does not exist: resources/android/icon/drawable-hdpi-smallicon.png

#Error: Source path does not exist: resources/android/icon/drawable-hdpi-smallicon.png

# editing gitignore file to add the resources

npx cordova prepare

# ignoring the warning about conflict

npx gulp

npm start

# This is an optional step to check in a browser and verify.

# waiting till transpile started etc. then Ctrl+C

# If we wait till the end - nearly an hour on a 4 GB system? it opens in the default browser.

# https://docs.moodle.org/dev/Setting_up_your_development_environment_for_Moodle_Mobile_2#Compiling_using_AOT

cp -v "changes made to moodleapp files/inside node_modules dir/@angular/platform-browser-dynamic/esm5/platform-browser-dynamic.js" "node_modules/@angular/platform-browser-dynamic/esm5"

cp -v "changes made to moodleapp files/inside node_modules dir/@ionic/app-scripts/dist/util/config.js" "node_modules/@ionic/app-scripts/dist/util/config.js"

# edited gitignore to not ignore this config.js, did git add -f.

npm run ionic:build -- --prod

# this takes a lot of RAM and a lot of time. 

# if possible, avoid swapping by closing all apps and clearning memory before doing this.

# REMEMBER to use nvm use 11 if doing this in a fresh terminal

#PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND 

# 3682 mac       20   0 3142252 2.215g  28948 R 131.5 59.6   6:34.48 node

#   build prod finished in 3128.04 s - this was on a 4GB machine

# on a 16 GB GCP instance, it took up around 1/3rd of memory - 5 GB+ - 

# and finished in just 5 or 6 minutes instead of 50 minutes.

# npx cordova run android

# Could not find an installed version of Gradle either in Android Studio,

#or on your system to install the gradle wrapper. Please include gradle 

#in your path, or install Android Studio

# So, I just imported the platforms/android directory using the Import Gradle project option in Android Studio 4.1.3, and built from there.

/dev/kvm permission denied fix for Android AVD unable to open in Android studio

I did not go through this, since my machine had only 4 GB RAM and was probably too RAM constrained to run the virtual device, but

https://16shuklarahul.medium.com/how-to-fix-kvm-permission-denied-error-on-ubuntu-18-04-16-04-14-04-f04a6e23c0cd

- install qemu-kvm and then give appropriate permission to the current user.

$ sudo apt install qemu-kvm

% add your user to the kvm group.

$ sudo adduser <username> kvm

% And then 

$ sudo chown <username> /dev/kvm

uninstall apk before installing with different signature

While developing android apps, we find that sometimes the apks we build don't get installed - the package manager just says Not Installed. The reason is apparently that if the same app is currently installed with a different signature, it will not get overwritten. 

Uninstalling via long press on the home screen on my LG Q6 was not sufficient.

To uninstall cleanly, the way was as listed at this page:

Settings - Apps - Select the app to uninstall - Choose Force Stop
Then choose Storage - Clear Cache - Clear Data
Return to the app screen - choose Uninstall

After doing this, the apk will install. If it doesn't there may be something wrong with the signing etc. which will be the subject of another post of mine. 

signing release build apk files in android studio

The procedure to sign apk files - if you just click on Generate Signed Bundle / APK, the wizard prompts you to create a keystore (or use an existing one) and create the signed build. We have to save that keystore safely, since we will need it for signing any updates to the app on Google's Android Play store (app store). 

But that is not enough - the apk would not install. We've also got to set the signing configuration in the Project Structure, check both v1 and v2 signature versions, and also look for any of the issues which are listed at this page. Mainly, as in this link - 
Project Structure - Modules - Signing tab - add a configuration, and under Build types tab, release build, under signing config, choose the one which we have added. 

loading software on a few hundred tablets

There was some discussion about loading software on some Android devices which were to be shipped to schools. 

Initial thought was to create ids like
OurPrefix.TAB.2021.01@gmail.com
to log on to the play store for the tablet which has asset id OurPrefix-TAB-01 etc.

We could then print out the password and put it in the box for the teachers to change if necessary.

This is required, since this is not a factory install, and factory install process is very different from manual install :) For a manual install, we have to log on to the play store with a particular account, etc etc. 

If we are to not create such ids, installing and then logging out of the account used to install can work for our free apps, but with the drawback that the apps will not update. For updates, the users would have to uninstall and reinstall. 

The next idea was to install all these apps using pre-downloaded APK files, so that the internet connection would not be a bottleneck in the prep process which was planned to be done on a single day for several hundred devices - taking the help of a score or more of volunteers. This too had the drawback of the apps not updating automatically from the Play store. 

Finally the idea of links to the Play store was mooted. 

• Connect via USB cable to the computer, Choose the "File transfer' USB usage preference, copy the tabletslinks.txt file to the Download directory on the tablet. 
• Navigate to the txt file using Google Files on the tablet, click on it and choose to open with Chrome. 
• Copy each line (each link), open a new tab in Chrome, paste in Chrome, hit Enter (while offline)
• Click the three vertical dots on the right hand side of the Chrome address bar, choose "Add to Home Screen"
• Write an appropriate short link title like Pdf for the adobe pdf reader link, Office for the office link, Teams/Meet/Zoom for the respective links.
• Drag and drop the link to the home screen.

The whole process was timed at less than 5 minutes from the time the tablet was booted up. And this was the final solution chosen.

web push api and running our own push notification server for Moodle - link dump

https://developers.google.com/web/ilt/pwa/introduction-to-push-notifications#push_api

https://docs.moodle.org/310/en/Messaging_settings#Mobile_notifications

https://github.com/airnotifier/airnotifier/wiki/Installation-3.x

change default app in Android

https://www.techadvisory.org/2014/01/how-to-set-a-default-app-on-android/

Open Settings - Apps - the relevant app

Scroll down and choose "clear defaults" 

Android studio Git checkout

For checking out at a specific commit - 
https://stackoverflow.com/questions/36517118/how-do-i-checkout-an-old-git-commit-in-android-studio

right-click relevant commit on the Git tab's history, choose 'Checkout revision <hash>'

keeping Windows and Linux dual-boot times correct

This 2011 post has the Kubuntu 10 way, and for current machines running systemd, the way to do it is, if choosing to make Linux use the hardware clock as local time, 
timedatectl set-local-rtc 1

fixing Filezilla error about LC_CTYPE environment variable

As described on this page, the solution was

sudo nano /usr/share/applications/filezilla.desktop

Find  the line
Exec=filezilla
and replace it with
Exec=env LC_ALL=en_US.UTF-8 filezilla