Monday, April 25, 2022

fixing an error in a moodle ad-hoc query

One of the ad-hoc queries in one of our moodle servers was returning "Error reading from database" when executing certain queries with certain parameters, working with other parameters.

Troubleshooting - The general issue is that with ad-hoc queries, only the generic error "Error reading from database" is shown, and not the exact cause of the error, which would be some problem with the SQL query. Running the same query using DBeaver, (I had to find/replace { with ourprefix_ and } with a space) 
I get the more helpful error message

SQL Error [1242] [21000]: Subquery returns more than 1 row

Commenting out all except the first subquery, I still got the error. So, there was an issue is with the first subquery.

Looking at the table ourprefix_feedback_value I saw that there are some entries where the name field with match
like '%Chapter%' but the typ field is not a textfield.

Assuming that this is what is causing the problem, I modified the query by adding an additional line in bold below
where
fc.timemodified = fc2.timemodified
and fi2.name like '%Chapter%'
and fi2.typ = 'textfield') as "Chapter name",

This also did not solve the issue. Then, checked the ourprefix_feedback_completed table looking for duplicates in the timestamps like
select fc3.timemodified, count(fc3.timemodified)
from
ourprefix_feedback_value  fv3
left join vv_feedback_item  fi3 on fi3.id = fv3.item
left join vv_feedback_completed  fc3 on fc3.id = fv3.completed
where
fi3.name like '%Main%Script%'
GROUP BY fc3.timemodified
HAVING COUNT(fc3.timemodified) > 1

found 1646480361 was repeated twice. Two feedback items had exactly the same timestamp. Changed the second one to 1646480362 and the problem was solved.


 

Saturday, April 23, 2022

troubleshooting a php framework / cms based site

A couple of websites went down after a server upgrade. Checking out the contents of public_html, found some directories named fuel and codeigniter. Initially thought it was a codeigniter-based site, 

Later, found some function calls like fuel_set_var() which on searching github and google led to FUEL CMS - 

My first impressions were that upgrading from the existing 0.9x version to 1.5x was not so straightforward - the migration tool, mentioned in the documentation above, did not run. Then there is the part of the documentation which says that it needs MySQL 4.x + - 
Hopefully it works with 5.x? Will have to try manually capitalizing filenames and class names as needed for codeigniter 3+. 
"If you have a current installation of 1.3x or less and are wanting to upgrade, there are a few things to be aware of. FUEL 1.4 uses CodeIgniter 3.x which includes a number of changes, the most prominent being the capitalization of controller and model names. Additionally it is more strict on reporting errors."


Friday, April 22, 2022

changing cpanel theme for all accounts in WHM

There was the notification about CPanel Paper Lantern theme being deprecated and replaced with Jupiter theme. Changed the theme for all accounts using

- changed themes for all by going to WHM - Edit a package - and editing all the defined default packages. 

fixing wordpress errors after server upgrade

A wordpress site was not loading after a server upgrade. Just a blank screen. To troubleshoot, first checked database connectivity, database password in wp-config - that was working. Then, tried to get more information about the error by changing the define( 'WP_DEBUG', false ); - changed to true. Then the error showed up - Fatal error: Allowed memory size of 41943040 bytes exhausted (tried to allocate 65536 bytes) in /home/websitename/public_html/wp-content/plugins/woocommerce/includes/class-wc-ajax.php on line 1101

Fixed it using the solution at 
by adding 
ini_set('memory_limit','128M');
to the top of wp-config.php

Problem solved. Set the WP_DEBUG variable back to false. 

Wednesday, April 20, 2022

OpenCV with Qt creator

K asked for pointers for this issue - I am trying to integrate OpenCV into the C++ code that I wrote for gamma acquisition. I am facing a problem while integrating it into the software Qt Creator. The headers are included without any errors, but when I initialize an empty matrix it returns "undefined reference to 'cv::Mat::~Mat()'". 

My reply was - 

"Undefined reference in the linking process doesn't mean, that it can't find the declaration in the header file, it means that it cant find the definition/implementation in any of the provided object files."

So the issue is most probably due to some path issue in the Makefile - it depends on what you are using to build the project. If you are using qt creator, then it would be qmake.

The easiest way to solve this would be perhaps a google search on getting started with OpenCV and QT creator.
Perhaps
if you are using windows.

This is an old version for opencv 3 on ubuntu,
but you may get some ideas from it.

He finally solved it himself - "I could link the library and use its functions now. I believe there are a lot of ways to link the library and I figured one way that worked for me.

I have discussed the way I solved it here 
so that it will be helpful to whoever compiles the OpenCV with Mingw 32-bit in Qt Creator."

So, in short, he "had to link the libraries by right-clicking on the project folder on the left-sidebar, selecting Add Libraries and choosing the External Libraries option (added one by one).

win32: LIBS += -LD:/opencv-build/install/x86/mingw/lib/ -llibopencv_core320.dll
win32: LIBS += -LD:/opencv-build/install/x86/mingw/lib/ -llibopencv_highgui320.dll
win32: LIBS += -LD:/opencv-build/install/x86/mingw/lib/ -llibopencv_imgcodecs320.dll 
INCLUDEPATH += D:/opencv-build/install/include
DEPENDPATH += D:/opencv-build/install/include

////NOTE: imgcodecs library is required for imread to work//////

Friday, April 15, 2022

improvMX for domain email forwarding

In a previous post, I had mentioned using our domain registrar's free email service. Now I have moved away from that to improvMX, for a hopefully more stable performance - emails were taking around 20 minutes to be forwarded using our domain registrar's mail severs, and this was causing problems with one-time passwords (OTPs) sent by Github, for example. There is a way to send emails via gmail smtp servers, too (but it displays "via ourdomain.tld" when I use my account@ourdomain.tld ).

Sunday, April 10, 2022

PBX with Raspberry Pi - RasPBX - or with the cloud - AWS

 https://www.youtube.com/watch?v=n_txukfW3uE
How to turn Raspberry Pi into FreePBX-based GSM gateway
(needs 3g modem)

Or, if willing to pay for the SIP trunk termination instead of buying a 3g modem, AWS based method -
https://www.youtube.com/watch?v=n_1wX7kKx7k

Quite interesting. 

Friday, April 08, 2022

youtube live streaming possibilities

Wednesday, April 06, 2022

bulk replacing descriptions

In our local database, using phpPgAdmin, got an sql prompt by choosing a Select and Edit SQL, then the following query worked:

SELECT "fileid","description","languageid" FROM "public"."file_information" WHERE "description" LIKE 'Our Descrip%' AND "languageid" = '1'

Edited SQL to

update "public"."file_information" set "description"
REPLACE(description,'Our Descrip', 'Prefix to Our Descrip') 
WHERE "description" LIKE 'Our Descrip%' AND "languageid" = '1'

(trying this with a view instead of a table did not work, 
ERROR:  cannot update a view
HINT:  You need an unconditional ON UPDATE DO INSTEAD rule.)

For the remote database, tried a similar query, but it failed. phpMyAdmin had a find/replace option, so used that, and the corresponding query, which has single back-quotes, was:
UPDATE `our_table_name` SET `description` = REPLACE(`description`, 'Our Descrip', 'Prefix to Our Descrip') WHERE `description` LIKE 'Our Descrip%' COLLATE utf8mb4_bin

Tuesday, April 05, 2022

moodle minor upgrade

An update for doing minor point upgrades to moodle - much simpler than what I have listed at 
https://hnsws.blogspot.com/2021/08/detailed-steps-for-moodle-upgrade-using.html

Just
cd /var/www/theLMSgit
sudo -u www-data /usr/bin/php admin/cli/maintenance.php --enable
git pull
sudo chown -R azureuser:www-data .
sudo chmod -R 775 .
sudo -u www-data /usr/bin/php admin/cli/upgrade.php
(type y when prompted)

sudo -u www-data /usr/bin/php admin/cli/maintenance.php --disable
cd our-notifier-folder
sudo ./startupscript.sh
(can check out.log to see if server has started)

We can then test notifications using Site Administration - Messaging - Mobile - Check and test push notification configuration - that page asks us to make sure that "your devices are connected to the Internet and that the mobile app is not open (since push notifications are only displayed when received in the background)."

Wednesday, March 30, 2022

changes needed for migrating show creation from Windows to Linux

As noted in an earlier post, 
avisynth works on Linux under Wine, with certain changes.
  • need to run Virtualdub with
    cd ~/windows/VirtualDub
    wine VirtualDub
  • need to add the line 
    Loadplugin("/home/myusername/.wine/drive_c/Program Files/AviSynth 2.5/plugins/ffms2.dll")
  • need to use FFvideosource("in.mp4") instead of DirectShowSource()

Tuesday, March 29, 2022

streaming our own youtube videos as a live stream on youtube

Implemented as a bash shell script using ffmpeg and the latest youtube-dl fork yt-dlp, at https://github.com/hn-88/live-stream-from-youtube-to-youtube 

References are mentioned in the readme. Usage is in the comments.

Sunday, March 20, 2022

trying out the free email check api

Mailboxlayer.com offers a free tier for their email verification api, 100 requests free per month - I signed up for free, and could easily use their api - via apilayer.com.

The api (from https://blog.apilayer.com/the-anatomy-of-a-great-email-address-validation-api/) in the format
http://apilayer.net/api/check?access_key=4aMyAccessKey6f8ca4&email=thisIsWhatWe@WantTo.Check&smtp=1&format=1

The smtp_check value (in the returned data) being false would indicate email not existing. This is probably the older version of the mailboxlayer api, since the current api has a syntax like
curl --location --request GET 'https://api.apilayer.com/email_verification/customercare@apilayer.com' \
--header 'apikey: YOUR API KEY HERE'

Saturday, March 19, 2022

turning the alarm off on HTC-1 clock humidity temperature meter

As seen in this video, in order to set the alarm, short-press Mode, then long press Mode till minutes start blinking. In order to turn alarm on or off, press mode again, set the hours, press mode again, and now when in Alarm setting mode without the numbers blinking, press the Adj button to toggle through Alarm on, (tone every 15 minutes I think) and Alarm off. If the unit is just left alone for a minute, the current time display returns. Or we can short-press Mode again to come out of Alarm set mode. 

So, in short, if we just want to toggle Alarm on or off, just short press Mode and then short press Adj repeatedly to toggle through the Alarm on or off modes.  

mass emailing with Google Apps scripts and validating email lists

One of the ways in which we can send "Mail merge" equivalent customized emails to a large number of users would be using the sample script at
https://developers.google.com/apps-script/samples/automations/employee-certificate
- this seems to be a much better solution than GMass (free option) which I had earlier posted about, but requires some coding. 

This works quite well if all the emails are pre-verified. Our non-profit accounts have ~1500 emails per day as reported by
var emailQuotaRemaining = MailApp.getRemainingDailyQuota();
Logger.log("Remaining email quota: " + emailQuotaRemaining);
 
But the important thing is that the list of emails must be accurate. If multiple emails bounce, the whole process will get derailed, and the user sending the emails will get suspended. We should probably also go slowly so that rate limiting does not kick in. 

One of the good ways to ensure valid emails might be if all these emails in the list are obtained using the users filling up some google form which requires google sign-in. 

A youtube video for validating email lists uses SMTP check being done with api from



We could also add the email ids to a google group - that would also validate their email to some extent, with at least hotmail, yahoo and gmail ids being caught if invalid. 

Edit: Some interesting reading about spam filters and "warming up" - 

Wednesday, March 16, 2022

webpage with links to ebook using RSS and Calibre

Calibre has the feature of "news feed" which will fetch the links listed in an rss feed and create an ebook out of them. Though "custom recipes" can be written to scrape pages and get links from there, I tried to make things easier for me by using an automated tool to create RSS from HTML, and then use calibre's "add new recipe" to "create a recipe from scratch" using just the RSS feed. The Hindu's Young World archives did not have an RSS feed, but I could create one quite easily with feed43 ("Feed for free") using the simple item search pattern
<h3><a href="{%}"{*}class="">{%}</a></h3>
with
Item link template = {%1} and 
Item Title template = {%2}

feed43's free feed is limited to 20 items, so for all the older links, I would need something else. Found mkfeed, which uses the same syntax as feed43. So I just needed to install mkfeed and slightly tweak the provided example to create an RSS file locally - 
URL="https://www.thehindu.com/topic/The_Hindu_Young_World/"
wget -q -O - "$URL" | mkfeed \
    --pattern-item '<h3><a href="{%}"{*}class="">{%}</a>' \
    --feed-title 'YW Archive1' \
    --feed-link "$URL" \
    --feed-desc 'Older Young World articles' \
    --item-title '{%2}' \
    --item-link '{%1}' \
    --item-desc '{%2}' > yw1.rss

Calibre didn't seem to be able to fetch local files as RSS feeds when I gave the local path as /local/path/to/yw1.rss as the feed url. So, ran a simple webserver using python as described here, but changing the port to the less privileged 8000, with
import os
from http.server import HTTPServer, CGIHTTPRequestHandler
# Make sure the server is created at current directory
os.chdir('.')
# Create server object listening the port 8000
server_object = HTTPServer(server_address=('', 8000), RequestHandlerClass=CGIHTTPRequestHandler)
# Start the web server
server_object.serve_forever()

Then it was just a matter of pointing a new recipe in calibre to http://localhost:8000/yw1.rss and calibre did the rest. This might be an easier set of steps to do compared with my earlier manual method using HtmlAsText.

Saturday, March 12, 2022

phone photos as thumbnails in Nemo

In order to manage the photos on my Android phone, it is convenient to connect it via USB and browse the filesystem via the PC (Linux Mint) File Manager, Nemo. If I want to see the image files with their thumbnails as icons, have to change Edit - Preferences - Preview - Show thumbnails from Local files only to Yes. Then, it will show thumbnails for previewable files even for remote filesystems like the Android filesystem. Also have to change the "Only for files smaller than" setting, for which the default is 1 MB. 

Edit - and then increase the size of the icons using the slider on the bottom right of Nemo's status bar, like below - 


google apps script for adding to a google group

Since we have a non-profit edition of Google Workspace, we can use google groups with group email id like email@ourdomain.tld for a group created using the admin console, and users can be added to such groups using the API. My trial scripts have been added to this github repo, Google-groups-GAS-tests. Some points to note - 

Sunday, March 06, 2022

google apps script to turn blog posts into an ebook - no oauth required

It turns out that all the complicated steps I went through in the previous post were not necessary. Blogs on blogger provide a feed url which returns xml or json as you prefer, with no need for authentication for public blogs. My code is at https://github.com/hn-88/bloggerToEbook and the feed url, for reference, is like 
https://domainnameofblog.tld/feeds/posts/default/-/relevant-label?max-results=100&alt=json

If we want to narrow down results via posted date,
shows us that we need to add the following to the url above - 
&updated-min=2008-03-16T00:00:00&updated-max=2008-03-24T23:59:59&orderby=updated

After downloading the Google Doc as an EPUB, I did a couple of tweaks to make it read better on the Kindle when converting to MOBI with Calibre. Turning the green text to black, by ticking the 'Color' box to ignore,


and also auto-generating Table of Contents,



I had also increased the number of links to add to Table of Contents from 50 to 500 in the screen above, since I was making ebooks with 100 or 150 posts each. 

Edit: Bloxp.com is one of the high ranked google results for turning a blog into an ebook. But right now it doesn't seem to be working - tried both with Blogger feeds as well as rss feeds from https://www.thehindu.com/rssfeeds/

Edit 2: Another way to turn blog posts into an ebook, mentioned at
is

This seems to be dead now. archive.org shows that it redirects to pothi.com now. 

Edit 3 (2025): There are updated posts about turning blog posts into ebooks, at 
https://ravisiyermisc.blogspot.com/2023/06/google-apps-script-to-create-blogger.html and

Saturday, March 05, 2022

google apps script to turn blog posts into an ebook - calling blogger api

Google Docs have an EPUB export option. So, if we get the html content of the blog into a google doc, we can export as an ebook. Some blogger blogs have good content, but have some sort of html / CSS / char encoding issues so that merely doing

var response = UrlFetchApp.fetch(urls[i], { muteHttpExceptions: true });
 var ablob = response.getBlob();
      var AssetGDocId = Drive.Files.insert(
        { title: ' temp' + i + '.html', 
        mimeType: MimeType.GOOGLE_DOCS },
        ablob
      ).id;

results in a google doc with a single character per line!

So, going the blogger api route. But there are certain complications.

Blogger is not one of the services which are directly available via the "Add a service" dialog box,

When following the example at
needed to make a couple of changes as mentioned at
making the headers variable
Authorization: "Bearer " + ScriptApp.getOAuthToken(),
and adding 
"oauthScopes": [
    "https://www.googleapis.com/auth/drive",
    "https://www.googleapis.com/auth/blogger",
    "https://www.googleapis.com/auth/script.external_request"
  ],
to the manifest. To add to the manifest, we have to make it visible in the editor via the settings tab,




Further, the default apps script project does not have the blogger api automatically enabled as for other advanced services - we have to create a project and add the blogger api (and drive api in this case). We can ignore creating credentials - the app script will automatically create the oauth credentials. 

Once we create a new project at console.cloud.google.com, we have to add the required apis using the APIs and services tab on the left-hand side - Blogger api and Drive api in this case. And then associate the google apps script with that project under the script settings, Change project. The project number can be found from the Home -> Dashboard page, in the project info card of console.cloud.google.com.




A caution here - if creating credentials or editing the consent screen - sometimes there are errors creating credentials, and one of the triggers seems to be the use of multiple logins. If we are logged in to the cloud console by appending &authuser=2 (or something like that) to the url, like 
then credentials creation can run into all sorts of unexplained errors. "There was an error creating credentials...."
It would be safer to log in with only one account, perhaps by using incognito mode. Or maybe this firebase trick or other issues in this post can help in some cases.

After creating the consent screen, we have to add the scopes needed - again, drive and blogger apis in this case. And also add our own email id as a test user. 



Finally, after getting the api calls to work, I need to find out how to get all the posts without timing out. One way might be to use the search call instead of the list call and get "all posts labelled with 'suitable label' in September 2013" or something like that. That would probably be covered in a separate post here.

Edit: Getting all the posts with a particular label or a particular date range is a good way to narrow down the number of posts, and that is how I implemented. But it turned out that all this work of adding the blogger api etc was not required after all - the blogger atom feed api doesn't need any authentication for public blogs.