some notes on using supervisord to start a flask web app on Raspberry Pi or Ubuntu-based systems

Using flask on Raspberry Pi - https://towardsdatascience.com/python-webserver-with-flask-and-raspberry-pi-398423cc6f5d

One of the methods suggested to start a python flask web app on boot was to use supervisor. Since I installed it with apt, the suggested method to create a configuration file did not work.

Also, it is apt install supervisor, and not apt install supervisord.

https://www.google.com/search?client=firefox-b-lm&q=supervisorctl%3A+command+not+found

sudo apt install supervisor

https://github.com/Supervisor/supervisor/issues/376

https://forums.raspberrypi.com/viewtopic.php?t=126950

looks like /etc has a supervisor directory and a conf.d directory inside it!

sudo service supervisor status

not supervisord

Also, supervisor seems to search for and use supervisord.conf files, so we must ensure wrong or backup files are renamed to something else. 

INFO exited: flask_app (exit status 1; not expected

This was apparently due to not using the user=ouruser in the config file.

Final configuration was just by appending the following to the existing /etc/supervisor/supervisord.conf file.

[program:flask_app]
user = ouruser
command = /usr/bin/python ourapp.py
directory = /home/our/path/to/WebServer
autostart = true
autorestart = true

Then, to add a favicon, we need to add a route:

https://stackoverflow.com/questions/40696745/adding-a-favicon-to-a-flask-server-without-html

Authentication, or running on android - 

https://www.digitalocean.com/community/tutorials/how-to-add-authentication-to-your-app-with-flask-login

Flask on android | How to run flask server on an android | python web development | we make geeks - YouTube

https://www.youtube.com/watch?v=eDlauNGVYf4

UFW and limit ip address would be easier.

https://pimylifeup.com/configuring-ufw/

(Had to install python-is-python3 on one of my systems for using python nameof script.py instead of python3 nameofscript.py)

 

Edit: problems with this approach, and alternate solution in another post.

notes about controlling "smart lamps"

The final solution was something like this,
https://github.com/hn-88/python-tuya-fadein-web

Connecting to the web app running on our internal network using our mobile browsers to have local controls. 

Making the flask python app start on booting the "server" - using supervisor. I'll write a separate post on configuring supervisor on Raspberry Pi (in general, Ubuntu-based systems).

Link dump:

https://www.makeuseof.com/build-diy-alexa-smart-switches-home-automation/

Add Dimming or Brightening Bulbs to a Routine - Amazon Customer Service

https://github.com/beville/flux_led

flux_led shows no devices. And there is no ip address for the wipro smart batten 

(Actually it did have an ip address, our router was not displaying the client list properly - it would show only those clients which have recently requested ip leases.)

https://www.makeuseof.com/tag/control-philips-hue-lights-arduino-and-motion-sensor/

Apparently philips hue uses a bridge, which does have an ip address.

on alexa app, info by long press is "Connected via Smart Life"

https://www.google.com/search?q=smart+life+lamp+bulb+control+with+python+

https://ardha.dev/blog/controlling-tuya-smart-bulb-using-python

https://stackoverflow.com/questions/73062266/stuck-at-1004-sign-invalid-from-tuya-api

Local Control of Tuya Devices // No Cloud Needed // LocalTuya - YouTube

TradeFace/tuyaface: Python Tuya client that allows you to locally communicate with tuya devices (github.com)

from

https://github.com/rospogrigio/localtuya

which is mentioned in above video.

Will try the nmap solution at

https://www.reddit.com/r/homeassistant/comments/nvtwh3/how_to_identify_tuya_device_on_network/?rdt=40848

nmap -sU -p 137,5353 --script nbstat,dns-service-discovery <<your target range here>>

Install localtuya custom integration and when trying to add devices,
it will list found ones with their ip addresses and IDs.

sean6541/tuyaapi (github.com) - but this seems to be 5 years old, may not work.

https://www.quora.com/What-protocols-are-used-by-Tuya-smart-devices

->

https://github.com/codetheweb/tuyapi

->

https://github.com/clach04/python-tuya

->

https://github.com/jasonacox/tinytuya

Trying tinytuya.

https://github.com/rathmerdominik/tuyactl/blob/master/tuyactl

also uses sleep.

But in testing, even if the set_brightness_percentage() calls are not separated by sleep calls, it still works synchronously, taking more than half a second per call.

Exploit for jailbreaking tuya devices (not used)

https://rb9.nl/posts/2022-03-29-light-jailbreaking-exploiting-tuya-iot-devices/

via

https://github.com/jasonacox/tinytuya/issues/249

fade in and fade out lights using "smart lamps"

Testing out various options, found that these two smart lamps do the job for us.

Wipro Next Smart Wi-Fi 20W CCT+RGB LED Batten | 16 Million Colours | White Tunable | Dimmable |Scheduling |Scene Creation |Smart Grouping of Lights |Amazon Alexa & Google Asst. Compatible |Pack of 1 

https://www.amazon.in/gp/product/B08D19X3LS/

Amazon Basics - 9W Smart LED Bulb with Alexa, Google Assistant & Wi-Fi Enabled (Pack of 1, Cool White, B22 Holder)

https://www.amazon.in/gp/product/B0BBMRCFGW/

Apparently these are Tuya compatible, since they are controlled using the Smart Life app. 

We could control these using either the Smart Life app or the Alexa app, but in case future updates cause them to need a subscription or something, I thought of writing python controls for them with the help of tinytuya.

 Some notes. 

The Amazon basics lamp - for pairing, need to put on and off exactly 5 times: doing 10 times does not seem to work. First need to set up in smart life app, only then can we set up in Alexa app.

1. Smart Life App -

• way to add - add a 'home', add a 'room', add devices
  
  Click on the + in the top right to create "scene" (fade in fade out etc)
  Tap to run
• minimum delay between actions is 1 second
• does not seem to support drag and drop to rearrange actions in a scene.
• does not seem to support "local" control - the control type seems to be set to 'cloud' only.
  

2. Alexa App - 

• When adding device in Alexa app, choose Smart Life as the device type.
• After adding to "office" group, More... for custom routine.
• minimum wait time is 5 seconds. For shorter wait, a workaround as suggested at reddit is to make Alexa say something with 0 volume.
  
  https://www.reddit.com/r/amazonecho/comments/qtz9ob/create_1_second_wait_in_routine/
  
  Action: +
  >(Device Settings) Set Volume 0
  >Next>Action: +
  >(Alexa Says)>(Customize)>Count 1
  >Next>Action: +
  >(Device Settings) Set Volume 5
  >Next>Save<  

3. Writing a python script: I'll write a separate post on my notes for writing this set of scripts.

 

Excel as a sport

Interesting concept - Excel as an E-sports event!

https://www.theverge.com/2023/8/3/23817220/excel-esports-espn-the-ocho-championship-start-time-watch

passport renewal from a rural location

A follow-up post from my earlier post on applying for a passport.

• Since I had a "Post office passport seva kendra" or "POPSK" in a nearby town of Hindupur, I could avoid going all the way to Hyderabad.
• In the online application process, I had to change my Passport regional office from Hyderabad to Vijayawada, since Hindupur comes under Vijayawada. But for doing this, the procedure is to create a new account - the old account's office cannot be changed. I checked this with the toll-free helpline too.
• After filling up the online form, after the last step of making the payment, there is no option to choose a date and time of convenience - the appointment date and time is intimated. So, if we need an appointment on a certain date, we have to keep checking the available dates at their website. Booking an appointment in the evening after 8 pm, like I did this time, seemed to result in an early morning appointment.
• The only required documents shown at the passportindia.gov.in website were the old passport and a proof of identity, for which Aadhaar was available. I took my Voter Id card (EPIC - Election Photo Identity Card) too, and also the old copies of all the documents I had taken for the initial passport application. That turned out to be useful. The passport officer asked me for proof, since I was born out of India and was a citizen by descent and not by birth. Note to self - take the old passports of parents on next renewal too!
• The officer also asked me to take fresh xerox copies of the first and last page of the passport on a single sheet, as well as page 4 & 5 (blank pages). I wonder why. Perhaps as proof that I had not gone out of the country after receiving the first passport booklet in 2014?
• The officer also refused to recognize my class X SSLC certificate (which is a booklet as they used to issue in Kerala in 1990), saying that it is Transfer Certificate and not X class certificate(!) and finally took my MSc certificate as proof that I'm a matriculate for "non-ECR category"!
• But overall, very smooth. Only 2 people in a tiny office - the passport officer and a lady to take the passport photos and biometrics - thumb/other fingerprints. As soon as we entered at 9:35, he started working with us, since ours was among the first appointments, even though the appointment time was 10 am. 
• And the drive to Hindupur was very pleasant, via a National Highway to Lepakshi. Left at 7:45, reached Hindupur POPSK at 9:15 after stopping for coffee and a restroom break at AP tourism hotel next to the Lepakshi bull. On the return, started at 10:10, reached back at 11:45 after taking the non-toll road, since we had to wait at a level crossing for a goods train for a few minutes. 

pdf find and replace

 There was a request to find/replace some boilerplate text in a few thousand pdfs. One option was to use  https://pdfreplacer.com/

and see if it does the job with a watermark.

(You may need to find/replace one line at a time).

Just for interest, looked at scripting options, found it was quite complicated. Some notes below.

Learning more about pdf

https://automatetheboringstuff.com/2e/chapter15/

Visual studio

file 

open with 

binary editor

But does not find the text, because it is zlib compressed.

Is there a C++ library to extract text from a PDF file like PDFBox for Java? - Stack Overflow

Code to extract plain text from a PDF file - CodeProject

podofo/podofo: A C++17 PDF manipulation library (github.com)

https://pdfreplacer.com/

($30 for no watermarks)

libreoffice - Command line tool to search and replace text on a PDF - Ask Ubuntu

Repository search results (github.com)

google search github for pdf tools

Hosted docker-based,

https://github.com/Frooodle/Stirling-PDF

(but no find-replace text in api

https://app.swaggerhub.com/apis-docs/Frooodle/Stirling-PDF/0.10.3#/Convert/processPdfToRTForTXT

)

https://www.google.com/search?client=firefox-b-lm&q=github+pdf+bulk+find+replace+text

https://gist.github.com/artlbv/ec1153feb8644c613b4c7110b25e49ed

https://coolaj86.com/articles/extract-text-from-pdfs-even-protected-ones/

above gist does not work in our case, since the text is not encoded as ANSI/ASCII text but using the font's encoding, in hex.

font subsetting

https://stackoverflow.com/questions/9871585/how-to-find-and-replace-text-in-a-existing-pdf-file-with-pdftk-or-other-command

 

How to replace text in PDF file with Embedded Subset Fonts? | VeryPDF Knowledge Base

pdftotext seems to be from xpdf and not pdftk

how to extract text from pdf with embedded subset fonts - Super User

list of pdf tools and what they can do

PDF Tools and How to View and Edit the Code of a PDF File in Linux | Baeldung on Linux

exploring mutool

and qpdf too

4. Text - Developing with PDF [Book] (oreilly.com)

to see how we can convert the Tj Tf etc to text

pdf generation - How to convert the PDF content code to the type like "(<0034>) Tj"? - Stack Overflow

The <0065> does not just mean 0x65 in ASCII - 

pdf generation - PDF Tj command with angle brackets? - Stack Overflow

python possibilities

Search and replace for text within a pdf, in Python - Stack Overflow

Blog post with 2 ways to replace Replace Text in a PDF File with Python - ThinkInfi

Both are not useful.

Method 1 changes the formatting

Method 2 makes the pdf into an image.

https://www.codingunit.com/c-tutorial-binary-file-io

since xpdf pdftotext seems to work, checking out the code

https://www.xpdfreader.com/download.html

PDFDoc.cc

Page.cc

https://www.codingunit.com/c-tutorial-binary-file-io

https://stackoverflow.com/questions/2094253/editing-pdf-with-xpdf-or-with-something-else

Although not a library in traditional sense, Pdfedit has scriptable editing capabilities. But it requires QT. PodoFo probably fits best at your requirements. There's also PdfHummus.

http://pdfedit.cz/en/pdfedit_windows.html

(unmaintained)

https://github.com/podofo/podofo/issues

(maintained, but unstable)

https://github.com/galkahana/pdf-text-extraction/tree/master/TextExtraction

(apache license.)

IDR solutions guide

page 16 says

The Font encoding object specifies how to translate this value into a character (and these are defined in Appendix D of the PDF Reference specification)

But these seem to be similar to Unicode / ASCII, with d being  0x64 - not 0x47 as seen in our document at an offset of 18020 or so.

pg 27

RUPS free tool for diagnostics of PDF

Diagnostic tool for updating PDF Syntax and debugging PDF code | iText PDF

Using RUPS

Thence to find the encoding Identity-H

Identity-H encoding - Adobe Support Community - 10400841

https://community.adobe.com/t5/acrobat-discussions/identity-h-encoding/td-p/10400841

So, looking for a ToUnicode CMap

found the stream,

to understand bfchar and bfrange in cmap

How are Embedded CMAP tables defined in a PDF File? (idrsolutions.com)

https://blog.idrsolutions.com/how-are-embedded-cmap-tables-in-pdf-file/

we see a pair

<0047> <0064>

So, 47 in the pdf is mapped to unicode 64?! Yes!

We can use

http://brendandahl.github.io/pdf.js.utils/browser/

instead of RUPS to browser (but RUPS can also edit/change the PDF.)

Unfortunately, there is no space 0x0020 mapped? So we can't just replace the characters with 0x0020 or something similar.

The "correct" way to do it might be to use the "stamp" function to overwrite the offending text with what we want.

The "hacky" way might be to replace the characters with something else and make them not visible, if that is feasible.

creating and using an "outage page" using github pages and cloudflare redirect rules

Easy to create a page on github pages for outages, and then point to that page using cloudflare redirect rules. Once the outage is resolved, we can simply disable the rule for the site to work again normally. 

openapi

Interesting concept - Design API first, code later

https://support.smartbear.com/swaggerhub/docs/integrations/github-sync.html

problem with dimming LED lights

We purchased a cheap dimmer off Amazon which advertised "suitable for LED", but when we hooked up some of the lights we have, the lights started flashing instead of dimming. Dimming worked OK for incandescent bulbs. Looking around for solutions, found these videos which gave some good info.

The problem with dimming AC LED bulbs - https://www.youtube.com/watch?v=BhQWOv10Oag

Apparently the issue is that those LEDs which are sold as "not dimmable" (or in our case which specifically don't say "dimmable") contain an SMPS with a capacitor across it, so the pulse width modulation of the dimmer circuit causes it to flash instead of dim. 

Apparently we can dim such bulbs using a capacitor - https://www.youtube.com/watch?v=JH68wBXO5j0 . But then we can only have discrete dim levels, and probably the light will just cut out completely at some point. 

Using duct tape, parchment paper etc we can physically block the light and thus dim it too,

https://www.youtube.com/watch?v=NtdJ8pwweMs

 

But in our case, creating a motorized mask like the above seems more difficult than just buying some "dimmable" LEDs. So the next step might be to explore such products.  

Edit: my post about using "smart lamps" for this is at
https://hnsws.blogspot.com/2023/08/fade-in-and-fade-out-lights-using-smart.html

 

finding created time of a file on linux commandline

 Following the different methods given at https://kodekloud.com/blog/file-creation-time-linux/#

On one of our Azure Ubuntu 20.04 servers, only the debugfs solution worked. 

sudo debugfs -R 'stat /path/to/our/file.txt' /dev/root 
... etc 

crtime: 0x64afbcf8:ea7086a8 -- Thu Jul 13 08:59:36 2023

-----------------------------------------
ls -l --time=birth ourfile.txt
ls: invalid argument ‘birth’ for ‘--time’
Valid arguments are:
  - ‘atime’, ‘access’, ‘use’
  - ‘ctime’, ‘status’

--------------------------------------
stat additionalplugins.txt 
... etc ... 
Change: 2023-07-24 05:00:13.991923547 +0000
 Birth: -

gripes about online services from Govt-run undertakings

Canara bank was fined for non-compliance, and is now planning to overhaul its IT infrastructure. Something similar might be needed for LIC also soon. Some points from my recent experience with the LIC, their customer portal and phone-based customer service - 

1. Money was credited to my account. But no intimation about which policy etc. Digging around in the portal showed one of the policies' "survival benefit" processed 2 months ago - that seems to have been credited now. 

2. Asking by email or via phone support about which policy credited the amount - no reply to email, and phone support wanted me to enter policy number, but gives too short a time to enter the 9 digit number! Tried without entering the number, but the agent said I have to enter the number for them to help me. Phone support also pointed me to the wrong section on website for checking which policy credited the amount. Instead of Premier Services -> Policy Claim History, the agent on the phone was guiding me towards Basic Services -> Claim Status. The latter seems to show maturity etc in the future, while the former shows the past credits, which is what I had asked for.

3. "Service request" - OTP kept being rejected "Wrong OTP..." on Edge browser. Finally tried Chrome browser, that works. Phone support initially asked me to email portalsupport at licindia dot com, no reply. I filled up a new ticket as mentioned on the home page of their new portal, again no reply. 

4. After going through all those hoops, "Could not connect to server, try again later".

5. Luckily, when I tried again the next day, it worked. This "try later" business is applicable to the "Jeevan Saakshya" app also. Sometimes it just doesn't work. Trying again later, after a few hours or days - it may then work. 

6. Some of their warnings to policy holders given on the website seem to be at odds with the instructions given to the people in their offices. A random person rang up me up saying that he's an agent, and I could send the policy to him so that the survival benefit could be credited to me! This, in spite of the NEFT registration already done on the portal for that policy. Finally, their office rang me up, clarifying that it is their policy to inform "good agents"(!) about maturing policies so that users can re-invest. Anyway, I could send an email to them with the form they wanted me to to fill up, along with a picture of the policy, and though they said that the money would be credited the next day, it actually took a week.

Maybe this sort of bad IT infrastructure is not limited to Govt-run undertakings - TCS comes to mind with their terrible websites in the past. Hopefully improving now. 

Just a random link about issues with govt offices, which are much much worse than their online systems - https://old.reddit.com/r/Kerala/comments/rbvpue/what_are_some_of_the_worst_experiences_youve_had/

 

Edit: August 2023 - Credit where credit is due. Calling / emailing the "customer zone" at Kozhikode for LIC gave a very different experience. The person taking the call / answering the emails was very helpful, was ready to give a printout of the required amount credited details, gave all the details I asked for by email when I said I was out of town. Probably this is a direct employee of LIC, as against the contract workers manning the toll-free number.

deleting lots of files - ftp vs Plesk

Bulk deleting all files on a website - found that deleting via Filezilla ftp client was taking a long time, since each file delete command was being sent via ftp. Logged on to Plesk, chose the directories to delete and chose the Remove button, opting to not move to Recycle bin - this went much faster, since Plesk would just use the OS command to delete the entire folder and all files in it.

linux mint upgrade from 21.1 to 21.2

The upgrade from Linux Mint 21.1 to 21.2 went surprisingly easily. For a change, I just clicked on the graphical update manager / system reports prompt which said that a new version is available. I did read through the release notes as prompted, but overall, the upgrade did not force me to take a timeshift backup etc. I just started the process and continued working on Firefox - the upgrade went on in the background, and finished after an hour or so. It took an hour because (a) my internet speed was limited to 4 Mbps and (b) the machine is an old Macbook Pro.

After a reboot, the only thing removed by apt autoremove was "vera backgrounds". I went into Update Manager, changed to local mirrors, and then via View -> Linux Kernels -> Remove Kernels , removed the 5.19 kernel in favour of the installed 6.2 kernel. After going through the menu again, removed 5.15 kernel too.

W3Schools Spaces for testing out static html pages

I've been trying out test html pages on W3Schools' new free offering, https://spaces.w3schools.com/

Useful for quick tests of css/html changes. Easier than setting up a web server on every local machine I work on.

exposed git repo risks

Recently I got an automated email from reporter at repo-lookout.org warning of an exposed git repo on one of our sites. According to this,
https://iosentrix.com/blog/git-source-code-disclosure-vulnerability/

I fixed it using the method for Apache 2.4

<DirectoryMatch "^/.*/\.git/">
    Require all denied
</DirectoryMatch>

LIC premier services application

If we log out after printing the generated application form, we can either wait for a week for the application form to get reset, or go to Service Requests Tracker, where the application form is noted as incomplete, which we can then resume. Perhaps this is shown only after a day or so. (Phone support asked me to email portalsupport at licindia.com but there was no response from them.) 

Edit: As noted in the other post about issues with the website etc, had a good experience with LIC's customer zone, Kozhikode, via telephone. Very helpful, unlike the toll-free phone support.