Work Stuff

Maciej Pocwierz, a senior software engineer Semantive, writing on Medium: A few weeks ago, I began working on the PoC of a document indexing system for my client. I created a single S3 bucket in the eu-west-1 region and uploaded some files there for testing. Two days later, I checked my AWS billing page, primarily to make sure that what I was doing was well within the free-tier limits. Apparently, it wasn't. My bill was over $1,300, with the billing console showing nearly 100,000,000 S3 PUT requests executed within just one day! By default, AWS doesn't log requests executed against your S3 buckets. However, such logs can be enabled using AWS CloudTrail or S3 Server Access Logging. After enabling CloudTrail logs, I immediately observed thousands of write requests originating from multiple accounts or entirely outside of AWS.

Was it some kind of DDoS-like attack against my account? Against AWS? As it turns out, one of the popular open-source tools had a default configuration to store their backups in S3. And, as a placeholder for a bucket name, they used... the same name that I used for my bucket. This meant that every deployment of this tool with default configuration values attempted to store its backups in my S3 bucket! So, a horde of misconfigured systems is attempting to store their data in my private S3 bucket. But why should I be the one paying for this mistake? Here's why: S3 charges you for unauthorized incoming requests. This was confirmed in my exchange with AWS support. As they wrote: "Yes, S3 charges for unauthorized requests (4xx) as well[1]. That's expected behavior." So, if I were to open my terminal now and type: aws s3 cp ./file.txt s3://your-bucket-name/random_key. I would receive an AccessDenied error, but you would be the one to pay for that request. And I don't even need an AWS account to do so.

Wednesday, May 01, 2024

OCVWarp on Google Cloud Shell

Checked and found that OCVWarp works (though slowly - 5 fps for XVID encoded 1080p output) even on Google Cloud Shell, after installing the dependencies.

wget https://github.com/hn-88/OCVWarp/archive/refs/tags/v4.01.zip
unzip v*.zip
cd OCVWarp*/build
wget https://github.com/hn-88/OCVWarp/releases/download/v4.01/OCVWarp-4.01-x86_64.AppImage
chmod +x OCV*mage
mv OCV*mage ocvwarp
sudo apt install libopencv-dev

(after uploading the input file)

./ocvwarp OCVWarp.ini 30fps.mp4 output.mp4

Friday, April 26, 2024

Moodle upgrade path - 4.1 to 4.5

An evaluation of requirements and timelines for upgrading Moodle 4.1 to 4.3 or 4.5.

https://moodledev.io/general/releases/4.3

Moodle 4.1 -> 4.3

Min PHP 7.4 -> 8

PHP extension sodium is required

(seems to be built in to 8

https://serverfault.com/questions/1118165/how-to-install-php-sodium-on-ubuntu-22-04

)

PHP setting max_input_vars must be >= 5000.

PHP variants: Only 64-bit versions of PHP are supported.

MySQL 8.0 (increased since Moodle 4.1)

Moodle 4.4 release date is 22 April 2024

so probably should upgrade to that - OR -

Next LTS is 4.5. 7 October 2024.

https://moodledev.io/general/releases

Moodle 4.1 -> 4.4

Min PHP 8.1

Ubuntu 22.04

PHP 8.1.2

Mysql 8.0.28

Ubuntu 24.04 has:

PHP 8.3

Mysql 8.0.36 (via this link to get the url, and https://web.archive.org/web/20240502044149/http://archive.ubuntu.com/ubuntu/dists/noble/main/binary-amd64/Packages.gz )

How to upgrade from Ubuntu 20.04 to 24.04 - Probably will need to do it as a two step process -
https://linuxconfig.org/ubuntu-upgrade-to-24-04-noble-numbat-a-step-by-step-howto-guide

https://linuxconfig.org/how-to-upgrade-ubuntu-to-22-04-lts-jammy-jellyfish

20.04 is supported till Apr 2025. So in Oct 2024, we should be able to upgrade the OS and Moodle together.

Friday, April 19, 2024

Filezilla TLS errors

When using Filezilla on Linux Mint at work, connecting to an FTP server, after logging in, I get some errors like the following:

Command: LIST
Response: 150 Opening BINARY mode data connection.
Error: GnuTLS error -110: The TLS connection was non-properly terminated.

Some file transfers go through, while some get 550 errors - when I close Filezilla and open it again, the transfer succeeds.

Later, when using Windows and downloading using a domestic BSNL fiber line, there were no errors.

According to this set of posts on serverfault, it may have been due to either the firewall gateway blocking passive mode FTP or it may have been something to do with implicit/explicit TLS.

Wednesday, April 17, 2024

methods to download from Amazon S3 bucket (if we own/manage the bucket)

Some ways are mentioned at

https://bobbyhadz.com/blog/aws-s3-download-folder

Cyberduck can provide a UI for Windows or Mac - "Cyberduck is a libre server and cloud storage browser for Mac and Windows with support for FTP, SFTP, WebDAV, Amazon S3, OpenStack Swift, Backblaze B2, Microsoft Azure & OneDrive, Google Drive and Dropbox."

Saturday, April 13, 2024

workflow_dispatch only from default branch

I like to set Github Actions to only run when manually triggered, that is, using workflow_dispatch

As that page says, "To trigger the workflow_dispatch event, your workflow must be in the default branch."

But we can change the default branch - like we may want to do for forks etc - Repository --> Settings --> Default branch (choose by clicking the "choose another branch" button which has two opposing arrows as an icon)

comparing the contents of two directories on Windows

How to Compare Two Folders on Windows 11 and 10 (howtogeek.com)

robocopy "C:\first" "C:\second" /L /NJH /NJS /NP /NS

Thursday, April 11, 2024

reminder to take a break - using cron

Wanted to create a pop-up message or reminder (and was later requested an audio reminder) for N to take periodic breaks from the computer. Since his usage was mostly passive, workrave did not work for this purpose - workrave's timer only progresses when the keyboard or mouse are active.

Looking for solutions, reminder apps on Mint/Ubuntu seemed to have only flatpak versions which I did not want to install.

Finally thought it would be simpler to use cron.

But, to pop up messages on the screen with cron, we need some special handling.
https://unix.stackexchange.com/questions/307772/pop-up-a-message-on-the-gui-from-cron
https://unix.stackexchange.com/questions/111188/using-notify-send-with-cron

And for audio, we would need another line to be added to the script. Working script -

#!/bin/bash
export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
export DISPLAY=:0
export XDG_RUNTIME_DIR=/run/user/1000
# this last line above is for sound to work
notify-send -u critical "this is the reminder to take a walk"
# the -u critical is supposed to show over full screen apps, but doesn't always work
/usr/bin/mpg123 /home/username/Downloads/Walk.mp3 > /dev/null 2>&1

For help with setting up cron, cron.help is useful. Eg.

*/30 * * * * /home/username/walkremind.sh

for every half an hour.

For getting the environment variables' values, used

printenv

And for installing notify-send,

sudo apt install libnotify-bin

using a laptop as a secondary monitor using Miracast

Apparently Windows now has built-in support for Miracast, so we can use a Windows laptop as a secondary display for a Windows desktop, for example.

https://www.tomshardware.com/how-to/use-laptop-as-monitor-for-another-pc
that is if both run Windows.

Miracast client for Linux - miraclecast - to use as sink,
https://askubuntu.com/questions/1254998/how-do-i-launch-miraclecast-after-installation

https://alternativeto.net/software/miraclecast/
https://letsview.com/screen-mirroring
(but not on linux)

https://github.com/albfan/miraclecast

Wednesday, April 10, 2024

RAM usage - Edge and Firefox browsers on Linux Mint

I've installed 'Graphical Hardware monitor' to show RAM, Disk read and Disk write on the panel of the old Macbook Pro running Linux, which has only 4 GB of RAM including VRAM. Curious to see RAM usage, tried opening Gmail + GDrive + a spreadsheet on Firefox and Edge browsers separately. With Firefox, total RAM used was 1.92 GB. With Edge, it was 1.63 GB. So, Firefox was using around 300 MB more than Edge in this (not too rigourous) test.

image sequence support for OCVWarp

Prompted by this feature-request for command-line support, OCVWarp version 4.00 now has basic support for image sequences, too. The documentation in the wiki has been updated with caveats. And here is a link to all posts about OCVWarp.

PS. And in December, a test build with MacOS X succeeded on github actions using homebrew to install OpenCV.

Sunday, April 07, 2024

using blender online

As noted at https://digitalarthub.net/3d/how-to-use-blender-online-without-downloading/

"Do note that most of these apps run outdated versions of Blender and they’re pretty laggy",

OffiDocs (Requires no account) - seems to be ver. 2.8 or so.
Turbo.net (Requires account)
RollApp (Requires account)

Since Blender doesn't run any more on the old Macbook running Linux - https://www.blender.org/download/requirements/ - I can use the online versions for quick screenshots and so on.

Friday, April 05, 2024

Windows defender finding a Trojan in build from Github actions - false positive

Artifacts from some builds of OCVWarp were being flagged by Windows Defender as having a trojan, Wacatac. As mentioned in this page, https://github.com/RPCS3/rpcs3/issues/15309 this is probably a false alarm, as some other builds with minimal changes were not flagged. So, tried Add an exclusion to Windows Security - Microsoft Support - that worked well.

Start --> Settings --> Privacy & security --> Virus & threat protection --> Manage settings --> Exclusions --> Add or remove exclusions.

shifting a blogger blog to Google Analytics 4

There was an email from Google, "All Universal Analytics services cease to function starting July 1, 2024"

Googling "use analytics with blogger", found this support article - and found that this blog was not automatically transferred from the earlier Universal Analytics to Google Analytics 4. Copy-pasted the "Measurement ID" as mentioned in that support article, finding the "Measurement ID" via this article. Now let's wait for a couple of days and see if it works.

Thursday, April 04, 2024

CPU-based stable diffusion

In my previous post about Stable Diffusion and how I could not generate 4k fulldome images with it, I had mentioned running out of RAM as the reason it failed. I thought perhaps if there is a way to run it on CPU instead of GPU, I could use a more recent faster machine - and perhaps use virtual memory and run for a longer time - to generate 4096 x 4096 fulldome images. But trying out rupeshs/fastsdcpu: Fast stable diffusion on CPU - found that even if I do an img2img using a 4K fulldome image as the input, the output was only 1024 x 1024.

There may be ways to run google colab or something like that, and generate 4K fulldome images - but that is something to be tested later. At first glance, it seems that even Dall-E is limited to 1024x1024, and people have to stitch generated images together to make bigger canvases, as mentioned in this reddit thread. And this thread gives a Stable Diffusion guide which uses upscaling - which has issues as mentioned in my previous post.

super-resolution experiments

I thought of doing some super-resolution experiments directly with ffmpeg or something similar for virtualdub or avisynth or avidemux, but this detailed post makes me pause - mostly not so much better than Lanczos, as they say - scale - How do the super resolution filters in FFmpeg work? - Video Production Stack Exchange

Tuesday, April 02, 2024

stable diffusion experiments

Now that I have access to a GPU, I tried out some AI-based image resizing, video upscaling and image generation, using stable diffusion as noted in a previous post. This post notes some of the pros and cons of the technology as it stands now, in April 2024.

1. There doesn't seem to be a way to incrementally correct images as with ChatGPT-generated code. We need to fine-tune our prompts if we need better results, and run the generation once again.

2. On this machine - 1050TX GPU, quite slow - stable diffusion with the webui takes around a minute to generate a 512x512 default image with an img2img promt and the default 20 steps.

3. Following this guide on fddb, my results with generation were not so great. Changing "little girl" to "businessman with briefcase" did not result in a briefcase in the 4-5 iterations I tried out. Additionally, scaling up the image showed that the skyscrapers were not realistic at all. Perhaps this can be fixed by generating in the higher quality instead of first generating in 512x512 and then scaling up - but I can't do that, since I run into 'CUDA out of memory' errors.. Edit - further experiments in this post seem to indicate 1024x1024 seems to be the upper limit for most models.

Example - part of the fddb example image, upscaled to 4k using Lanczos resizing,

and using R-ESRGAN-4x+, we see the cartoonish quality,

4. Trying to upscale a video which had a series of stills - something like a slideshow - resulted in lots of image flicker in the upscaled video. The reason for the flicker is some horrendous hallucination, close-ups from a couple of frames shown below.

Original -

Upscaled -

and the next frame has the shading which causes the flicker,

Decided to use Lanczos instead of AI in that case. But single image upscaling can give good results, especially if the image is a generic image and not an exact likeness of someone. In case of an exact likeness, some hallucination of features is seen. Example close up, something like spectacles is seen on the bridge of the nose, not present in the original.