troubleshooting dot net server crash on AWS

First, checked if the AWS dashboard has any troubleshooting info for me. But "This user does not have permissions to view AWS Health events" - so I wouldn't get much help from it. Used my earlier post on investigating another Linux server crash as well as ChatGPT for going step by step.

ChatGPT suggested these,

dmesg -T | grep -i oom

journalctl -k | grep -i oom

which had no results. No out-of-memory events seen with those commands.

du -sh /var/log showed 4 GB of journal

journalctl --disk-usage

also showed nearly 4 GB of current and archived logs.

But the system had enough hard disk space left, 

df -h

showed around 30% remaining for the OS disk. 

Checking authentication log with

tail /var/log/auth.log -n4000 | more

found an entry

2026-01-25T14:38:58.186744+05:30 ip-10-0-0-73 systemd-logind[600]: New seat seat0.

But ChatGPT said that's normal, this is apparently done for each login.

Then, copy-pasting the lines just before the crash from syslog, using T13 since the crash was around 13:30

sudo cat  syslog | grep T13 > /home/theadminaccount/syslogsnippet.txt

ChatGPT immediately found the issue. 

ourtestapi.service: Scheduled restart job, restart counter is at 101203

and the reason the service was failing was

The application '/home/thepath/www/ourapi/ourtest.dll' does not exist.

Stopped the service, disabled it, and deleted it from /etc/systemd/system

sudo systemctl stop ourtestapi

sudo systemctl disable ourtestapi

sudo rm /etc/systemd/system/ourtestapi.service

Additionally, as ChatGPT suggested, set up some limits for journalctl's logging by editing

/etc/systemd/journald.conf

and uncommenting these lines and adding values for them - under

[Journal]

SystemMaxUse=900M

SystemKeepFree=1G

RuntimeMaxUse=100M

MaxRetentionSec=30day

and restarted,

sudo systemctl restart systemd-journald

Also vacuumed (removed archived logs) with

sudo journalctl --vacuum-size=3000M

(0 bytes removed - archives were smaller than that)

sudo journalctl --vacuum-size=300M

(500M of archives were removed.)

see who has signed an app on MacOS

Via https://www.youtube.com/watch?v=1UDUtO7LY0k

 Apple Menu > About this Mac > More info button > (Search for System Report on left-hand panel)

In System Report, click the System Report button, then Software > Applications on left-hand panel.

Scroll down to the app in question.

Our self-signed OpenSpace builds show up as "Unknown". Sheepit Mac client also shows "Unknown", so there is a chance that I can edit the manifest of the app to run a script on double-click in order to allow the Mac build to run with a double-click on the app file.

allowing a non-root user to restart a service on Ubuntu Linux server

As described by ChatGPT, the preferred method was to create sudoers rules for the relevant service(s).

sudo visudo -f /etc/sudoers.d/ourapi

and inside that file, write

deployuser ALL=NOPASSWD: /usr/bin/systemctl restart ourapi.service, \

                         /usr/bin/systemctl start ourapi.service, \

                         /usr/bin/systemctl stop ourapi.service, \

                         /usr/bin/systemctl status ourapi.service

where ourapi.service is the service and deployuser is the non-root user.

Here, we double-checked the location of systemctl using 

which systemctl

After doing this, on logging on as deployuser, we can do things like

sudo systemctl start ourapi.service

where it will prompt for deployuser's password and then carry out the command.

But a caveat - we need to type exactly the same command as mentioned in the custom sudoers file - even equivalent commands like

sudo systemctl start ourapi

will not work unless we add that to the sudoers file also.

some quirks of my KVM setup and Apple keyboard layout

Currently using this product, 

HDMI KVM Switch 4 Port - 

there were some quirks which I had to work around. 

1. The Dell desktop running Linux Mint needed the USB keyboard to be mapped to it while booting, or else it would boot into Windows instead, even though the default boot sequence was set to Linux. Probably because the "fail-safe" boot option is different, when it beeps to indicate "No keyboard detected."
2. The KVM switch defaults to number 4 on power up, not number 1. So, the Dell desktop needs to be connected to 4 and not 1, for booting up without having to hit the KVM switch.
3. The Mac mini can boot OK without having the KVM mapped to it, but if we leave it on and not logged in, the login screen blanks and we can't get the screen to come back to life with the keyboard or mouse - we need to short-press the power button.
4. The Mac mini default keyboard mapping was showing the Indian rupee symbol ₹ instead of the backtick symbol ` for the key to the left of 1 (perhaps after the upgrade to MacOS Tahoe). Since I need ` for markdown on github etc, changed the keyboard layout to "US" from "ABC India" which was the default. For this, following this,
   chose Apple menu > System Settings, then searched for Keyboard in the sidebar. There, Input sources > Edit button, the small + button at the bottom left to add Input sources. Quite easy to miss.

adding a 'slow-send' functionality for Google Apps mailing list

I've updated the github repo Google-Apps-Script-mailing-list with a script file which does 'slow-send' for large mailing lists which can be sent over several days. 

Slow-send-one-by-one.gs

Features - 

• time-based trigger, programmatically created and removed
• sends emails to each recipient individually, ramping up from sending
  24 emails per day (once an hour)
  to 96 (once every 15 minutes)
  to 144 (once every 10 minutes)
  to 288 emails per day (once every 5 minutes). This could be increased further, too.
• uses GmailApp instead of MailApp, so we can have features like name (of sender), alias, reply-to, etc.

sheepit renderfarm statistics for the M4 Mac Mini

Compared to the earlier stats, the CPU speed is nearly double, but lack of NVidia GPU prevents many projects from rendering. Copy-pasting from the session information page (My account > Connected machine > (clicking on the machine) - 

OS	macm1
CPU	Apple M4 x 10
RAM allowed	11.0 GB
RAM available	16.8 GB
Max render time per frame
Power CPU	208 %
Scheduler	default

running OpenSpace on Apple Silicon and modern Intel Macs

As written in the documentation, OpenSpace uses OpenGL double precision floating point operations extensively, and Apple Silicon does not have hardware support for it. Checking if it will downgrade gracefully to software rendering, found that OpenSpace would just crash, with Claude.ai telling me that the crash report 

Crashed Thread:        0  CrBrowserMain  Dispatch queue: com.apple.main-thread

Thread 0 Crashed:: CrBrowserMain Dispatch queue: com.apple.main-thread

0   libsystem_kernel.dylib                   0x190856388 __pthread_kill + 8

1   libsystem_pthread.dylib                  0x19088f848 pthread_kill + 296

2   libsystem_c.dylib                        0x1907989e4 abort + 124

3   libGLProgrammability.dylib               0x1f0ea63e4 glpPrimitiveTypeGetScalarType + 136

4   libGLProgrammability.dylib               0x1f0e8af20 glpLLVMLoadVector + 120

5   libGLProgrammability.dylib               0x1f0e7c8f0 glpLLVMCGLValue + 1160

6   libGLProgrammability.dylib               0x1f0e771b0 glpLLVMCGAssign + 112

7   libGLProgrammability.dylib               0x1f0e7788c glpLLVMCGCommaExpr + 268

8   libGLProgrammability.dylib               0x1f0e7788c glpLLVMCGCommaExpr + 268

9   libGLProgrammability.dylib               0x1f0e771b0 glpLLVMCGAssign + 112

10  libGLProgrammability.dylib               0x1f0e7788c glpLLVMCGCommaExpr + 268

11  libGLProgrammability.dylib               0x1f0e7788c glpLLVMCGCommaExpr + 268

12  libGLProgrammability.dylib               0x1f0e771b0 glpLLVMCGAssign + 112

13  libGLProgrammability.dylib               0x1f0e7788c glpLLVMCGCommaExpr + 268

14  libGLProgrammability.dylib               0x1f0e7b5f8 glpLLVMCGBlock + 516

15  libGLProgrammability.dylib               0x1f0e722c8 glpLLVMCGNode + 632

16  libGLProgrammability.dylib               0x1f0e7b234 glpLLVMCGFunctionDefinition + 2888

17  libGLProgrammability.dylib               0x1f0e70f30 glpLLVMCGTopLevel + 1444

18  libGLProgrammability.dylib               0x1f0e985f8 glpLinkProgram + 10092

19  libGLProgrammability.dylib               0x1f0eb1ea0 ShLink + 208

20  GLEngine                                 0x1f104a3b8 gleLinkProgram + 1140

21  GLEngine                                 0x1f0fc6350 glLinkProgramARB_Exec + 180

22  OpenSpace                                0x1021f8aa4 glbinding::BasicCallHelper<void, unsigned int>::call(glbinding::Function<void, unsigned int> const*, unsigned int&&) + 44

23  OpenSpace                                0x1021f87d0 glbinding::Function<void, unsigned int>::call(unsigned int&) const + 184

24  OpenSpace                                0x1021f3a24 glbinding::Function<void, unsigned int>::operator()(unsigned int&) const + 32

25  OpenSpace                                0x1023470c8 gl::glLinkProgram(unsigned int) + 32

26  OpenSpace                                0x101e5dc2c ghoul::opengl::ProgramObject::linkProgramObject() + 44

27  OpenSpace                                0x101e5e668 ghoul::opengl::ProgramObject::Build(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::__fs::filesystem::path const&, std::__1::__fs::filesystem::path const&, ghoul::Dictionary const&) + 1456

28  OpenSpace                                0x100a5dbac openspace::RenderEngine::buildRenderProgram(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::__fs::filesystem::path const&, std::__1::__fs::filesystem::path const&, ghoul::Dictionary) + 316

29  OpenSpace                                0x1016ce084 openspace::globebrowsing::RenderableGlobe::recompileShaders() + 6156

30  OpenSpace                                0x1016cc82c openspace::globebrowsing::RenderableGlobe::initializeGL() + 208

31  OpenSpace                                0x100b47b88 openspace::SceneGraphNode::initializeGL() + 584

32  OpenSpace                                0x100ae4bb0 openspace::Scene::update(openspace::UpdateData const&) + 132

33  OpenSpace                                0x100a5a6d0 openspace::RenderEngine::updateScene() + 196

34  OpenSpace                                0x1007eb428 openspace::OpenSpaceEngine::loadAssets() + 972

35  OpenSpace                                0x1007ed234 openspace::OpenSpaceEngine::preSynchronization() + 156

36  OpenSpace                                0x1004303ac (anonymous namespace)::mainPreSyncFunc() + 80

37  OpenSpace                                0x100ce8b90 sgct::Engine::exec() + 288

38  OpenSpace                                0x10042d170 main + 14684

39  dyld                                     0x1904eeb98 start + 6076

indicates that 

The crash is happening in Apple's OpenGL shader compiler (libGLProgrammability.dylib) when OpenSpace tries to compile shaders for the globe rendering. Look at the stack:

Frame 26-29: OpenSpace is compiling shaders for RenderableGlobe

Frame 19-21: Apple's OpenGL implementation tries to link the shader program

Frame 3: The shader compiler encounters something it can't handle and aborts 

Now, this could possibly be due to some filesystem issues, restrictions due to running OpenSpace from command-line from a sub-folder of my home folder instead of Applications folder, but I got a similar crash when running OpenSpace inside UTM on Ubuntu 24.04 (using the daily build deb file yesterday) if I enable hardware acceleration. 

Unfortunately, mesa on MacOS doesn't seem to enable software rendering - 
brew install mesa

# Set environment variables to use Mesa's software renderer

export LIBGL_ALWAYS_SOFTWARE=1

export GALLIUM_DRIVER=llvmpipe

# Run OpenSpace

./OpenSpace.app/Contents/MacOS/OpenSpace 

But the same crash is observed. Claude says,

Unfortunately, Mesa on macOS is limited. The Homebrew Mesa package primarily provides: OSMesa (off-screen rendering) and some utilities

But it won't override Apple's OpenGL framework the way it does on Linux. macOS applications are hardcoded to use /System/Library/Frameworks/OpenGL.framework. 

Then, tried disabling hardware acceleration and running the same OpenSpace deb file on Ubuntu 24.04, and that works. Around 8 fps, and the frame-rate does not seem to improve much beyond 20 fps or so even if we use something like onlyEarth.profile or loading individual assets to the empty profile, etc. But this could be still useful for offline work, validating single-precision builds and so on.

And so now to single-precision builds - I'll try to build for Mac Apple Silicon and see what happens.

Also, maybe I can try building for Intel Macs with these patches - with XCode for MacOS26, many of the incompatibilities and unsupported implementations are now ironed out, so we need far fewer patches. And github now has MacOS26 public runners.

Edit: 11 Jan 2026 - Finally OpenSpace for MacOS - Apple silicon - is up and running. Lots of shader patching, especially a tricky one here - Mac build - crashes when shader is called · Issue #75 · hn-88/OpenSpace-AppImage

tuning lossless ffmpeg command-line with Claude

The ffmpeg command

ffmpeg -y -r 30 -i in_%05d.jpg -i ../NormPeak-6dBENG.wav -c:v hevc_nvenc -preset lossless -global_quality 18 -pix_fmt nv12 -c:a aac -b:a 192k ~/Downloads/4kframes.mp4

gives the following warnings - 

[aist#1:0/pcm_s16le @ 0x59f7e96fbfc0] Guessed Channel Layout: stereo
....

[in#0/image2 @ 0x59f7e96f4e40] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)

[swscaler @ 0x59f7e97ae500] deprecated pixel format used, make sure you did set range correctly

    Last message repeated 3 times

[hevc_nvenc @ 0x59f7e9701880] The selected preset is deprecated. Use p1 to p7 + -tune or fast/medium/slow.

[hevc_nvenc @ 0x59f7e9701880] Using global_quality with nvenc is deprecated. Use qp instead. 

Claude's suggestion to fix - 

ffmpeg -y -r 30 -i in_%05d.jpg -thread_queue_size 512 -i ../NormPeak-6dBENG.wav -c:v hevc_nvenc -preset p7 -tune lossless -cq 0 -pix_fmt yuv420p -c:a aac -b:a 192k ~/Downloads/4kframes.mp4