Thursday, April 03, 2025

run unsigned binaries on MacOS

 I was not sure if unsigned binaries would run on MacOS, and whether one can override its behaviour. So, apparently MacOS 10.15 Sonoma onwards, unsigned binaries won't run by default, but we can specifically white-list any binary we want to run, and then it will run.

https://github.molgen.mpg.de/pages/bs/macOSnotes/mac/mac_procs_unsigned.html

https://discussions.apple.com/thread/254786001?sortBy=rank

Either via the GUI - 

* Try to open myFancyBinary in Finder. This will fail.

*Open System Preferences, choose the Security control panel, select the General tab.

*Look for the message: “myFancyBinary was blocked from opening because it is not from an identified developer.”

*Click the Open Anyway button to the right of the message.

or via the terminal - 

spctl --add /path/to/myBinary

And for getting the full path, and easy way is to drag and drop the binary from Finder into the Terminal window.


Posted by on No comments:

Stable Diffusion tests and benchmarks

I had tried out Stable Diffusion on Windows last year, now I wanted to try it out on Linux and if possible, using Docker. 

Unfortunately, the docker method failed. Searching for stable diffusion docker with web ui



talks about
(I had to run everything with sudo prefix.)

sudo docker compose --profile download up --build
roughly 12GB of data to be downloaded.

Unfortunately, after 2 hours of downloading, using the Automatic1111 profile,

sudo docker compose --profile auto up --build

Error response from daemon: could not select device driver "nvidia" with capabilities: [[compute utility]]

So, probably I need --profile auto-cpu.

But that gave an error,
File "/stable-diffusion-webui/webui.py", line 13, in <module>
etc
ImportError: cannot import name 'TypeIs' from 'typing_extensions' (/opt/conda/lib/python3.10/site-packages/typing_extensions.py)

Deleting the dir, cloning the repo and starting again - again downloading 12 GB, 

And again failed.
sudo docker compose --profile auto-cpu up --build

Mounted styles.csv
auto-cpu-1  | Mounted ui-config.json
auto-cpu-1  | mkdir: created directory '/data/config/auto/extensions'
auto-cpu-1  | Mounted extensions
auto-cpu-1  | Installing extension dependencies (if any)
auto-cpu-1  | Traceback (most recent call last):
auto-cpu-1  |   File "/stable-diffusion-webui/webui.py", line 13, in <module>
auto-cpu-1  |     initialize.imports()
auto-cpu-1  |   File "/stable-diffusion-webui/modules/initialize.py", line 23, in imports
auto-cpu-1  |     import gradio  # noqa: F401
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/gradio/__init__.py", line 3, in <module>
auto-cpu-1  |     import gradio.components as components
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/gradio/components/__init__.py", line 3, in <module>
auto-cpu-1  |     from gradio.components.bar_plot import BarPlot
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/gradio/components/bar_plot.py", line 7, in <module>
auto-cpu-1  |     import altair as alt
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/__init__.py", line 649, in <module>
auto-cpu-1  |     from altair.vegalite import *
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/vegalite/__init__.py", line 2, in <module>
auto-cpu-1  |     from .v5 import *
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/vegalite/v5/__init__.py", line 2, in <module>
auto-cpu-1  |     from altair.expr.core import datum
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/expr/__init__.py", line 11, in <module>
auto-cpu-1  |     from altair.expr.core import ConstExpression, FunctionExpression
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/expr/core.py", line 6, in <module>
auto-cpu-1  |     from altair.utils import SchemaBase
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/utils/__init__.py", line 14, in <module>
auto-cpu-1  |     from .plugin_registry import PluginRegistry
auto-cpu-1  |   File "/opt/conda/lib/python3.10/site-packages/altair/utils/plugin_registry.py", line 13, in <module>
auto-cpu-1  |     from typing_extensions import TypeIs
auto-cpu-1  | ImportError: cannot import name 'TypeIs' from 'typing_extensions' (/opt/conda/lib/python3.10/site-packages/typing_extensions.py)
auto-cpu-1 exited with code 1

Then I thought I would try chainner, https://chainner.app/download
gives a deb file, for ease of installation. It has a dependency manager also. 

But the dependency manager does not install stable diffusion! So, trying

(
current python version is 3.11.5 on this Linux mint based on ubuntu 24.04 - which may run into problems,
- we're supposed to run on python 3.10-venv
)

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacty of 3.94 GiB of which 31.50 MiB is free. Including non-PyTorch memory, this process has 3.73 GiB memory in use. Of the allocated memory 3.60 GiB is allocated by PyTorch, and 77.43 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF


Stable diffusion model failed to load

Can try lowvram command, since we have the NVidia 1060 -  

export COMMANDLINE_ARGS="--lowvram"
./webui.sh

Success. Does a 512x512 generate in ~ 1 minute, 4x ESRGAN upscale resize in ~30 sec.
this has a listing of how to make prompts.

My primary use case was for upscaling. I took a picture which I had clicked with the mobile camera in its "50 megapixel" mode, which shows quite a bit of "softening" when viewed at 100% - then cropped to 4096x4096, scaled down to 512x512 (1k images were causing out of VRAM errors) and then scaled up to 4096x4096. Interesting to see the results for a jpg show lots of ringing, while the results for a png seems to make things sharper. 

A portion of jpg input on the left, with output on the right, processed as 
scale down to 512 px, save as jpg, and then upscale to 4x - this shows ringing. Click on the image to view larger size.

A portion of jpg input on the left, with output on the right, processed as 
scale down to 512 px, save as png, and then upscale to 4096 - this shows an interesting pointillistic effect, and has made edges if the image sharper - clearly seen in the foreground trees.  Click on the image to view larger size.

I'll write a separate post on Easy Diffusion, since this post has already become quite long.
Posted by on No comments:

Tuesday, April 01, 2025

experiments with LLMs and docker

Tried out some "easy to use locally" Large Language Models (LLMs) with the motivation of being able to use them with our own documents. 

Looking for easy ways to install and run them, via Run Local LLMs on Hardware from $50 to $50,000 - We Test and Compare!

ollama list
ollama pull llama3.1:latest (~4 GB - the download went at nearly 4 MBps on the Airtel 40 Mbps connection, and around 6 MBps on Airtel 5G - 12 minutes)
ollama pull llama3.2:latest (~2 GB, and runs faster on low-end hardware)

(run as ollama run llama3.1:latest --verbose to get tokens/sec.
)

With 3.1, it was going at ~4 tokens per second on the desktop. CPUs, I think. With llama3.2, it was going at 23 tokens / sec - quite an acceptable speed. On the Windows laptop, running on CPU, llama3.2 at 9 tokens / sec.


Wanted to try the docker route - installed docker via https://docs.docker.com/engine/install/ubuntu/ by setting up docker's repository. All my docker commands needed to be prefaced by sudo, since I didn't go through the post-install steps.

Trying to run open-webui with the commandline given in the youtube video, 

sudo docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama 
has the steps.

sudo docker pull ghcr.io/open-webui/open-webui:main

For Nvidia GPU support, add --gpus all to the docker run command:
Single user mode - 

sudo docker run -d -p 3000:8080 --gpus all -e WEBUI_AUTH=False -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

That also failed. "Unable to load models". Then, found that we need to start ollama separately in another terminal window, from How to Install Ollama, Docker, and Open WebUI on Windows - YouTube
and we need a different set of parameters in the command line if ollama is already installed on our system. So, got the openwebui to work with two modifications:
ran from administrator cmd (on Windows)
without -d (so that I can see the errors if any - do not detach)
and must start ollama first in another window with
ollama run llama3.2

 docker run  -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

But connecting stable diffusion to open-webui seems to be complicated - 

So, will run Stable Diffusion (for image generation and upscaling) from its own web ui instead.


Posted by on No comments:
Subscribe to: Posts (Atom)