u/mtbMo

Hab hier auch eine Variante mit Garage s3 als backend - zickt manchmal , nicht ausgeschlossen dass Garage die Ursache ist

Take a look at johndpope turbo/planar llama fork, saves a lot of vram for ctx and may allow parallel sessions

I do use two m4000 combined to run 3b/9b models.
one P40 runs up to 35b class models. This inference machine is around 600w running all three GPUs parallel

Did you looked into GPUstack platform? Running this since v1 for my various backends. Llama.cpp cuda/rocm, as well for cloud provider upstream proxy

Für initiale batch processing Miete dir die GPU compute bei hetzner oder cloud provider of choice. Retrieval search etc kannst du dann local hosten. Je nach Anzahl docs wirst du mit local compute nicht weit kommen oder es ewig dauern.

Build myself two Ai Workstations based on AMD Rocm 2x 16Gb and cuda p40 24Gb.
Rocm node runs Nemotron-Lightning at 75-85t/s and 900pp/s - quite impressive for this legacy GPUs. To be fair, custom kernels based on arte-facts fork.

Yeah I know therefore I got my 80gb VRAM GPUs collected and run LLMs on them 😉

Yes you can ;)
But also didn’t looked into it, until i may receive fiber 🙃

Minimax M3 is my workhorse model for most of my agents through tokenplan 20€
Alibaba tokenplan for Qwen deepseek and Glm models 18€.
At scale, cloud provider will handle this more efficient than any homelab selfhost can do.

I measure all my pdu with Shelly plug-s and can exactly tell which machines consumed the juice. 39ct€/kwh 🙃 - sep 26

https://preview.redd.it/7vwqnoduq8vh1.jpeg?width=1119&format=pjpg&auto=webp&s=1b0f5c73f12d8f1e95eb5040a12a446438d54b99

My Ai Servers don’t run 24/7, nightly shutdown for 7h. Both servers Dell workstation Xeons are draining around 1kw from the wall under load, GPUs are gfx906/p40/m4000 and share 800W, remaining watts is storage disks hba, etc.

If I would by a reliable server and hardware, i would preferred host this in a data center collocation.
Moved all my servers to the basement, bc of noise and heat reasons.
Basement got enough air volume and open space to not rely on AC.

Hi there,
Running two Xeon based workstations, with various cards range from 8-24GB - NVIDIA Rocm. Engine choice is llama.cpp.

These custom amd/nvidia forks are maintained by 20$ Minimax tokenplan and Claude sub, provide a still decent performance and efficiency on these 7-8y old GPUs

Workloads are mixed, from openwebUi to coding agents based on Pi Agent.

These m4000 were shipped with the workstations I bought, so why not use them. Saved them from ewaste

Can confirm. I run two m4000 8GB GPUs, serving embedding and 3b ministral. Slow as hell, but for background tasks it’s fine and faster compared to cpu based compute

Combine this with zfs snapshots and file clones ;)

Hi, what inference engine do you run?
Im maintaining a custom fork with johndpope TurboQuant kv-compression- enables up to 3x context by minima tradeoff.

Mi50 Rules ;) got myself two 16GB cards. Runs my 35b class models with 256k context, with turbo3 arte-facts kernel compression. Up to 3x contex with max 20% overhead PPS

Dann macht die nächste HF Platform auf, gibt glaube schon 1-2 Alternativen

Wow, nice to hear. In the mean-time I am now maintaining two llama images with optimized mi50 gf906 kernels, these cards cranking 80-90tps on Nemotron-Lightning.

I am tempted to get another pair of GPUs, to bump to 64gb vram.

You can try johndpope llama turbo quant - running this on my p40 sm61

Is there still a supply of Mi50 32GB? May I ask what the stack cards did cost?

Checkout johndpope planar/turbo kv-cache compression fork. It’s amazing!

My setup ist dual mi50-16GB and single P40 - runs Gemma4 up to 70tps. Very happy for the ancient GPUs

Should checkout other planar/turbo/iso kv-Cache quant compressions. This enabled me to run up to 256k ctx on my GPUs