u/rainbyte

Well, you can get 27B working with less hardware.

A single GPU will be enough for 27B @ INT4, and with two GPUs INT8 at good speed, ie. +3k t/s prefill and +50t/s decode.

QFN will require more than that, eg. 2 GPUs and a chunk of RAM, to end up with 300 t/s prefill and 20 t/s decode.

QFN could be better in some cases, but 27B allows faster iteration on agentic workflows.

It might not be the bandwidth but the PCIe topology and atomics.

I noticed R9700s and ROCm have more strict PCIe requirements compared to CUDA.

Gain becomes smaller with newer PCIe versions.

Here I compared gen3 16x/1x with gen3 16x/4x, and the later makes tensor parallel possible, but going from gen3 16x/4x to gen5 8x/8x didn't make visible changes.

Also beware, it might be model dependent. I'm using qwen3.8-27b

Es el nuevo status quo, trabajar código generado por el usuario que ya hace la parte "fácil" pero necesita limpieza e implementar todo lo que el robot no pudo

does rx6600 even have matrix cores? maybe that's causing the slow down, because I think only 7000 series qnd higher have matmul

AUR recipes are maintained by community members, as they can be uploaded by anybody registered the quality varies a lot, and they can also be deleted.

For the specific case of llama.cpp I would suggest installing the package available at the official Arch repo or just compiling manually.

Compiling is the probably the best way, as it is basically 2 commands: git pull and cmake build

AMD R9700 is missing!

Other Intel combos are also missing support, so I guess AMD has been strategically including it for Radeon GPUs.

Edit: I had trouble with older Z490 and Z590, completely solved after switching to AM5 B650M and X870 mobos

At least it can be solved by switching mobo, which is cheaper than switching the GPUs with current numbers o.o

Here the 9070 is cheaper, but appears on 12Gb variants. I would try to stay on 16Gb and up if possible, given the current model sizes.

9070XT is like a R9700 with less vram and cheaper, so it also supports new ROCm stuff and FP8 quants.

I think it is a hardware limitation, because Intel cpu+chipset combi doesn't support PCIe atomic ops, which can be checked on dmesg logs

Most Intel chipset consumer boards will cause trouble, as they don't support PCIe Atomic ops (check with dmesg | grep Atomic) and their topology makes GPU-GPU communication more difficult.

Better look for AMD chipsets, probably AMD made sure they work with Radeon GPUs. Here I used B650M and X870 AM5-based chipsets, both fine. Not sure about older AM4 ones, investigate before jumping there.

NVIDIA doesn't have this problem because CUDA is older and was developed before Atomic ops were even there, but I guess it could be difficult to adapt ROCm, because AMD has other goals.

TLDR: avoid Intel chipsets, just prefer AMD chipset + PCI 8x/8x or server-oriented boards

EDIT: I found info here, on github, and googled a bit... you can also ask here before buying something

Ahora que la moda es el AI-slop parece que la gente se olvido del Human-slop, jajaja

Beware! R9700s are really nice, but they need to be paired with a compatible motherboard!

Even if some mobos support PCIe 8x/8x, they might be missing other requirements or have incompatible PCIe topology, so be careful with that.

Here I recently configured a 2xR9700 setup and made a mistake of using an Intel chipset mobo, which doesn't support the PCIe atomic ops required by R9700s to work together, so Llama.cpp was extremely slow and vLLM didn't even start.

I had to wait some days until I could get money for a better AMD chipset motherboard, and only then everything worked perfectly. Now it runs vLLM with ROCm 10 at good PP and TG levels.

Other people here mentioned 3090, which are also good, but they lack native FP8 support and have less vram, so they might not be ideal depending on the models you need/want to use.

Feel free to ask any question :)

A few electrons moving somewhere

Yup, I do this all the time, so I ended up adding a subagent with a faster model.

Have you tried turning off reasoning? I have been switching between off and low, with low as default in vLLM launch command. It works pretty well :)

I'm currently in the process of building a dual R9700 setup, and ROCm went pretty well after adding 7.xx repos.

Older versions of amdgpu and ROCm are not so good, as new ones gained so much performance and stability.

Beware, PCH attached PCIe ports could cause trouble on GPU communication! Here I'm waiting for a newer motherboard because of that.

edit: also make sure to have a recent kernel, 7.x series work better than 6.x ones for R9700

No sé los demás, pero yo no subo código que no entiendo o que no revisé antes, ni aunque me apuren. Parte del laburo es saber cuando decir "no".

Yo opino lo mismo. La persona delante del teclado es responsable de lo que pushea, y del otro lado debe haber alguien responsable de hacer PR review. No se puede andar subiendo ni aprobando código que ni se entiende que hace, y menos cuando toca areas fuera de scope.

Had some issues with podman compat, which required some changes (maybe I could do PR about that), then also noticed iommu was disabled on uefi so I went to enable it.

Finally it looked promising given that vLLM appeared, but in the end it seems I encountered a bug with ROCm, as it errored with "HIP failure operation cannot be performed".

I will investigate a bit more as soon as I'm again in front of the machine, as there are some issues reported with similar error message.

I'm testing this right now, let's hope it goes well, fingers crossed :)

Radeon GPUs became a good option, but I would add installing updated kernel (6.x vs 7.x), mesa (25 vs 26) and rocm versions, because the difference between old and new is night and day.

Of course, the message should be an improved version, not exactly the one included in this post