u/mr_zerolith

Did the approve any alternatives?

Wrong technology to support customers with, most big corpos tried it and rolled back, unlikely you'll have different results

Same case here, i couldn't squeak a performance gain out of it.

We replaced rented AI with Deepseek v4 flash at my shop when it came out. It was quite expensive but we are happy with it and haven't thought about what frontier labs are up to since.

Typical; layers are different sizes and llama.cpp can't subdivide layers so that last few gb or so is always going to be a crapshoot

Hey thanks for this tip, i run a server with 2 slots available and am not using the 48gb RAM for anything so i think we are about to have a good time.

This is an AI generated explanation but it's generally correct:
https://github.com/ggml-org/llama.cpp/discussions/20574

5090 doesn't need to offload if you run Q6, which is a fine compromise for speed purposes with a single card setup.

AMD has a 32gb GDDR6 card and it's still going for about $1500.
Your average person could easily save over the course of months to buy one, and have a pretty good starter setup!

Not unusual for a number of strange things to happen when you're so close to the limit. llama.cpp will typically attempt to allocate a little more memory on the first few prompts.

Ended up buying a RTX PRO 5000 for a total of 176gb to run deepseek v4 flash!

oh, i used to run inference on a PCIE 3.0 machine with a 10th gen intel ( almost passe at this point ) and 5090.
Moving up to a zen5 + ddr5 setup only gained me 1-2 tokens/sec.
For a single GPU setup it's amazing how much you can cheap out on everything but the GPU.

I guess we're finding out how many people didn't read the terms and conditions now, huh?

485b params.. so you really need 256gb vram to run a midsize Q4

"engram n-gram lookups"? Hopefully that means that some of the model sits on the disk..

It's okay, you're up next. I felt like this with a 128gb vram setup for ~5 months

I paid $11.5k for it because i bought the low and added a RTX PRO 5000 recently.
Pretty nice, it generates code with Deepseek V4 Flash at about 125 token/sec average.

Faster than most commercial endpoints and worth every penny :)

I've seen some demos posted of this and been wowed.
https://miaai-lab.github.io/DeepSeek-v4-1-Flash-100-HTML-Files/021-liquid-metal-ui.html#c-buttons

https://preview.redd.it/ivkhad271doh1.png?width=1014&format=png&auto=webp&s=e101799ff2e38d70406c4ff8eaed2221b5a35ae2

Too bad they didn't read the terms of service

Yep that's where i got the idea from.
Enjoy!

It's completely useless at this point.
llama.cpp has a great web based and terminal based chat engine inside.

I like LMStudio still because it offers fast tuning via the GUI. I usually do some of the tuning in LMStudio then run in llama.cpp. :)

Completely normal for most models.

In llama.cpp, i add: --draft-p-min 0.6

This lowers the penalty for non-code output & and your hit rate will go way up because it will terminate speculation early if it's not close enough..

If there is a vllm equivalent, this would allow you to increase the num_speculative_tokens, and get away with it :)

Yep.. i picked Deepseek V4 Flash because it thinks really quick on the default high setting.

Even if the token output is slower, we get there quicker.
Also once you slap DSpark on it, you're outputting code at 125 tok/sec average easy on 5090-class hardware. When we combine this with quick/short thinking, it's very snappy.

I haven't liked Qwens for a while because they overthink so much to get good results.
It is cool what they can achieve in a ~72gb VRAM setup though.

what kind of boost are you seeing with the existing mtp?

https://preview.redd.it/2460im54jcnh1.jpeg?width=1059&format=pjpg&auto=webp&s=cc7998fe5ab74c7c706e1b2e95c80b8a2f60f6c2

My solution..

I basically don't trust any other source at this point.