u/bennmann

This is an interesting comparison. I am on a 9070 xt + ddr4, at fresh 0 ctx I get 700pp and ~22tps sustained. 350pp over 100k ctx later though.

I guess there could be some long horizon utility in this too. Provable tool chains via first principle execution or something like that, almost like a virtual execution protection. But I'm not smart enough to formalize that. So your "just for the fun of it" is way better for me too lol, good comment

Whoever lies first wins propaganda games.

Whoever releases open datasets + provable modals wins egalitarian games.

Don't play propaganda games.

Compaction resilient. Compaction is almost a solved task for deepseek, glm, and Qwen, Gemma does not as gracefully compact context and continue tasks today.

Output/ unused inventory will have a market price on eBay too. Clever acquisitions lead to mean time between failure calcs, excess GPUs in backup MTBF inventory will also be sold over time by every player that is smart enough to do so. There will always be a trickle through back market channels of enterprise hardware.

Depends on how many times openAI buys up so much of the wafer capacity like they did. I believe they can do it for multiple years, precisely because it makes local expensive.

I have a 9070 xt and 6900 xt, just about to play around with rocm on Linux. Expecting a bumpy road for mixed gfx arch. You know or read of anyone with similar setup?

Costco still let's you call the food court and pre-order a pizza by phone in 2026. Not bad too.

Welcome to Costco, I love you.

I never learned these rules for this specific punctuation in high school, though these symbols enrage me. Native English speaker here. What arbitrary (seemingly) style choices.

With n-grams....

Do these have wifi 7 built in? Might be able to RPC 3x of these for a relatively cheap Qwen flash next hub over fast wifi

Reseat all your cables, reseat the card, clean out all dust, watch your temperature transients with watch -n0.2 'nvidia-smi', consider a repaste of the card itself. As others say I am sus of your power supply (it's weak)

Try rocm-smi --set-fan 99 I forget what values it takes though so read the man/--help docs for the flag too

Thunderbolt 4 hubs exist and can act as switches, sort of. I am unaware of the exact constraints.

You can find a 5 port USB4 hub for $150 on Amazon.

Might be the lowest friction solution.

anyone have the back of envelope math on how many GB the KV takes at full ctx?

I think it was pr 29030 that relates to an idea I hope gets implemented, but I don't see a set of flags that does this specific thing. It would speed up PP up to 50% for those of that barely cannot fit the n-grams.

Will you please work on a PLE tensor row split method that allows -lzm on but with some of the rows (not all) in ram?

Mistral-vibe harness has "auto-compaction" and 130k is fine.

Will run summary/compaction at my user configured 110,000 tokens and keep working. Not super efficient but finishes work.

Maybe hermes has that setting somewhere named different? Also I use vanilla llamacpp so maybe server harness interaction might fail for strata.

couldn't one use clever use of sleep/wait logic to simulate slower SSDs? EDIT: also you could make a fork of llama.cpp implement this, and make an Issue on the official repo, the community would be more likely to help reproduce in an issue on GH that way.

i was unable to get your ot flags to work from powershell windows 11 tag 0.6.0 testing today. but could be user error. also i only have 64GB ddr4 and mlock kept OOMing :(

Local llms like Qwen benefit more.

I gotta try this in Vulkan, maybe cuda specific bug....

Not gonna lie, even if you don't find a 3-5x pp increase here or elsewhere, that v620 8x is my dream upgrade to play around with.

What's your pcie connection speeds?

What are your llama.cpp flags? May still be low hanging fruit there.... Ubatch should probably be 512 and no higher, "-fitt 768" might help

With one small 60 step RPC process, you and your friends can run DeepSeek! /Sarcasm

Might be amazing for 4-8 friends and Qwen-flash-next though

Deepseek  4.0 vision exp iq2_xss might be worth trying

I have a 9070 xt and 64GB 3200mhz ddr4, please do make your changes public and remember me :)