u/Flashy_Management962

It may, but they wouldn't include deepresearch Q&A if it wasn't also the scope of the model

Deep Research q&a is basically agentic rag no?

You could try using exllamav3. Its quantization is miles ahead compared to llama cpp. The q4 is almost the q5km of llama cpp qualitywise

which quant are you using?

Oh boy, hab dasselbe über staatliche Kulturförderung gehört. Da wurde dann der Satz gedroppt "Wenn es sich nicht selbst erhalten kann, dann ists quatsch."

it's definitely very intelligent to do so! Especially with the new -sm tensor / -sm graph (for ik). My speeds are pretty good as well

Use bf16 cache for k and v for qwen models, do not use --fit-target on dense models, use -sm tensor

Those things are SO insanely different. This is like saying "I do not trust my car mechanic because he couldn't tell me the first 5 digits of pi"

2x rtx 3060 with 24gb vram total

thank you for your work btw! One little question though, is it normal that I get with your iq4nl gemma 26b this perplexity "Final estimate: PPL over 576 chunks for n_ctx=512 = 26296.2393 +/- 532.75059" - with the bartowski i get around 200.

I love the speed but it takes SO insanely much more vram with it, I can't run it on dual rtx 3060 with 24 gb total

I currently use qwen 3.5 4b on my shitty laptop as an agent, if this is faster/better I'm sold

yes, it does. I use the 27b for coding on a daily. fiddle around with those flags and do not forget to add --jinja: 

-sm graph -amb 64 -sas and depending on the pcie speed, grt can help improving speeds

you should get way faster speeds than that. i get around 750 t/s pp and 22-24 ts tg at ~50k with 2x rtx 3060 12gb. You should check out ik llama cpp

never ever does qwen coder 30b outperform 80b in realworld tasks

Ich weiß nicht, was du mit normalem meinst, aber ja

Du nimmst 500ml milch, 30g schoko whey, 10-15g dunkle schokolade und 35g Maisstärke - bester Proteinpudding und 100x günstiger

10 Jahre Training, 29 Jahre alt, alltime natty und wiege 118kg auf 1,83 Kraftwerte: 270,5kg Beuge, 185kg Bank und 270 heben.

id love to see qwen long l1.5 on this benchmark, it also claims to reach gemini pro 2.5 performance while being 30b 3a

https://huggingface.co/tencent/HY-MT1.5-1.8B

Dieses komische um jeden Preis sich selbst zerstören und man nur dadurch wächst ist totaler humbug. Viel mehr auf den Körper hören und dadurch rausbekommen, wie viel man verträgt. (Von einem Menschen, der nur einen Satz jeweils Beugt und Hebt in der Woche)

don't, use dry sampler instead. Repeat penalty really decreases tok/s