u/while-1-fork

Reading doesn't cause any significant wear. Writing does, specially many tiny writes.

I don't think most inference engines write to the SSD but if you for example configure a large amount of swap and hit it hard 24x7, you can kill a SSD in a couple of weeks (I mean real hard like swapping 64GB and doing reads and writes all over it).

Even if not directly writing, running tight on ram will cause some swapping so what I would do is setting up zram.

Being so tiny and asm only you could potentially do some insane things like running it directly from a bootloader without an os of even embedding it directly into a coreboot/libreboot bios.

Looking at the notes about the 4 bit quants for more speed, you may want to look a bit into an old technique SWAR (SIMD with a register) as I think AVX does not support 4 bits directly that may allow you to eek out a bit more prefill vs regular unpacking to 8 or more bits (generation will be bandwidth limited by bandwidth ).

Be sure to get a debug one , or to modify one. Only one of the ends has power connected and the data pairs should be crossed over.

And you would need to write a driver for it emmulating a network, ideally replacing the driver that presents it as a tty instead of building on top to reduce latency and improve bandwidth.

I'd love to hear how it goes. I am not in need of a bc-250 but I may suggest it to friends looking to get into this.

I'd be interested in knowing if bc-250 supports usb3 debug capabilities as I believe that they could be hacked into a faster data transfer thing with crossover debug cable. Could you run this to see if it is enabled?

ls /sys/bus/pci/devices/*/dbc

If you can upgrade the system ram to 16GB a heavily quantized qwen 35B A3B may do it with ncmoe running the experts on cpu and the fixed part on gpu (the mac mini won't because this relies on the gpu vram). With 32GB it would work way better. It would still run on 8GB using mmap but likely unusably slow.

Other models like ling 3 tiny or lfm 2.5 are not smart enough to do coding on an harness but may be good to answer some coding questions and provide snippets of code.

One model that is small and works quite decently in a test I did this week is the qwen 3.5 9B MiMo 2.6 distillation. However it is a dense model and cpu offloading it is gonna hurt. I did run the IQ4 on a 8GB gpu and it does 2k prefill and 50 tgen maxing the vram at 134K context with q5_1 KV cache. I'd put a bit over Haiku 4.5 in smarts and in this setup it is fast enough and from what I have seen it is crazy persistent, checks its own work and retries until it thinks it is ok. On yours it would be way slower but maybe still usable. Specially if you try -ncffn which runs the MLPs on CPU keeping attention on the GPU, I have not tried that very recent flag so I don't know how much it helps.

I am running that one on a 64GB gpu and it seems equivalent to unsloth in intelligence but the size difference means 240k context , ub=2048 for faster pp and only needing to run the mmproj on cpu which for me is over 2x as fast as unsloth with 150K context and some cpu offloading with ncmoe.

Batch 4096 won't increase vram but I think it does increase system ram.

I am not using MTP, waiting for it to be merged on mainline but I will need to reduce ub or context, likely a bit of both.

If you want f16 KV cache, double my KV cache numbers to get an idea. 72GB won't be enough for the unsloth one but as I said not all IQ4 quants are the same, they all use a mix of tensor sizes and roughly slap a label the one doing the quant thinks fits it well. One problem with this one is that a good chunk of the model is not compatible with k or i quants.

Not all IQ4 are the same.

I started with the unsloth one and you won't get full context at 64GB without offloading unless future llama.cpp improvements find some big vram savings or unsloth manages to do a smaller quant. It is not short by a ton as this model KV cache is small.

I can fit it fully on 64GB with -ub 2048 too but with only 15K of context with q8_0 KV cache.

262K context only costs like 3.8GB in KV cache but there is also the compute buffer that scales with both -ub and -c which is 548MB when using ub=16 but that performs terrible in prefill. With default ub = 512 the compute buffer grows to 1.8GB. With ub=2048 which helps prefill a lot, the compute buffer needs 7.6GB.

After messing with UD_IQ4_XS I started looking for slightly smaller quants and ended up with Qwen3.8-Flash-Next-APEX-I-Compact . It averages 3.85BPW and I can fit 240K context with Q8_0 KV and -ub=2048 on 64GB. I could scale ub back a bit (does not need to be power of 2) and fit 262K or even be able to extend with yarn if I scaled ub back to defaults.

I used a single GPU and sm=None, multiple ones have some extra overhead.

I am sure that there are others Q4-ish that would fit, the ngrams table makes difficult to tell from the file size alone.

That is not really true. You can distill the pretraining too.

If they distill from logits as they should given than they share tokenizer and architecture training goes way faster. When you distill from text you are training one output per token. When you distill from logits you train the whole vocabulary (248K outputs per token), learning directly all alternatives plus what is unlikely. When you do that some extra knowledge that isn't in the text generated by the teacher gets transferred too, it isn't only quicker convergence.

Also they can not start from a random initialization but from pruned larger models.

They may even do an initial intermediate layer matching or attention score based distillation as initial steps if not starting from a pruned model as those are cheap and quick. Though effectiveness is limited unless they share an intermediate dimension and on their own don't result in a coherent model but it would if they do some logits KLD on top that would go even faster as the model is already close to where it needs to be.

Not saying that distillation is supercheap or fast. It will still be expensive but less than from scratch training. It isn't even easy to do it well starting from a pretrained model.

Not exactly a thing to try but could you ask Valve about the expansion port pinout? We know that it has pcie 1x and MIPI but do they share the same pins or both could be potentially be used at the same time? Are there any other interfaces in there? What about power pins?

I wouldn't be surprised to hear that even more is possible. Tried on one of mine and it is stable at ndiv 70 which in theory is 1935MB/s but in practice in OpenCL-Benchmark I am getting 1758MB/s so my card has worse memory. I guess you got ndiv 76 like in the repo?

But looking at the results in the 170tune repo they discard (Claude does) most timings as a lever because tightening the timings does not increase bandwidth or token generation in any significant way but that shows that Claude does not understand overclocking!

In practice token gen or bandwidth being timing insensitive shows that you can probably relax the stock timings and run a higher memory clock and that it should translate into higher bandwidth and tgen. I still have to experiment with that but I plan soon-ish.

However what the 170Hx is more starved of is compute. I managed to get the tensor cores to 203TF tensor compute from the stock 183 using 170tune try +190 1600 (higher either scores lower or is not fully stable) but undervolting limits the overclocking and we could likely go much higher if we could push the frequency higher without reducing the volts. I likely won't do in a good while, at least some months until seeing if some software voltage unlock happens or if more SMs are unlocked and power becomes the constraint but doing and old school voltage mod with a potentiometer on the voltage regulator feedback has crossed my mind.

I wonder about how the PTQ1_0 would do as that one could do decent amount of context even on 8GB cards.

Plain llama.cpp built with (there is also a 5060 but currently it only runs and embedding model in another instance):

cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="80;120a" -DGGML_LTO=ON -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON -DCMAKE_C_FLAGS="-march=native -mtune=native -O3 -flto=auto -mprefer-vector-width=256" -DCMAKE_CXX_FLAGS="-march=native -mtune=native -O3 -flto=auto -mprefer-vector-width=256"

Environment="CUDA_SCALE_LAUNCH_QUEUES=4x"

c = 262144

n-gpu-layers = 999

cache-ram = 0

fa = on

sm = none

ts = 1,0

lv = 4

fit = off

models-max = 1

jinja = true

ub = 4096

b = 4096

mmproj = ./models/mmproj-Qwen3.8-27B-BF16.gguf

image-min-tokens = 1024

image-max-tokens = 8192

spec-type = draft-mtp

spec-draft-p-min = 0.5

spec-draft-n-max = 3

The IQ4 seems to be a bit slower than Unsloth IQ4-XS even if it is smaller. I tried on an unlocked 170Hx but looking at the graphs it is a bit slower on a 3090 too.

I have been using your 3.6 35B A3B since it came out (on a 3090 and even on my 170HX it is still my first go to due to speed) and that one has been awesome so while this one does not seem to be for my setup I am looking forward to future ones.

Any plans of doing this for 3.8 flash Next?

A second verifier LLM would work but I was thinking that just when a certain threshold is reached we can inject text directly in the same model context so it reconsiders it(maybe in a thinking block). I am not sure if it would correct course, double down or even erroneously correct course on false positives. But I'd hope it would work to some extent.

Probably terrible but I think that what could work is detecting and either surfacing it to the user or maybe even injecting into the model context something on the lines but wait, I am not sure this " whatever was likely hallucinated" is correct, let me double check.

And quite clearly your online approaches work, the cache hits are good and don't seem to cost much in terms of cpu even on a smartphone cpu that is already bogged down doing inference.

What I was asking was more on the line of is there a way to use your sofware on desktop but with the ability to pass any flags we want to the underlying llama.cpp and would you consider a mode in which the same logic not only prefetches from disk but moves the experts to the gpu? (not sure if the last one is even doable without forking llama.cpp). The last one would also benefit some smartphones I believe, not for token generation but for prefill.

This one works great on smartphones : https://github.com/Helldez/BigMoeOnEdge

I have not tried it in desktop but it is also supposed to work. The mechanism seems elegant because it is not a fork but hooks into llama.cpp from the outside. In smartphones it doesn't support GPUs and I have only messed around with the GUI so no idea if in desktop it could use GPU and the full feature set of llama.cpp. Maybe u/dai_app can tell us.

The status of speculative decoding on a second card is similar to last time I tried:

• MTP: ignores --speculative device

• DFLASH: tries to use second device but crashes due to not being able of creating the output tensor (not oom). Maybe due to the different generation of cards.

• Eagle3: Works but it is 25% slower on average than MTP on a single card and less than half speed peak while also hitting prefill 20% harder.

At least with 27B gaining much more with speculative decoding in general, it gains a bit with Eagle 3 so it is not a net massive loss like it is with 35B A3B. But I don't see much reason to use it if I can squeeze MTP on a single card given the noticeable performance difference.

He implies that there is one to wait for and if it is for those without the hardware I'd imagine that if it isn't 35B-A3B it may be trying to be slightly better in some way.

I can imagine 3 ways: activating a few parameters more (a very cool thing they could do is a model trained to work well with changing expert counts so we can set it to whatever in a range), being a bit smaller than 35B, or an architectural change to attention to have an even smaller KV cache.

I love 35B A3B but it is slightly annoying that you can't have everything on a single 3090: full context length, mmproj, mtp, a large -ub of 2048 or more and 4 bit quant. So you need to compromise on some, have a 5090 or multiple cards. If they shaved a few parameters and made attention a bit more efficient we could have it all for a single card speed demon. So hope that they are doing that. Activating more parameters would make smarter but slower, for smart and slow we have the 27B, although admitedly a 35B-A5B would still fit even on 6GB gpus with cpumoe I believe.

You can always disable thinking, or at the very least set a reasoning budget to a small value or 0 but I wouldn't be surprised to hear that it lobotomizes it into being unusable.

Have you tried LFM 2.5 8B A1B? I have been impressed with it, it is slightly worse in the benchmarks but not by a huge amount and is is seriously fast. While the ling tiny seems quite slow in the benchs in the PR. I may try myself some time as LFM has half the context length and I am using LFM for deep research which would likely benefit from extra context.

I have a 3090+5060 and so far the only real speed preserving ways to share the load has been running the mmproj, the display and an embedding model I use for rag on the 5060. Every other thing I have tried does cause some slow down.

My original idea was using the 5060 for speculative decoding but you can't move MTP to it and so far other methods that do honor the speculative device settings have caused slow downs rather than speed ups.

Granted that all of the spec decode experimentation has been on the 35B A3B and I havent tried in over a month.

This weekend I did setup router mode to have both 3.6 35B A3B and the 3.8 27B which benefits way more of speculative decoding (about doubles going from almost 30 to 56 vs the 35B that goes from 170 t/s to 210t/s in my setup). The 27B may be more friendly to methods other than MTP and to moving them to the 5060 but I didn't try yet. Maybe later today or tomorrow.

Yes you can run 4 bit if you use -ncmoe to run some experts on CPU. The performance hit is not as bad as layer offloading. You can already try it with 3.6 exact same architecture, the only difference is training.