u/terminoid_

looks nice, good job!

i'm not sure which is better, i've only tried lora so far. the lowest learning rate i've found to be usable so far is 7e-4 for somewhere around 30k samples i think (rank 128/a 128 on the 1.7B). i'm still running non-stop finetunes, it's going to take a lot of fiddling i think to figure out the best LR.

what on earth are you basing that on? you don't even know what i'm using to do QAT with. torchao is dumb with tensor dims that aren't divisible by 128, and MoEs aren't officially supported. for future runs on TPUs i'll try to make a sane qwix config which will hopefully work fine.

edit - i see now, you meant basically don't do QAT at all, just finetune the QAT model like normal. that's not a terrible idea i guess (since i'm not doing FFT)

no, but there's no easy way to do QAT for MoE yet. i have some hacky code that might work, but didn't want to risk it

i've got a big creative writing finetune for gemma4 26b cooking on an mi300x right now, 30ish hours left to see if it sucks or not

yep, a million times this

why wouldn't it?

if you run into Evennia's limitations give my codebase a try, it's directly inspired by Evennia: https://github.com/electroglyph/atheriz

because i hate myself and i hate proper tools and i hate getting things done

someone forked it for gemma4 moe and used for it that

kimi 2.6 is such a downgrade from 2.5. it blathers on and on without doing anything when it's stuck on a problem.

did any of the techniques use norm-preserving biprojected abliteration?

here's new graph by Daniel, enjoy. also, for the record, sol4ra on discord manages to be pretty obnoxious in both the LocalLlama server and Unsloth's server.

https://ibb.co/LX624Zm1

top

no, it works with something waaaay better: opencode

no.

there used to be a frog/toad website up that was pretty good at crushing all toad and frog misconceptions. you can't tell just by looking at them unless you've memorized em all

definitely a little bit of weirdness going on. i'm tempted to see how it's affected a benchmark or 2

it's either a poor naming choice or a genius one

RTX4500

those are pcie 4.0 btw

do some QAT finetunes of popular models and upload them

https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.latest-advancements-list

ernie kinda sucked a bit. still happy to see new models tho!