u/Dathide

Thanks, that is interesting. I see exl3 in the release notes. The readme is terrible and doesn't mention exl3, but I like that it's made specifically for Ampere.

That's an insane improvement. BF16?

I want to wait but I might give in within a week. I've been using bf16 models lately. I think the unlocked capabilities still run on the main die, and I will monitor that and the VRAM temps.

Thanks, I stopped making my own presets months ago

Thanks

Thanks! I'll try those later

25-30 makes me think I didn't configure it well, I have 2x3090s with an nvlink Edit: and 96gb ddr4 3200

thanks, i'll double check the offloading stuff

good to know, i've seen hot experts mentioned before and wondered if it was just a strata thing

Thanks, that's a good idea. Maybe it can configure TabbyAPI itself.

I would've guessed that some of it is vibecoded. Do you think they used Claude? I'm not sure how much effort it would be to vibe-code a fork of exllama. I might need to know a lot to help sometimes.

Thanks

Is running non-greedy as simple as sending temp = 1 with requests?

Misinfo. Do better next time.

There's a key word in there that reinforces my point

Why are so many acting like this is more than a draft? I'm guessing there are dozens of Chinese drafts that never pass. EDIT: Looks like it may be good for open: https://www.reddit.com/r/SillyTavernAI/s/nWdXEFRC63

Interesting

That's a huge increase just with x16/x4

Thanks, I'm glad both can be focused on since many consumers are stuck with potatoes

You need to make it more clear that these optimizations only work on 50 series