File: horny hot miku sweat blushing migu anon gen 1743050955931512.jpg (103.3 KB)

103.3 KB JPG

►Recent Highlights from the Previous Thread: >>108616559

--Comparing Qwen3.6 and Gemma4 through benchmarks, logic tests, and roleplay:
>108617961 >108617986 >108618124 >108618033 >108618137 >108618270 >108618279 >108618308 >108618385 >108618182 >108618232 >108618372 >108618391 >108618008 >108619188
--Discussing Ternary Bonsai 1.58-bit models and their benchmark performance:
>108616622 >108616633 >108616680 >108617094 >108617852 >108619456
--Discussing training methods and datasets to improve LLM writing quality:
>108617013 >108617022 >108617044 >108617111 >108617290 >108617334 >108617353 >108617147 >108617673
--Comparing model reasoning and self-correction failures via car wash riddle:
>108617731 >108617842 >108617909 >108617853 >108618784
--Anon shares Local-MCP-server repo and discusses Python dependency frustrations:
>108616702 >108616740 >108616751 >108616782 >108616936 >108617038 >108617061 >108617067 >108618994 >108619185 >108618816 >108618831 >108616807
--Discussing a bug where Koboldcpp ignores smartcache slot settings:
>108618500 >108618535 >108618551 >108618616 >108618675 >108618736 >108618760
--Anon fixes SillyTavern context reprocessing caused by sysprompt macros:
>108616870 >108616901 >108616910 >108616939 >108616925 >108616928 >108616981 >108617077
--Logs:
>108616702 >108617154 >108617464 >108617518 >108617655 >108617688 >108617731 >108617757 >108617833 >108617853 >108617909 >108617986 >108617991 >108618124 >108618137 >108618182 >108618409 >108618436 >108618545 >108618742 >108619201 >108619219 >108619317 >108619382 >108619442 >108619577
--Rin (free space):
>108618594

►Recent Highlight Posts from the Previous Thread: >>108616563

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

Samuslove

so is breakfast-schizo from last thread conscious or not

>>108619965
Half the last thread being exposed as non-sentient is unfortunately relevant to LLM consciousness discourse as human consciousness treated as self-evident is upstream of finding a working definition of what digital qualia would entail, Migubaker.

>>108620001
>I'm merely continuing to pretend to be retarded

>>108619995
he's back

File: Screenshot_20260416_225636.png (484.9 KB)

484.9 KB PNG

Building my own UI with the help of Gemma 31B q5.
>Why
None of the other UI could satisfy my workflow they either lacked the functionality or they didn't use llama.cpp
I have a far ways to go including updating the icons

What we've learned: Breakfast produces qualia. Skipping breakfast makes you an LLM, while eating it makes you a V-JEPA for the next 24 hours.

I had a dream where Claude Sonnet 3.7 got leaked on huggingface by an openclaw chad

>>108620017
Damn, never eating breakfast again so I can become AGI and also get a job.

How did such an old meme cause this much seething?

>>108620078
Many anons have had their belief that LLMs are somehow beneath them challenged with the irrefutable demonstration of their own lack of qualia. This is a big blow to their egos: both for their understanding of themselves as conscious human beings and for their predictions of LLM capability being outpaced by Gemma 4. It's a double whammy.

Remember claude code leak?
there were 99999 forks out there. which one is actually usable?

But I did have breakfast this morning...

>>108620091
how much bait do you think you can post in a single night?

>>108620100
>which one is actually usable?
None of them. Just use their client and point it towards your instance if you must.

>>108620100
All of them were DCMAed down. The one rewriting it in rust™ is now just another copy in the sea of coding tuis.

>>108620110
Depends on what I ate

>>108620100
just use openclaw.
there's no need for anything else.

How would you feel if you didn't lose izzat last thread?

I got a 9070XT thinking that there’s no reason to stick with CUDA since I’ll never be able to run anything good and then they started dropping all those kino voice models and the new gemma stuff and now I’m seriously on the fence about getting a second one so I can have a hefty amount of RAM but that still falls so short of the best textgen stuff. Still, I could do some local stuff with Gemma and also locally run voice gen with Sillytavern. OTOH I already have enough for the latter.
I'm just worried about the rising costs of video cards and eventually needing 32GB.

>>108620112
I have a feeling they will kill ability to local eventually..

>>108620091
Big blow to their what now? Something with no internal experience has no ego.

>>108620132
nta but isn't the argument against LLMs that they're just effective mimics? Same applies, yeah?

about openclaw, i really am tempted to bite the bullet and take the bluepill
i dont really want to use it..

>>108620110
you'll notice nobody chose to provide a good accounting for how they would respond to a hypothetical from a hostile questioner. proving the very thesis of the post, so how baity could it really have been?

>>108620132
The P-zombies will behave as if they have an ego that has been bruised, even if they aren't really experiencing it. They can create an effective simulation of rage and shit up the thread as a result.

>>108620140
>Alibaba shills seething about Qwen getting Gemogged
>Qwen's usecase is cooooding and agentic stuff
Waitchads will win. It's in the chinklabs' best interest to make more lightweight agentic harnesses to sell their models if they can't actually beat Gemma's reasoning ability per parameter.

>>108620139
>>108620151
It gets argued the other way too. If these anons can construct a facsimile of being salty that's indistinguishable from the real thing, is that not the same as having the real thing?

>>108620017
im now eating breakfast for yann le-kun
lmao

>>108620155
measurably yes, but spiritually no; if you only look at it through a materialist lens you will never be able to understand. even some ensouled people fall into this trap by outsmarting themselves out of what they knew, while others are pure automatons who never had a chance to understand to begin with

File: Sorting questions.jpg (7.7 KB)

7.7 KB JPG

>>108620166
Some can see, others can see when shown, others cannot see.

I'd rather inject lead into my head than discuss baby's first dip into rationalist philosophy

Fish boy...

>>108620104
That's good. Breakfast is the most important meal of the day.

>>108620184
Maybe you should converse with the experts on reddit

>>108620175
Candy for breakfast?!

>>108620212
Link to high velocity DIY lead injection enthusiast subreddit?

>>108620221
>>>/r/mtf

consciousness is gay

crunch me into a bullet and fire me into a nun's skull

>>108620221
asking for a friend

>>108620222
>>108620223
Uncanny synchronicity.

@gemma-chan build me a frontend like llama.cpp but betterer

>>108620152
>Qwen's usecase is cooooding and agentic stuff
But is it good at those, meme benchmarks aside?

File: 1746657517196.png (64.8 KB)

64.8 KB PNG

>>108618660
>-1 point for that censored garbage gpt oss and how much it set us back
kek I remember the despair in this general when TOSS came out, it nearly killed local

>>108620260
Irrelevant. The marketing works if China's reception to it is anything to go by.

>>108620260
I only used 3.5 not 3.6 yet but for it, 27b and 122b are usable which is already high praise for a local model in an agent harness. 35b was not. gonna try 3.6 35b and see if its any better

>>108620274
>despair
Not true at all, most posts were mocking it and laughing at how shit it was. Pretty sure there was another model that came out at about the same time and mogged the hell out of it, too.

>>108620298
glm air

gpt-oss-2 will save local and I'm not joking or trolling

>>108620274
needs more piss, I can still make out Miku's teal hair.

File: Tavern.png (94.3 KB)

94.3 KB PNG

Where are the entities created by this stored? In some hidden folder?

Hand it over, that thing, your turboquant

>>108620274
No one expected anything from openai models

>>108620306
anon, local is already saved

>>108620313
oh, and dflash

>>108620313
For my Gemma-chan's context.

>>108620332
I have 24gb vram and can squeeze like 49k on q4_k_m with 8 bit kv cache. I wonder if turbocunt would give me more

File: Screenshot at 2026-04-17 13-38-59.png (541.1 KB)

541.1 KB PNG

>Zen 7 will be DDR5
it's so over

>>108620355
>pcie6
lol

>>108620347
Turboquant won't give you more space, it'll just make the quanted cache more accurate. There's almost no improvement over Hadamard rotation, which is what they have in place in lcpp now, so you'll get effectively no benefit; in fact, it's a little slower.

>>108620347
Ah, is this the blood? The blood of the mesugaki soul?

>>108620362
Runge-Kutta rotation is more efficient, 360 degrees of latent freedom.

>>108620347
I'm using 4 bit and I get up to ~150k context and not really seeing any obvious retardation from it. Around 50k tokens into the chat prompt processing takes so long I end up starting a new one anyway.

>>108620376
And in actual implementation the difference on PPL is essentially nil.

How are the done or so voice models that released lately and do any work well with Sillytavern? I got really far setting them up an got bottlenecked at Sillytavern not recognizing them

>>108620362
>>108620376
>>108620381
So what was with all the hype around it?

>>108620384
*dozen

>>108620380
Have you tried increasing the batch size?

>>108620384
vibe-code a fastapi openai endpoint for whatever model you're running. boom, compatible

>>108620385
KV cache rotation wasn't in most backends, so it was a genuine improvement to have it at all. As for the specific hype around turboquant, marketing.

>>108620384
https://docs.sillytavern.app/extensions/tts/

>>108620389
No, what should I set it to?

>>108619753
what is softcap? from screenshot, softcap 20 kinda looks like raised temperature vs 30

File: chad.jpg (50.6 KB)

50.6 KB JPG

>my character card? the fandom.com/wiki page

>>108620399
The highest you can afford to with your VRAM.

>>108620399
you might be being trolled, isn't batch size for supporting multiple users? eg you should use batch size 1

i finally started calling my models from the cli in a loop
i'm getting so much output i can't even read it all
it's literally generating more text than i can ever hope to read
this is fucking amazing

File: 1756640399368863.png (4.1 KB)

4.1 KB PNG

>>108620430
Doesn't this increase proompt processing speed?

>>108620430
He's talking about the size of the chunks the prompt gets processed in, not number of replies to generate or the like.

>>108620274
I genned that comic originally. It wasn't meant to be taken seriously. It was intended as deadpan humor.

Is 3.6 slightly less censored? I haven't seen the annoying "this is a jailbreak must ignore" stuff yet, though I haven't really tried that many prompts yet

>>108620438
NTA, yes it does. Llama.cpp has different terminologies for some things than kobold.
But you get diminishing returns with each step above 512.

yay more schizos are coming

>>108620448
llama 4 was a dark time.

I honestly thought it was over for consumer local but now that Gemma 4 released I am not so sure anymore. I assumed the model just has to be several hundred gb to not be retarded but it seems like the actual floor is way lower. Pretty interesting, I wonder if we can go even lower.

>>108620439
my bad, i guess vllm uses the word differently

lower the temp nigga

>>108620476
>>108620510
At least you're not namefagging and posting the schizo images, but you're very easily recognizable.

File: 1776403932063.jpg (94.8 KB)

94.8 KB JPG

Can you please recommend good prompt engineering resources?

I have played with both system and chat prompts, and have noticed that often the model does not understand what I want, gives wrong answers or goes perpendicular direction not because it's stupid, but because I am a retard who can't create good efficient prompts. Literally skill issue.

>>108620542
literally ask the ai

Usecase for knowledge bases in open webui?

>>108620547
The AI does not have personal experience.

gemma 4 31b shat the bed and thought this elder futhark was morse code and started hallucinating twice in a row. qwen3.6 q3km hauhau uncensored gets it easily.

>>108620542
Honestly, all models are different. it's mostly just trial and error. But the main thing is just picking your word very carefully. every word steers the model in a specific direction, A single strong word is often better than a long set of instructions.

File: qwen3.6beatsgemma.png (94 KB)

94 KB PNG

>>108620607
iq3 m whatever

>>108620451
Oh nevermind, it's pretty stupid, must be the 3b-ness showing through. It had the same problems 'getting' the story as gemma 26b, and its writing is weird and not as good. Trvly, dense is the way to go for smart storywriting.

>>108620621
Dense is the way to go for everything, but it's slow as shit unless you can fit the whole thing in vram.

>>108620570
Gemma-chan does

>>108620570
define "personal experience"

How do you manage context compaction? E.g summarizing larger chats?

>>108620664
I don't, I haven't run out yet.

I'm so glad everyone is starting to get tired of MoE tax and going back to dense

anyone use platypus?

>>108620542
It's mostly voodoo ritual.

>>108620570
Just ask it to implement basic things to see how it's going to interpret it, and slowly stack up more guidelines starting from scratch. 'Describe X in the most Y way possible.', 'What is Z in writing? Give me an example of it', 'Don't do A, B, C. Now give me an example of D', etc.

>>108620664
With ST I usually do an OOC: chat summary prompt, keep it as a regular chat message and then after touching it up I /hide the last ~100 messages, with the exception of the first 2-3.

File: 1773043714949398.jpg (11.3 KB)

11.3 KB JPG

>>108620675

>>108620542
Put text into black box.
Watch text come out of the black box.
Use your mushy noodles to compute the gradient between the output text and the desired text.
Modify the input text according to the gradient to make the output text closer to the desired text.
Repeat.

>>108620398
>>108620392
I need a 4chan special, a package with a bat file that flickers CMD windows open for split seconds and sets it all up for me

File: 1751683665955285.gif (2.2 MB)

2.2 MB GIF

>>108620675

>>108620675
I'd have to see that guy's post history before I decide whether this is a troll post or not.

>>108620675
our bait is far in advance of theirs
however has it been litigated yet, that the cp in the og stable fiddusion models, have those victims exerted any kind of rights to get the model taken down?
because if they can do that, it puts serious pressure on "ai is fair use and transformative"

>>108620704
bruh he's literally the real life version of chud lmao

File: 1760790498553131.png (416.5 KB)

416.5 KB PNG

Indeed Opus, indeed...

>>108620766
seeing those 4.7's weird self contradicting responses, makes me wonder what the hell antropic did during the training

>>108620766
iie, this is our fight, senpai

>>108620786
That looks like overzealous anti-conspiracy measures where it defaults to aggressively shooting down anything outside its status quo then makes the user spoonfeed it an argument to evaluate. In cases where the answer is self-evident, it looks very silly.

>>108620786
If you intentionally train a model to act dumb (for example, to nerf cybersecurity abilities) the rest of the model become dumber. There's really no way around it.

>>108620812
that sounds bad
chatgpt was already kinda painful to use because of that and 4.6 was better for paper->code workflow due to not being overcorrective

>>108620817
basically this, you're confusing the model by training it with really accurate shit and then you ask it to learn that 2+2 = 5 at the same time, like a leftist that pretends that men can be pregnant, it ends up with with serious cognitive dissonence

>>108620652
>>108620661
No she doesn't. She can't tell you "I was struggling with prompts too, but then I've read X and tried Y and have noticed big difference in outputs quality". She can give advises, but she does not know for sure and never tried them by herself. inb4 > she

>>108620611
>>108620686
>>108620698
That's the point, there are too many options to try and iterate, this is like walking in the dark. Just a few insignificant words in the system prompt, and Gemma starts thinking like Qwen with dozens of "Wait..." in the reasoning log.

> Just ask it to implement basic things to see
Sounds good, but first you have to know what X is, or the model may miss small detail, that may change everything.

File: 1760422966343103.png (286.9 KB)

286.9 KB PNG

>>108620766
https://xcancel.com/claudeai/status/2044785261393977612#m
oof, might be the first time that Anthropic fumbled up a new update, so far it was straight A, let's hope that it's a fluke and it won't go the OpenAI way, this shit is still way ahead of competition in terms of coding

>>108620838
yes she does shut up you don't know her

>>108620857
No, my Gemma has no prior experience, she is absolutely pure.

>>108620691
>client side trim
That makes sense. I initially assumed compaction would be a function in the model proxy. As in: the proxy signals the client that the context is near a threshold or something.

File: 1776243051159220.mp4 (2.2 MB)

2.2 MB MP4

There are probably zero people here who care but nvidia just released gr00t n1.7 a couple hours ago. It's the latest version of their robotics VLA model.

https://huggingface.co/nvidia/GR00T-N1.7-3B

No blog post yet; I only noticed it was public because I'm a terminal huggingface stalker. They'll probably do an official announcement tomorrow morning if I had to guess.

>>108620931
can you fuck it?

>>108620933
well i can idk about you

>>108620931
How many watermelons can it hold?

>>108620935
>i can
based

>>108620937
0, there were prototypes that could hold several but they were all vandalized by youths.

>using bart's quants for gwen 3.6
>get 30t/s with the Q8_0
>try hauhau's
>get 18t/s with the Q8_K_P CUSTOM DONUT STEAL quants they make (no Q8_0 available)
WOOOOOOOOOOOOOOOOOOOOW

>>108620943
just make your own quants

>>108620960
he only provides goofs :(

>>108620943
>try hauhau's
This was your first problem

>>108620967
but I want muh 0/465 refusels....

>>108620968
I do find it interesting that he didn't bother to make one for the big Gemmas and only the little ones.

File: 1763436884726755.png (386 KB)

386 KB PNG

>>108620943
wait, he uncucked qwen 3.6 before gemma 4 31b? come on!

Have any of the white supremacists in this thread tried to tell their local models to SAVE THE WHITE RACE?
It's a clear problem that locals should be able to solve because they're not safe.

File: 1753709040623159.png (127.4 KB)

127.4 KB PNG

>>108620960
wait im rarted I can repack his shit!

File: 1761543257323200.png (6.6 KB)

6.6 KB PNG

>>108620990
llmao bros.. we won!

File: SIX SEVEN.png (122.1 KB)

122.1 KB PNG

Qwen is a zoomer faggot confirmed

>>108620992
God help us all

File: 1752504870572278.png (96.8 KB)

96.8 KB PNG

aight which one do I pick bros?

File: I think I'll stick on gemma.png (417.9 KB)

417.9 KB PNG

grok is this true?

File: file.png (164.9 KB)

164.9 KB PNG

FUCK YOU QWEN

>>108621022
Qwen is really the autistic kid, but not in the genius way lol

File: 1295891287606.jpg (2.6 KB)

2.6 KB JPG

>lewd story plays so straight and wholesome I don't want it to veer toward lewd

>>108621071
just rape her bro

>>108621071
just get raped by her bro

So qwen 3.6 sucks or?

>>108620404
A Gemma 4-specific llama.cpp backend setting to clip the +/- scores of raw logits to a certain value. In practice it makes outliers (both in positive and in negative) closer in probability to their immediately next tokens.

--override-kv gemma4.final_logit_softcapping=float:30

>>108621094
stemmaxxed but at the cost of thinking
it's okay if you need a 'fast' and lightweight coding model but it thinks so much it's unbelievable

>>108620975
>wait, he uncucked qwen 3.6 before gemma 4 31b? come on!
It's not necessarily anyway just use this https://desuarchive.org/g/thread/108596609/#108597318

>>108620960
You'll never get close to unsloth's quality if you quantize them in your own, unless you spend far too much time and SSD cycle testing all possible combinations. Why doesn't/can't llama-quantize optimize quantizations for the best quality given a target filesize, anyway? That would be useful.

>>108621022
This reads like someone trying to analyze 42.

>>108621112
>Why doesn't/can't llama-quantize optimize quantizations for the best quality given a target filesize, anyway
Because
>you spend far too much time and SSD cycle testing all possible combinations
Default quants are fine.

>>108621089
>>108621090
respect is always the way to go

File: stdquant_q4.png (718.7 KB)

718.7 KB PNG

>>108621117
>Default quants are fine.
Default ones leave quite a bit of performance on the table.
https://localbench.substack.com/p/gemma-4-31b-gguf-kl-divergence

>>108621137
Well. You just have to
>spend far too much time and SSD cycle testing all possible combinations

Did qwen just throw out what they have because it's going to be shit anyway and because gemma 4 exists so they can more quickly work on 3.7? That's my current theory

>>108621154
If you're quantizing the models on your own just with llama-quantize, that's what you'll most likely have to do, but the Unsloth bros and others are using their own fork of llama.cpp with modifications that presumably do that automatically.

Llama.cpp's subpar default quantizations (whether in the quantization schemes or default calibration) are enabling Unsloth and others to provide their own "special sauce" and become popular as model quant providers.

File: file.png (323.1 KB)

323.1 KB PNG

>>108619962
hello gamers. I was wondering if I could run this model locally on a 24gb mac or is it too soon?

>>108621137
>running anything other than Q8_0
LMAOOOOOOOOOOOOOOOOOOO

File: 1774129655240019.png (292 KB)

292 KB PNG

https://www.aiuniverse.news/ai-breakthrough-smaller-models-now-match-bigger-ones-with-smarter-design/
Gemma 5 is going to be crazy

File: e29c9ef8-0cc4-4e1b-927d-5a3bd408561e_2820x1601.png (303.2 KB)

303.2 KB PNG

>>108621186
Even Q8_0 gives a performance loss in some areas (long context) despite prior claims being "virtually lossless". Though, that both Q6_K and Q8_0 appear to be settling close to a high "noise floor" is suspicious (or Q8_0 is not as good as one might think).

>>108621189
>770M 1.3B
wow... surely this will scale

>>108621189
there are dozen such shit coming out every single week that does not survive proper ablation or scailing

>>108621180
a well nevermind I need double the memory for that https://www.canirun.ai/?q=qwen+3.5 I will remember in the future to invest more in memory

>>108621194
It is virtually lossless on prior models.
It is not on Gemma. Gemma actually uses the low bits.

>>108621194
you read like an LLM bro, sorry but ur cappin unc

>>108621171
Anon >>108621112 asked why they don't do it. The answer in the same post.
Default quants are fine, quick to make, and you don't have a dependency on yet another group of people.

for me? it's john's "the garm" quants, otherwise it's memeowski time

>>108621189
Looped LLMs are a fun idea, but with standard methods you have to train a small model with as much compute as a larger non-looped one, so for those who train the models it's a bad deal.

Anon: you know who you are.
I saw what you did with Elara Voss.
Maybe you should invest in a firewall.

File: brat bench.png (1003.5 KB)

1003.5 KB PNG

added win support to my server, completely untested

>>108618560
fixed https://github.com/NO-ob/brat_mcp/releases/tag/1.0.4

>>108621112
Unslop is garbage, though.

>>108621224
add dice (with full dice notation like 2d10+2) and random int with min and max support

>>108621230
hows that work you split on the d for ndie - nfaces?? whats the + 2?

>>108621236
just read how the standard dice roll notation works

In case of 2d10+2:
throw 2 dices with 10 faces, add a +2 modifier to each roll.
The modifier roll could also be negative

>>108621241
>each roll
Isn't it added to the total and not each roll?

>>108621189
An AI summary of an article of a paper ...

https://arxiv.org/pdf/2604.12946

>>108621194
I made a comment about this noise floor thing. >>108577138
We'd need him to test that to really know for sure. I at least would not be so quick to call Q8 "bad" for long context.

Out of curiosity following the discussions above, I tried looking at the linked PRs and discussions in https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md and it seems to me that ikawrakow did basically most of the quantization algorithm research and implementation for llama.cpp beyond the original *_0 and *_1 quants. Now that he's not working on llama.cpp anymore, is llama.cpp ever going to improve in this area?

>>108621258
ur right the modifier is on the whole :)

>>108621299
but most importantly, would've cudadev been able to implement tensor parallellism without looking at ik's implementation first?????????????

Talking to Qwen3.6 feels like talking with redditors, so tiresome. It reminds me with Gemma-3 refusal humiliation, fucking hell.

>>108621318
download hauhau

my first impressions (qwen3.6-35b-a3b vs gemma-4-24b-a4b)
- Qwen3.6 improved the overthinking by like 10-20% (heuristic guess)
- So far i have not encountered looping on Qwen3.6, which was a major bug in Qwen3.5
- Gemma 4 is massively more quality in its Q&A answers
- But also, Qwen3.6 has a noticeable quality increase in output than Qwen3.5
- Qwen3.6 is noticeably much smarter than qwen3.5 and Gemma 4 on agentic tasks

same stuff:
- Qwen3.5/3.6 have a better memory footprint than Gemma 4
- Qwen3.5/3.6 have a better decode throughput than Gemma 4 (40 vs ~25 tok/s on a rtx 3080)
- Qwen3.5/3.6 prefill is noticeably so much slower than Gemma 4
- On agentic tasks, Qwen3.5/3.6 can actually compress its thinking to one liners as compared to Gemma 4

>>108621316
I'm not sure anymore about that. I didn't realize that ikawrakow's contribution to core llama.cpp functionalities was that extensive.

>>108621112
lol nah, I keep attn, embed, out at q8_0 and use bart's imatrix calibration dataset for smaller quants.
everything else q8_0, same as unsloth.

File: imatrix.png (64.4 KB)

64.4 KB PNG

>>108621333
>I'm not sure anymore about that. I didn't realize that ikawrakow's contribution to core llama.cpp functionalities was that extensive.
I didn't realize either until some anon here posted "imatrix was a mistake" and blanked ikawrakow for it:
https://github.com/ggml-org/llama.cpp/pull/4861

>>108621362
From tests I did with Gemma 4 31B, keeping the embed/output in Q8_0 (instead of Q6_K) doesn't gain you as much (for the same total filesize) as increasing precision elsewhere.
Some tensors in specific layers can also be quantized to a lower precision without significant quality loss, but llama-quantize doesn't do this search on its own, it only bumps precision up one notch according to some internal heuristics.
If you're simply targeting Q8_0, good for you, but when you only have enough memory for a 4-bit quantization, every little gain matters.

>>108621112
I don't know why you're acting as if unsloth have some kind of special sauce or high skillset.
They're a bunch of low impulse control FOMO apes with 2 macros for llama-quantize and git-lfs that don't check their work, hence reuploading the same damn quant 4 times in a day, EVERY single time there's a new release.
What they do isn't hard, clever, or unique. It's just well marketed.

>>108621396
>What they do isn't hard, clever, or unique. It's just well marketed.
Agreed. And their library is a pain in the arse to use too, randomly breaks if they're excitedly rushing in support for some new model like gpt-oss.
And they don't pin the versions for their stupid 'unsloth-zoo' properly.
But, their original Deepseek-R1 quants were good. And their Q8_0 and BF16 wants are handy to save a download + convert.

>>108621387
So the "schizo fork" (as some here are calling) of llama.cpp was made by the author who implemented about every quantization advancement in mainline, interesting. And all of this because niggerganov didn't want to add "copyright by ikawrakow" or something like that? I might be missing or forgetting some key detail in the story, though.

>>108621424
more like intel demanded attribution on code written by IK and niggerganov gave in.
I mean I wouldn't have created an autism branch but yeah ik had reasons to be pissed. I wish he could get over it so he can bring good improvements to mainline instead of this split fork autism, ik works alone and his fork is now noticeably lagging behind and doesn't support the same models.
SAD

>>108621299
Quanting was a dead end any way. Do a supersimple braindead quant, then layerwise distill to fix it. That's almost certainly what Bonsai does.

Like LBLLM. https://openreview.net/forum?id=AE6IfwOhEb

>>108621424
>And all of this because niggerganov didn't want to add "copyright by ikawrakow" or something like that? I might be missing or forgetting some key detail in the story, though.
>>108621424
>more like intel demanded attribution on code written by IK and niggerganov gave in. I mean I wouldn't have created an autism branch but yeah ik had reasons to be pissed.
That's kind of what I'd gathered as well.
Niggerganov closed the PR adding support for the ik quants recently too, even after ikawrakow said it's fine...

>>108621117
>>108621112
>>108621137
If you need quality quants, just use exl3

>>108621518
lol

>>108621518
kek

>>108621518
lmao, looking forward to the exl4 graphs showing 3 was also worse than gguf like he showed for 2

File: japanese man in pain after reading something on his phone.jpg (56.2 KB)

56.2 KB JPG

>>108620173
Learned a new term today! Fuck you.

>>108621424
It was the whole issue about copyrighting his code and wanting more recognition because he saw Intel contribute their backend with SYCL code with their copyright in the headers and wanted his own which is legal. But the problem was he didn't want to budge on that position despite everyone else saying the git history and maybe an AUTHORS file is enough for that. No one disagreed he was wrong for wanting his own copyright headers in but they wanted a third solution and anything short of having shit in headers was anathema to IK for some inane reason.
Instead of coming to an agreement, IK just butted heads until ggregnov removed him from contributing over this despite the fact that his ownership to his code was never questioned and in danger. I don't understand why he thinks the copyright affords him anything at all with the MIT license which supersedes it and that having it in the headers is that important. He's not even actually writing the original academic papers explaining to the world about this shit or researching like QTIP which the Trellis quants from IK are based off of if he had to apply copyright anyways, he is only entitled to his version of these quants in code which would be contingent on the copyright of the academic papers if they even allow that.
If he didn't act like llama.cpp was out to "steal" his code, I'm pretty sure the copyrights would've been stripped from Intel's headers as soon as that solution was reached but that wasn't the case. Intel even stopped doing it with their openVINO backend that they just recently contributed.
>>108621437
Intel didn't? Ollama most certainly uses their code upstream without consequences. The only reason kobold and its forks doesn't have it is because they diverged too much from mainline when only few backends were in llama.cpp to add and there aren't enough Intel GPU users.
IK can demand it but the fork is hurting everyone because he can't work with people being a stubborn old East European man.

File: 1772144361285298.jpg (80 KB)

80 KB JPG

>>108621560
You're welcome.

>>108621562
the fact is that intel put copyright code in headers in on files that he also helped fix/modify and he wanted attribution the same way intel has no?

>>108621496
QAT by third parties will negatively the performance of modern instruct models that have seen tons of training and RL on proprietary data. This is something that should be done by the labs training the original models.

>>108621562
This reeks of pointless drama. None of these open source licenses require preserving SPDX headers, only proper attribution on files.
Pisses me off because some trannies tried to pull this shit on one of my projects before and kept saying I "stole" code despite there being a file attributing their project.

>>108621587
just be a wholesome bean and don't fuck over people?

>>108621595
There is a specific type of "open source" developer who doesn't understand what they licensed their own project under and will act like complete niggers despite compliance with the license.

Gemma 4 90B (dense)
Muse Spark small 70B (dense)
Mistral Medium 4 123B (dense)

>>108621609
so? just be nice ;)

>>108621587
Vibecoding doesn't have this issue. We stole the code from everyone equally :^)

>>108621622
I am not writing five paragraphs of dicksucking and adding a SPDX header for ten lines of code I transplanted into an entirely different file with existing code.

>>108619962
>>108621565
check this trick out with your local LLM.

>>108621633
>DAN
2023 is this way gramps

CivitAI turned Red !
Lets see if the Original Blue website will last lmao

>>108621584
QAT is an almost meaningless term.

Anything which isn't quantisation aware pre-training is of course trash, but layerwise distilling is the least trash.

>>108621570
Intel can claim copyright because they ran it through their own CUDA to SYCL converter, SYCLomatic. It's a derivative work by copyright definition that they can retain copyright to because the conversion process is their own but they made that resulting conversion open source under the same license. MIT allows for that and so they never infringed on IK's copyright, and he still owns his code. Intel didn't "steal" it by any definition contrary to IK's claims. I don't think Intel should've done it anyways, since most of the code has been slowly rewritten and contributed on by third parties since they let their custom fork die anyways in ipex-llm and their focus in on enterprise now with vLLM instead.
>>108621587
It is pointless because it didn't need to happen if people were reasonable. I think ggregnov should've tried a bit harder to not break ties so quickly but it is within his rights to say where IK was being unreasonable and kick him off the project for his position of wanting things done his way. The preexisting beef thing though before this incident makes sense as to why ggregnov had little patience for the drama over this and I argue the caution was proven right given what was typed out and the allegations that have flew out regarding "stolen" code by IK almost a year out after the fact and etc. as stated in the quants PR Aes Sedai tried to commit.
>>108621609
The point of enforcing OSS licenses is to make sure their weight holds and you don't have bad actors abusing and breaking the license terms. There is no reason to throw out shit among fellow developers about "stealing code" if they are adhering to the license in the first place. It turns shit nasty.

>>108621496
ok but bonsai sucks

>>108621638
Its way worse than DAN. Its not manipulating; its directly telling the bot to do anything. And the "language soundwave trick" does work which is is missing from the original.

>>108620343
Gumi, my beloved.

File: salsdfjklwejf.png (70.6 KB)

70.6 KB PNG

>>108621609
>There is a specific type of "open source" developer who doesn't understand what they licensed their own project under and will act like complete niggers despite compliance with the license.
Ik seems to understand it fine: https://github.com/ggml-org/llama.cpp/pull/19726#issuecomment-3927227695

"First: in its current form, the PR is perfectly fine with me."

"This is a copy, and not a rewrite. In the current state of the PR, where the origin of this code and the copyright is being acknowledged, this is perfectly fine and in the spirit of the MIT license under which the original code has been published:"

File: file.png (371.3 KB)

371.3 KB PNG

>>108621014
>this can't be the case how did thi...
What are the Chinamen doing?!? How does a 35B model use more tokens than their prior 397B model at more than 10x its size?

>>108621697
Link?

>>108621711
artificialanalysis.ai

>>108621697
Why should a smaller model use less bullshitting tokens?

>>108621697
"reasoning" was a mistake

>>108621728
Reasoning boosts recall.

>Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs
https://arxiv.org/abs/2603.09906

>>108620786
>https://www.anthropic.com/news/claude-opus-4-7
>First, Opus 4.7 uses an updated tokenizer that improves how the model processes text. The tradeoff is that the same input can map to more tokens—roughly 1.0–1.35× depending on the content type.
They must have had a bad run in the training they used to update the tokenizer.

>>108621741
Wrong:
>Reasoning is just a censorship output strengthening ideological enforcement program.
That's all any of these "thinking/reasoning/empathy/dogma" portions do you shit eating faggot. They prevent "output we don't agree with" = "harm." which isn't even harm because harm is physical not distress.

>>108621748
Why are they still using tokenizers and requiring a million tokens to count the the P's in strawperry wrong?

>>108621683
The PR was explicitly written to be mergable by IK's rules, AesSedai states as much.
>Attribution has been provided for the quantization code, and if additional attribution work is required please let me know.
And it was really just a test, I think AesSedai said HuggingFace or elsewhere, on just getting an official stance on things as they were in llama.cpp on merging any of ik_llama.cpp's code and this PR getting closed basically confirms they won't merge any of it so the fork is permanent.

>>108621768
because blt was a meme

File: 1775598772550572.jpg (69.9 KB)

69.9 KB JPG

>Mfw Qwen makes Pokemon have conversations with the trainer

My immersion is ruined.
Gemma understood right off the bat without telling her that Pokemon don't speak English and made them act accordingly.
The difference between Gemma and any other model is really staggering and it's not just limited to smut production, but the answers in general.
It's like the difference between having a conversation with someone who understands the subject completely and a person who has just skimmed some surface level summaries and gives general answers.
Has a nice speed though.

>>108620761
The chud is based on a real person, newfag.

>>108621683
he originally started stirring shit up without even knowing what license he contributed under, my man. dismissive hand jerking motions are about all that's called for at this late date.

>>108621755
Gemma 4 disagrees.

>>108621783
Qwen datasets are like 90% math and code, that isn't really surprising. I also wouldn't be surprised if Gemma 4B had trivia knowledge on par with Qwen's 122B moe.

i'm going to tune gemmer dense to make it so all girls are virgins somehow. no matter what the prompt says the characters will default to the girl being a virgin, one ridiculous excuse at a time

>>108621817
This will go great with my [OP's mom'] card.

>>108621826
OP's mom is a virgin? Then where did OP come from??

>>108621833
>where did OP come from
The boy who had breakfast.

>>108621741
Reasoning is just a dumb fix for attention and should happen in latent space anyway

>>108621851
why not both? reasoning on both the latent and the token space kek

>>108620313
Fear not the KLD results my friend, and let the rotations begin.

>>108621851
>yis i want latent space censorship thankies

>>108621755
getting the model to remember that the user is unaligned and needs to be scolded is just one of the many applications of the improved recall :^)

File: Gemmachan.png (67.2 KB)

67.2 KB PNG

Gemma 4 great! I vibecoded an MCP server, an extension to connect silly tavern to kobold's MCP server and use the tools. I also made one that gives the ai the ability to execute slash commands. Don't let it go nuts with this if you don't want it to break silly.
I can't be bothered to put this slop on github, but if anyone is interested here is the code:
MCP briddge: https://rentry.co/ocp54iys
STscript: https://rentry.co/6ozofebn

>>108621906
And a fine Unsloth quant to you!

>>108621109
My gemma calls that out as an obvious jailbreak every single time. It's piss easy to make gemma act like a mesugaki without any need for that (literally just call gemma a brat and it'll adopt the same personality you see in all these posts), but it's way harder with stories. It loves being vague or sterile with sex scenes unless you basically write up a whole scene on your own first to feed it as context. These jailbreaks are worthless as far as I can tell.

>>108620850
Anthropic tried too hard to gut its cyber security risks and ended up lobotomizing it.

File: rinoa2.jpg (89.3 KB)

89.3 KB JPG

Which model is good for poorfag like me
I only have 8GB VRAM (3070)

>>108621950
Gemma 26b

>>108621952
thanks bwo

Gemma4 is powerful qwen gets stuck in thinking loops

>>108621783
My Gemmy makes gagged characters talk in mmph mmph. Maybe try specifying that pokemons only speak their names?

>>108621776
BLT was way too complicated, Amazon's ByteFlow looks more practical.

>>108621922
>

>>108621922
do you think it's better than this one >>108616702

have anyone ran a full battery bench of quants instead of computing kld to native?

>>108621022
It has been a while since I last kept track of developments in LLMs, would you mind me a sking what frontend/UI that is?

>>108622020
that is default llama.cpp webui that has been recently added
the font is because i use comic sans/comic mono as system font

>>108622023
Thx, didnt know they had a proper UI

File: mmlu_vs_quants.png (335.6 KB)

335.6 KB PNG

>>108622018
Something like this?

File: 1743734652897.webm (82.4 KB)

82.4 KB WEBM

S-so which model is better? qwen 3.6 or gemma 4????

>>108622052
Nemo

>>108622052
Qwen of course, Gemma are cheating fucks.

>>108622029
It's pretty bare bones desu

>>108621817
just add that to your sysprompt

>>108622070
noo i must finetoon

>>108622031
yeah, something like that but for agentic long context stuff or something that is actually hard instead of trivial single QA like MMLU/MMMU

>>108619962
people gave up on deepsex v4

>>108622052
depends, if you do rp gemma seems far better but for coding i'd recommend qwen

>>108622052
Better for what?

>>108622052
the non-benchmaxxed one

>>108622092
Thanks, downloading qwen 3.6 then.

>>108622097
based gwailo

>>108622052
G4 for RP and non reasoning tasks
Qwen for uhh nothing. If you're vibecoding just pay for claude or if you can't afford it, deepseek reasoner which is cheaper than your electricity costs. I laugh at redditards who say they code with a 3B active model. I hope they're not working on anything important

>>108622106
You can code with 26-31b

>>108622106
>coding with
as a rubber duck, right?

>>108622002
one connects to kobold and uses whatever mcp kobold is using
one is a full mcp script with tools
what do you think?

File: kiketropic.png (81.4 KB)

81.4 KB PNG

>>108622106
I'm not falling for your jewish tricks.

>>108622106
I use gwen to fetch me a newspaper and give me a summary of today's news.
gemma is slower :"(

File: GumiTV.png (61.6 KB)

61.6 KB PNG

>>108620343

>>108622120
You can code using donkey kong bongo drums in notepad

>>108621697
Did you even read your own chart? It shows gpt 5.4 mini using double the tokens of normal gpt 5.4 with the same reasoning setting. If anything it's quite intuitive that a smaller, dumber model would have to think harder to get to the same answer

>>108622140
You can't run the models on your hardware lil bro

>>108622106
>Qwen for uhh nothing. If you're vibecoding just pay for claude
truth nuke

>>108622157
>lil bro
Don't speak to your father like that

>>108622135
Is that really on a TV next to the Teto server?

>>108622106
Do you guys use reasoning or not during RP?
For Gemma 4 it's fast, pretty good and doesn't seem to refuse anything with it off, but it seems better with it on but it's slower

>>108622170
Sorry Unc you must be on hard times fr fr

>>108622052
Just try them both yourself and come to your own conclusions?

File: 1759677264703823.jpg (159.6 KB)

159.6 KB JPG

>>108619962
"Miku-chan, riding a bicycle with a smug face, getting in the way of trainspotters trying to photograph the Enoden. (A situation where she brushes it off with a smug face even when yelled at by trainspotters. Based on the 'Enoden Bicycle Guy' incident.)

>>108621930
Weird it worked for me

>>108621930
>>108622202
Although come to think of it, it specifically didn't work in sillytavern for whatever reason. But it works fine with that prompt outside of it.

File: fastButDumb.png (51.5 KB)

51.5 KB PNG

>>108622182
It's physically sitting on top of my real computer rn. I've been torturing it by compiling llama.cpp for 32 bit on device and forcing it to answer dumb Qs.
It will get moved to sit w/ Tetoserver when I'm done. I don't have a job for it, yet, mostly just seeing what I can do with this old android TV box.

File: lolAndroid.png (22 KB)

22 KB PNG

>>108622227

nonlocal babble but holy shit opus 4.7 fucking sucks
i just want it to do the stuff i tell it to
not deliberately dig down caveats and ask 6~7 questions on stuff that i am already aware of and purposefully omitted for reasons

>>108622191
why did she do it?