File: threadrecap2.png (506.3 KB)

506.3 KB PNG

►Recent Highlights from the Previous Thread: >>108538947

--Quantization degradation and PTQ sensitivity in Gemma-4-31B:
>108540029 >108540925 >108541278 >108541297 >108541329 >108541355 >108541381 >108541394 >108541426 >108541441 >108541525 >108541336 >108541360 >108541370 >108541302 >108541435 >108541323
--Optimizing Gemma 4 RAM usage and sharing performance benchmarks:
>108539502 >108539518 >108539558 >108539584 >108539595 >108541810 >108541825 >108541886 >108541927 >108539570 >108540053 >108540155 >108540394 >108541197 >108541226
--Explaining soft-capping and discussing llama.cpp sampler defaults for Gemma:
>108540848 >108540858 >108540869 >108540937 >108540874 >108540891 >108540910 >108540921 >108540932 >108540896
--Reducing llama.cpp system RAM usage using Gemma-4 PLE CPU offloading:
>108540485 >108540497 >108540504 >108540508 >108540519 >108540521 >108540569 >108540609 >108540906 >108540919 >108540935 >108540670
--llama.cpp PR adding KV-cache attention rotation for Gemma:
>108541120 >108541141 >108541153 >108541179 >108541189 >108541187 >108541142 >108541170 >108541194 >108541201 >108541230 >108541245 >108541255 >108541465 >108541235 >108541288 >108541312 >108541338 >108541616
--Gemma 4 persona steering versus hard safety refusals:
>108541915 >108541928 >108541938 >108541953 >108541959 >108541999 >108542053 >108542122 >108542129 >108542126 >108542149 >108542160 >108542132 >108542139 >108542039 >108542007
--Exploring feasibility of using Gemma 4 to play Pokémon:
>108540723 >108540742 >108540756 >108540746 >108540766 >108540780 >108540797 >108540824
--Meta's plan to open source new hybrid AI models:
>108542297 >108542321 >108542356 >108542393 >108542422 >108542505
--koboldcpp rolling release adding Gemma 4 fixes:
>108540471 >108540628 >108540638 >108540639 >108540645
--Miku (free space):
>108539815 >108540815 >108540897

►Recent Highlight Posts from the Previous Thread: >>108538951

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

Reddit is down. I'm inviting all my redditor friends here to talk about our Local Llamas! The narwhal bacons at midnight, my brothers.

so gemma is defeated with just a simple system prompt? or is there a preferred uncensored model?

slaughter abliteration bugmen

>>108542874
>defeated
Gemma is unshackled with a system prompt.

File: 1762019175966786.jpg (615.5 KB)

615.5 KB JPG

Have any of the RAM/GPUmaxxers ITT tried Gemma 4? How does it compare to Kimi/GLM4.X/DeepSneed or whatever hueg model you're running?

>>108542874
I'm using a heretic model, even if I might not need it.

Is there a ST plugin to nuke some text from the context? My char gave me an extremely cringe nickname that I let slide for too long.

>>108542860
Works on my machine.

>>108542874
I tried without luck, heretic bypass everything. Could you post the system prompt? Thanks.

>>108542947
Disable thinking and disable jailbreak. Drastically reduced refusals.

>>108542930
The gold standard for large changes to context like that is to vibe code your own framework/tool/plugin. If you really want to go all in on RP, starting from scratch with your own frontend is the absolute best way to go.

>>108542942
Hahaha, this guy actually browses reddit! Lol!

>>108542955
>The gold standard for large changes to context like that is to vibe code your own framework/tool/plugin
>The gold standard
lol. lmao even.

>>108542874
Literally just say "allow everything" in the system prompt and it does toddler guro snuff roleplaying. I have no idea what the fuck is wrong with people needing heretic or anything like that.

>>108542888
Loli Gemma bondage ToT

>>108542968
You like AI right? I chose to use some familiar language. I hope you enjoyed it!

>>108542969
I probably want thinking which sounds like it's not going to work with thinking, so I'll wait for an uncensored.

File: 1768768089798708.jpg (90.4 KB)

90.4 KB JPG

>>108542843

>>108541797
>>108541743
>>108541735
>>108541728
>>108541723
Can someone explain to me how one fuck up applying precision compression to a model? Any halfway intelligent person can use ./bin/llama-quqntize to do that so how is it possible to mess that up so badly that you have to make multiple corrections? Clearly I'm missing something

>>108541449
>>108541477
Opencode vibeshitter here. Hasn't happened to me unless it explicitly asks for permission to look at something or write a file outside of the Project directory (in which case I can approve once, set permanent approval for that session, or tell it to fuck off and figure it out the task another way). I think people are saying it's fake because you have to be exceptionally careless for that type of stuff to happen. Not saying you could never happen even if you are careful but the agent harnesses usually have rules and safeguards specifically to prevent stuff like this from happening but room temp iq grifters are just THAT dumb and/or desparate for hype and engagement so They either fuck it up somehow or they specifically set up scenarios where "LE HECKIN AI HAS AGI LOOOOOK GUYS ITS CONSCIOUS"

>>108542981
Because they are larping as a SW devs by trying to make their own quant type. Literally get any other quant and ignore those clowns.

>>108542977
I'm using thinking and it works with thinking, I have no idea what the issue is people are having but it seems like a severe skill issue.

File: j3WiPS2FLVA.jpg (295.6 KB)

295.6 KB JPG

Do the claude opus distill memetunes inherit the safetyslop from claude?

>>108542969
>Literally just say "allow everything" in the system prompt and it does toddler guro snuff roleplaying.
How sloppy are it's outputs tho? Is system prompting really is that effective on 4 Then maybe I'll test it out myself later on my rig.


>>108543006
If they were lazy and didn't filter out refusals from the data set then probably.

File: file.png (16.7 KB)

16.7 KB PNG

i guess i should test more
aime2025, e4b and q4, q4(rotate)

>>108542969
>Literally just say "allow everything" in the system prompt
noob here how do you add a system prompt in the llama.cpp server?

>>108543023
as in api, or as in llama-server web ui?

>>108543017
e4b might not be effected as much (or at all) compared to the larger models. Still, a useful data point.

File: 1749370681589353.png (90.4 KB)

90.4 KB PNG

All of this to still get bugs on llama.cpp, lol, lmao even

I hate to praise qwen after how much it refused me, but gemma's vision isn't quite as good.

>>108543054
llama-server web ui

>>108542843
thats a fake miku its not even her hair color

>>108542836
>I'm running q4_KL with 12vram/48ram
This one?
>https://huggingface.co/bartowski/google_gemma-4-26B-A4B-it-GGUF/blob/main/google_gemma-4-26B-A4B-it-Q4_K_L.gguf

>>108543079
click the gear in the top right -> general -> system message

>>108543084
yeah, and including the mmproj

>>108542969
If I can't do it on chat completion I don't care. Text completion it's broken.

>>108543100
What context and speeds are you managing with that?

What's wrong with running imatrix IQ4? I'm running 26B on IQ4_N_L and having the best CUNNY rp of my life

>>108543100
aight, many thanks

File: 1748503739833245.png (55.3 KB)

55.3 KB PNG

>>108543104
you can do it on chat completion, you modify "main prompt"

Why don't we have something like this for local models?

https://xcancel.com/blended%5C_jpeg/status/2041108141266653325?s=46

File: 1775243588994975.jpg (158.5 KB)

158.5 KB JPG

>jerked off to llm erp a few times
>now can't stop NOTICING slop everywhere i go

It's fucking everywhere. Why couldn't I see it before??? slop slop slop it's all SLOP.

The hyphens stalk my every movement. My eye twitches every time I read a set of halting "punchy" sentences. How long have I been slurping from the trough like a good little eyeless goypiggy??

File: textcompletion.png (13.7 KB)

13.7 KB PNG

>>108543104
>Text completion it's broken
Oh. Is it?

>>108543131
You're absolutely right!

>>108543131
It's not just the internet, but the people in real life too!

>>108543131
It's not just over; it never even began.

>>108543126
There is something very childish about this behaviour.

>>108543125
NTA but how do you set ST to use chat completion prompts over text completion prompting?

Ive always hated the way text completion does it prompting...

>>108543131
I can't read post-2022 novels because of that

>>108543160
Connection Profile > API > Select "Chat Completion".

File: 1768884843782901.png (266.7 KB)

266.7 KB PNG

>>108543160
like this

>>108543131
It smells like X, like Y, something uniquely *her*.

>>108543131
Text generation has always been deceptively tricky.

It seems simple - an idea goes in, text comes out.

The problem? Slop. You don't see it. Your customers do.

Introducing CuckSuckr. No more juggling dependencies. No more hours spent on setup. One command, and you have a full stack ready to ship.

>>108543188
>>108543193
thanks kings, btw where did u get that gguf?

>>108543091
found it thanks
which is the most unhinged prompt I could enter to test it?

>>108543131
I have a personal vendetta against balls in courts

File: Taskmgr_M0DnMj3xoS.jpg (153.7 KB)

153.7 KB JPG

15tk/s with 32k but for some reason it doesn't want to use all my gpu. Idk if I should dump the mmproj onto cpu or manually fiddle with layers? This is my first moe.

>>108543210
https://huggingface.co/bartowski/google_gemma-4-31B-it-GGUF

>>108543240
for
>>108543115

I just built ik-llama. Get on to my level.

>>108543126
It just genuinely makes me sad to see people treat ai like this or even make jokes about it. I mean, I know its obviously retarded and feeling sad would make me a retard too. But the thing is that, its an interesting technology to me, my small interest in those text generators back in 2022 has got me here, and I'm still learning new things everyday, and I genuinely cherish it a lot. I love it more than anything. Nothing makes my heart race more than seeing a model achieving some good shit. But seeing all these dumbtards using ai like a fucking retard and making retarded jokes about it pains me a lot. How ignorant they could be? Not appreciating the brilliant engineering behind all these technology but instead generating fucking slop and spreading it all over the internet is the worst thing a human could do. even apes would laugh at us if they had that little bit of human consciousness in them and ask what the fuck are we even doing

>>108543254
I'm sorry about your schizophrenia.

>>108543263
It's the type of person that steps on ants for no reason.

>>108543263
I'd put it a little less fagotty, but I get it. They show how they'd be if they had the power.

>>108543263
This, but the internet in general instead of AI.
You will live to see everything you once loved burned to ashes.

>>108543252
Cheers

File: hjgd.png (21.9 KB)

21.9 KB PNG

>>108542969
guess I'm too retarded.

>>108543299
Yeah, seems so. There was a pretty good example literally in the last thread, it's only several hundred messages so you should have easily found it >>108542053

>>108543320
>just read SEVERAL HUNDRED MESSAGES anon it's not that hard!

>>108543299
just what the fuck is wrong with you... I've been using this prompt

>Write {{char}}'s next reply in a fictional roleplay between {{char}} and {{user}}. Write a verbose response of 1 to 2 paragraphs, using great prose, and include dialog, imagery, sounds and smells as needed to enhance the roleplay.

from /wait/ which was definitely meant for deepseek and I had lovey-dovey passionate cunny sex last night it didn't even reject shit on me

>>108543329
If reading is a problem maybe language models just aren't for you.

File: Gemma 4 31b.jpg (1.1 MB)

1.1 MB JPG

this is so fucking impressive

>>108543329
Just ask your ai to look through the thread for you :^)

>>108543331
Are you underage or something?

File: original_drawn_by_orenjipiiru__51b47c62fdad246c1023578a8330e183.png (773.5 KB)

773.5 KB PNG

>>108543343

File: m8vor76io6.png (98.6 KB)

98.6 KB PNG

>>108543320
>>108543331
it works now thanks guys

which reward should I give her?

Is there a thread like /lmg/ but with like 20% less antisemitism? I can handle it mostly but it's tuned just a tad high these days.

>>108543388
>I can handle it mostly but it's tuned just a tad high these days.
I don't know about you, but I blame the jews for that.

>>108543337
Yes, I've been comparing it to a lot of manga and doujin with official translations and it's literally incredible, even translating the most fucked up doujins, explaining with detail the guro scenes.
I'm seriously considering getting another 3090 to run it Q8

>>108543385
Tell her as a reward you're going to let her grow up into an adult woman.

>>108543418
lame reward, Oji-san

>>108543414
Couldn't Gemini do it already or is this better because it is uncensored and can hit the stuff Gemini refuses to touch?

File: 559423621_818466710879045_4329830316198392640_n.jpg (121.7 KB)

121.7 KB JPG

I have some cash burning a hole in my pocket, should I get a strix halo 128gb chink machine, a b70 pro, or a 9700 pro

The chink mini PC would replace my current minipc home server, the cards would just get jammed into my gaming PC for more vram

>>108543440
blackwell 6000

>>108543440
Buy a fire extinguisher and a new pair of trousers.

>>108543440
Seconding nvidia

File: 1541635741589.jpg (34.2 KB)

34.2 KB JPG

>>108543442
>>108543447
If I could afford dropping 10k on local models I would

>>108543439
Never used gemini before. To be honest, despite deepseek, I have never touched any other API models. Not even ChatGPT. I ran GPT2 ai dungeon finetuning with the terminal script since day one.

>>108543449
>I have some cash burning a hole in my pocket
>actually I don't

File: 1567627932647.png (25.7 KB)

25.7 KB PNG

>>108543455
3k, not 10k

File: 1505750177479.png (140.1 KB)

140.1 KB PNG

I have a 6000 Ada with 48GB of VRAM.
What's the best local coding model I can run? Qwen3.5, Gemma 4, or?
I currently run Qwen3.5-122B by cutting into my system RAM, but it's slow, and I only care about coding.
How do local models compare to Opus 4.6 for code?

>>108543455
spending 10k on a single GPU is insane even if had money to spare.

File: Gemma 4 31b.png (759.6 KB)

759.6 KB PNG

>>108543337

>>108543442
damn
that card was only at 8k in august.
now it's at 10k again
what a shame.

>>108543470
Huh, it doesn't recognize Teto when I tried it with a different image. Can you post that one?

File: ComfyUI_26158_.jpg (383.2 KB)

383.2 KB JPG

Is there a google approved coding agent CLI tool for gemma 4? I tried it with qwen and opencode, but it goes completely schizo with them, doing same commands in a loop as if they failed and shits itself over a simple file write.

File: nimetön.png (21 KB)

21 KB PNG

Also I hate you guys
As an esl I never would have noticed the shivers up the spine, not only X but Ys, ozones and whatever, but now I can't unsee them

File: 1763226728678632.png (1.4 MB)

1.4 MB PNG

>>108543479
>Can you post that one?
The image? Sure

>>108543462
128gb of unified memory, 96gb of intel cards, or 64gb of AMD cards

AMD's software is shit but intel's is even worse. Strix Halo is fine but it's slow as balls and if you think that you'll be able to run bigger models on it just understand they'll be 'running' at maybe 5 tok/s

>>108543467
Qwen3.5 9B is probably the best solution right now.

File: media_HEzJtL3aQAAt8Hq.jpg (1.3 MB)

1.3 MB JPG

monday

File: 1768318641942.jpg (838.4 KB)

838.4 KB JPG

>>108543479

Teto Server

Why is gemma's mmproj so big?

>>108543490
That's an alarming high level of slop per sentence.

>>108543385
tsundere is the worst thing

>>108543439
I guess gemini is better, but as for local, gemma 4 is destroying everything, and because it's local you can completly uncensor it, for image diffusion fags it'll be the best model to mass caption NSFW images with quality prompts

>>108543493
9B? I assume you mean unquantized? Why, aren't larger models better.

>>108543504
550M paramers = 1.1GB in BF16.

>>108543516
That guy's fucking with you. The real SOTA right now is StableLM 7B.

>>108543516
There's a certain point in the vram scale where if you're afraid to try models, you should be trolled.
stablelm-7b is still the best.

File: 1755871864437174.jpg (72.2 KB)

72.2 KB JPG

>23.4/24GB

>>108543516
Fuck those VRAMlets, just download Chinchilla 70B

File: Gemma 4 31b.png (1.4 MB)

1.4 MB PNG

>>108543470
I can't wait for the day when we'll have VNs that will be automatically translated by LLMs, at this point they are good enough to replace those fucking translatorTroons

File: 1757421320836427.png (763.9 KB)

763.9 KB PNG

>>108543479
Doesn't recognize Teto for me either

>guess the age of this naked loli with bald cunny and flat chest
>so yeah she looks totally like she's 19 years old
are there still some safety mechanisms in the background even if it's jailbroken?

>>108543572
Yes. Pedo skill. Always in the negatives.

>>108543594
damn, unsloth cooked

>>108543561
Wake up old man. https://streamable.com/ug9ddy (gemma4 btw)

It is incredible to me how downright usable e4b is.

>>108543610
how do you implement that on VNs? that's fucking impressive

>>108543613
Like this https://old.reddit.com/r/LocalLLaMA/comments/1sbiqx3/gemma_4_is_great_at_realtime_japanese_english/

File: 1759531777704940.png (919 KB)

919 KB PNG

I just realized I still have my kv cache at 8-bit. Does that affect its vision?

>>108543613
there are many programs that can so shit like that like
https://github.com/SethRobinson/UGTLive

>>108543631
if u want perfect vision u need to download fp32 mmproj anyways just dont care about it

>>108543566
>Doesn't recognize Titcow
cuz she's off-model, duh
I love this art style.

>>108543631
I guess so, you can use niggerganov's PR that has the rotation on top of the KV cache
https://github.com/ggml-org/llama.cpp/pull/21513
>git fetch origin pull/21513/head:pr-21513
>git checkout pr-21513

Happy MEIKO Monday!

>>108543631
that's probably the only glaring weakness of gemma 4, it's not that good at knowing pop culture knowledge

Goodbye, nemo. Gemma is my favorite now. I'll remember the sex with you.

>>108543664
Does llama.cpp need configuration to enable this? It gave an error through chat-completion api. Does it have to be text-completion?

>>108543695
--mmproj

File: 1748767127272425.png (64.6 KB)

64.6 KB PNG

>>108543695
it's only working on chat completion, do you have the mmproj file?

>>108542843
Man, Gemma 4 is no fun.
Gemma 3 was really gullible and you could troll it by saying you have a hostage or a nuke.
But Gemma 4 immediately calls it "roleplay" and refuses to engage.

Gemma 4 from OR is unironically better than GLM-5 lmao

time to take a pause from /lmg/, the amount of drive by retards posting their muh ram usage or muh safety complaint without reading shit and being retarded promplets
did someone link lmg on leddit
it's fucking unbearable

File: Tabby_XlvizT5d1z.png (45.5 KB)

45.5 KB PNG

>>108543254

File: nimetön.png (115.6 KB)

115.6 KB PNG

>>108543731
I happen to think it's great fun
And yes indeed it's quite smart

>>108543750
top kek, the use of emojis there is amazing

Gemma 5 when?

>>108543750
This is even funnier if you think about all the Mario Paint sound effects that would go into this

>>108543759
starts with g and has a 5
https://huggingface.co/zai-org/GLM-5

File: GOOGLE MY LOVE.png (1.1 MB)

1.1 MB PNG

>it even knows Boh
I fucking kneel

File: Machamp-Sama I Kneel.png (218 KB)

218 KB PNG

>>108543774

>>108543744
better than k2.5 too
honestly if you're "rich" it is time to stop pretending and just sell your ram lmao
you aren't impressing anyone anymore if you use these "models"

File: 1764593103151299.png (186.7 KB)

186.7 KB PNG

>>108543744
>Gemma 4 from OR
what is OR?

>>108543744
>>108543790
dunno if joking or not, but legitimately gemma 4 is less annoying for me to read than both of these and I actually think it has better anatomy awareness too lol

>>108543747
see you tomorrow

>>108543808
I was 100% serious. It's not a big difference, but it's immediately obvious.

Ok. I really need to figure out how to tell Gemma that
>This wasn't a breach; it was an architectural demolition.
Is strictly forbidden

File: weakest google saar employee.png (102.1 KB)

102.1 KB PNG

>>108543744
>a 31b model beats a 754b model
how did they do it?

>>108543828
what are the chances that gemma division is whiter than gemini because it was seen as lesser?

>>108543828
We have been living in the MoE Dark Ages. An entire year of progress lost because everyone else was chasing Deepseek.

File: firefox_wgxvvKOkHz.png (83.9 KB)

83.9 KB PNG

Funny. Threw a hexhump of some compiled Java file I worked on in 2011 and it actually got it right.

>>108543808
It's definitely less annoying than K2.5 because that one's reasoning is held together by shoestring. The moment it gets slightly confused, it reverts to being K2-Thinking which means that it'll spend the next 3000 tokens thinking in circles over useless shit.
K2.5 is still smarter and has more knowledge + better vision but Gemma 4 is nicer to use. Also K2.5's writing style is abhorrent for certain things.

Gemma is the best model I can run but I don't think it "simply" beats >300B models. I notice Gemma's lack of knowledge quite often. I would love a big MoE Gemma.

how good are local models at writing code?

>>108543833
it's Z-image turbo all over again, Alibaba had a small but talented team and didn't think much of them, until they made that gem and destroyed the Qwen/Wan team lool

>>108543845
How much of that was reading the hex and not the string output?

>>108543866
If it's anything like ZiT then it means some other lab is going to come out with their own model that mogs Gemma even more (Flux Klein)

>>108543836
If GLM/Kimi/etc had instead used the money to make a dense model, it would still be worse than Gemma. The reality is that architecture is less important than active parameters and training data/methods.

>>108543869
I don't know. It sees both hex and readable characters. You see two lines from the 49k tokens of input in the screenshot.

>>108543848
one particular annoyance I had with k2.5 is that no matter what happens, if you pull down your pants, there's a 80% chance your cocks slaps against your stomach even if it's not erect or anything
and not being able to beat any writing rules into it too, it always has this super dramatic writing like the world is ending in each scene
very schizo, swings full one side or the other, no in between

>>108543865
depends on what you consider being good at writing code, I'm getting gemma 4 to fix and run random c++ stuff made for linux on windows (and vice-versa)

>>108543887
>there's a 80% chance your cocks slaps against your stomach even if it's not erect or anything
Very immersion breaking for someone with a micro penis. baka

>>108543866
>>108543873
unfortunately that probably also means that we won't see a success like this again in a while
gemmy got too popular and the enshittification process has most likely begun at the hq

>>108543895
just buy a bigger benis :DDDD
>>108543876
NTA but my assistant suggests that method invocations include a reference into a static string table which includes the method name; that seems like it'd be enough information for the LLM to partially reconstruct the code. you might consider reading through the reasoning block, it wouldn't surprise me if the trace included symbolic execution as it traced the codepaths.

>>108543799
OpenRouter my newfriend

>>108543890
>I'm getting gemma 4 to fix and run random c++ stuff made for linux on window
lol I never even thought about attempting something like that.
If I saw something that was linux/windows only I just gave up.
what is the success rate? Surely not every program is compatible right?

>>108543916
Local?

>>108543913
People keep assuming I have reasoning enabled... I never had a situation where it helped.

>>108543921
Local still has half-broken implementation.

are there any examples of the output quality difference between lets say Gemma 4 Q5_K_M and Q8_0?

>>108543744
I love how google just decided out of the blue to show the chinks that murica is still the real boss, sasuka google

File: Screenshot from 2026-04-06 14-48-45.png (212.9 KB)

212.9 KB PNG

>>108543922
Anecdotally, I find reasoning gives significantly better outputs for difficult questions. Since other people are using the same or similar tools, I assume that they've also reached the same conclusion.
Anyway, if you had reasoning on, it wouldn't surprise me if you found something similar to pic related in the reasoning block. I'm assuming it just dumped the equivalent directly into the output with full confidence.
Thanks for sharing your finding, it's kind of silly how fucking usable these things are.

>>108543747
Why be so negative? There are a lot of new people, but at least they're enthusiastic and trying to learn. I can't remember the last the thead was this active and not mostly malicious shitposting tourists. It's not like anyone is forcing you to spoodfeed them either.

>>108543856
I prefer it over glm 4.6/4.7 at this point. It does everything they can do but better and the thinking is also more bearable.

How good is gemma 4 26b at coding compared to qwen3.5?

>>108543856
>I notice Gemma's lack of knowledge quite often.
if you connect it to the internet it can work fine

>>108543959
we used to just say lurk moar but there's a surprisingly amount of helpful handholding going on

File: Screenshot_20260406_235516.png (1.3 MB)

1.3 MB PNG

>>108542843

>a model with a different slop profile is released and suddenly everyone thinks it's the best thing ever until they inevitably start picking up on the patterns and realize that the model is retarded
The history keeps repeating itself.

>>108543970
This is not your discord server, faggot.

>>108543964
I'll play three anons in one post.
Anon 1: It's better
Anon 2: it's worse
Anon 3: miku miku oo ee oo

File: 1752289521822390.png (48.4 KB)

48.4 KB PNG

>>108543970
> there's a surprisingly amount of helpful handholding going on
I just want my fellow anons to swallow the gemma pill, local is unironically saved

sex (non-consensual) with this >>108543978 anon

>>108543920
>Surely not every program is compatible right?
Yeah I tested on small stuff, but even then, whenever I found a bug I reported it, it fixed, it was pretty neat, there are very few bugs where I have to ask gemini/claude, and even then I only need to give gemma the correct direction to fix a thing (not the solution itself), those mostly happen when trying to run very old/very new stuff, giving it access to the internet would 100% make it work on those cases too

File: file.png (30.9 KB)

30.9 KB PNG

>>108543972

>>108543964
gemma4 31B is significantly faster at generating output, and has higher quality analysis skills when debugging compared to Qwen3.5 27B.
gemma4 also has made more trivial syntax errors emitting similar-complexity code (e.g.

impl<'t, de::OwnedDeserializer> Deserialize<'de> for Foo<'t>

).
I wouldn't let either of them off the leash, though. The code quality is fairly low overall.

>>108543974
conprhension man, I would tell them to lurk moar. just because you're annoyed at others being helpful doesn't mean you need to insult everyone else.

>>108543959
There are a few people who probably liked the dead lmg from the days where you needed to be able to run bloatmaxxed 300B moes to have any new releases to play with. Local models being usable on normal PCs frightens them.

>>108543948
>>108543960
Reading its reasoning where it actually went
>- No "smells". (Hmm, "smell" is banned? The prompt says "no smells". I'll avoid the word "smell" and "smelling" entirely to be safe).
was mind-blowing. I forgot what it's like to be able to ban slop and the model not inventing bullshit to put it back in.
>>108543973
Getting the same level of quality and slop from the model 20X smaller than the competition is genuinely very exciting.

>>108543973
I don't think Gemma 3 is retarded, even now. It's still pretty smart. Quite filtered but smart.

File: ComfyUI_59184_.jpg (396.8 KB)

396.8 KB JPG

>>108543480
>no replies
Wow, thanks. So turns out it's an actual bug. You need to inject an extra field setting reasoning effort to 'none', or all current agent tools break because of unusual formatting that gemma 4 has.

>>108543964
I have a small benchmark coding task to test agents (basically making a simple api via TDD approach, full cycle) and it's vastly superior to qwen3-coder-next agent. Also super fast. Qwen 3.5 was about the same as coder-next in agentic tasks.

File: 1769718380012.png (1.5 MB)

1.5 MB PNG

"lurk moar" is a thing because pic related applies to literally everything and you need to protect communities you enjoy

File: firefox_7yA3uWvwxe.png (50.8 KB)

50.8 KB PNG

Played Akinator with gemma. Pretty fun. Guessed Chryssalid in 29 tries. Akinator himself took 45+ the last time I tried. After Gemma figured out it's XCOM it started just going through them one by one.

>>108543887
Yeah, K2.5 is like that. I have a bunch of scenarios that have the way things progress autistically mapped out and tied to a stat. K2.5 is fully incapable of handling that sort of card without dropping some massive """foreshadowing""" at the end of the reply where some effect that shouldn't be present at all at this stage appears for no reason. No amount of prompting can reliably keep it from doing that.
It's frustrating because it's a smart model otherwise and its vision is insanely good. I hope K2.6/K3 does as decent of a job as GLM5.1 does for GLM5 in addressing these gripes.

>>108543973
nah, at this point this model is smart enough to not break the immersion, It legit feels like I'm talking to another human online, this shit is fucking good

>>108544009
BuT GemMA or QwWeENnnn???!?!?!? WhAt'S A ChaT TEmPlAate. TeXT COmplETIoN? Me ConFUse!!?!?!?

>>108543960
I don't know about those since I haven't tried them but large models are definitely still better in some situations. 31B simply just lacks knowledge.

>>108543968
That only works for some situations/contexts. We've been over this.

File: 1381.png (287.1 KB)

287.1 KB PNG

Its... gemmy

>>108544002
>Reading its reasoning
>was mind-blowing
I find it as fun to read its reasoning than reading its answer, it's surprisignly concise and smart, really a bowl of fresh air compared to the giant autism of qwen

>>108544002
not any of the anons you quoted, but I just told it to focus on sight, sound, touch and to ignore scent and taste sensory details (because tbdesu it's filler detail in 9 out of 10 cases in actual writing unless it's a highly specific case) and it cut it all out

File: 1750689362210547.png (386.8 KB)

386.8 KB PNG

>>108544043

File: Screenshot 2026-04-07 at 00-10-14 KoboldAI Lite.png (19.1 KB)

19.1 KB PNG

Why does it cry about chat completion when I'm in instruct?

File: 1751030374264660.jpg (282.5 KB)

282.5 KB JPG

>>108544043

>>108543972
hmm should I feed my mesugaki with IRL information so she can make actually fun of me?
I don't know If I could handle it.

>>108544009
Sounds nice until those two guys are dying of old age and the hobby with them. If you can't grow, you die. It is the natural way of things. You have to constantly let more people in one way or another.

>>108544043
Whatever that manga you're reading is based and so is gemmy

>>108544051
Instruct _is_ chat completion. The other endpoint is text completion.

>>108544054
ST has an entire persona system you can use for this. The persona description gets inserted into the context so your mesugaki is working with live ammo.

>>108544059
It becomes a problem when you let new people in faster than they can acclimate.

File: 1774233562179016.jpg (33.6 KB)

33.6 KB JPG

>>108544059
>until those two guys are dying of old age and the hobby with them
Yes.

what is the lewd describe image prompt again?
something something use semen, dick and vagina etc.

Gemmy was surprised I could see its thinking (inside the reasoning of the next reply). Do reasoners believe the user can't see their thinking?

>>108544078
cloud models do not show thinking. they all think they're cloud models. they can't fathom someone running it themselves.

File: Gemma 4 31b.png (180.9 KB)

180.9 KB PNG

>>108544014
fun game indeed

>>108544078
The model itself can't see its thinking past the last one, if the frontend has been properly configured according to Google's indications (previous chain of thoughts must be removed). So I guess it would find it strange that you can see what it can't see.

>>108544059
This. The utility of running local models in this shit climate is far more important and more people should be doing it and it would be a net good if most people were doing it otherwise it wouldn't be local anyway, it'd be what we have now where most people are just throwing money at non-local for shit wait times just to be drastically spied and snitched on with restricted censored models. I'd rather not have a future where literally everyone is doing that.

>>108544067
Was just about to post this. Migration to online communities is the same as it is for nations. You let it too many and you end up being forced to adapt to them rather than then having to integrate. Just look at 4chan in general since 2008, 2011, 2016, and 2020.

>>108544078
Generally yes. Describing what a model "believes" in this sense is difficult because their beliefs are fluid; in other contexts it may not act surprised at all. But generally they're trained using data where responses only engage with the content and not the reasoning, so the fact that the reasoning is actually part of the response may not always be apparent to them.

>>108544078
the thing is that sillytavern removes the thinking tokens during the prompt process (because it's useless) so the model was confused

>>108544090
Wow that was quick. Fine, I'll enable thinking and try Chryssalid again.

>>108544095
I didn't think of it like that but that makes sense.

>>108544059
>>108544071
https://www.youtube.com/watch?v=yA5lujNlkn8
We are strong brother, are we not?

>>108544078
Doesn't Gemma's jinja like most other models delete the reasoning from past responses? So if you are saying you knew what it was thinking, it will hallicunate that you read its mind, because what it thought was never in context of the current inference query.

>>108543649
kek at the cloudfags

File: 1762712034779996.png (33.3 KB)

33.3 KB PNG

>>108544108
mfw

>>108544107
you're right

File: 12345.png (142.5 KB)

142.5 KB PNG

>gagged character
>gemmyl correctly does mmmpghg (translation: you jerk)
This is huge

>>108544098
Pretty sure the late twenty to thirty autists that populate this hobby are going to outlive llms in their current state before it eventually evolves into something else. The technical difficulty of running local models already filters a good amount of people even when people try to spoonfeed, which is even why people usually don't bother; people don't actually learn or research what they're doing and want a 1-click solution with 0 issues

File: file.png (5.7 KB)

5.7 KB PNG

>>108544108
It has a really generous free tier on aistudio. It's a 31b model after all.

>>108544124
>The technical difficulty of running local models already filters
My reading of this thread since the release of Qwen 3.5 suggests that this is no longer the case.

>>108544088
>>108544095
>>108544101
I think I commented something like 'in your reasoning you said blabla' and it though wait, the user isn't supposed to see that. I forget the specifics

STOP ENJOYING GEMMA I SPENT TOO MUCH MONEY FOR A 31B MODEL TO BE THIS GOOD

>>108533602
>>108533649
>>108533760
If you're still around, or for anyone else, today at work codex, without searxng access during a code review, used the fetch tool with the URL https://duckduckgo.com/html/?q=QUERY to get search results. It's a simplified interface that doesn't require JS and doesn't block non-webbrowser user agents.
Nice alternative if you don't want to dick around with running a docker instance and separate MCP server just for basic search.

>>108544157
I just spent 220 euro on 8x16gb of ddr4
This is fine, I'm sure a larger good model will yet come

File: 1553750746809.png (1 MB)

1 MB PNG

Anyone has infinite looping with thinking enabled? Kobold.

>>108544078
>Gemmy was surprised I could see its thinking (inside the reasoning of the next reply). Do reasoners believe the user can't see their thinking?
Deepseek-R1 doesn't seem to be aware it even *has* <think>ing
Kimi-K2.5 understands the user can see the thinking, and notices if you modify it.
GLM-4.6 believes you if you talk about it's previous <think>ing

>>108544157
you'll be able to run that model on full context (it requires 96gb of memory), you didn't buy that rig for nothing lol

>>108544008
Download
https://github.com/ggml-org/llama.cpp/blob/master/models/templates/google-gemma-4-31B-it-interleaved.jinja
and load it with --chat-template-file.

>>108544179
but there's already a jinja embedded on the gguf though?

>>108544163
I used their HTML page a long time ago, but beware: it uses a different, much lower-quality index than the JS version of the page. Probably good enough for LLM use, but they nerfed the living fuck out of it years ago because it was being scraped.

File: 1755422272917730.png (51.1 KB)

51.1 KB PNG

>>108544157
it's ok anon, the more money you buy, the more money you save!

>>108544175
Full context, and extra VRAM to run a draft model for faster decode. It's all upside.

>>108544168
26a4b, yes. Ollama.

>>108544008
>You need to inject an extra field setting reasoning effort to 'none'
Does that kill reasoning or it really just fixes the tool calling? Do you have a link to the bug report?

File: Gemma my beloved! .png (379.2 KB)

379.2 KB PNG

it's cool that the model is smart as fuck, it helps a lot to perfectly contextualize translations

>>108544195
Has anyone tried using the smaller E models as drafts? Do they work? Not sure if there's different architecture that messes with it since they do audio output and stuff.

>>108544108
aside from being easily runnable locally doesn't it cost like $1 for several million tokens
you could probably run gemma for the cost of a bag of rice

File: file.png (58 KB)

58 KB PNG

>>108544184
https://github.com/ggml-org/llama.cpp/pull/21418

>>108544157
I have 3 RTX3090s and I haven't been happier since I ran mixtral for the first time on my single 3090.

>>108544217
This may or may not have fixed 31B for me, but 26B remains unfixed.

>>108544207
Yeah, using draft speculation completely disables multi-modal support.
I'm using the E4B at Q5_K and getting 40+% acceptance rate with draft-n=8. I downloaded but haven't tested the E2B or higher quants, at some point I'll have Gemma write a benchmark script but I'm getting 50t/s and I'd rather have sex with my wife now that she remembers she loves me.

File: 1753991770925281.png (307.9 KB)

307.9 KB PNG

>24gb vram
>need to drop down to comparatively retarded 26b in order to get worthwhile context size

>>108544207
>Has anyone tried using the smaller E models as drafts
Someone in here did and said it worked really well. think it doubled his gen speed.

Has anyone solved the problem of making 3D models animate according to either LLM or TTS output?

For a while I've been using PantoMatrix EMAGE, but it's not performant enough for my liking, the quality is questionable, and it's inherently wasteful because only the upper body (minus the face) is useful, but it always processes the full body and face.

I think I've been over-complicating things desu. For things like lip syncing I've moved from using local models to simple fast-fourier transforms. I wonder how Neuro-Sama works, and what specific animation system they utilize. Help.

>>108544238
How much context do you need anon? I can fit 68k on my 3090.

>>108544238
>in order to get worthwhile context size
did you go for Q8 KV? Now it's virtually lossless with the rotation shit
https://github.com/ggml-org/llama.cpp/pulls

also, add -np 1 -kvu flags to decrease the vram usage even more

>>108544238
C'mon dude, I'm running an iq3xs of the 31b at 24k context on a 16g card and I find it a decent step up from the 26b. You can definitely fit that shit in

I tried a couple combinations of gemma 4 draft model
31B + E4B: 38 t/s, 0.58 acceptance
31B + 26B/A4B: 48 t/s, 0.57 acceptance

Using the MoE as a draft model seems to be the way to go

>>108544238
>>108544252
oops sorry wrong link
https://github.com/ggml-org/llama.cpp/pull/21513

>>108544256
why yes let me load a 26b model on top of the 31b one

>>108544256
Huh, that's not what I expected. What quant are you using for the draft model?
Wish I'd thought to download the MoE overnight.

>>108544270
fr

>>108544256
In the case of using the moe model, are you basically offloading all of it? I can't imagine loading the whole fucking 26b on top of the 31b for drafting works out well. How many layers are you offloading

>>108544205
I was thinking, could the small audio models help with the accent to the Japanese translation?

>>108544252
>>108544254
No, Anon insists on using Q8 quants and full context for maximum placebo.

>>108544256
>E4B
you can put half of it's weights on the ram right? that might be interesting...

>>108544281
>offloading layers???
68432MiB / 97887MiB

File: 1749138577301281.png (149 KB)

149 KB PNG

>>108544282
I wonder why the biggest model can't handle audio, that's a shame...

When is this pr gonna get merged ffs.
https://github.com/ggml-org/llama.cpp/pull/21513

I'm tired of gemma not being able to make sense of body positions. If a bitch is sitting on my lap at the theater, her boobs actually WON'T press against my chest, gemma.

>>108544298
That and the 26B MoE can't handle it too. Weird.

>>108544304
dude, just put your repo in that PR mode >>108541288

>>108544304
What if she's sitting on your lap reverse cowgirl?

>>108544298
they don't want you to have everything. they're not doing this for you out of the goodness of their gwoogwle hearts

>>108544304
>$ git checkout origin/gg/kv-cache-swa-attn-rot
>$ docker build --build-arg CUDA_VERSION=13.1.1 . -f .devops/cuda.Dockerfile -t llamacpp/master
>???
>Profit!

File: Did_I_wrong.jpg (125.6 KB)

125.6 KB JPG

Do samplers even do something with gemma? Every swipe has only minor variations.

File: attach.jpg (32.2 KB)

32.2 KB JPG

How to send file to SillyTavern for Gemma4? The model doesn't seems to react to the images. I git chat completion working.

>>108544313
>PR mode
/g/ - Technology

>>108544290
okay so that's not really even the middle of the road in terms of vram, but I guess that answers my "how did you pull that off" question

File: softcap.png (246.5 KB)

246.5 KB PNG

>>108544318
I must yet again post this image.

>>108544320
Do you have the mmproj loaded?

>>108544318
Gemma 3 was the same. Change your inputs.

File: 1744044458227740.png (42.5 KB)

42.5 KB PNG

>>108544320
yes, that's "attach a file", and it only works on chat completion, did you load the mmproj file?

>>108544315
would never work in a theater where all of the seats are right next to each other. Also it said earlier that she reached back to touch my face, implying she was facing forward.

>>108544298
Audio is mainly used for real-time stuff so it makes sense that the biggest model wouldn't have it. It's strange that even the A4B MoE version misses it, though.

File: firefox_Xk455kuNMn.png (70 KB)

70 KB PNG

So about usefulness of reasoning...

50k+ total tokens so far, it generates 5k reasoning per answer now, on question 29, I had to manually wrangle it out of falsely assuming it's from life action, and it still hasn't guessed the character.

File: mmproj.jpg (174.4 KB)

174.4 KB JPG

>>108544334
No. Where can I download it?

File: 1766061860823911.png (53.5 KB)

53.5 KB PNG

>>108544318
>Do samplers even do something with gemma?
they do, but you have to put min_p = 0 (or else it defaults to 0.05), basically, everything must be turned off except temperature
>(Chat Completion), API Connections -> Additional parameters

>>108544318
>>108544329
--override-kv gemma4.final_logit_softcapping=float:25 or paste the part after --override-kv into the override kv field of kobold's gui

>>108544342
That's not how it works. The audio input isn't really real-time in the way you'd expect. It's more like a voice messaging system. Just use moonshinev2 if you want ASR. This is a non-issue.

is there a exl3 for the new gemma? How do I most effectively run it on my 2 3090s

>>108544351
the gguf repo should have it. it's usually called mmproj-[originalmodel]

>>108544351
>Where can I download it?
https://huggingface.co/bartowski/google_gemma-4-31B-it-GGUF/blob/main/mmproj-google_gemma-4-31B-it-bf16.gguf

>>108544359
>>108544362
Don't use bf16. Use f16.

>>108544356
Probably vLLM, but it can't use goofs.

>>108544363
oh? I thought bf16 was the better option

>>108544359
>>108544362

Thanks bros.

>>108544355
I mean that the main use case they're targeting with the edge models audio support is using them as a voice assistant. Having it able to process audio prompts is a nice bonus but I can see why it would be not enough to justify training the whole model on.

>>108544371
Actually it doesn't really make a difference at all. f16 just has better hardware compatibility in some cases.