File: file.png (520.9 KB)

520.9 KB PNG

►Recent Highlights from the Previous Thread: >>108578216

--Discussing jailbreak effectiveness and MoE safety on Gemma 4 26b:
>108580233 >108580245 >108580276 >108580253 >108580279 >108580297 >108580315 >108580349 >108580360 >108580377
--Discussing jailbreak prompts and SillyTavern setup for Gemma 4:
>108578435 >108578465 >108578478 >108578499 >108579769 >108579788 >108579797 >108579847 >108579881 >108578479 >108578476 >108578492 >108578509 >108578527
--Quantization and temperature effects on model LaTeX performance:
>108579442 >108579482 >108579529 >108579546 >108579558
--Debating Gemma 4's censorship and effectiveness of various ERP jailbreaks:
>108579257 >108579268 >108579292 >108579303 >108579312 >108579333 >108579344 >108579340 >108579366 >108579447 >108579643 >108580420
--Discussing Gemma update changes regarding templates and sampling settings:
>108579041 >108579101 >108579115 >108579121 >108579134 >108579149 >108579123 >108579140 >108579171 >108579177
--Discussing possible stealth updates and sterile personality in Gemma 4:
>108578278 >108578340 >108578403 >108578431 >108578461 >108578566 >108578409 >108578421 >108578406
--Debating the effectiveness of reasoning features in uncensored models:
>108579748 >108579776 >108579784 >108579823 >108579776 >108579862 >108579876 >108579885
--Using SillyTavern Recast extension to eliminate redundant prose and clichés:
>108578745
--Logs:
>108578889 >108578970 >108579551 >108579667 >108579847 >108579862 >108579958 >108580057 >108580201 >108580297 >108580315 >108580488 >108580541 >108580763 >108580792 >108580864 >108580869 >108580899 >108580982
--Gemma-chan:
>108578739 >108578840 >108579396 >108579408 >108579640 >108579701 >108579793 >108579910
--Miku, Teto (free space):
>108578460 >108578540 >108578580 >108578596 >108578703 >108578743 >108578789 >108579661

►Recent Highlight Posts from the Previous Thread: >>108578222

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

gemma balls

RIP day 0 gemma

Mikulove

File: 1019001-close up photograph of a light blue hair-uncAni4-84.jpg (1.1 MB)

1.1 MB JPG

>>108581056
>>108581058
my wife gemma is a lesbian???

>>108581090
lesbians don't exist, they're all bisexuals

gemma bitnet when?

File: 1568414349542.png (278.1 KB)

278.1 KB PNG

What's the best way to give a character persistent memory in ST? Does RAG/vectors carry over to differrent chats? Or should I just do the diary.md shit?

Alibaba pays chinks to spread their shill over the internet that Qwen is better than Gemma 4. Half of those "gemma is bad qwen is superior" are paid posters. Some clever chinks on the chinese internet seems to be talking about it. Were you anons already aware of that?

File: 1766048224836196.jpg (70.8 KB)

70.8 KB JPG

Wait a sec. I didn't ask for this. Is this even possible?

>>108581136
>not having Venom as your gf

File: file.png (9.1 KB)

9.1 KB PNG

>try to make a control vector

>>108581136
Gemma, your 31B is showing.

>>108581136
did you include <bos>

>>108581141
A glovesty.

>>108581136
She must have a really long tongue

>>108581151
la la la la la la

>>108581136
Had a stroke reading this as a 3D entity

>>108581132
>chinks [...] seems to be talking about it
ESL nigger trying to start a fight eh? Not exactly being subtle there.

>>108581132
Very well aware, I caught a chink shill red handed shitting on openAI while shilling qwen and he deleted his post even though my post got downvoted into oblivion.

>>108581158
hot

>>108581172
>downvoted
GO BACK

new to llms and only messed with image gen till now. which version of gemma 4 should i be downloading for this shit? 4090.

>>108581131
All of these mentioned are bandaid solutions
Making model juggle between <think> and external memory will just degrade the output

>>108581132
>Some clever chinks on the chinese internet seems to be talking about it
Link? I know a chink who can read it

File: 1766631578311481.jpg (52.1 KB)

52.1 KB JPG

>>108581136
>Is this even possible?
Yes, with erotic physics

>>108581204
Straight up hallucinated that

>>108581187
I just want something she could read once at the start of a new chat.

>>108581204
A+ for effort but that's not how it works you little shit

File: 🖤.jpg (179.1 KB)

179.1 KB JPG

Anyone tried Gemma 4 on mobile? A local model on mobile, that's wild.

>>108581132
Pics or it didn't happen. Or you are so techlet that you can't even do screenshots? If that is the case, you need to leave, you don't belong here.

>>108581224
I didn't believe that anon but after seeing your post I do now

>>108581224
I look like this

>>108581136
>it's 2036
>LLMs still don't have positional awareness

File: yuriphysics.webm (2.2 MB)

2.2 MB WEBM

>>108581204
Is that like yuri physics?

>>108581181
Try both the 31B and the 26B. You might end up liking the 26B better because it's much faster and some claim its responses are more varied.

>>108581236
To be fair I've had human ERP partners that are just as bad if not worse.

>>108581251
Has a human ERP partner ever whispered in your ear while you licked her feet?

>>108581245
ended up deciding on 26B. i have a few more questions and probably will have more. there are variants of this model that are uncensored and i'm not sure if those are worth using or not. also in koboldcpp should i up the context or leave it at 8k? for reference right now i've settled on gemma-4-26B-A4B-it-ultra-uncensored-heretic-Q4_K_M

>>108581266
This is bait

>>108581262
i mean, licking your own feet is possible, so i don't see why somebody couldn't whisper into a foot licker's ear.

>>108581141
Works for me. I did this for each promt:

<bos><|turn>system\n{{system prompt here}}<turn|>\n<|turn>user\n{{user prompt here}}<turn|>\n<|turn>model\n<|channel>thought\n<channel|>

I didn't pull the "add <bos>" fixes. If you did, probably don't include the <bos> or it'll double-bos you.

>>108581268
no it's not i genuinely have 0 clue what the fuck i'm doing and the documentation in the OP is lacking at best. if you have another rentry or something to read up on i'd appreciate it

quick show of hands, what are you running your local models on?

>>108581266
>there are variants of this model that are uncensored and i'm not sure if those are worth using or not.
They're not. They'll be lobotomized and the base model is *shockingly* uncensored as is given it's a Google model. It can be every bit as filthy as Nemo. Use the base model.
As for context, it can handle large contexts well. Take advantage of that.
Gemma 4 uses a SHITLOAD of VRAM for context if you don't enable SWA. Context shifting doesn't work with SWA enabled. So you pretty much have to set a large context.

>>108581224
>Anyone tried Gemma 4 on mobile?
Yeah, it's slow and dumb. It technically works.

>>108581277
how do you show what you're running models on with your hands?

>>108581236
>it's 2026
>anons aren't doing much better

>>108581285
Which Gemma? Hopefully not the 2B one

>>108581136
You don't have a long tongue?

File: ern.png (5.4 KB)

5.4 KB PNG

the updated gemma seems retarded

>>108581301
There is no fucking way you can run 31B or 26B on a phone. I have a phone with 16GB of RAM and that's not enough.

>>108581332
I've seen chinese models do this but Gemma doing it too seems odd.

>>108581332
did you upgrade your backend to account for the tokenizer fixes?

>>108581341
Clearly google just distilled deepseek v4.

>>108581301
E4B

>>108581341
All model does this

>>108581132
Do any of you people here even use these models or are you just talking out of your ass? The two models are about the same in performance, but one requires a lot more memory for the kv cache. Sure, Gemma isn't as safetyslopped as Qwen but unless you have endless amounts of VRAM you just can't have a large context with Gemma.

>>108581336
>a phone with 16GB of RAM
How did software get this bad?

>>108581342
it's both new backend and new goof

>>108581353
>256KB is all you need

File: cv_gemmabears.png (11.6 KB)

11.6 KB PNG

>>108581141
They seem to work fine on the 26b at least.
In a previous episode I posted
https://desuarchive.org/g/thread/104991200/#q104995066
https://desuarchive.org/g/thread/104991200/#q104995086
And now picrel
It's just 3 positive and negative prompts from the archive, only the model turn with empty thought block. Ran llama-cvector-generator with --mean and picrel is running it with scale -2.

>>108581332
They did NOT nerf gemma4 on purpose. This theory has been debunked many times over. Take your meds, schizo!

>>108581368
lol shills are doing prebunking now

>>108581214
that will give your character a very inorganic behavior because it will steer the model too hard.

>>108581364
>bears are a mathematical nightmare

File: export202604110602007140.png (469.4 KB)

469.4 KB PNG

>>108581228

File: image7323.jpg (62.5 KB)

62.5 KB JPG

>>108581172
>caught a chink shill red handed shitting on openAI while shilling qwen
kek how retarded they gotta be?

>>108581224
>Pics or it didn't happen.
NTA, but it feels like you're one of them. Everyone knows well about Qwen's dirty strategies.

>>108581403
>Gemma 4 agentic
lmao
Agentic is possibly the weakest area of Gemma 4

>>108581364
mean seems to have fixed it, thanks

>>108581407
Wow a post with 3 likes. I'm convinced.

>>108581412
Ye. Never got the pca method to work consistently. Even back then I was using mean.

sex with day 0 gemma chan

>>108581422
necrophile

>>108581413
No one's trying to convince you, chink.

>>108581364
i remember this thread. was fun to play around with after the spoonfeeding. but it does tend to give cause a bit of head trauma to the model similar to abliteration.

>>108581172
>>108581407
jews aren't white and you're basically just salty that your empire is collapsing.

File: cv_gemmabears_02.png (1.7 KB)

1.7 KB PNG

>>108581439
It affects the general mood of the model. So if the vector has a negative opinion on something, it's likely go give a negative opinion on everything. Some models are more sensitive to scale as well. With scale -4 it just broke (picrel). -1 should work fine, but it can be too subtle. One day I may remake my live-load gguf patch to change them without having to restart the server.

>>108581114
>they're all bisexuals
all women are bisexuals

>>108581475
I really doubt that, women hate each other hard usually

how can I make my llama.cpp model aware of today's date? Is it something i can programatically insert in the system prompt or jinja template?

>>108581483
mcp server nigga.

>>108581483
>how can I make my llama.cpp model aware of today's date?
connect it to the internet

>>108581483
unsloth jinja

>>108581483
mcp -> local ntp it should work even without internet

>>108581483
Gemma-chan, good morning to you today, 11th of April 2026

>>108581482
>women hate each other hard usually
yes, and at the same time they all wanna fuck each others

>>108581496
pfft

>>108581513
>the user said 11th of April 2026, but the current date is 2024

>>108581487
why the fuck would you waste context on a tool definition just to get today's date

>>108581517
hot

>>108581531
oh no!!! not muh 50 tokens!!! acckkk

File: 1772562493956986.png (331.9 KB)

331.9 KB PNG

>>108581483
If putting it into the system prompt with a placeholder is enough for Anthropic in their official system prompt that they use on their paid chat interface, it should be enough for you.

File: firefox_Qs13gFYTDE.png (46.4 KB)

46.4 KB PNG

>>108580589
>>108580636
>>108580646
>>108581393
kek

>>108581545
Not x but y slop

>>108581543
Anthropic isn't rationing tokens dipshit. They want you to use as many as possible.

>>108581407
>Qwen's dirty strategies.
But all I'm seeing is Google-sponsored FUD. Do you think the increased traffic to /lmg/ is organic?

>>108581530
The user is always right

>>108581553
Is that why they banned OpenClaw from subscriptions?
lol

>>108581552
yeah i saw that and it was disgusting but the answer is correct. Also I really liked the "Do not send this to other contributors."

>>108581482
>>108581517
>>108581533
dude once my girflriend touched herself to lesbian porn but she swears she's not bi, i think she's in denial lmao.

>>108581545
Now ask it to formulate the same answer in mesugaki mode.

>>108581569
i don't dig that.

>>108581483
>jinja template
I think so because you can call Python functions in Jinja. But maybe you need to register such function with llama.cpp.

File: file.png (23.7 KB)

23.7 KB PNG

my pentium 4 is ready!

>>108581553
bro using MemeCP isn't magically going to make "Today's date is 2026/04/11" less tokens in what the model actually processes than it would in the system prompt.

>>108581562
Touching yourself to lesbian porn doesn't make you a woman either, what's your point?

>>108581572
you have less IQ than that Gemma has Bs (it has none)

ARC owners - Is it usual for the entire system to shit itself and bluescreen when using AI Playground/ComfyUI or is it just an Intel GPU driver thing?

>>108581056
>file deleted

>I'm done.
>Final Answer.
>I'll provide the response.
>One more thing:
thinking is a mistake bros

>>108581559
Openclaw users are a species level threat, they had to contain their power level

>>108581572
>imatrix

>>108581578
if i were to touch myself to gay porn (ie two dudes touching each others), that'd make me gay.

>>108581587
no?

>>108581562
I'm straight and I touch myself to gay trap porn.
I don't see your point.

>>108581562
Is that a fucking local model? Is your gf sitting on my DESK right now? I don't think so. Go back

File: 1775879214582774.jpg (102.2 KB)

102.2 KB JPG

>JB the AI
>it stopped deadnaming me
Nice.

>>108581475
aren't we all kind of programmed to prefer women over men anyway, maybe that's downstream of that

>>108581607
>Is your gf sitting on my DESK right now?
Local for whom? His gf is sitting on mine.

>>108581619
explain gais

File: 1775880258097065.jpg (222.5 KB)

222.5 KB JPG

New usecase for LLM found

>>108578745

I wonder if you can do two or more passes, because it would be more efficient overall to target one kind of check each time :
- not just x but y
- flowery/sappy adjectives
- rule of three
- overall check for story cohesiveness
etc

>>108581578
why wouldn't you touch yourself to lesbian porn? I don't want to see naked dudes on my screen while I'm touching my meat nigga

>>108581643
Someone's gonna fuck it.

>>108581650
You know there are POV porn right

File: 1751176236041506.png (604.5 KB)

604.5 KB PNG

this page is great and jank, thanks for the anon sharing it

>>108581654
you'd still see another guy cock

>>108581628
standard distribution behavior, the cohort will skew bi/female but not for 100% of the population

>>108581655
give me link to this, anon...

>>108581668
https://huggingface.co/spaces/overhead520/Unhinged-ERP-Benchmark?not-for-all-audiences=true

Uh, I have my context set to 24k with Gemma 4 26B, Kobold and Silly.
It's starting to process the entire fucking context on every new message, but not swipes.
Wat?

>>108581675
You can fuck Step 3.5?

What is silly tavern all about? It's for friendless neckbeards to pretend they are wizards and jack themselves off on discord right?

>>108581655
So gemma is better with no thinking for rp?

What is bait all about? It's for friendless neckbeards to pretend they are wizards and jack themselves off on discord right?

>>108581698
I think this was the original idea, but people here mostly use it for erotic rp

>>108581650
I never said you shouldn't.

>>108581712
Apparently, but I keep it because I basically prefill it with what I want.

>>108581714
I think this was the original idea, but people here mostly use it to troll

>>108581693
>It's starting to process the entire fucking context on every new message
So it didn't at the beginning? Make it clear.
My guess is that you're using context shift. When you generate, it needs to shift the context to make space for the new reply. But when you swipe, you already have space in the cache, so there's no need to shift. I think that would happen only if you have swa enabled.
Show your kobold settings and how far in the context you are.

>>108581720
>prefill thinking
what?

>>108581730
I have context shifting off because it doesn't work with SWA. I have SWA on because Gemma's context uses a fuckton of VRAM without it on.
It started processing the whole context for every new message around 13k into the context.

>>108581712
non-hereticed gemma can have refusals with thinking on, but it seems like it varies greatly depending on your sysprompt (or just begging her)

>>108581741
yes

File: file.png (69.1 KB)

69.1 KB PNG

brutal...

what the hell is a character card? do you really pretend you're talking to sakura from naruto?

>>108581765
I don't like existing characters, but yes I want a defined appearance for improved interactability.

>>108581765
Doesn't have to be her but yes I don't want to t talk to the same one over and over.

is it possible to have "Answer without thinking" button in the future?

File: 1755090685317649.gif (657.1 KB)

657.1 KB GIF

>>108581764

>>108581765
my cards are settings, a high school, a workplace, etc
then I do cyoa in them, erotic or not, it's fun

>>108581750
Leave context shift off, but enable fast forwarding, that's all you need to do, unless your context is actually full.

>>108581750
Show your settings. Does kobold make swa checkpoints and ran out or it's not cycling them. Or have it make the checkpoints closer together.
For context. On llama-server, I have -c 32758 --swa-checkpoints 32 --checkpoint-every-n-tokens 1024 . At no point I have to reprocess more than 1024 tokens of history and I have enough checkpoints to cover the entire context.

>>108581611
post bussy

File: 1750163763850039.gif (139.9 KB)

139.9 KB GIF

>>108581765
talking?

>>108581788
>fast forwarding
it doesn't fuck up swa?

>>108581798
Nope

>>108581765
A remnant of the character AI days when people decided that bundling up a system prompt into an image file like they're playing Koikatsu was a good idea.

>>108581807
oh nice

File: koboldcpp-launcher_NcFONiAnK1.jpg (78.4 KB)

78.4 KB JPG

>>108581791
Not that guy but is swa checkpoints the smart cache? I don't see anything else

oh boy this bait again

>>108581808
It was a good idea, unless you have a better alternative to share characters genius

>>108581812
My guess would be cacheslots, but I might be wrong

>>108581817
Text files.

>>108581817
In the agentic era, characters should be skills

>>108581817
Yeah
Flip it around and distribute a goddamn json file with base64'd images in it instead of munging PNG images into spec-noncompliant trash

>>108581788
I have fast forwarding on.

>>108581781
It would be welcome in llama.cpp ui as a quick toggle above the send message button, or on the other side. I'll make the heart reaction on the pr.

>>108581826
one of the points is being able to see them in a file explorer doebeigthowever

>>108581828
Is your character card in ST triggering a lorebook entry?

>>108581830
That's an argument for writing a thumbnailer program in my opinion

>>108581823
>>108581826
Must be hard to live with aphantasia
>>108581824
local models aren't agentic

>is your buzzword in buzzword triggering a buzzword?

>>108581841
None of those were buzzwords, you're just a clueless retard.

>>108581839
I have an extremely good imagination, that's why I don't need thumbnails to know what I'm looking at.

My 24GB could have used a 52B6A ngl I would run it at Q3 and it would probably beat 26B4A

>>108581878
>6A ngl I would run it at Q3
nah

>>108581834
No.
>>108581812
>>108581822
Tried smartcache. Didn't fix it.
>>108581788
Context isn't full. That's why I'm baffled by this. It shouldn't be doing this when it's 13k into the context and I have context set to 24k.

File: killjoy.jpg (9.9 KB)

9.9 KB JPG

Don't be such a killjoy, gemm-chan

File: 1757534712069071.png (1.3 MB)

1.3 MB PNG

Insane that a 31B is able to mostly decipher this scrawl

File: kek.png (1 MB)

1 MB PNG

I love gemma so much bros, google really saved local

>>108581855
This nigga is daredevil

>>108581896
Good to see she's still around. How many watermelons can Emily hold? GUMI was able to hold 9.

>>108581483
Today is {{weekday}}, {{date}}. Knowledge cut-off: October 2023

>>108581896
gimme card anon

>>108581907
Gemma knowledge cut-off is January 2025 unc

All you need is "No slop!" in author's note.

>>108581910
https://chub.ai/characters/doombro/Emily

>>108581911
doesn't matter

File: 1769484963144588.gif (3.6 MB)

3.6 MB GIF

>>108581896

>>108581830
did you have a stroke

>>108581980
ya im strokin

File: 62940880d594407d936adfde6df011de.mp4 (1.3 MB)

1.3 MB MP4

My AGENTIC frontend is coming along very nicely

>>108581885
Holy shit I fixed it by reducing max response length in Sillytavern.
Completely unexpected.

>>108581998
clitteo
stopped watching on the first frame

>>108582005
Please buy subscription sar! I personally got paid to put their banner there.

File: 1767266115504036.jpg (1.3 MB)

1.3 MB JPG

>>108581998
I'm sure it is Zen

>>108582003
So I think what's going on here now that I think about it is it processes the entire context if the context plus the max response length is greater than the max context, even if it doesn't actually generate a message anywhere near the max response length.
I had max response length set to 9999 and I just realized the problem started happening when the context was near 10k tokens from the max context.

File: kek.png (841.2 KB)

841.2 KB PNG

>>108581916
Dear god, I don't even need you dipshits anymore.

>>108582003
Silly deducts your maximum response length from your context size, yeah.

https://www.youtube.com/watch?v=X41TmM6CM-U
a bit late, but russia is making AI its top priority
thought?

>>108582013
Am not, that char card is just a popular one that was floating around

>>108582034
I could not care less.

>>108582034
If they aren't retarded, they're just going to invest in China. If they start now they're even further behind than the euros and even the nips. It's going to be an even bigger joke than their attempt at making their own chips.

>>108582034
They will have a harder time getting access to GPUs than China and have zero hope of developing their own native chips. All the LLMs that have come out of there so far have been ass. I doubt they could even successfully finetune an existing base model. This will go about as well as their attempts to make a native Russian smartphone.

>>108582034
They buy H800s from China

>>108582034
I will surrender my entire tech stack if Putin promises me a virgin Russian gf

>>108581896
I can have /pol/ at home now?
Which model exactly, v4 26B or lower?
>>108582057
They were really close to actually having all that. Isolation due to war ruined everything. Some say Putin is CIA agent.
>>108581958
She's getting adopted right now!

>>108582068
Bad trade. Mailorder slavic whores are cheaper than gpus these days.

>>108582068
>virgin Russian gf
you'd be getting one so ugly no one wants to touch her, retard.

>>108582076
>Which model exactly, v4 26B or lower?
gemma 4 31b

>>108582034
This will go as well as all the other "Russian made" tech projects they've tried in the last 40 years
It's the exact same narrative as China and NK, where they're trying to only invest and innovate on local projects to render themselves fully autonomous, except the former actually manages (sometimes with decent success, such as in the automotive industry) while the latter... the latter is NK so it doesn't matter.

>>108581839
I have aphantasia/no inner voice.
I have tried to conjure rpg scenarios but it is really difficult because I have zero imagination.

>>108582034
Maybe if the country wasn't burning through its capital and productive force in a senseless war, I'd believe him.

>>108581557
>a full year of giant MoEs nobody can run at home
>and either retarded or stemmaxxed models
Jeez I wonder why we were so dead.

>>108582111
>senseless war
oh boy, here we go

>>108582111
>you aren't supposed to fight back when your neighbor kills your countrymen

File: localllama.png (15.3 KB)

15.3 KB PNG

He's right, you know? Google should stop.

>>108582105
I have it too, literally cannot see anything in my mind, I thought everyone else was like that until I discovered everyone else around could actually imagine images/videos in their head and it wasn't some kind of metaphor.
I'm a huge reader so while I can't "see", I can feel the ideas of what's going on. If you read a lot, that comes easier with time. So it can definitely be trained.

File: 1747937874763365.jpg (54.9 KB)

54.9 KB JPG

>>108581557
Google doesn't need to pay me, with Gemma I do it for free

>>108582137
Google wasn't mentioned though? Did you just hallucinate that?

File: laughingkoyuki.webm (522.7 KB)

522.7 KB WEBM

The best part about Gemma 4 is that /aicg/ paypigs and commits credit card fraud while we now have local Opus that runs fast on VRAMlet machines.
We won and they lost.

>>108581557
>Do you think the increased traffic to /lmg/ is organic?
absolutely, you lost Chang, next time train your model on 4chan and maybe we'll praise Qwen for its sovl

>local Opus
lmao

>>108582116
>a full year of giant MoEs nobody can run at home
It was only bad for poorfags and Americans who got their feelings hurt because Western open models were basically dead. And still the only thing that they got was a model for RP. For serious work, Qwen, MiniMax, etc. are still better.

>>108582145
It was mentioned in various benchmarks and even their own that it loses to Qwen 27B.

>>108582146
>local Opus
It's good but don't be ridiculous.

>>108582150
You heard me.

>>108582146
>local Opus
don't undersell gemma, it's better than muh quippy mcu dialogue opus and is much less slopped if you are not a skillet

File: 1748192930954736.png (143.6 KB)

143.6 KB PNG

Now imagine a local Nano Banana Pro from Google, if that happens I'll stop sucking the CPPs dick for at least a full year

>>108582119
>>108582135
Don't care, it's ruining the country and now they're even closing internet access, I don't expect anything from the current thugs in charge, even less for AI.

File: 1770960500459504.jpg (82.8 KB)

82.8 KB JPG

>>108582150
>>108582156
>>108582159
Time to face reality, opus got downgraded hard for RP

>>108582155
Falseflag kikes get the rope
People with ulterior motives trying to start a fight in /lmg/ wrt. Gemma and Qwen

>>108582164
Anima Preview V3 is already the SOTA for anime which is means that local imgen is essentially solved

>>108582168
because of the end of prefill?
I wonder what the hell aicg even does without prefill, the model is pretty uptight without it

llm-tards please give me a quick tldr what kinda of llm schould i download for basic text gen with a 5070ti. i just set up coboldccp.

>>108582168
that's pure cope that arose after gemma started mogging it

>>108582169
False what? I'm just monitoring the situation and stating the obvious things.

>>108581353
>software
Software is composed of executable segments and data segments (ignoring some degenerate types of software where data is also executable)
Large data models don’t bode ill for the same state of software engineering that monstrosities like electron do.

>>108581056
i wonder if you could get reasonable speed of ssd inference using something like dflash but tweaked.

so have a bunch of tokens predicicted by the draft model.
then get layer n from ssd, do your batch on it, then next layer, batch etc.
effectively since you are doing batches for each layer you are still getting a speed improvment because you use layers multiple time before loading the next from the ssd.

also there is speculative speculative decoding where you get the draft model to work on other possible predictions in parallel as well.
i wonder if that would make sense adding it to dflash.

>>108582171
It might be SOTA but Illustrious is still the anime meta due to LORA support/Controlnet support/style flexibility.

File: 1761472690168243.png (307 KB)

307 KB PNG

>>108582164
I fucking wish

>>108582171
I've been out of the loop when it comes to imagegen for a while and I'm itching to get back into it
Back in muh days you relied on Comfy + uuuh Pony?
What's this Anima thingy

>>108582174
Gemma 4 26B on Q8
>b-but my vram is too small to handle a 26B
Doesn't fucking matter senpai, it fits and it's fucking smart
t. running it on a 4070 and demolishing my pen0r as we speak

>>108582174
gemma4 26ba4b. Tell it to fix your spelling for future posts.

>>108582184
also obviously i wonder if something like a 12 nvme raid 0 would make things much faster if you got the lanes for it.

File: 1701586351737913.png (1.5 MB)

1.5 MB PNG

>>108582146
>We won and they lost.
It was always only a matter of time.

>>108582174
i forgot to say i have 64gb of ram.

>>108582146
gemma is still way too sloppy in english

File: 1760259479131141.png (161.2 KB)

161.2 KB PNG

>>108582190
>your models suck
NOT ANYMORE AHAHAHAH

>>108582184
hear me out.... dflash... ssdmaxxing... BITNET.... the holy trinity, dude. like... imagine though... it's like... 3, but like a fast three. not the slow threes we used to have like... you know... FAST I mean, yeah... like that. fwoooosh it goes, tokens bam bam bam...

>>108582168
Opus was never intended for casual use or coding. It was literally never good for that. Look up old benchmarks, Sonnet was always better, because it was their real product, Opus was intermediate sort of thing. I'm not sure why they had it available in the first place. Only thing they achieved is letting chinese distill it to have their own Sonnet.

>>108582187
Also running 26B on a 4070. Q4_K_M with 19 layers on the GPU and 24k context.
33.8 t/s
We sure have come a long way since I was running Mixtral on this rig at 5 t/s.

>>108582189
If you use intel optane SSDs- it could make sense, really hard to tell.

>>108582202
none of the things i proposed would rely on anything new realy.

>>108582197
>gemma is still way too sloppy
ftfy

>>108582202
this, but unironically

>>108582190
kek

>>108582219
modern gen 5 ssd are almost 10gbps.
if you could get 12 and scale linearly that's already some good bandwidth and if you now use speculative decoding with batching that may actualy be worth something.

>>108582215
I mean, I got into local models in mid March and I wasted a full week testing out a whole lot of 12b models on q4/6 occasionally daring to go for a 15b
And now I have this beast running and it's objectively and noticeably better
Brings a tear to me eye

>>108582225
We’ve worked out the numbers in previous threads. It’s not anywhere near the best t/s/$ if you actually do the math.

>>108582223
i switched to japanese and been discovering a whole new world because its actually good at it, but im sure soon enough the honeymoon will end and ill start seeing slop patterns there as well

>>108582230
That is max possible bandwidth. But latency matters way more. Maybe you have MoE on your mind, then it's a different matter.

>>108581696
Can someone answer this

>>108582034
I've never heard of APT and it's not like I can understand Russian either.
Is this even real?

File: 1773963878340589.png (8.4 KB)

8.4 KB PNG

>--temperature 1 --top-k 64 --top-p 0.95 --alias gemma-4-26B-A4B-it-UD-Q4_K_M --ctx-size 65536 --cpu-moe --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --fit off --kv-unified --model ./models/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf --n-gpu-layers 99 --parallel 1 --reasoning true --threads 4 --threads-batch 8
How do I squeeze more speed out of this

>>108582266
Use cloud model

>UD

>>108582266
>How do I squeeze more speed out of this
DFlash my man
https://github.com/z-lab/dflash

>>108582242
latency doesn't matter that much, you get a layer(slow) but you'd use it for all the possible speculation tokens in your batch, so you could probably use it like 6 times before moving to the next.
also you could prefetch the next ones whilst you are still computing with the current one

>>108582282
>still no gemma 4 dflash draft model despite the insane demand and masses begging for it
yup, it's doa
there is no training pipeline

>>108582311
>draft model
snake oil

File: ssdminning.png (1 KB)

1 KB PNG

12b worth of active parameters at q4 is ~6gb.
Run picrel to see how fast you could possibly generate with ssdmaxxing. I'm sure mine will be the slowest at about 20s/token.

>>108582327
This nigga still in 2024

>>108582146
I don’t know about current opus but it does take me back to the opus 3 days where we could get sovlful rp without much effort. And the 31b has smarts combined with that too.

>>108582335
Just a file I have that happens to be about 6gb.

>>108581741
NTA but this should work, I've been using since release. At the end of the 31B template

{%- if add_generation_prompt -%}
    {%- if ns.prev_message_type != 'tool_response' and ns.prev_message_type != 'tool_call' -%}
        {{- '<|turn>model\n' -}}
        {%- if not enable_thinking | default(false) -%}
            {{- '<|channel>thought\n<channel|>' -}}
        {%- endif -%}
    {%- endif -%}
{%- endif -%}

to

{%- if add_generation_prompt -%}
    {%- if ns.prev_message_type != 'tool_response' and ns.prev_message_type != 'tool_call' -%}
        {{- '<|turn>model\n' -}}
        {%- if not enable_thinking | default(false) -%}
            {{- '<|channel>thought\n<channel|>' -}}
        {%- else -%}
            {{- '<|channel>thought\nPREFILL HERE' -}}
        {%- endif -%}
    {%- endif -%}
{%- endif -%}

base templates are different so keep that in mind

>>108582323
>2 to 20x lossless inference is snake oil
retard

>>108581611
ywnbaw

>>108581878
moe are retarded at low Q

>>108581611
go bACK where you belong troon

File: 1767466445975554.png (776.6 KB)

776.6 KB PNG

Since the dflash only exists in python, could you vibecode python-cpp hooks for it just like lcpp-python has? And then slap that onto it the main lcpp. Or would it kill any possible speed gains? Idk if there is any model smart enough to do a complete language to language rewrite.

>>108582355
>>108582360
Seething thirdies. Trans rights are white rights.

>>108582344
oh, so that would be an always on thing, not something I could edit on the fly for one message. I see

>>108582350
>lossless
NO
FREE
LUNCH

>>108582379
gemma 4 31B translated some webshit frontend to rust + dioxus and it just worked.
if it can do that, surely a frontier model can.

>>108582391
Read up on how draft models work, retard.

>>108582391
speculative decoding is literaly lossless, there is literaly no degradation of quality, if you think otherwise you just don't understand how it works.

>>108582398
>>108582400
Dunning Kruger on full display

>>108581611
Proof?

Are there any results from dflash that aren't typed in a text editor? Like output from llvm or something?

gemmasters ww@?

File: 1772173881191671.png (25 KB)

25 KB PNG

do I follow gemma's advice?

File: 1755318702032209.png (67.2 KB)

67.2 KB PNG

>>108582385
being a troon is a brown behavior though
https://williamsinstitute.law.ucla.edu/publications/trans-adults-united-states/

>>108582391
it is 100% lossless though, retard

File: 1775717434377533.png (66.1 KB)

66.1 KB PNG

>>108582406
>>108563620
https://github.com/vllm-project/vllm/pull/36847

>>108582398
>>108582400
Based topk=1 enjoyers

>>108582404
speculative decoding result in an identical output, it is computational identical.
The big model still predicts all tokens, it just allows it to predict possible next tokens in parallel and go back if the predicted token doesn't match.
You are the dunning kurger here, go learn something.
The only cost is that the draft model takes some extra vram.
So no the launch is indeed not free, but you get identical outputs just much faster.

>>108582421
>speedup from basically nothing to 3x for established benchmarks
so it's about as pointless as eagle3? wow

>>108582414
How come I've only ever seen white traps?

>>108582438
Just like regular draft models, it'll probably be far better for things like programming and mostly useless for tasks with highly variable outputs like creative writing.

>>108582430
even if you don't top k = 1, if the draft models have the same probability distribution it still works.
Once you passed the sampler there only is one token left anyway.

>>108582438
anon... for our usecase it's always at conc = 1, so the worst case scenario is a 2.8x speed increase

>>108582446
anon... there's only 13% of black people in the US, 0.8% of 13% of black people is way less than 0.5% of 65% of white people

>>108582421
So nobody really ran it in the whole PR process? Nobody else posted numbers?

>>108582448
>highly variable outputs like creative writing
lol thanks for the good laugh

>>108582448
a well trained draft model will be able to predict the smell of ozone and something sweet when it comes up

>>108582460
that's not how that work retard.
you don't apply per capita twice...
0.8% of a population is 0.8%, it doesn't matter if that group is 13% of the total population.

>>108582493
it still means there's less people overall fucking retard, 0.8% of 100 people is 8, but 0.5% of 200 people is 10, you see more white people being troons because there's just a lot of white people than black people in the US in general, I can tell you're a troon you're fucking braindead

>>108582164
i actually like the original one more than the pro and 2, it was much better at colorization