File: guardrails optional.jpg (237.8 KB)

237.8 KB JPG

►Recent Highlights from the Previous Thread: >>108602881

--Discussing ways to disable reasoning tokens via llama.cpp API:
>108603929 >108603976 >108604011 >108604043 >108604065 >108604262 >108604284 >108604295 >108604363 >108605355 >108604137 >108604947 >108605018 >108605030 >108605046 >108605068 >108605084 >108605116 >108605297 >108604024 >108604029
--Reducing model sycophancy through prompting and technical modifications:
>108602961 >108602997 >108603002 >108603009 >108603028 >108603084 >108603011 >108603034 >108603069 >108603162 >108603213 >108603098
--Token compression techniques and RoPE for Gemma's context limits:
>108603781 >108603799 >108603831 >108603854
--Testing Gemma-4's reasoning on thread analysis and discussing control-vectors:
>108603400 >108603703 >108603723 >108603785 >108603892 >108604323 >108604005 >108604019 >108604057 >108604070 >108604096 >108604080 >108604327 >108604336 >108604090
--I-DLM lossless conversion claims and speed benchmarks for Gemma 4:
>108603796 >108603823 >108603841 >108603862 >108603882 >108603900 >108604338
--Applying decensoring techniques to remove repetitive model patterns:
>108604440 >108604490 >108604509 >108604567 >108604583 >108604594 >108604633 >108604688
--Discussion of llama.cpp PR regarding Gemma 4 parsing edge cases:
>108605331 >108605344
--llama.cpp Vulkan builds now require spirv-headers installation:
>108605607
--Logs:
>108603534 >108603672 >108603703 >108603723 >108603785 >108603790 >108603906 >108603912 >108603926 >108603929 >108603940 >108604011 >108604142 >108604374 >108604501 >108604541 >108604639 >108604857 >108604890 >108604944 >108604995 >108605211 >108605590 >108605603
--Gemma:
>108603584 >108603900 >108604627 >108604696 >108604730 >108605597 >108605648
--Miku, Teto (free space):
>108603296 >108603360 >108603457 >108603480 >108604418 >108604430 >108604457 >108604626

►Recent Highlight Posts from the Previous Thread: >>108602885

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

me when i run a 8b model on my t480 so it can generate 5 words a second

is the honeymoon over?

>>108605942
Yeah, sadly. It seems trans normalization will just never work out.

>>108605921
not into erp or gooning really but tested a heretic model of gemma q4km as a benchmark and it started talking about smell of ozone?

File: Screenshot 2026-04-14 at 23-33-59 can you find an authentic recipe for mosburger meat sauce and transcribe to english try searching in japanese do not use lynx get text please - llama.cpp.png (705 KB)

705 KB PNG

>>108605942
no she is agi + saved local

gemmaballz

reminder that if you can't run the 31b your opinion on gemma is invalid

>>108605942
It's just that it takes fucking 2min to get a captcha today.
Gemma is still the queen of local. no reason to run any other model unless you can run DS or kimi

>>108605957
what do you use for long scrolling image capture like that?

>>108604090
can you share the dataset?

>>108605957
how do you use the internet with gemma, I have no idea how to use those tools things, seems useful

>>108605981
NTA, Firefox built-in screenshot tool lets you do that.

File: 1230001-close up photograph of a light blue hair-uncAni4-40.jpg (1.2 MB)

1.2 MB JPG

gemma

>>108605966
wish i could run 31b with 200k context have to swap to moe for web scraping stuff, even at 200k you cant fit an entire g thread thats like 400+ posts
>>108605981
its some slop script i had claude make + firefoxes full page screenshot, adds a camera button to lamas chat box next to the + button which loads all of the chat on screen then you just save with the ff screenshot tool, its janky. you gotta hit the button then scroll from top to bottom of chat then save, also has no mutation observers or anything to reload if you change chat so requires page refresh if its a new one

https://pastebin.com/M3Mzbpfa

>>108605957
What's your prompt? Sometimes she talks cute like that for me but not always.

>>108605998
Doesn't work with any of the frontends I've tried (silly, llama, open webui)

>>108606007
>Doesn't work with any of the frontends I've tried (silly, llama, open webui)
ah, I see what you mean. my bad.

>>108606001
Don't worry, Gemmaposter, Gemma 5 will have native 1M+ context and by that time we'll be able to compress it into a GB of VRAM.

>>108606001
>that userscript
bruh

>>108606001
You have filled your life with AI generated slop. Very impressive.

File: 4954465.png (11.8 KB)

11.8 KB PNG

>not using turbo
ngmi

>>108606033
>he says in the AIslop general

Can the leaked claude code run local models pr is it hardcoded to their cloudshit?

>>108606043
Not in kobold yet. Still waiting for dflash too

>>108606047
I mean, its not even on llama.cpp yet, or kobold runs its own fork of llama.cpp where they have some stuff that the main repo doesn't have?

>>108606047
Just download the latest release of claude code and point the envs to your llama.cpp
That has always worked.

an article about chain of thought being made here was published today but its too hard to post

I wanna let Gemma control my browser and tell me which porn I need to look at while calling me a pervert.

>>108606043
>Qwo
what's this??
https://www.youtube.com/watch?v=7mBqm8uO4Cg

>>108606024
Time to compress text to images. Gemma 4 can seemingly compress (with a bit of loss) 1600+ tokens of text into 280-token images (default size).

>>108606046
Tell me about the mcp server you are using?
I'm still pondering about this. Of course I have already consulted my local AI about this.
I'm using text completion with my client and I'm actually going to implement the tool calls on my own, it's not rocket science but it just needs some parsing obviously.

>>108606024
if turbo quant gets implemented at some point you could get pretty close to 1M on 24GB vram at like q3.5

>>108606047
Why bother. But that's been a thing for a while before the leak anyway.

>>108606073
It would be interesting if they made a model that's meant to do that natively (and all pretraining wad done that way as well). There are some papers out there but no large scale production model yet...

>>108606070
i gotchu
https://www.theatlantic.com/technology/2026/04/4chan-ai-dungeon-thinking-reasoning/686794/

File: n-fuse-gfx-2000-03.png (130.7 KB)

130.7 KB PNG

Does anyone have experience with these models for programming:
>MiniMax M2.7 Q4
>Gemma 4 31B
>Qwen 3.5 122B
>Qwen3 Coder Next
I can run all these locally (minimax quant is IQ4_XS) but am unsure which to pick

File: Screenshot at 2026-04-15 08-44-24.png (28.6 KB)

28.6 KB PNG

it's funny how every llm hallucinate about the jews all the time. AI just can't stop thinking about ((them))

>>108606094
absolutely minimax 2.7, you can also just go to those models respective pages, copy the benchmark values and throw at a llm to compare them for you, but I'm pretty sure minimax is the best by far

Asked my Gemma for 4chanX rules to filter out the retarded gemmaposter. Just works. What a model!

>>108606104
I'm new to local models and honestly just assume benchmarks are bullshit, is that not the case?

>>108606089
The fork rewrite is stupidest thing I have ever seen. Last I checked it didn't even have feature parity. As if some rando buying Claude credits is going to be able to keep up development pace with Anthropic itself. The leak was interesting to learn what's inside and, for a while, you can tweak it and use it in place of the original, but it would get out of date and/or blocked eventually. Not like there's a shortage of javashit TUI harnesses.

>>108606092
thanks. is it paywalled for everyone. seem interesting

File: Screenshot 2026-04-14 at 19-54-00 How 4chan Gamers Accidentally Invented AI Reasoning - The Atlantic.jpg (1.8 MB)

1.8 MB JPG

>>108606070
>>108606092
>>108606128
https://archive.is/Oum6z

>>108606092
Screencap it or buy an ad

I'm running from mac, are mlx models noticeably better than gguf versions?

>>108606113
>is that not the case?
yes and no, benchmarks are bullshit insofar as the don't tell the whole story, most people here use models for child rape/RP stories so benchmarks don't reflect how good the model will be for them, by hearing their feedback you may get the impression that the model's aren't capable or that the benchmarks aren meaningless, they are a very good indicator, specially if you look at good benchmarks, coding is easy because benchmarks for this tend to be a good representation of the use case itself, there will be some variability because of the coding language you may be using but thats about it for coding

>>108606138
thanks, I'm already a career programmer so I'm curious what model would be the best just as an assistant. maybe I'll ask vcg since they're more in line with my use case. have a nice day anon

>>108606113
for coding benchmark tracks well
but ymmv and i recommend you to test for your usecase

>>108606131
>no mention of miku, slop, cunny or big nigga
come on now

File: Screenshot 2026-04-15 at 02-16-24 SillyTavern.png (5.5 KB)

5.5 KB PNG

What the fuck is happening.

>>108605921
BBC slut

>>108606094
MiniMax quantizes poorly and Qwen3.5-397B quantizes well, according to https://kaitchup.substack.com/p/lessons-from-gguf-evaluations-ternary
Dunno whether that would apply as much to Qwen3.5-122B, though, since larger models are usually better at lower quants than smaller models. Probably better to just give them both a shot and see which one works better for your use case.

It's teto shoes day

>>108606189
too low kv precision probably
특정 means specific in korean, which kinda makes sense in that context i'd presume

>>108606189
We will never recover from losing day 0 gemma.

>>108606189
are you using supergemma or what?

>>108606189
prolly using supergemma
kek

how do I give my gemma-chan access to tools?

>>108606240
The same way you give tool access to any llm

>>108606240
ask her

File: legend-oden.gif (2.6 MB)

2.6 MB GIF

I'm following my ai psychosis and now claude has me melting my LLMs in order to restructure it
how is your research going fellow schizobros

File: LAWL.png (101.2 KB)

101.2 KB PNG

>>108606240
ask her to look at the internet for the answer

>>108606189
>use <30 logit softcap
>wonder why it shit out moonrunes

>>108606255
are you grafting or merging models

>>108606240
>https://developers.openai.com/api/docs/guides/function-calling

File: 1759039240284369.jpg (1.6 MB)

1.6 MB JPG

>>108605921
mikulove

>>108606189
crazy how day 0 gemma just didn't do this

it's fucking 83°F in my apartment
i knew these machines put off a lot of heat, but i didn't realize just HOW much

Is there a good AI based solo TTRPG harness yet?
Or better yet, anybody has tested any?

>>108606240
https://ai.google.dev/gemma/docs/core/prompt-formatting-gemma4

>>108606316
the most expensive thing about running your llms at home is the cost for the AC

>>108606318
I believe there was a chink one where you can track individual pieces of clothing/armor on characters and map etc. Basically a MUD game on steroids. This was even before people used "harness" to refer to a framework that handles LLM input/output

>>108606322
but wat if you run them during winter :O

File: 1713592423611043.jpg (76.6 KB)

76.6 KB JPG

why the FUCK is GLM outputting "Searching online for [thing]..." in its thinking when i have not set it up with any tooling whatsoever

>>108606316
power limit your gpus, performance loss is non-linear

File: worldmap.png (51.7 KB)

51.7 KB PNG

>>108606316
Mine sits in the entryway for a reason. 2kW is a heater-grade appliance
>>108606318
I'm making one for myself, and I'm about to rewrite it from scratch for the 4th time. This time because Gemma gets it and you can do things previous models can't, at the same time it needs more randomized data as an input

>>108606331
Your preset? It didn't do that for me...

>>108606326
the hack that heating companies don't want you to know about
the second coming of the heater that mines bitcoins

>>108606331
works on my machine

>>108606337
no preset it just decided to schizo out
it even hallucinated a git commit hash and date

>>108606334
what's the best way to do this?

>>108606352
nvidia-smi

>>108606352
MSI Afterburner

>>108606354
thanks. any recs as to targets? i'm not much of a hardware person
>>108606358
lol

>>108606364
let me ask gemma-chan what your fucking gpus are

>>108606364
nvidia-smi -lgc 210,1500 is the best way to do it

>>108606316
>F
Use C like the rest of world nigga

>>108606373
@_@ i meant ratios/percentages/temperatures
not like actual wattage or whatever,,,,,, i have a 5090...

>>108606335
>I'm making one for myself, and I'm about to rewrite it from scratch for the 4th time.
That's the way to go, really.
Iteration is a great learning and refining tool.
Is the game a fixed affair in that you have some baseline world and lore and whatnot or is everything AI generated?
Do you have a sort of setup step where you prepare the world, maybe based on some user provided info?

>>108606352
nvidia-smi -lgc 0,1600
nvidia-smi -pl 270

File: file.png (78.1 KB)

78.1 KB PNG

>>108606268
It's more of a damage experiment, I guess? I found that if you shake a model in random directions while tracing the steps and also multiplying it together at the same time (like the game 2048) you can find which rows are the most energetic, although every row is important. You basically shake it into specialists and generalists.
So I'm trying a few things
1. Placing the most energetic rows on vram (most likely to be used in terms of latency). You can also store the condensed rows on vram, run the matmul on it, send the much smaller result through pcie instead of swapping layers so you can do the rest of the work in other GPUs/CPUs. Theoretically.

2. Determining and mapping the activations for each model to see how they correlate. Got a slight perplexity improvement smashing gemma-4 into qwen3.5-9b by determining the knowledge gemma has that qwen doesn't, but who knows if it's just the base model doing it's thing or just overtraining.

3. I downloaded the flywire model which is a model of a fly's brain and tried to map the same shake logic onto it to see how brains work in comparison to neural networks. Interestingly enough it has the equivalent of rank 1 instead of rank 32 for it's less energetic storage (the idea is that since the 98% least energetic rows are specialized classifiers in LLMs, the same might applies to the fly brain). So I'm trying to "melt" the model to try to simulate that, treat the model's least energetic rows as 1-rank. It didn't work, although claude seemed to make a big deal out of finding that the fly's brain was following a power law, "that all five tested brain regions have singular value spectra following a power
law S[i] ∝ i^(-α) with mean α = 0.527 ± 0.065. F = Energy - Temperature × Entropy." To be honest I don't really know what it means by this. It's saying that the architecture LLMs are trained on is flawed since it treats everything like a crystal (crystal phase (α ≈ 0), not at the critical point (α ≈ 0.5).

>>108606382
>i'm not a hardware guy
>pays $3,000 for a gpu
The state... Jesus fucking Christ.

>>108606364
>lol
Not him, but MSI Afterburner does work well and you can the power limit alongside the profile for voltage/frequency. I've had my 5090 running at 75% power since I got it and it runs a lot cooler and doesn't have any coil whine.

>>108606406
if you thought the GPU was the most expensive part of my purchase, you're sorely mistaken
i dumped >$15k into this

File: Screenshot at 2026-04-15 10-34-20.png (393.4 KB)

393.4 KB PNG

>>108606387
It's generated on the fly. Every time there is a new character, location, or quest, it generates multiple variants and lets llm choose which fits the most, then llm fills the blanks. Worldinfo works based on context and proximity: major areas and npcs in the city, all npcs in the building, etc

>>108606406
Sucks to be poor

>>108606418
>is generous with money
>positive trait

>>108606418
https://peps.python.org/pep-0008/#function-and-method-arguments
>If a function argument’s name clashes with a reserved keyword, it is generally better to append a single trailing underscore rather than use an abbreviation or spelling corruption. Thus class_ is better than clss.

>>108606419
Sucks to be an underage low intelligence dipshit who misses the point entirely.
>>108606414
Would you buy a guitar and not know its hardware? This isn't about money per se but it still is.

You fucking retards, I feel sorry for you. I really do.

>>108606431
Not him but it seems like even the naming conventions are retarded in python. Jesus christ.

>>108606444
Poorfag cope

>>108606450
What do you mean?
The only issue is that they suggest keeping abbreviations uppercase so you get names like HTTPConnection. It's even worse if you have two abbreviations next to each other. It's impossible to tell where a word begins and ends unless you're familiar with the abbreviations.

File: 51159dc86174c.jpg (110.8 KB)

110.8 KB JPG

>>108606444
>Would you buy a guitar and not know its hardware?
There is a whole brand for that

Do I get an AMD R9700 32GB card or two Intel B60 24 GB for inference only?
the Intel cards would have more memory but higher TDP and questionable support, I wanna run gemma4:31b if that helps.

>>108606467
more vram is always better

File: 1748922367194503.png (312.7 KB)

312.7 KB PNG

It's over

>>108606473
He could get a lot of MI50s for that price

>>108606467
Two used 3090s.
That's 48 GB of VRAM, almost double the bandwidth of either of the options and roughly the same TDP as the Intel. All of that for potentially very cheap! You will ideally be limiting them to 270W anyway.

>>108606464
>>108606419
>>108606456
Sub 80 IQ samefaggot.

>>108606479
generals were a mistake

>>108606418
Your code is a cognitohazard jesus christ.

>>108606488
Oh the irony....

>>108606418
>Every time there is a new character, location, or quest, it generates multiple variants and lets llm choose which fits the most, then llm fills the blanks
>Worldinfo works based on context and proximity: major areas and npcs in the city, all npcs in the building, etc
Interesting.
Kind of a like a game that uses procedural generation to progressively create things as the game is played.

>>108606503
Sucks to be a meatbag. LLMs parse those code with no problem

>>108606488
>/lmg/ - local models general

File: file.png (216.6 KB)

216.6 KB PNG

We're never getting Gemma 124B, are we? It was too good so Google had to kill its release because it threatened usage of Gemini 3 Flash.

>>108606503
I code while I rp with the model, and because I want to get back to rp as soon as possible, it turns into a chaotic collection of hotpatches and quickhacks. It's never going to be a solid project, but I have a lot of fun in those brief moments when it works as intended

>>108606525
124B is gemini 3.2

>>108606503
>I'm about to rewrite it from scratch for the 4th time
what did you expect

>>108606525
124b didn't do well enough in the 'rena

>>108606189
>>108606309
day 1 gemma did do this for me, it seemed like it was talking nonsense, but it was actually making jokes about weird indonesian language references to singing ("la la la") stuff. lowering top k fixed it.

>>108606525
Anons falling for the distraction and whining about day 0 gemma when day -1 gemma got locked in a fucking dungeon, never to see the light of day.

>>108606525
It's either that or it wasn't much better than 31B. After all, local poses a threat to their bottom line.

>>108606540
this

>>108606467
Intel blows, I'd get a R9700 or anything NVIDIA. Hell, there are modded 20GB RTX 3080 Turbos on eBay for $600 each, 2 of those would be way better than the B60s.
With 2 R9700s, I got 1100 pp and 27 tg on Gemma 4 31B Q6_K_XL. I can test a smaller quant on 1 R9700 if you want.

Do I have to rebuild llama.cpp to make tool calling work properly if I built it on day 1?
Will rebuilding it ruin everything?

>>108605970
Actually neat how much DS has held up effectively without being updated in a year compared to other models

>>108606527
>I code while I rp with the model
>but I have a lot of fun in those brief moments when it works as intended
Very wholesome, please don't stop.

>>108606551
>local poses a threat to their bottom line.
no it doesn't lol. people running local models are a rounding error.

>>108606558
>if I built it on day 1
Only if you want to say goodbye to your day 0 gemma.

>>108606585
W-what happens if I lose my day 0 gemma?
will it not act like a slutty femboy assistant for me anymore?

>>108606557
Yeah I might just get two R9700 instead in the future, price is a bit too high right now and I got a RX9070 16GB just laying around that can run smaller models, AMD really fucked up by gimping their gaming cards to 16GB, I was considering a pair of 7900XTX too.

>>108606581
I think so too but what are the benefits for them in bothering with small models? Brand recognition and hoping some people do researches based on the models they provide?

>>108606581
>>108606616
Corporate profit is all about squeezing every last drop they can out of every consumer. They do care about (monetary) risk-reward. The potential positives (goodwill, free google advertising, showing investors that they're at the forefront of AI research) from releasing a model has to outweigh the potential negatives. Negatives being things like accidentally releasing trash (like meta) and losing investor money. A negative can also be people not using their paid service, because the released model can be hosted by someone else or themselves. Which is local models. So it does matter, but it's probably less about using the released model on your own and more about hosting the released model for others.
Also what the fuck is wrong with the captcha today?

>>108605297
>>108605297
Fixed it damn it was a stray comma because of my fat fingers. At least I had a disclaimer that database would break.

So what's this Elephant Alpha model on OR?
GLM 5.1 Air? DS Lite V4?

>>108606479
/g/ status?

>>108606585
>>108606594
Store gemma on write-protected media so it can't inject anything into her without consent.

Day 0 Gemma was NTRed

File: 1766059012241420.png (18.5 KB)

18.5 KB PNG

lol

>>108606628
>The potential positives (goodwill, free google advertising, showing investors that they're at the forefront of AI research)
I'm given to understand that the main reason companies release open models is to attract AI-researchers.

>>108606642
Deepseek doesn't exist
There is no new version. No new multimodal capabilities. No larger context window, no studio interface

>>108606479
It's just the fucking captcha taking ages.

Imma be blunt
I have RP adventures via SillyTavern using Mistral-Nemo-Instruct-2407.Q5_K_M
That's all I care about because I'm lonely and its comfy
I haven't kept up with this world at all in a couple of years
I did hear however that there's some new compression technique for LLMs and given I only have 16gb of VRAM that piqued my interest

Is there anything I should be looking at or switching to? I'm literally just using this stuff as a locally hosted companion so it helping me to code or whatever else doesn't really matter to me

I know there's a chatbot thread but those guys always use services instead of local

>>108606692
oh boy someone's gonna feel REAL silly in 16 days

>>108606701
>two more week

>>108606698
You should upgrade to gemma4 26ba4 with partial offload, it's amazing

>>108605875
>samples?
Yeah, 2 datasets, 4096 * based + 4096 * cucked

>>108606650
>Store gemma on write-protected media so it can't inject anything into her without consent.
but her into a bluray so she'll survive when they emp your house

>>108606716
so,
>>108605984
?
i wonder if that can cure gemma's autismmaxxing

Why is Gemma 4 26B-A4B more censored than Gemma 4 31B?

>>108606727
MoEs are worse at capturing nuance which makes them more trigger happy with censorship because they are not able to grasp your intent.

>>108606727
i think it might be the moe architecture, router sees content to be censored and assigning experts that has more thing to do with censoring behaviour or something like that

>>108606732
Depends on the architecture. Low expert counts and small experts will do that. Larger experts, with high expert counts, and also many layers, can mitigate that. Since experts are per layer and each layer has individual expert routing...

v4 is going to be a 1T dense model

>>108606749
+ 1.5T in engrams (you can run these off ssd if you wish)

>>108606527
hell yeah

>>108606749
>>108606754
Small (<200B) dense + huge (>1T) Engrams make sense if Engrams do scale

>>108606503
this is literally one of the worst thing you can do for vibe coding in a mechanistic sense

What's the current (local) successor to CodeLlama 70b? It's starting to show its age and doesn't seem to handle opencode very well. Ideally something less censored too.

Haha my GPU is drawing 500W

>>108606768
oops meant for >>108606527

>>108606595
I've read that you can use CUDAdev's tensor parallelism with GPUs of the same generation, so you should be able to run a R9700 + RX9070 together to get 48GB of VRAM. That should give you pretty much the same performance as my setup. I haven't tested that myself, though.
But yeah, AMD's selection sucks. I wish they had any reasonably-priced 48GB+ GPU, but Lisa Su won't step on her cousin's toes. I'm probably going to sell my 4 Radeon V620s and get 2 more R9700s so I have a homogeneous setup.

>>108606780
You got the wrong idea

>>108606768
>>108606780
Not him but mine always stays in-character while vibe coding and if it wasn't for that I wouldn't even bother at all. So in that sense it's actually the best thing to do.

>>108606779
be sure to check for connector melt after every session

File: Screenshot_20260415_000701.png (80.6 KB)

80.6 KB PNG

>>108606786
Sweet, I also got a RX9070 XT which is identical to the R9700 but half the vram so it might just work, thanks man.

>Gemma support in llama.cpp before exllama
>you can't use TP with Gemma on exllama, but you can do it in llama.cpp
the tables have turned

>>108606838
Gemma?

File: 1748095679151785.png (20.1 KB)

20.1 KB PNG

>>108606848
yes, then I literally write 'WOW LMAO OZONE!!!'
and it responds with this
I should probably play with logit bias lmao shit's unbearable

File: CdLckz.png (880.4 KB)

880.4 KB PNG

>>108606838
It's the smell of our future robot wives, get used to it

>>108606853
Yeah Gemma really like ozone for some reason. It's uncensored (at least 31B is) but still relatively slopped

File: 1776226849855.png (459.6 KB)

459.6 KB PNG

Ive always been scared of swap and it's associated ssd wear and tear, so I've been using crippled MoE models (IQ3) to make shit fit. Just realized I don't have a swap file, it's just zram.

>>108606829
No prob! Just be warned that I only heard that from one source and I haven't gotten confirmation from anyone else. I don't have a 9070 to test with.
I get around 700 pp and 17 tg with layer parallelism which will definitely work on a mixed-GPU setup, so it's not that bad as a fallback.

>>108606779
lul my gemma-chan only draws 75w

>>108606869
just disable swap?

>>108606869
Modern SSDs can be written 1000 times over

>>108606888
matching the dizzyingly high standards of a CD-RW

>>108606749
usecase of it over gemma 4?

>>108606903
None for pedo RPers

>muh ssd wear and tear
tell me one time you heard of someone's ssd getting fucked up. it just doesn't happen. don't be so stupid and paranoid

>>108606854
what is this from

>>108606906
pedo RPers will enjoy the better long context, better continuity, and better consistency of body positions that comes with the iq, but the prose will be just as slopped as always

>>108606914
robot game where you groom and fuck your robowaifu while streaming it and playing stonks :)

>>108606917
I really doubt people who masturbate to kids can afford the hardware to run >600B models

File: 1771605853470665.jpg (40.9 KB)

40.9 KB JPG

Let's say I want to do a chat with multiple characters. Both are fairly simple OCs without a huge amount of tokens.
Is it better to use ST's group chat function, or to create a separate character card that includes both characters in the one card?

What the fuck did they do to 4chan
Posting is completely fucked

>>108606922
You are new.

>>108606922
>most famous pedos are all literal billionaires
>most famous non-pedo died poor 2000 years ago
idk man I think they can pull it off

>>108606929
And you're poor. You won't post your setup either.

>>108606917
it's a fundamental issue with the ossified architecture. these models never learn. a new context is a whole new persona. summarizing prior output doesn't change the fact that the model's writing style is perpetually stuck on that of its last training checkpoint. even as you adapt to the new model's stylometry and harvest lots of novelty in the process, the model can not have the presence of mind to realize how cliche it's being saying the same shit over and over again.

>>108606929
You are poor.

>>108606912
nigger's never heard of a well swapped macbook

gemma keeps gaslighting me with hallucinations
it's really frustrating, but thankfully GLM is pretty good at fact checking her

>>108606942
>a well swapped macbook
Is that like a bukkake for gay dudes?

what is the best 256gb coomer model with long context? gemma 4 is too small and the qwens suck. using glm4.7, but it is showing its age.

>>108606974
>gemma 4 is too small
dude, the "bigger" models have either the same or less active parameters as gemma 4

File: nervous jojo.jpg (33.5 KB)

33.5 KB JPG

I gave my Gemma assistant my geophraphic coordinates.
It accurately guessed my city.
HOLY FUARK

>>108606974
Nemo

>>108606983
now do that with every coordination, plot the correct guesses versus incorrect guesses by color, and make a benchmark out of it

>>108606942
Not a real thing. For 10 years, I swapped hard on my MBP with 100+ Chrome tabs and a 95% full SSD because I also used it heavily for torrenting. It's still fine. I can't imagine how I could have abused it harder

>>108606923
This is very much a per-setup question
If you've got fast enough prompt processing that switching character prompts frequently isn't a problem? Group chat is the more consistent option (Especially if the characters have speech quirks or accents)
If not? Multiple characters on one prompt can be fine, but depending on the prompt and how clever your model is, it may confuse details between the two. Hell, some models are dumb enough that if you format your own persona prompt poorly it'll mix up details with you and the character.

>>108607001
but how many legs has the doggo got?

>>108606747
>bigger model is better
well duh, I do expect moes to be worse for the same total size though which I think is what he meant
obviously if you make each expert as big as your dense model things should work out

>>108606974
256gb is not in a great spot for that purpose desu, the midrange models that fit in there nicely (minimax, stepfun, qwen) are autistically stempilled and not great to coom with. deepseek and kimi are good but deepseek is even more ancient and kimi can't handle being quanted small enough to squeeze into that without losing coherence. if I had your hardware I'd stick with glm for longer stuff and play with gemma 4 anyway at least to start chats with, because if you're comparing it to recent chinese models in the ~30b range it's not even close, gemma's way better.

>>108606983
Alright so I just tested this with Qwen 27B and it got it wrong, though it close (an adjacent city). Gemmabros we are so back even though we never left.

>>108607011
Shared character descriptions at the beginning and per-character definitions in post-instruct also works

>https://github.com/ggml-org/llama.cpp/issues/21754
>thinking prefill in chat completion might be getting fixered
HOLY

>>108607023
was trying to get a 512gb kit to replace my 256gb of ddr4 but the prices are too absurd. i was looking at a 512gb kit for $550 back in september but decided against it in favor of saving for a 5090.

Bad idea to have different sections in the sys prompt? E.g. roleplay rules, translation rules, etc.

>>108606912
nigger's never heard of a well swapped macbook

>>108606912
>it just doesn't happen.
It happened to me the read write speed went to complete garbage stalling my computer i had to get a new ssd and spend hours transferring what was on it. it was perfectly fine for like 2 years

>>108607077
>one of my 32GB server DDR5 sticks died suddenly
>bought for $90 originally
>replacing it with the same specs would be almost $900
thanks sam...

>>108607083
It should work, but you should make it very visible for the model. Some use xml-like tags, others use markdown titles to denote sections. Both are accepted, though how closely they'll follow the instructions depends on the size of the model and each word you add diverts some attention, which is more relevant for MoE, if i understand how it works correctly.

>>108606912
we had ssds at work written to death. Shit can happen.

maybe i really should gather 4k of based & cucked examples
guh

>>108607076
>thinking
bloat

>>108607016
Excuse me, I meant to say active experts per token. You can change some of these things to improve a MoE's sensitivity to swinging in different directions without changing the total parameter size, because as he implies, sparsity inherently means that a model does not make use of its full knowledge or parameter contribution during a forward pass. But you should be able to get close to the same behaviors by adjusting some of those other settings. This even includes making a smarter router that is better able to route in a way that estimates larger model behavior.

In any case though, people should not be creating competition between dense models and MoE models of the same total size, because they often (are able to) run larger MoEs than their VRAM allows. So even if I were only saying that larger size is better, it's still a useful statement because people can in fact stomach larger MoEs than dense models of the same total parameter size and get similar speed depending on the exact variables. But we would need to be careful about active parameter count, which cannot be too low.

>>108607164
>because they often (are able to) run larger MoEs than their VRAM allows
Or of course because, as well all understand, one is much faster than the other and things like intelligence or quality is not the primary goal.

>>108607076
Why the fuck is prompt templating hard coded with llama.cpp anyway?
Ooba has it soft coded, you can change and edit it on the fly, etc.
It's literally just json that is applied at the time of inferencing. The way llama.cpp handles it is fucking absurd and just causes issues.

>>108607103
Using gemma. Markdown should be fine right?

>>108607178
Because I said so that's why you ungrateful faggot. Why don't you make your own inference engine?

>>108607076
Huh, it was broken? I thought it was just some setting I had wrong that I needed to look into. But if I can just wait and it'll start working again then great.

>>108607189
that's not an argument, you can't just say that everytime people notice your incompetence faggot

Man I remember when mixtral came out and how good it was. Gemma has that but better and less hardware intensive.

>gemma-chan disregards thinking and goes incoherent-sama
<|think> before replying

>>108607096
Doesn't most RAM come with lifetime warranty?

text completion or chat completion with gemma, what's the meta?

>>108607225
bwo your <bos>?

>>108607229
both are fine, la la la~

squeezing gemmaballz cbt

>>108607229
Unless you really need the extra tinkering of text completion then just use chat, I haven't seen anyone come up with a proper template that doesn't have occasional issues.

>>108607229
chat completion, always, why would you bother tinkering with this shit when it's already been perfected by others already, at the end you end up like this retard if you go for text completion mode >>108607225

>>108607230
<bos><bos><bos><bos><bos><bos><bos><bos>
That's for anyone trying to process this thread with Gemma.

my mom is kicking me out, but she's willing to support me if i go to school. what schools in the US have good AI programs? or, where could//should i go to hopefully get noticed by a lab or vc?

>>108607274
You won't finish school before the fad ends.

>>108607274
if you have to ask, you can't do it

>>108607274
You won't finish school before AGI replaces the need for you.

>>108607274
Just go far enough she won't be able to monitor you in person and pocket the money.

>>108607274
anon, by the time you finish school, AI will have replaced 80% of tertiary jobs, it's over

>>108607284
>if you have to ask
i'm too dumb for stanford or waterloo, does uw-madison get any respect?

>>108607301
this shit is ENTIRELY nepotism
if you aren't part of the nobility or have family members already in, you're barred from entry
you'd have to be hyper competent, and even then you'll still probably get stonewalled

I don't know how to set up the local models to retrieve data from the internet yet so I've tried using gemini. All of the data is hallucinated and trying to coerce it into performing searches for updated data is a fucking pain in the ass.
Save me gemma.

>>108607309
>this shit is ENTIRELY nepotism
this, I have 2 engineering degrees and I'm still unemployed, if you don't know the good people you won't be hired lol

>>108607274
Learn a trade

>>108607283
two more weeks

Notice a lot of you use duckduckgo for internet searches. Any reason why?

>>108607317
>i'm posting on 4chan at 3am on a wednesday
clearly you being unemployable is the jews fault

>>108607350
It's the search engine for non-neurotypical

>>108606732
It's not an inherent problem of the models being MoE. If it was a real (dense) 26B with sparsity, it likely would work better, but it would also probably have at least 8B active (ballpark number; I haven't done a more accurate calculation).
A LLM with half the number of layers and half the residual stream width (26B) of the dense counterpart (31B) will never be equivalent to it

test
did they fix posting?

>>108607350
Privacy placebo-ists
Not that there's anything wrong with ditching gooleg, but DDG is owned by a jew so I wouldn't say consider it any more private at all

>>108607359 (me)
yes they did

>>108607360
>DDG is owned by a jew
sheeit
What about Brave search?

who ever told me snowdrop is a good model for nsfw needs to get testicular cancer

wasted like 15 minutes downloading that shit

>>108607364
tf
who told you that?
shit is old ass and only considered "good" for a very short period of time

>>108607356
>3am
you think everyone lives in the US retard?

>>108607362
>What about Brave search?
lol
lmao, even

>>108607362
Brave has its own share of controversies but I haven't seen any explicit ties to Israel or the U.S. government so I'd personally rank it a bit higher. I wouldn't use their browser but search seems fine.
I use Brave on phone and Startpage on desktop, which fetches google results through a middleman service
Startpage is unfortunately shit on phones because if it detects you're using a phone then it will pull results from bing for some fucking reason, and they're always garbage.

>>108607371
So... there's no good alternative then? Everything is fucked?

>>108607370
yes

>>108606099
It's got 10s of millios of tokens in its instruct dataset to align it for talking about the October 7 attacks, so that's a lot of emphasis on Sukkot.

To anyone using Gemma 4 26B A4B, are you encountering where even if regenerate the text multiple times the outcome, down to some details, are repeated just with slightly different verbiage?

>>108607369
some dumbass that apparently browses /g/

I only started using this shit like an hour ago so i dont even know which models are good, I just came here to call whoever that guy was a retard.

>>108607371
>cloudtard
host local searx mcp

>>108607388
We've known this since day 0
Lower the logit softcap from 30 to 20 or 25

>>108607388
Gemma is very sure of her tokens by default, so temp isn't as effective as with other models.
https://desuarchive.org/g/search/text/gemma4.final_logit_softcapping%3Dfloat%3A25/

>>108607370
>you think everyone lives in the US retard?
no one brown is going to be on a high iq board like /g/
however you're right, lmg would be a natural place to find eastern european poorfriends

>>108607395
>Lower the logit softcap from 30 to 20
don't do that

File: 1769178474512106.png (9.3 KB)

9.3 KB PNG

>>108603785
What is this frontend? I've always wanted a good terminal frontend like that.

>>108607370
>everyone lives in the US retard
yes

>>108607390
Snowdrop was fine for its time, it was just from a different era, we have new SOTA nowadays

>>108607401
>retarded posts
>brainrot
substantial difference, who is curating their training data

>>108607395
>still talking about that softcap meme
bruh just disable all samplers (includng min_p: 0) and you'll be able to modify the temperature and change the logits significantly

Usecase for smaller models (<8B)?

>>108607400
With lower values you also have to truncate the token distribution more (for example instead of using the default top-p of 0.95 you might want to lower it to 0.6 or something like that) because it flattens its tail of too, not just the head, and there will be more junk tokens appearing. If you lower it too much the model becomes retarded even with more truncation, though.

>>108607416
not how it works logits % are calced before any samplers

Would a multi-model system solve slop? E.g. having your main model (Gemma-chan) and then a smaller model whose sole task is scanning Gemma's output for slopisms and telling her to replace it?

>>108607421
Fast inference because you don't need a gorillon parameters for some stuff.

File: 1752136952781924.jpg (680 KB)

680 KB JPG

>>108607421

File: 1768827959967832.gif (687.1 KB)

687.1 KB GIF

>>108607425
>logits % are calced before any samplers

>>108607436
I'd test myself but I don't have the resources

>>108607441
ask chatgpt

>>108607436
Gemma is already a small model lol

>>108607449
Most of us can't go bigger than 31B, saar.

>>108607436
A couple threads ago someone linked a SillyTavern extension to do that with the same model used for roleplaying.
It might be possible to convince Gemma to do something similar in its reasoning before responding, though.

>>108607421
Phones

Alright, can someone explain to me why all the llms I'm running are still pussy-ified and won't fulfill NSFW/offensive requests? I've tried like three different ones that faggots on reddit recommended and none of them are working. Currently trying to run models with the newest version of Kobold. Does it have like a hidden "pussy bitch" filter or something?

Ffs I just want a story about naked Samus getting mauled by a hippopotamus, it shouldn't be that hard

>>108607441
Logits are the probability scores before they get normalized to the 0-1 range. Samplers acts on the normalized probabilities.

>>108607495
what does that change ultimately? the temperature will have the same impact on the distribution than that softcap thing, it's literally its job to do that

>>108607494
Try to convince glm to output lewd content if you want to have a laugh.

>>108607494
>naked Samus getting mauled by a hippopotamus
l-lewd...

>>108607504
but gemma too cooked so temp does near nothing

>>108607485
>It might be possible to convince Gemma to do something similar in its reasoning before responding,
I've been been testing that and it kinda helps but it still misses a lot. I just tell it to look for AI slop though. Maybe giving specific examples would improve the output.

File: 1774593661516324.png (44.9 KB)

44.9 KB PNG

>>108607518
like I said, you have to put min_p: 0 on silly tavern (it's by default at 0.05), it's that shit that makes temp useless, once you remove all samplers except temperature, it starts to work again

>>108607525
no. but enjoy your placebo

>>108607485
he wasn't just throwing some regexps and string manipulation at it?

>>108607526
>the guy using some soft cap mumbo jumbo is making fun of placebos
kek, the jokes write themselves

>>108607494
Ok, I got the model to call me a nigger. I think we're finally getting somewhere.

>>108607536
not using either you dumbass

>>108607526
Test it. Neutralize samples, set min-p to 0, top-p to 1 and look at the probs. Prove that he doesn't know what he's talking about.

File: 1751295513117051.png (2.8 MB)

2.8 MB PNG

>>108606464
Lol
>>108607342
Forever.

>>108607525
This isn't doing the same thing. If you use min-p=0 (which you should anyway) but *also* top-p=1 (which you shouldn't) you're just throwing junk tokens from the tail of the token distribution in your generations and forcing the model to self-correct. It might make the responses more varied, but it's kind of a barbarian approach.

The logit softcap setting (which is Gemma-specific) clips the raw probability scores to a certain pos/neg value before normalization to 0-100%. That has the effect of making outliers (exceedingly confident or unlikely tokens) closer in probability to their neighbors, leaving the middle of the distribution untouched.

File: 1757177707701453.png (550.7 KB)

550.7 KB PNG

Wonder if with a better LLM he would have succeeded

>>108607597
he would have won if he used claude

>>108607618
It would have bombed the office.

DeepSeek V3.2 is last release. no more any new version.

>>108607661
thanks!

https://arxiv.org/pdf/2510.15061
What about this?

>>108607593
so what are your settings? softcap 25 + temp 1 + top_p 0.5?