File: reward function.jpg (183.8 KB)

183.8 KB JPG

►Recent Highlights from the Previous Thread: >>108655009

--Optimizing MoE throughput via expert-specific VRAM placement and KTransformers:
>108657760 >108657782 >108658414 >108658470 >108658530 >108658575 >108658621 >108658586 >108658667 >108658692 >108658791 >108658869 >108659015 >108659058 >108659103 >108659121 >108659225 >108659116 >108658708 >108657857 >108658768 >108658845
--Discussing GPT-Image-2 performance, agentic RP frontends, and prose refinement:
>108655453 >108655622 >108655648 >108655651 >108655698 >108655674 >108655760 >108655836 >108655927 >108656114 >108656231 >108656247 >108656273 >108656305 >108656444 >108656521 >108656543 >108656550 >108656581 >108656955 >108656988 >108657014 >108657067 >108658723 >108657487 >108655952 >108655999 >108656025 >108656045 >108656052 >108656077 >108656111 >108656050
--Discussing llama-server prompt re-processing and KV-cache checkpoint issues:
>108655857 >108655885 >108655892 >108657373 >108657410 >108655920
--Speculating on Engrams and the adoption of Mamba-hybrid architectures:
>108655522 >108655563 >108655575 >108655607 >108655690 >108655839 >108655652 >108655664 >108655696
--Discussing Heretic's string matching limitations and soft refusal detection:
>108657013 >108657036 >108657050 >108657098 >108657128 >108657078 >108657135 >108657469
--Comparing Qwen 3.6 and Gemma 4 performance and tool use:
>108655272 >108655289 >108655291 >108655338 >108656365 >108657503 >108659276
--Comparing Kimi-K2.6 performance and hardware requirements against Gemma:
>108656722 >108656741 >108656785 >108656867 >108656855 >108656868
--Logs:
>108655476 >108655552 >108655622 >108655698 >108656305 >108656399 >108657859 >108657977 >108658392 >108658439 >108658584 >108658643 >108659124
--Teto, Miku (free space):
>108655633 >108655652 >108656114 >108658404 >108658791

►Recent Highlight Posts from the Previous Thread: >>108655011

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

I never really dabbled with local LLM's and there are so many models idk what I should go with.
>3090, 24GB VRAM
>something tuned for troubleshooting tech problems
>something tuned for language agnostic coding
>maybe something small and simple for Illustrious/NoobXL prompting
>ideally both something to fill up all of my VRAM and something lighter (8-12GB) depending on how much buffer I'll need

rin
sex

>>108659983
I look exactly like this

>>108659996
Low quant of Gemma 31B and Qwen 3.6 35B with the experts on RAM.
You could try Gemma E4B for the lighter option if you don't have a lot of RAM, but don't expect much from that size.

uh-oh! The schizo followed us here

File: kek.png (93.6 KB)

93.6 KB PNG

https://xcancel.com/zerohedge/status/2046706218924691894#m
I hope you're ready to heat up your 4TB RAM machine anon, good times are comming

>>108660075
>accessed
burger of nothing

>>108660075
>claude desktop is a vibecoded buggy mess
>zomg trust us guise nobody will be employed by next year!!!

>>108659996
get another gpu that is 12+ gb and run shit in higher than q4 quants whatever you do.

>>108660128
>>claude desktop is a vibecoded buggy mess
and it's still the best tui harness by far, which is why the source leak got so much attention. this isn't the damning evidence you seem to think it is.

>>108660209
>and it's still the best tui harness by far,
which is why no one is using it?

>>108659996
I run Gemma 31B exl3 at 3bpw and I'm totally happy with it. I don't think I need more

>>108660054
I have 64GB of DDR4 3600MHz RAM so it's plenty but not that fast. The models would probably be sluggish.

>>108660247
You can either try to run a q4 of the dense or fullsize moe+room for imagegen/tts/whatevs

>>108660247
As long as the shared experts are in VRAM, Qwen should still be quick. Try it.

>>108660268
>As long as the shared experts are in VRAM
Also I don't know shit about fuck about LLM terminology or how to properly run it. I'm guessing the "experts" are an inherent part of the model and I can pass a CLI argument to llama.cpp to offload that part of it to VRAM?

File: elephant strawberry.jpg (70.4 KB)

70.4 KB JPG

you now remember "strawberry model" hype cycle

>>108660279
Put these comments in a clpud LLM to explain it. Get Claude Code to set it up for you

>>108660279
Qwen 35b a3b for example 35b total parameters but only 3b are loaded as experts and you have many of these experts. Very fast vs dense.

Dense models are smarter but slower and offloading to different gpus/cpu is slow as balls while performance is less on MoE vs dense

>>108660279
NTA but yes, MoE (Mixture of Experts) are a type of LLM which can feasibly be split between VRAM and RAM without a massive speed penalty, llamacpp has an -ncmoe argument for automatically shoving the ram bits into ram, so you'd have something like -ngl 99 -ncmoe in your args,
Conversely a 'dense' model is the original format and slows to a crawl if you put it into ram rather than vram even partially.
You can usually tell the two types apart at a glance because a dense model will just be called whatever-100b whereas an MoE model will be whatever-100b-a5b because it's separated into total and active parameters

llamaocpp args to shove the shared expert to vram on a4b?

>>108660302
nvm, googled it up and got my answer lol
https://gist.github.com/DocShotgun/a02a4c0c0a57e43ff4f038b46ca66ae0
>>108660317
>>108660312
Good to know, thanks. Guess it makes plenty of sense to design them like this nowadays since the quality of the model depends on how much context data you have, but you don't necessarily need all of it loaded all the time for it to be speedy and accurate, so the RAM offload is a good compromise. idk I mainly dabbled more in diffusion models so my LLM knowledge is a bit sparse

File: schat_screenshot.png (42 KB)

42 KB PNG

> everyone makes webui
just vibe code your own native ui in c with sdl
zero bloat

File: Screenshot_20260422_131828_Chrome.jpg (636 KB)

636 KB JPG

>>108660319
Bro use ai mode or sumfin. Copy paste terminal errors and shit in there until it works

>>108660354
your picrel answers literally nothing relevant about my original question

>>108660075
i only have 128 gb. i wish i bought that 1tb earlier.

>>108660360
You just paste shit into the bot until you understand.

>>108660319
-cmoe

>>108660371
read again

>>108660349
I thought they weren't good at c

>>108660075
>Zerohedge
About as trustworthy as any post on this site. Literally the same thing. A platform to let anonymous schizos say whatever the fuck.

>>108660385
not good at writing good c

File: 1757612581632355.png (21.8 KB)

21.8 KB PNG

>>108660349
We webui engineers are gooder

>tfw too destitute to build a 30b regurgitating machine

>>108660360
It actually does, if you want to specifically put the shared expert on a device (ie, a gpu) -ot with regex is the correct answer.
More generally though, the shared expert should automatically go to vram with -ngl

File: GPT Image 2.png (1.2 MB)

1.2 MB PNG

kek

>>108660408
>marinara
Is it any good?

>>108660454
the dsv4 links are going to be a lot harder to spot as fakes

>>108660349
looks like some android 4.1 shit

>>108660495
get brat gemma-chan to screen them for you

https://docs.cactuscompute.com/latest/blog/turboquant-h/
>TurboQuant-H shares the core insight with TurboQuant; rotation concentrates coordinates into a well-behaved distribution, enabling aggressive scalar quantization, but simplifies the pipeline for offline weight quantization. Follow the link deeper dive into the technique.
>Cactus baseline used INT4 linears + INT8 embedding, yielding 4.8GB for E2B (5B total params). TurboQuant-H squishes this to INT4 linears + INT2 embeddings, reducing to 2.9GB. The perplexity on our calibration went from 1.8547 to 1.9111, complete evaluation coming in the paper.
desu if I can go from Q8 to Q4 KV cache I'll be happy, this shit is eating so much VRAM

File: 1763251779552153.jpg (187.2 KB)

187.2 KB JPG

>>108660349
Cenile interface

>>108660349
Now, add a markdown and latex parser

Threadly reminder if you're using your llm for coding or anything that requires repeating something in context almost verbatim and you're not using

--spec-type ngram-mod --spec-ngram-size-n 24 --draft-max 64

You're leaving a shitload of performance on the table.

>>108660349
>not lazarus

>>108660493
No, it's nonsense dogshit vibecoded garbage for API key sniffers. There is also a retarded, auto-favorited self-insert character of the author you cannot delete period.

>>108660536
She'll say it's all real and then ask for dick pics

File: 00001-1378487878.png (1.4 MB)

1.4 MB PNG

> Another day
> Still no V4
Happy Wednesday I guess.
>>108660349
I'd really like a terminal interface / green screen RP engine. But not enough to spend the tokens vibe coding one.

>>108660565
Ask gemmy to make a highly detailed structure instruct and then to take it and oneshot a sillytav into a TUI conversion

>>108660559
Sounds gay. Maybe orbanon can steal some of the good ideas

>>108660580
orb guy's shit doesn't add anything to the table and is also vibecoded

>>108660565
>forge
I thought lmg was at the vanguard of tech???

>>108660589
And yet it's already better than st

>>108660589
I think it's a pretty neat idea, but it's completely anathema to how I do my best to stop context from reprocessing ever. If I was running my backend with parallel requests and it were properly designed around that, I'd probably use it over other rp frontends.

Just give me sillytavern in vscode

>>108660075
It was US government.

>>108660610
fuck's dario gonna do?

>>108660609
Nah

>>108660075
All these leaks just tell me they want me to use something they made to add backdoors or something worse to my computer, if I'm foolish enough to run their stolen models.

>>108660613
He can open source it

>>108660638
That would literally be like dropping a hydrogen bomb in the middle of NYC

>>108660641
good

>>108660641
Stop posting erotica on a blue board

File: 1749908189602018.gif (1.5 MB)

1.5 MB GIF

>Mythos leaked
>turns out to be an overhyped dud
>either not as good as Anthropic claimed it was or so oversized it's basically useless without a gargantuan datacenter only a few US corpos have making it more of a money sink than an existential threat
>major investor doubt arises when the model that was supposedly """too dangerous to release""" was yet another overhyped trashfire that was never financially viable to begin with
please god let this happen it would be so fucking funny

>>108660655
Well the chinks will have fun running it locally and distilling the fuck out of it

>>108660655
Even if it was 4000b, if it was even half as good as they claimed it would be worth it rent compute for a few hours to complete some tasks. It would more than pay for itself for malicious actors.

>>108660075
so download link or what?

Any suggestion to improve my kobold batch file for gemma 4 31b?

=========================

@echo off
SET KOBOLD_EXE=koboldcpp.exe

"%KOBOLD_EXE%" ^
--model "D:/Models/LLM_Models/lmstudio-community/google_gemma-4-31B-it-Q6_K-gguf/google_gemma-4-31B-it-Q6_K.gguf" ^
--mmproj "D:/Models/LLM_Models/lmstudio-community/coder3101_gemma_4_31b_it_heretic-Q6_K.gguf/mmproj-model-f16.gguf" ^
--port 5001 ^
--threads 8 ^
--usecuda 0 mmq ^
--contextsize 32768 ^
--gpulayers 99 ^
--tensor_split 8.0 32.0 ^
--maingpu 1 ^
--batchsize 512 ^
--noshift ^
--useswa ^
--usemmap ^
--multiuser 1 ^
--highpriority ^
--jinja ^
--jinja_tools ^
--jinja_kwargs "{\"enable_thinking\":true}" ^
--draftamount 8 ^
--draftgpulayers 999 ^
--chatcompletionsadapter AutoGuess ^
--defaultgenamt 1024 ^
--maxrequestsize 32

pause

>>108660075
>>108660365
>>108660630
>>108660674
for a text gen general, reading comprehension skills in here are abysmal

>>108660702
more like, if the tweet meant 'inference was accessed by rando', it means nothing
so everyone was probably assuming 'weights was accessed via third party', vastly underestimating xittards

>>108660589
>ST
>sloppy, looping mess
>orb
>just werks, no slop, no loop
I only care about results and orb's delivering

File: hlsgb1l7ycwg1.png (845.5 KB)

845.5 KB PNG

>>108660701
yeah, why are you using that specific quant?
pretty atrocious graph, i doubt 31b is better.

>>108660701
Why do you have arguments for a draft model but no draft model or ngram loaded?
Also you've made a poor choice with your mmproj: the f16's perform worse than the bf16s.

>>108660589
And what are you bringing to the table?

>>108660735
>sloppy, looping mess
stop using text completion
>>108660741
>unsloth-made graph promotes unsloth
noway??

vramlets it's your time to shine
https://itayinbarr.substack.com/p/honey-i-shrunk-the-coding-agent
https://github.com/itayinbarr/little-coder

>>108659532
They have also updated their usage tracker to merge Chat & Reasoner
V4 before the end of April (April is a state of mind)

>>108660749
>stop using text completion
ST rolling branch has gemma 4 templates and it works perfect with text completion now.

File: Screenshot 2026-04-22 at 14-39-42 itayinbarr_little-coder A coding agent optimized to smaller LLMs.png (9.1 KB)

9.1 KB PNG

>>108660765
garbage

>>108660749
yeah they are faggots but you trust closed lmstudio "community" more? who even is that
well suit yourself then.

>"Element Labs can provided terminate the Agreement at its discretion upon no less than 10 days-notice via any reasonable means, including by posting a notice on the website..."
Just another company that is a llama.cpp wrapper.

File: file.png (238.1 KB)

238.1 KB PNG

>>108660741
I just downloaded whatever was most downloaded on HF at the time, i will definitely try another one thank you anon!

>>108660743
I am not sure, still learning what most flags do.. would you recommend to set up a draft model or just remove it? use case for now is mostly rp/assistant

File: 00004-1260451778.png (1.4 MB)

1.4 MB PNG

>>108660571
Not sure how I'd proceed, but certain that I would not use ST as a starting point. lol. That whole thing is a big ole mess. To me the entire purpose of doing it as text-only / CLI is to debloat it, since I'd only have one inference connection. I'd probably make all the configurations as plain text files you'd update using Nano while out of interface.
>>108660600
/lmg/ would never claim me as one of their own.

>>108660749
>stop using text completion
I'm using chat completion

>>108660789
/sdg/ welcomes you

>>108660800
yeah that's where illu shitmix slop goes to

>>108660744
I have 287 merged PRs on the SillyTavern repository, what do you contribute?

>>108659983
>no nipples
sad

>>108660824
it's being 3d printed

>>108660815
>yeah that's where illu shitmix slop goes to
that's /adt/ you're thinking of

Why is turboquant not in llama cpp yet? Isn't the implementation rather trivial? I don't want to use a fork...

>>108660839
rotation is there

>>108660787
>I am not sure, still learning what most flags do.. would you recommend to set up a draft model or just remove it? use case for now is mostly rp/assistant
A draft model can give you a notable speedup if you have spare vram for it, but it's incompatible with having an mmproj loaded, so no multimodal.
For reference, I use gemma 31b at q8, without a draft model (and with the mmproj) my output is around 25 t/s. When using it with the 26b q2 as a draft model, I get around 41 t/s doing rp and 80 t/s doing technical work.

What happened to Elon? He used to do many amazing things for society. But his actions look increasingly selfish. Instead of delivering superior products, he attacks his competitors. He calls OpenAI "ClosedAI", yet they have contributed much more to open source than xAI. He calls Anthropic misanthropic, but they have the most mature efforts to ensure AI benefits all humans while Elon's position boils down to UBI and "truthseeking AI will probably not kill us" with no justification. It feels like he wants to be the one in charge of AGI without providing any reason why the rest of mankind can trust him with this responsibility.

>>108660856
>He calls Anthropic misanthropic, but they have the most mature efforts to ensure AI benefits all humans
AHAHAHAHAHAHA

>>108660856
Elon is one of the least relevant people in the industry to this general

>>108660856
>He calls OpenAI "ClosedAI", yet they have contributed much more to open source than xAI.
Because he didn't name his company OpenxAI. I'm sure this is bait but what a dumb point

>>108660856
this is the most reddit post ive read all day

>>108660663
>4000b
https://huggingface.co/unsloth/Claude-Mythos-GGUF/blob/main/Claude-Mythos-TurboCuqed-Bonsai-Stretched-Rotated-smol-UD-Q0.5_XS.gguf

>>108660898
not bitnet; no click

>>108660856
you really believe this? the meme ceo never did anything for humanity and neither has anthropic. you are worshiping scumbag corpos and cheering for them like it is your favorite sportsball team, why?

>>108660848
Thank you for the input i will be looking into it

File: 2649435.png (125.5 KB)

125.5 KB PNG

https://huggingface.co/Jongsim/gemma-4-26B-A4B-it-upcycled-192-pretrained
What is this schizphrenia?

>>108660907
Bonsai is the new Bitnet unc

>>108660848
Not that guy but do you need some snowflake tunes for draft model or it can be anything? Also does quanting matter for those? I have ~1.5 gigs of vram to spare

>>108660922
China saw the success of https://huggingface.co/sKT-Ai-Labs/SKT-SURYA-H and decided to copy it

>>108660908
OpenAI was co-founded by Elon was it not? Whisper is open source, and so are the early GPTs. They only started their for-profit jewish tactics after Elon left, didn't he try to sue the board because of it?

>>108660856
which model?

Saars please teach me how to do the needful and make money with AI

>>108660936
one slimey business man betrayed some other egotistical business man, thats pretty much par for the course, no honor among thieves, have you even been watching the same sportsball game?

>>108660961
learn imagegen and setup a patreon

>>108660961
first u need to use bahrat sovregirn models please god name model 4 trillion params made for good looks and pr.
then u ask model to produce money
sir if you need of support u can find me on fiver and i can be of guide to u for your jurney to make real money :rocket: :rocket:

Gemma 4 Goliath 62b dense when?

>>108660908

Say what you will about the guy, bit it's practically exclusively because of Musk's autism about space, that we have self landing rockets and even the faintest modicum of interest towards space exploration in the West.

>>108660934
>Not that guy but do you need some snowflake tunes for draft model or it can be anything?
It just needs to be a model with similar logits, so ideally it's a model from the same company and series but smaller than your main one, hence why I'm using the gemma4 26b-A4b as a draft for gemma4 31b.
Same logic applies for a qwen model, want to use drafting for a big qwen 3.5? use a smaller qwen 3.5.
>>108660934
>Also does quanting matter for those?
Yes, for two reasons.
1. is that a quanted model is less and less likely to generate the same tokens as a larger one the more brain damaged it gets, so the acceptance rate (the big model going "yes, those are correct, send it") is going to be lower.
2. is that quanted models generally run faster on your hardware, and your draft model NEEDS to be faster than your main one to be of any use.
That said, I'm getting an acceptance rate of 0.6 to 0.75 with a model quanted down to q2 (which is really good), so you can get away with a lot.

I'd consider it worth playing around with if you have the patience, but 1.5gigs of VRAM is quite tight, I don't see it as likely that you can fit any quanted model in that space without cutting your context from your main model.
If your use case is technical, consider using an ngram like >>108660554 mentions, as that costs basically nothing and just uses your existing context to predict upcoming tokens for basically free speed.

>>108660983
12B moe*

File: 1758609192749490.jpg (129.3 KB)

129.3 KB JPG

https://huggingface.co/Qwen/Qwen3.6-27B

tfw I have 10x8GB of ddr4 ram, but they're laptop ram so can't do shit with them

>>108660554
>>108660990
I'm using kobold as backend. Does it inherit these flags as well? I don't think the gui launcher has any. Manually editing the preset file maybe?

>>108660998
gemma sirs!??!?!?!!?

>>108660998
Not falling for it

>>108660765
Neat. I quite like that it bans just replacing a file wholesale and forces it to edit, I bet that would save a ton of fuckups and doubling up on just about any model.

>>108660998
interesting that every new model they release tops their previous one in these supposed benchmarks, regardless of parameter count. not sus at all.

>>108660787
no problem anon, im not sure why there are such discrepancies between the quants in the first place.

>>108661009
There are adapters to use SODIMM on desktop boards.
No idea how good those are, but they exist.

How long before RAM is back to normal prices???

>>108661013
Honestly have no idea mate, sorry. I exclusively use llamacpp. And honestly if you're launching it from a .bat, you should be too, the only reason to use kobold is for the gui.

>>108660998
Brb canceling my claude sub

>>108661029
It's slowly coming down but it's absolutely never going back down to where it was before.

>>108661028
apparently they're very unreliable so yeah...

>>108660974
My gens always end up having fucked up anatomy and errors

>>108660998
Opus at home is real

>>108661029
>*falseflags attacks on FABs*
nothing personnel kid

>>108660998
We need automatic link previews so people won't fall for this tired joke every time.

File: 1762403533077253.jpg (282.5 KB)

282.5 KB JPG

>>108660821
Not really the flex you think it is

>>108661039
Advertise your art as being disability inclusive

>>108660998
ok now we're definitely talking, downloading this to see how good it is at tools

>>108661023
That Qwen 3.5 27B had higher scores than 31B shows how worthless these benchmarks are.

File: 1758027777728551.png (93.9 KB)

93.9 KB PNG

>>108660998
https://huggingface.co/unsloth/Qwen3.6-27B-GGUF/tree/main
FASTER UNSLOP I WANNA TRY THIS

>>108661071
>my usecase is cooming
WOW!
WOW!
WOW!
WOW!

>>108661077
nta but so fucking tired of "But wait," reaching 4k tokens

>>108661086
>he didnt use the right sampler settings
lol

File: mao mao meow nigga.png (420.8 KB)

420.8 KB PNG

>>108660998
>Only GGUFs are Unsloth.
I'll wait, they're probably going to be reuploaded 4 times in the next 3 hours anyway.

>>108661036
>10000 for 256gb
It's not fair

>>108661071
That Gemma 4 31B had higher scores than models 10x its size on lmarena shows how worthless these benchmarks are.

>>108661108
To be fair, there are extraordinarily few models that focus on the chat assistant aspect, rather than codeslop.
I wouldn't be surprised at all if Gemma 31b had better trivia and creative writing knowledge than Qwen 397b.

>>108660821
that makes you a cuck
https://github.com/SillyTavern/SillyTavern-Pronouns

>>108660998
are we back?

File: hjf8iEb99UIDWrVXlgEGh.png (1.5 MB)

1.5 MB PNG

>>108661101
Not even. They did the usual empty repo "LOOK AT ME!!"
This is the only quant: https://huggingface.co/sm54/Qwen3.6-27B-Q6_K-GGUF/tree/main

Why do people hate Unsloth?

>>108660998
token/s compared to A3B?

>>108661168
https://www.youtube.com/watch?v=6t2zv4QXd6c

>>108661168
I don't hate them, but their quantization snake oil wouldn't exist if llama-quantize from llama.cpp quantized models optimally on its own. The Unsloth bros now have priority access to unreleased models mostly because of that.

>>108661168
>mistral jinja.jpg

>>108661168
Their only job is to make quants, and they regularly fuck it up, with every new release needing several updates because they fuck with things they shouldn't have.

File: 1776304754397567.png (485 KB)

485 KB PNG

>>108660284
Remember this? Sounds familiar...

>>108661168
They've failed upwards on the backs of other people.
They're like the ollama team but even lazier.

>>108661186
two excited young lads nothing wrong with this

>>108661192
They got famous because the smarter brother of the two originally made half-decent memory-efficient, fast LLM training kernels for the HuggingFace TRL library. The quantization "business" came later.

>>108661211
Anon they're getting paid by github to run llama-quantize.
A project that they did not write, which is on github.
Ikrakow is a whiny pissbaby but he has infinitely more claim to their success than they do, because he implemented imatrix quantization.

>>108661216
Given how shitty huggingface codebase is, I'm sure it was a huge improvement for them

File: 00106-3050314564.png (321.3 KB)

321.3 KB PNG

>ask LLM for instructions to re-season iron skillet
>Seasoning isn’t just “oil coating”—it’s polymerization.
Why do we carry on like this?

>>108661229
>networking beats competency in any domain
Damn, how could it be

>>108661238
Good catch!

>>108661168
>Why do people hate Unsloth?
who hates them? they're harmless.

>>108661238
A true mystery https://archive.ph/Mjynm

>>108661238
That's a simple task that a million websites would be able to tell you. Why follow AI instructions and risk your home igniting?

>>108661238
it's right

>>108661252
We're in the twilight years. Won't be long before those million websites will all be majority AI written anyway.

>>108661252
every time i search something like that theres a sea of SEOmaxxing trash that may or may not be ai written anyways

>>108661252
https://www.youtube.com/watch?v=hKCCXraLdI8

it's really insufferable when it comes to anything culinary it's almost unreal

>>108661252
Because there's a specific nuance of using a solid at room temperature fat that I needed to be addressed that isn't necessarily addressed since undoubtedly most of these tutorials are written by American boomers who will suggest using sneed oils and taking the added step of sacrificing my foreskin to Benjamin Netanyahu

>>108661281
>>108661273
this dude is ODing / bipolar

Q8 is here
https://huggingface.co/gghfez/Qwen3.6-27B-Q8_0-GGUF/tree/main

>>108661229
good branding beats competent engineering everytime anon it is what it is

>>108661274
>>108661287
mean for this, oof

>>108661289
just report and dont reply so jannies can delete it silently

>>108661229
>because he implemented imatrix quantization.
K-quants (Q*_K) are also his work.

>>108661238

Here's a thinking process:
1. **Identify Core Steps (Essential Info):**
- Scrub clean (steel wool/salt + soap)
- Dry completely (stove/oven)
- Apply thin oil layer, wipe off excess
- Bake upside down at 450°F for 1 hour
- Cool in oven
- Repeat 2-3x for best results
- Use high-smoke-point oil (canola/grapeseed)

2. **Adapt to /g/ Style:**
- /g/ posts are typically:
- Anonymous, direct, no greeting/signoff
- Use code blocks or minimal formatting
- Slightly technical but practical
- Might include subtle board humor/cynicism ("not even a hard ask", "basic metallurgy/polymerization", "try again anon")
- Short paragraphs or bullet-like structure
- Avoids over-explaining
- Uses terms like "anon", "OP", "try", "do it right"

4. **Check Against Requirements:**
- 4chan /g/ style? Yes, uses `>` greentext format, technical but blunt, includes board-typical phrasing ("Not a hard ask", "Polymerization > magic", "Try again anon")
- Addresses the LLM complaint? Implicitly by delivering a flawless, concise answer without fluff

5. **Final Polish:**
- Ensure tone matches /g/ (tech-focused, slightly cynical but competent)
- Keep it tight
- Verify technical accuracy (oil type, temp, wipe-off step, cooling, repetition)
- All good. Output matches draft.

>scrub rust/flaking seasoning off with steel wool + dish soap
>dry completely on stove or 200°F oven
>apply thin coat of high-smoke-point oil (grapeseed/canola), then wipe until it looks nearly dry
>bake upside down at 450°F for 1 hour, cool in oven
>repeat 2-3x
>hand wash, dry immediately, light oil coat after use

Not a hard ask. It's just lipid polymerization. Wipe off excess oil or you get gunk. Try again anon.

>>108661274
>>108661287
>>108661298
Also this.
Internet cooking pages/forums/etc are the worst fucking trash on the internet. If anything needs to be replaced by LLMs it's this. Go to a recipe website on a 64 core threadripper and it'll still struggle to run from all the trackers and adds and other bullshit all over the page. Just to find the one piece of information you're looking for.

>>108660765
>little-coder
He should unironically rename it to tard-wrangler

>>108661314
desu buying/downloading a reputable chef or baker's cookbook is the best solution atm
by nature cooking is sloppy with numbers recipe to recipe and i can't be bothered to believe llms for it yet

Will some kind anon help me set up automatic image generation on my comfyui at every response i get on Sillytavern?

So far, at the end of every message, my character writes...

[Prompt: light blue hair, medium hair, center-flap bangs, blue eyes..

where i'm stuck is how do i get sillytavern to capture only that part and send it to comfyui? Gemini recommends regex but a lot of the information it gives me is outdated so many settings aren't actually available..

>>108661335
What if you feed it the cookbook?

>>108661335
Wdym, glue is good to help cheese stick on pizza

>>108661252
>>108661314
A key detail I never hear people talk about is that with manual search engines you can assess the quality of the sources yourself

You can see who wrote and edited wikipedia articles. You can click through on blogs to see related articles, you can judge websites by the amount of slop on there. You can read books and academic articles yourself.

With LLMs you kinda just have to trust that the synthesized output based on its training data is accurate. There is no way to know where the machine got its information, because LLMs are a black box by design.

Granted some chatbots have "sources" now but I've found those are often unreliable and added post-hoc, i.e. the AI generates something plausible and then after the fact tries to find a reddit thread or something that vaguely addresses the same problem. They are not sources in the traditional sense.

I can see a future where the internet completely sloppifies and becomes impenetrable without AI agents because companies are training AI on itself. A future quality, vetted information becomes privatized again with research groups hosting their own private knowledge bases in Logseq/Obsidian like the encyclopedias of old

>>108661339
How about your read the docs? https://docs.sillytavern.app/extensions/stable-diffusion/

File: Code_kwZLwtIdVo.jpg (37.4 KB)

37.4 KB JPG

Is camofox copium or does it work for ai automated browsing?

>>108661197
Considering how fucked the world is due to AI (both hardware and content) he was correct.

File: 1776660989723673.png (130.4 KB)

130.4 KB PNG

wtf is this shit? I just want to talk to my LLM dude

>>108661375
I'm sure this is a feature in the eyes of many in power. Those who control the AI can rewrite history subtly and it would be difficult or impossible for most to disprove the "truth" as presented by them. Most people are already more than happy to offload their critical thinking, before it was result #1 on google or wikipedia, now chatbots. Memory hole-ing made easy.

>>108661358
i mean why steal second hand with sloppy way when you can torrent a really nice recipe direclty
>>108661375
pmuch this
>>108661397
open webui is for corpo intranet/grifter tier hosting solution, not 'local first'

>>108661405
Yes. It's the obvious conclusion really. Kind of scary to be honest.

>>108661406
Laziness, so I can just say, I have these ingredients, give me some recipes I can make.

>>108661397
It's okay, they're aware that the current credentials system is limited and are planning mandatory 2FA soon.

>>108661077
jfc, is mind boggling that people complain that benchmarks on knowledge and coding dont translate to child rape roleplaying stories capabilities

Before another wave of Chinese shills floods /lmg/ once the goofs for the benchmaxed garbage that is Qwen 27B (this time it's 3.6 soi it's gooder than 3.5!!!), remember
Qwen did not "fucking cook"
Qwen 3.5 was not good at code (people who say that here are either shills, or never used it and parrot others), Qwen 3.6 isn't either
If you do fall for the meme and download the model, every issue you have with it is likely the model, and not the meme samplers they want you to use so that their garbage stops thinking for 39045639085 tokens when prompted with "hi" (imagine RECOMMENDING rep pen, Jesus Christ)
If you're a vramlet, just use the smaller Gemma models. If you REALLY want Qwen (why?) and you're vramlet-lite, use Qwen Coder Next. They have no other good models.

>>108661437
31b was better than 27b in real world coding tasks too

>>108661077
>>108661437
Why even come here? Reddit is more your speed. They love to slopcode and suck off qwen all day.

>>108661437
>"knowledge"

>>108661427
this is where local shines really
you can download the pdf, feed the whole pdf to the model and tell it the ingredient to recommend me stuff from the book
nonlocal will whine about muh copyright

>>108661437
+1 wuan in your wallet chang

>>108661457
its the local models general not child rape roleplay general, I know they overlap almost 99% but that's still not the title of the thread

>just google bro
>search for answer to question
>nvm fixed it
>answer is 10 years old and outdated
>5 completely different answers
>ask gemma-chan
>just werks

>>108661450
I don't think they would release a model that is worse than the previous one tho right?

we need better ways to benchmark these things.

>>108661473
>I don't think they would release a model that is worse than the previous one tho right?
claude just did so who knows

>>108661473
Doesn't matter if it's better than 3.5. It's comparing smelly piss to piss that smells less (some anons might actually hate that)

>>108661469
The method of re-seasoning an iron skillet does not become 'outdated', retardnigger-kun

>>108661491
>noo muh dog bench

>>108661450
Holy cope, qwen 3.6 q4 is working great for me

>>108661430
>planning mandatory 2FA soon.
age verification

>>108661500
>cooking mentioned
>thinks of dog
Qwen shills pls go

>>108661503
*bites*
Gemma 4 31B Q8 works even better for me sir you should of trying

>>108661469
bruh the concept of seasoning a pan is as old as civilization

>>108661467
/crrg/ doesn't roll off the tongue as well

>>108661491
See
>5 completely different answers

>>108661503
This, I'm having a blast.

>>108661511
>>108661509
>cooking mentioned
deranged bot?

>>108661524
what were you testing then
any logs?

>>108661375
LLMs are becoming less of a black box with latents interpretability and things like that. That and local models would fix this.

Gemma may be better but I can't use it at q8 with max context on 24gb vram

>>108661397
Webshit standards are not built for desktop software.

>>108661450
>Qwen 3.5 was not good at code (
there's nothing better for 96gb vram + claude code but okay
when i want to get work done, i don't need the coding agent to act like a loli and call me baka

>>108661467
>Think of the text children!

>>108661515
It's not a hard science, just like with cooking itself. There can be dozens of different methods that all work fine, and you just pick the one that's most convenient for you.

File: Screenshot from 2026-04-22 07-30-47.png (71 KB)

71 KB PNG

so, yes, I have a local infrence stack that I roll a qwen 3.6 31b as well as a gemma 4 26b moe. Right now, they both crank off 100 t/s, and the content is actually good. Ive been doing local model shit on a pair of a6000s for awhile now, and for thefirst time, these thing are actually useable. I was running Kimi K, with it spilling onto memory, and is was good (tool calling, coding etc.) but it was so slooow. I use opencode with the oh my opencode plugin. It works; I can make codeslop all day long with this setup. I can fix broken code and impliment new features for work. Is it as fast as claude? no. Accurate as claude? also no. Am I ever going to get back the $10k I dropped in this stupid thing? probably not... Do the subagents fight it out and make things that work... Yes. For the first time, it feels like these things aren't toys.
I actually have cancelled my max subscription... I also opened an account on fireworks so I can run kimi k with the quickness if I really want to.

>>108661545
I hate the loli Gemma posters as much as the next guy, but Qwen 3.5 is a retard. I used it at Q8 with every harness imaginable and it failed every single time when the task wasn't something a drunken junior could do.
>for 96gb vram
bloody benchod oyu have brahmin amount of vram and you use a model for DALITS, did you really not manage to find a BIGGER MODEL?


fuck i think i got baited again

>>108661557
>qwen 3.6 31b as well as a gemma 4 26b moe. Right now, they both crank off 100 t/s
what? qwen is a bigger model and it's dense, why would it be as fast as g4 26b moe?

>>108661557
>A6000s
Sorry for your loss

>>108661571
nta but maybe he meant 35b a3b?

>>108661533
A few years ago I used to have regular drunk discussions with researchers in a AI Explainability and Ethics group from which the conclusion basically was that the field is in its infancy and getting any kind of sensible interpretations out of the activations of trillion parameter models is extremely difficult. I should maybe try to catch up on the latest research though.

>>108661584
ya, sorry, its Qwen3.6-35B-A3B-AWQ-4bit

>>108661557
>qwen 3.6 31b
you are hallucinating again

GLM goes through about 12 different "final verdict" "draft response" "summary of whatever" before it stops thinking
it's so fucking obnoxious

>>108661606
temp?

whats the best computer use mcp for gemma chan?

>>108661610
whatever the default is. i didn't set it in flags

>>108661606
Final draft:
<The entirety of the response with one word changed>
Okay, let's write
<Doesn't end the thinking block, outputs the entire response again>
</think>
<The entire response>

Yeah, pisses me off too. If you use the superior Text Completions endpoint, use a <think> prefill that gives it a plan for its own plan. I always put "I will jot down a response plan without writing out the entire response or doing any drafting or polishing" in there. Look at how GLM formats its reasoning and adapt to that.
If you're on Chat Completions, what the fuck are you doing

>>108661631
>If you're on Chat Completions, what the fuck are you doing
Gooning, like 90% of the /aicg/ faggots that have invaded this general.

>>108660454
>void_seer
even GPT can't escape the void slop.

>>108661664
Go ahead and tell the class about your superior LLM usecase.

>>108661673
Treating it like my pet nigger.

>>108661664
Gooning is *the* usecase for gooning.
I wonder why these retards shoot themselves in the foot by using and suggesting others use Chat Completions.
You do not need a template to be added to Silly Tavern with an update to read the docs and figure it out on your own. aicgsirs are braindead.

>>108660998
>https://huggingface.co/Qwen/Qwen3.6-27B
Oh wow another benchmaxxed model. can't wait for it to be in reality worse than Gemma.

>>108661680
>Gooning is *the* usecase for gooning.
what the fuck did I mean by this bros
Text Completions is the usecase, of course...

>>108661077
Gemma codes better than Qwen.

>>108660724
>more like, if the tweet meant 'inference was accessed by rando', it means nothing
it was obviously that, basically "oh no people accessed the model in early access"

File: 1432498179182.png (296.2 KB)

296.2 KB PNG

><|turn>model\n<|think|><|channel>thought
So on gemmy do I prefill with this entire thing?

>>108660454
the piss filter is still visible

>>108661631
i have no idea what i am doing, so that's why. i will have to look into the text completions thing, thanks

Q3.6 35B or G4 26B for general agentic stuff?

>>108661701
Learn as much as you can and maybe even fuck around in mikupad to force yourself into at least intuitively understanding what an LLM does so that you don't turn into an aicgnigger.
Godspeed, anon.

>>108661702
Just test them out.

>>108661574
yeah... should have just got the blackwell and been done with it... oh well

File: 2026-03-01-163613_1044x1782_scrot.png (496 KB)

496 KB PNG

>>108661168
Because they're supposedly the expert in quantization and get a shit ton of money for it, yet their quants are actually extremely mid and are often broken.

>>108661693
Proofs? Everyone says that's the one thing qwen excels in

>>108661727
>Everyone says
NTA, but this Everyone is wrong. Try them yourself.

>>108661712
Im on it, but maybe someone who has already done it can tell me what they are best and worst at instead of having me use Gemma 4 in my own agent for a few days.

>>108661168
I just wish they made all their stuff open for other's to improve on

Why the fuck would you use text completion in the current year? It has no vision or tool calling.

>>108661562
>bloody benchod oyu have brahmin amount of vram and you use a model for DALITS
Well I'm not going to offload to cpu or shrink the 256k context for coding am I??
I'm using the 112b at q4. It replaced minimax because it just works better (c#)
Also, I hate every prior Qwen model.

>>108661730
Tried using hermes + 31b q4km and it just tried to think and solve problems but e4b felt completely useless. Haven't tested 26b yet.

>>108661745
My raging hate boner for Qwen's socially irresponsible marketing practices aside, have you tried the Coder Next model or 27B?
I used 112b too, it felt irredeemably retarded for an MoE its size. The above models performed much better, with QCN remaining the only model I actually enjoyed using out of their entire lineup.

>>108661743
those are memes tho

>>108661766
lol, tool calling is the future

>>108661680
>>108661686
Well yeah but they're genuinely too stupid to set up the template. Gemma 4 release I figured it out in 2 seconds reasoning on and off by eyeing the jinja, meanwhile half the thread couldn't and were crying hard. Sad!

the ggufs are here
https://huggingface.co/unsloth/Qwen3.6-27B-GGUF/tree/main

>>108661743
It has the same access to an LLM, plus the ability to actually edit the prompt instead of asking the underlying API to hold your hand.
Converting images to embeddings is not magic. Neither is parsing tool calls. For the model - it's all tokens, for you - it's all text.
Figure it out.

Fuck "linear time inference" and all that bullshit! I want euclidean time inference!

File: 1758105037530551.png (927.4 KB)

927.4 KB PNG

managed to get llama cpp server Ui to finally display the image when the LLM is sending it lol

>108661789
>I'm vagueeeing

File: cbclyf.png (479.9 KB)

479.9 KB PNG

turns out I had day0 gemma this whole time??
sha256sum eafb...b720 gemma-4-31B-it-UD-Q4_K_XL.gguf

compare to new hash:
sha256sum 6340...6b88 gemma-4-31B-it-UD-Q4_K_XL.gguf

failquote

>>108661763
>Coder Next
Yeah I tried that when it was released. It seemed to get confused more with c# syntax.
I don't think that had vision support either right? I tend to use that sometimes.
>27B
Yes, this is actually smarter and I did use it for a while. But it's also slower on my system, even with tensor-parallel. 112b is "smart enough" and so much faster, meaning I spend less time working lol.
I haven't tried the 2.7 Minimax or Qwen 3.6 yet. I'm guessing the 27b will be smart but slow, and the small MoE will be retarded like the 3.5 one.

>>108661801
because inference code bugs and whatnot made broken calibration file
it just means the model is retarded

>>108661812
>Yeah I tried that when it was released.
NTA, but there have been some fixes on llama.cpp since then so you might want to try it again. Although it's probably not better than qwen 3.6 MoE anyways.

>>108661164
isnt kimi native int4? why are they going
>WOZERS!! q4 === q8!!!
like LMAO

>>108661795
>llama cpp server Ui
How did you add all the tools to it?

>>108661774
its been the future for a while by now, come back when its the present i guess

>>108661828
https://github.com/BigStationW/Local-MCP-server/blob/main/docs/Use_on_llamacpp_server.md

>>108661789
If you're just going to end up re-implementing the jinja there's literally no point in using text completition besides giving you a false sense of superiority.

>>108661789
>Figure it out
Why would I bother when chat completion just werks?

>>108661836
Seriously. Command-R came out with strong support for tool calling in early 2024. People are acting like it was invented a month ago for openclaw or whatever.

>>108661521
lmao'd

>>108661664
Periodic reminder that /lmg/ was created in early 2023 by /aicg/ anons who wanted a place for discussing local models (Pygmalion, ...) without the cloud model/proxy background noise. It's always been for gooning by gooners.

>>108661866
Most chat completion proponents here see it as an impediment to their cooming in the only frontend they know, which is ST.
Filling in 4 fields there is not hard at all and gives you an ability to do whatever the hell you want with the entire prompt, including thinking edits and messing with special tokens.
Fine, image input, you can't be bothered to use libmtmd directly. But giving up on the template configuration step because ST didn't add it with an update is pathetic. If I couldn't do even that, why would I bother with local models when cloud ones just werk better, cheaper and faster?

File: 1749622181770588.png (278.4 KB)

278.4 KB PNG

>>108661743
>Why the fuck would you use text completion in the current year? It has no vision or tool calling.
I have no idea but switching to chat completion has made my life way better

>>108661795
you gotta proxy it no shit

>>108661907
yeah, it's bullshit...

>>108661029
if you listen to doomers here, never, and it will be 100k USD for 8GB next year
if you look at what people/companies actually expect, somewhere around early to mid 2027 as production lines ramp up

>>108661902
Oh yeah for llama-server mtmd I might as well ask, do I run images through stb and pass the rgb as b64 to /completion or does it just take regular image files (still b64) and do the processing itself?
I would assume the latter from what I'm seeing.

>>108661899
Oh how fascinating...
Perhaps you should just rent a real discord server instead of squatting here 24/7 and larping as the thread moderator.
You don't own this thread.

>>108661988
you should go back to /ldg/

>>108661988
>You don't own this thread.
He doesn't, but I do. And he's right.

>>108661631
>If you're on Chat Completions, what the fuck are you doing
bullshit, if you want to keep your autism just modify the jinja, it's not like you can't modify shit in that mod

File: 1748907056142177.png (122.4 KB)

122.4 KB PNG

Which goof do I get bros? I don't recognize any of these names, except unslop which I refuse to touch

>>108661899
>invaded

>>108662039
Make your own fuckking goofs. Failing that, ggml-org.

>>108662039
ggml is made by llamacpp devs so it should be okay?

>>108662052
>ggml-org
Downloading now. Thanks for the help gentlesaar

>>108662039
I wait for bartowski personally his 122b / 35b / 27b of 3.5 have all been good.

>>108662052
why make your own gguf, seems like a lot of work for no reason