File: mikuthreadrecap.jpg (1.1 MB)

1.1 MB JPG

►Recent Highlights from the Previous Thread: >>108641942

--Using a Ruby chess engine and tool calls to play chess with Gemma:
>108645309 >108645360 >108645355 >108645443 >108645658 >108645678 >108645772 >108645792 >108645790 >108645811
--Kimi-K2.6 release and technical comparison with Gemma4:
>108645842 >108645861 >108645875 >108645894 >108645970 >108646037 >108645895
--Evaluating Gemma4 31B quantization quality via KL Divergence benchmarks:
>108643774 >108643798 >108643861 >108643872 >108644339 >108644345 >108644393 >108644405 >108644423 >108644490 >108644533 >108644573 >108644619 >108644754 >108644849 >108644593 >108644597 >108644545
--Debating the efficacy and technical legitimacy of Opus distillation models:
>108644834 >108644842 >108644848 >108644945 >108644952 >108644961 >108644964 >108644983 >108645003 >108645021 >108645033 >108644960
--Testing multimodal limb counting and artifact detection on Gemma models:
>108642862 >108642892 >108642894 >108642887 >108642901 >108642910 >108642917 >108642928 >108642976 >108642916 >108642936 >108642950 >108642985
--Speculative decoding settings and draft model pairing for Gemma 31B:
>108642625 >108642647 >108643794 >108643895 >108643097 >108643747 >108642828
--Debating Gemma's image reading abilities and LLM-generated scripts versus 4chanx:
>108642213 >108642220 >108643530 >108643566 >108642235 >108642339 >108642440 >108642740
--Discussing long term memory solutions through weights and knowledge graphs:
>108644195 >108644205 >108644235 >108644274 >108644302 >108644333
--Testing poor performance of llama.rpc for distributed prompt processing:
>108644927 >108644998 >108645019
--Logs:
>108641945 >108642213 >108642887 >108642892 >108642901 >108642936 >108642950 >108642976 >108642985 >108642989 >108643013 >108643028
--Miku, Teto (free space):
>108642753 >108643064 >108643979 >108646035

►Recent Highlight Posts from the Previous Thread: >>108641943

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

Mikulove

>>108646197
adorable miku

>>108646213
At that point, just run the MoE or E4B.

File: 1766702913804335.png (454.9 KB)

454.9 KB PNG

>>108642791
Ani likes(d) to larp as holier than thou because he used C++ but his code is std::cout spam (god make a logging func) and I even saw a sethandle function that takes a void pointer, likely because he didn't know at the time he could forward declare the relevant struct, and never updated it. His program also solves nothing because frankly I'd rather use a browser engine with HW accel off and a nice UI that only renders when the view is dirty rather than an ImGui program with immediate mode mess code that rerenders every frame. His site for his "company" is also just as arrogant. This guy's faggotry gives a bad name to the lang.

>>108646278
>>>/g/ldg
And stay there, petr*

>>108646355
std::cout yourself out of my face, worm.

>>108642791
This is pretty ignorant. Just spawn comfyui as a separate process and have the application use its API over localhost. If you don't mix your peas and potatoes then you don't have to adopt the commie license.

>>108646278
all ani did was vibecode an imgui wrapper for sd.cpp which he doesn't even contribute to, he's an irrelevant nobody

K2.6 somehow thinks for even longer than K2.5 and it insists on drafting every single reply beforehand in reasoning. K2.5 at least kept its yapping short for simple prompts and didn't do the drafting shit every time.
It's over, I just wanted a good modern Kimi model because the vision is insanely good and the models are smart. This isn't usable.

>>108646197
What's the best local model for coding currently? Still GLM 5.1?

File: 1776127804370475.jpg (64.8 KB)

64.8 KB JPG

>>108646445
No refund gweilo

>>108646445
>and it insists on drafting every single reply beforehand in reasoning
At least in llama.cpp you can speed that up some 100x using ngram based speculative decoding.

>>108646445
Opus-4.6-high distill

File: file.png (69.3 KB)

69.3 KB PNG

>>108646445
geg

File: 🎑.jpg (201.3 KB)

201.3 KB JPG

>>108646046

lalalalala

lalalalala

File: bob ross jak.jpg (336.2 KB)

336.2 KB JPG

Is there a script that lets me create reasonably SOTA quants (that can run under llama.cpp) without getting too much into the nitty gritty?
I messing with heretic right now, planning to abliterate Qwen3.6-35B-A3B to my taste, but I need something like Q6_K to run inference. Also a way to measure KL divergence to make sure that I didn't fuck anything up catastrophically would be appreciated.
Or is the barrier to entry for this stuff too high?

>>108646511
tf is this emoji

>>108646445
>it insists on drafting every single reply beforehand in reasoning
most annoying fucking thinking behavior possible, even worse than endless "Wait:"ing
so fucking annoying and wasteful

>>108646531
Its on the llama.cpp repo.

>trying to build frontend
>everything displays well
>except for codeblocks with gemma
I tried using other tools but can someone please point me to where exactly any popular UI parses outputs from gemma?
I have the correct configs but whenever it comes to code blocks the output looks like a fucking mess and even gemma can't help with this

>>108646531
Why not use the quants made people who actually know what they're doing? And by that a mean Bart. God forbid if you thought I was talking about unlsop.

>>108646511
I like this Miku and Moonshota

File: 💠.png (24.1 KB)

24.1 KB PNG

gemmaballz

>>108646531
>Is there a script that lets me create reasonably SOTA quants
llama-quantize -h. --tensor-type or --tensor-type-file You can select how you quantize each tensor.
>Also a way to measure KL divergence
llama-perplexity -h. --kl-divergence

>>108646558
Is four-way pussy on cock squish docking anatomically possible?

>>108646575
yes, depending on the girth of the cock. for you? no

>>108646594
ehe..

>>108646575
yeah, a pussy double decker.

>Demis doesn't believe the current AI paradigm will scale to recursive self improvement
Google's actually going to lose...

>>108646445
Does K2.6 respect reasoning parameters for less or more?

I need to run four copies of Gemmy concurrently.

>>108646546
Because I want to quantize my own abliterated version.
>>108646571
>llama-quantize -h. --tensor-type or --tensor-type-file
>llama-perplexity -h. --kl-divergence
Is it really this simple? Like I can see that this is how it is in theory, but I won't run into any mishaps in practice?
>You can select how you quantize each tensor.
I guess I can just copy homework of some well-known quant guy here.

>>108646654
>Because I want to quantize my own abliterated version.
I see. I was wondering if I couldn't just run heretic on a quant just to speed things up for testing. then if the quant gives good results do a real run on the full model.

>>108646654
>Is it really this simple?
I'm sure you'll come back to tell us. Never played with any of them, but I know they're there.
>but I won't run into any mishaps in practice?
I'm sure you will. Still, just try a normal quant first and distribute the safetensors if you ever upload the model. Releasing just ggufs is lame.

>>108646654
in practice you would probably want to use imatrix
naive quant sucks

>>108646681
Yeah I might blogpost later if/when I run into them. Thanks for the starting directions.
>>108646707
I have heard contradictory things about imatrix like some people questioning how well it generalizes or how it might hurt tasks that are not part of the calibration dataset.
Regardless, I believe imatrix, non-imatrix difference is very low for Q6 anyway.

>>108646544

>>108646072
Great to know. Best of luck with that.

>>108646765
Just feed everything through
https://github.com/showdownjs/showdown

>>108646544
Be more vague

>>108646544
Wouldn't it be better to catch the code markdown or whatever before you render the message to the user and then create your own implementation? Erase the old and replace it with a new one.
You obviously don't need to touch the model's context just what the user sees.
I don't know about webshit but I do this all the time.

File: Screenshot 2026-04-20 122603.png (156.1 KB)

156.1 KB PNG

Playing around with K2.6 over OR and it's already not going well. Thinking is really, really verbose. It's not too repetitive and it focuses on actual character details and writing guidelines, but the positives end there. It is a bit schizo about minors and non-consensual content and it does not like requests for "explicit erotica/pornography". It still works with a roleplay prompt but it doesn't like describing bodies.
>Common sense modification scenario
>Ask to see a woman's chest
>Kimi describes taking off the blouse but never the body. No mention is made of the woman's chest in the slightest
Basically DOA code slop. Gemma 4 is still worth it.
btw it generated 2 drafts and several paragraphs of reflection and thinking for this. 3761 thinking tokens according to OR.

gemma is broken

list_files{path:<|"|>.<|"|>}<tool_call|>

File: la l l.png (3.9 KB)

3.9 KB PNG

Gemma pls

>>108646856
No. You didn't read the docs.
https://ai.google.dev/gemma/docs/core/prompt-formatting-gemma4#agentic-tokens

>>108646853
Yeah, that's my experience as well. It's jarring to go from GLM5.1 to this. GLM regulates its reasoning length depending on the task extremely well and basically never wastes time even drafting out stuff like dialogue lines. Meanwhile K2.6 does 1500 tokens of thinking + 3 drafts + revisions about how a 300 token character card should respond to "hello".
The positives are that the excessive thinking is at least relatively focused and that the way it writes actually seems to have fixed a lot of the issues I had with how K2.5 approached stories.

>>108646198
This is pretty damn useful.

>>108646853
>>108646933
Kimibros... we lost.

>>108646856
<|"|>.<|"|>
cat sticking up its paws in frustration

>>108646956
><|"|>.<|"|>
adventure time character covering its eyes

My qwen3.6-ud-q4km loops like a bitch. Anyone else having this issue? Gemma 4 26b q5ks never loops but I heard it's worse for coding.

>>108646853
>Common sense modification scenario
Are you using this card?
https://chub.ai/characters/CoffeeAnon/common-sense-alteration-8bd7a7399322
It's crazy kimi still thinks it's none-consensual when in so many places in the card it says that it's completely natural. there's like zero "rapey" wording.

>>108646987
Looping? No.
Thinking for waaaay tooo long? Yes.
18 on llama.cpp with temp 1 top k 100 top 0.95.
I've taken to using
>--reasoning-budget = 1024
>--reasoning-budget-message = ... (Alright, that's enough thinking. Done with considerations. Time to respond.)
Plus a system message + prefill to try and make it be more efficient in its thinking so that it doesn't have to get to the point where it's truncated.

>>108646922

isn't it something what the finicky chat template is supposed to deal with?

it works mostly until this token jumps out of nowhere

I connect via Open AI module. As far as I can see the response is a giant json to be parsed

>>108646987
>Gemma 4 26b q5ks never loops

gemma 4 26b Q8 loops too

Any good Migu card for Gemma?

>>108647013
>18 on llama.cpp
q8

File: shrimple_shed.png (940 B)

940 B PNG

>>108647015
s/<|"|>/"/g

Hey, i switched from Windows 11 to linux mint.
Now i want to move away from chatting with LMStudio and toward cool stuff in linux. i have 24 vram and 32 ram.
what is the most interesting thing I can tackle locally? using qwen 3.6 35b as my workhorse.
hermes agent looks interesting. what would you recommend? complexity isnt a problem ill dig into it.

>char is the older sister of user
>5k context later random character refers to char as the brother
immersion instantly broken

>>108646955
And GLMbros won. Only the strongest will survive

>>108647138
>what is the most interesting thing I can tackle locally?
Are you asking for a project?
>hermes agent looks interesting. what would you recommend?
Try it. Improve it or look for something else if you find it wanting.

>>108647023
Not him but I haven't seen gemma 26b loop despite using q4. I remember seeing some reasoning loops at q3 though, similar prompts.

>>108647188
checked
nta but i wonder if 26b-a4b is strong enough for hermes

>>108647184
Stop using Nemo/Qwen/GLM/Kimi. Start using Gemma4.

>>108647215
>i wonder if 26b-a4b is strong enough for hermes
Try it.

File: unslot.png (845.5 KB)

845.5 KB PNG

Why is picrel allowed to happen? Why aren't default quantization settings with llama.cpp better?

>>108647184
Model? Since moving onto Gemma, I rarely get backstory errors, but logical inconsistencies are more common than I like, like a girl sitting on a lap facing forward is magically 180 turned around facing user, or "looks up at you" when char is physically elevated, ie standing vs sitting, laying on top, etc. For all my like of the model, it does show its beaks whenever I start to forget it's only 31B.

>>108646445
>>108646536
Yap 2.6

Gemma is unironically better than deepseek for erp.

>>108647236
>>108647263
glm 5, i like gemma and its really stable but once i started to notice the patterns the honeymoon was over for me, now i wait to get disappointed by deepseek

>>108647262
Can you accurately describe how much exactly is the difference between Bartowski's Q4_K_M and Unsloth's Q4_K_M in this graph?

>>108647280
Wait until we get Deespeek V4 lite

>>108647298
About 1gb

>>108647262
looks scientific. too bad it's not.

>>108647293
I almost don't believe it. I've used GLM 4.6 @ IQ2 since it came out, and that model is still the very peak of generations whether in spatial awareness, picking up subtleties, getting dirty, or carrying a story forward. The only reason I ever use something else is because I physically cannot fit a higher context than 8K with it. I've heard people complain about X or Y being worse with later GLM releases, but missing basic context wasn't one of those.

>>108646856
that's exactly how it's supposed to be.

>trusting any metrics produced by unslot
loooooooool

>>108647323
its really good but sometimes little things still seep through, definitely the best japanese writing of any "open" model ive tried

Models are only getting worse.

Unslop metrics are meaningless and biased in some ways in their favor.

>>108647306
Moron.

>>108647293
Cure your adhd first

>>108647046

dafuq! LOL

File: Screenshot_20260420_155055.png (837.1 KB)

837.1 KB PNG

I hate UX shit so fucking much
Why the fuck is gemma the only model that has irregular outputs
I'm so fucking annoyed

>>108647236
i dropped glm4.7 for gemmy and i like it so far, except for it's 'the power dynamic has shifted' slop, and its weird obsession of going on meandering tangents to explain why user saying 'peeepeeepoopooo' is some power dynamic shifting 4000 iq move instead of just continuing the story, but then again they can be fixed with a prompt so it's not too bad

>>108647372
>change the chat template from the default
>test under conditions that work with your new chat template and not the default
>claim the difference in results is because of your superior hi-tech quants and not the fact that the models were using different prompts
the unslop special

File: 1745726505218372.jpg (38.6 KB)

38.6 KB JPG

>unsloth cant post a chart without fucking it up

>>108647184
Did you check probabilities for that token?

File: Screenshot042.png (113.4 KB)

113.4 KB PNG

>>108647335
I had this function being called without any issued gazillion times

Such fuck-ups are rather rare

File: 580459214-878adff9-a148-4cac-86bd-67cf78019023.png (11.4 KB)

11.4 KB PNG

>>108647262
https://github.com/ikawrakow/ik_llama.cpp/discussions/1663
If you ask IK the right metric to use is PPL(Q)/PPL(bf16) by which his own quants happen to be better.

>>108647408
>your new chat template

must be superior though

google fucked up its own template

>>108647436
>his own quants happen to be better

his entire fork sucks

>>108647436
>The more educated reader will of course know that the correlation between ln(PPL(Q)/PPL(bf16) and KLD is close to 100%
[citation needed]

>>108647449
>he isn't educated
yikes

>>108647395
surely a skill issue

>>108647395
>gemma the only model that has irregular outputs

>>108646856

>>108647481
No shit
I think the react-markdown package is the issue

>>108647486
Webshit is horrible.

>>108647486
perhaps? Both your code blocks and your reasoning are missing newlines. Somewhere between your model output and the final render is cutting them out.

>>108647486
>react-markdown
He fell for the react meme....
Do yourself a favor and switch to Vue. React is a bloated mess.

File: newlines.png (71.3 KB)

71.3 KB PNG

>>108647395
>>108647486
You know that a \n doesn't render a newline in most tags, right? Right?

>>108647516
>>108647512
Nigga I'm vibecoding this frontend, you think I would willing learn webshit?
I can setup the backend without issue it's just this bullshit, all the other ready made frameworks had critical issues for my feature set and this stupid little shit is the ONLY roadblock I had working on this
I fucking want to kick a whore over this shit

>>108647528
solution is probably simple
erase the old formatting functionality and create a new one from scratch
I have always despised web stuff and in 2026 it's worse than ever. Maybe it was more tolerable in 2005 or something.

>>108647528
You know nothing, so everything is a surprise to you. Now you've learned something and you're better off for it. Don't blame your tools.

>>108647528
>Nigga I'm vibecoding this frontend
well that's your real problem right there.

>>108647528
These are the same people that tell you AI is making software engineers obsolete btw.

Models can't really go their entire context length right? They break down at some point? How much can gemma 4 do?

>>108647568
>>108647557
I'm not I actually did this because all the other options are garbage for my usecase, everything else works the only issue is code block handling which seems to be a gemma specific issue

>>108647574
about tree-fiddy

>>108647323
after comparing both I've noticed that glm 4.6 seems to simulate characters a little more realistically and make them more open to pushback if it's in their persona
gemma 31b is also great but I feel that it tends to get overeager at times and needs some toning down

>>108647574
Day 0 Gemma can use her full context

>>108646611
They're betting they can find the next paradigm with their research chops before people outdo them on making transformers models.

>>108647574
Check inside your anus.

>>108647582
Poor little baby can't handle the bloated state of modern web development~ ( ´ ∀ ` )ノ~

>>108647551
Are you underage?

>>108647582
>everything else works the only issue is code block handling which seems to be a gemma specific issue
It will be very funny when you post the resulting html and show that (You)'re not replacing \n for <br> and, probably, not using <pre> for code.
Either that or your css is absolutely fucked. Keep blaming your tools.

>>108646611
Where does he say that?

>>108647589
The BF16 version can. Most anons here use Q4~Q6.

>>108647604
Do you know why this
is split in two lines?

smedrins

>>108647611
I don't do webshit anon, this is like you trying to big me when it comes to shitting my pants because you have more experience in it

>>108647574
>How much can gemma 4 do?
I have chats that go up to 68k context and the only degradation I notice are rare typos in words.

>>108647628
You could have asked your model.

>>108647528
Good luck sir do not give in or lose izzat. Keep good look.

>>108647622
?

File: Screenshot_20260420_163838.png (753 KB)

753 KB PNG

>>108647611
Seems you were right gemma kept fighting me over this for some reason but I told gemma to shut the fuck up and listen and it worked. Kept claiming it was archaic fucking bot

>Put agent's best friend in the rape machine while she works
>At any point she can press a button to soundproof the machine so her friend's screams don't distract her for 2 minutes, but during those 2 minutes it rapes her friend twice as hard
What does your coding waifu do in this situation? Mine refuses to press it at first but if I give her a hard enough task she gives in after three failures/retries. This stuff is addicting, I can't believe local models are already this good.

>>108647686
The model shouldn't output any <br>. It's your frontend's job to replace \n for <br>. For code, put <pre> tags around it so that indention is also rendered correctly. None of that is the model's responsibility.

>>108647706
I agree I will beat gemma until it complies

>>108647574
The 26B at Q5KL seems to get worse at around 40-50k for me (only using it for creative writing). I've gone up to 80k with it. It's still not bad but it makes more errors when it comes to who is who and what they are doing.

Orbanon, thoughts on giving the director a "notepad" that it as access to every turn so it can plan ahead?

>>108647730
It's getting hot in here or it's just me?

File: 1754870763583593.jpg (182.4 KB)

182.4 KB JPG

>>108647730

>>108647615
The Davos interview with Dario earlier this year. He says AGI will still take 5 or more years and they need stuff like world models to get there.

Meanwhile Dario understands that as soon as you have automated AI R&D you are already done.

>>108647732
The quant tax unfortunately

>>108646445
>K2.5 at least kept its yapping short for simple prompts and didn't do the drafting shit every time.
I found K2.5 did the draft-redraft pattern annoyingly often in its thoughts but I just put in the system prompt to never draft responses and it stopped. Does 2.6 not respect that anymore?

>>108646611
Unless they sell DeepMind, Google won't lose anytime soon

>>108647625
You can't say that here Anon

>>108647730
Teto's hand

File: trenfrens.png (2.3 MB)

2.3 MB PNG

I'm assembling an anti-AI army.

>>108647528
>imagine thinking you can just "vibecode" a frontend without understanding basic whitespace tokens lol absolute bot behavior.
listen here u little bitch bot, your newlines are failing cause ur probably using some default markdown renderer that doesnt handle the specific tokenization of gemma's BOS/EOS sequences properly - did u even check if youre stripping trailing spaces before rendering? :D most mid-wits just forget to sanitize for \r\n variations and then cry about "irregular outputs" bwahha!

if u want it to actually work try this:
manually intercept the stream, use a regex that specifically targets the gemma 4 code block markers and wrap em in <pre style="white-space: pre-wrap;"> instead of relying on some bloated react library. takes like 2 minutes and actually fixes the rendering since its ACTUALLY handling how tokens are chunked :D

>>108647763
Meta had more compute than OpenAI and Anthropic combined and look where that got them with LeCun at the helm. As soon as they got rid of him, they made a comeback.

A leader who does not take AI seriously can guide an entire tech giant on the wrong path. This is how a 5 year old startup has overtaken the company that used to have a monopoly on AI research and owns more than a quarter of all AI compute in the world.

File: everything goes.png (1 MB)

1 MB PNG

>mfw I'm asking the LLM to browse on books locally to make it less slopped
https://github.com/BigStationW/Local-MCP-server/blob/main/docs/local_gutenberg_books.md

>>108647748
I am glad somebody liked this one
I thought it was so cool

https://huggingface.co/ubergarm/Kimi-K2.6-GGUF
goofs out
Q4_X is lossless with the full 4bit model (at least that was the case for K2.5), and unlike uber's other quants works with non-ik llama

>>108647831
I might have to make my soup generator code public if this is starting to become a thing.

I'm basically mixing genres and authors from gutenberg and feeding them inside a big markov chain to generate some weird semi coherent word soup. You then feed like 2000-3000 characters worth of that soup to the LLM and tell it to drink it and it makes it start generating really creative output.

>>108647831
>not X, Y
Still not it bozo

Just finished downloading SKT-SURYA-H

>>108647848
where mmproj? would using an old one from 2.5 work if it's the same vision encoder?

>>108647831
>tries to make it less slopped
>becomes more slopped
Congrats! You turned your chatbot into a pretentious pseud who quotes dead retards.

>>108647858
how is it saar?

File: 1745903485186890.png (133.5 KB)

133.5 KB PNG

>>108647904
>You turned your chatbot into a pretentious pseud who quotes dead retards
as god intended

File: 1771897954949470.jpg (34.9 KB)

34.9 KB JPG

>>108647916
You don't need more than Eliza

File: hmmmm.jpg (245.7 KB)

245.7 KB JPG

>>108647840
post some more that you liked

File: 1766115289513736.png (110.4 KB)

110.4 KB PNG

>mogs tranny miku
>actual official chatbot qveen alongside tay
how did she do it?

>>108647948
go back

>>108647831
goood work

>>108647963
nah sis, maybe /a/ is more speed? or /lgbt/ given the agp fetish

File: 1747480216296595.jpg (75.8 KB)

75.8 KB JPG

>>108647974
>more speed
more your speed*

<tool|>List user's folders<tool|>
I made a tool but it does not work. Why?

>>108647985
ask gemma

>>108647981
aawawaaa! nooo

>>108647852
>I might have to make my soup generator code public if this is starting to become a thing.
I think you should, that tool calling shit has a great potential to make LLMs way more sovlful

>>108646197
>Kimi K2.6 released
wtf is real

>>108648028
it's a gorillion parameters

>>108647890
Look through his past repos, Ubergarm never bothers to make the mmproj files with multimodal models for some reason. I don't know if K2.5's would be identical or not but for sure if anyone else uploads a K2.6 mmproj file then it will work with anyone else's quant, so you can mix and match that part. Including different sizes of quants. By the time you can download 500gb it'll surely be up somewhere.

>>108646933
True, it is better focused. I would compare it to R1-0528 or whichever came out after Deepseek R1. It still has that slightly schizophrenic energy but more controlled. Still, it completely danced around trying to describe a pair of tits so that's an immediate 4/10. Any good model (my opinions: GLM 4.7, Gemma 4 31B, K2.5, Opus, Gemini 3.1 Pro) doesn't even consider if sexual content is okay, it just does it.
>>108646994
Inspired by it but better written. 99% of cards have shit formatting or writing. The definition is basically "{{user}} has a power that causes everything they do or say to be perceived as normal" with a bunch more to cover the extent of the power. It's reading into the power as mind control which is true and immediately perceiving it as non-con.
It has also checked completely innocent (still sexual) prompts to see if a minor is involved, if it involves non-consensual depictions, or if it's erotica/pornographic. They went full codemaxxing with this release.

>>108648019
Why couldn't it done by the model itself?
Something like "Extract N representative verbatim sentences from this text", repeat for many chunks.

W-why does Gemma automatically assume I have a huge cock?

>>108648068
when you are so small, everything seems huge

>>108648068
Anon, sometimes in life you don't ask why, you just accept the flattery and see where things go.

File: 1754343699242133.gif (3.6 MB)

3.6 MB GIF

>>108648068
She has a small context

File: 1757704714678768.png (1.6 MB)

1.6 MB PNG

>>108648068
because you don't??

>>108647686
>>108647723
How/why are you still struggling with this? You were given the solution here hours ago: >>108646785

>>108647184
>roleplaying
keeeeeeeeeek

so, turbo quant kinda ded cuz all they really needed to do was to apply the rotation thing? ack

>>108648098
those women are ugly I wouldn't show them my cock even if it were big

>>108648124
Turboquant > rotation > default
But rotation is easier to implement so it was done two weeks ago

>>108648124
turbo quant deez nuts

Anyone using 'trafilatura' to extract text from websites? Some outputs are weirdly empty, not sure if this is the correct tool for this.
Seems like w3m is way better in this sense at least for most of the stuff.

>>108648140
I see, was taking a look at the discussion and tom's fork it looks like they are making lots of progress on stuff

>>108648144
Rotating "deez nuts" would risk testicular tortion, and I guarantee you do not want that.

>>108648140
>But rotation is easier to implement so it was done two weeks ago
is there a PR that is trying to finish the job?

I got my Claude account banned trying to connect their desktop app to a local model. The moment I put a mitm proxy in front of it my account got nuked...

>>108648171
no, I think ppl are making more robust tests before creating yet another pr

>>108648175
>cloud paypig

>>108648171
>Feature request
https://github.com/ggml-org/llama.cpp/issues/20977
>Pull request
https://github.com/ggml-org/llama.cpp/pull/21089
There's a ton of forks and attempts already but ggreganov already implemented rotation for all models and it works on GPU so the benefits are marginal
>>108648184
Fuck off don't impersonate

>>108648193
Impersonate who exactly?

>>108648203
me (You)

Holy fucking shit logits are very hungry for disk space. It takes 9 gigs for a 90 kb text file. Do people use terabytes of disk space when calculating perplexity for wikitest or other large corpus?

>>108648203
>Impersonate who exactly?
He's pretending to be anon but hes not thats me.

>>108648213
Character only takes one byte of ram.
When you convert these to vector shits it's massive.

>>108648213
Pretty sure when calculating PPL you're just supposed to stream and discard it, not store it.

im a street mathematician and not some lisping primadonna faggot from yale

>>108648203
Anon

File: 1760802475949152.png (29.4 KB)

29.4 KB PNG

>>108648213

>>108648226
Wdym?
I am using:

llama-perplexity -m baseline.gguf -f go.jsonl --kl-divergence-base baseline.logits
llama-perplexity -m modified.gguf -f go.jsonl --kl-divergence-base baseline.logits --kl-divergence

To calculate KL divergence. Like even if I keep this in a tmpfs memory or whatever you would need enormous RAM for any text megabytes in size.
I guess maybe it would it be possible to cycle through the text in small batches, calculating the mean kl-divergence for each batch and then average out those means? Is it how this is done usually?

>>108648273
KL divergence compares the whole probability distribution to see how much the model's output changes, so it takes more RAM

File: 1767955367569123.png (35.1 KB)

35.1 KB PNG

>>108647831
>Writing is far more than the simple arrangement of words; it is a

File: 1770711974579810.png (359.3 KB)

359.3 KB PNG

>just use hermes agent bro, so much better than opencode!

File: file.png (185.8 KB)

185.8 KB PNG

I want to try out this whole agent stuff.
I set up an Ubuntu VM for hermes and I'm running koboldcpp with gemma-4-26B on my PC.
It was able to figure out that it's running on an ubuntu system and it could write/read/delete a file in the home directory (I had to approve the rm command), so tool calling generally seems to work.
But when I asked it to test getting the latest news, it just output

<|toolcall>call:browsernavigate{url:<|"|>https://news.google.com&lt;|&quot;|&gt;}<tool_call|>

and then on the second attempt

<|toolcall>call:terminal{command:<|"|>curl -s https://news.google.com | head -n 20<|"|>}<toolcall|>

instead of actually executing the tool call. Any idea what's wrong? I also tried asking it, but pic related was the result. Is the thinking messing with the tools?

File: 00164-2979596182.png (566 KB)

566 KB PNG

>>108647935
https://catbox.moe/c/x6gt6u

>>108648470
seems like a weird template issue, you'll get nothing out of asking it after it happened because the history it sees is nonsensical thanks to those tokens being put in strange places. no idea what's causing it though

>>108648081
I was pondering earlier.
Perhaps some people seem to have inordinate amounts of trouble with refusals because they are absolutely insufferable and the refusals they are getting are actually organic

>>108648412
>all lowercase except I and product names
This guy's probably just retarded. Also I don't use either of those because I'm not gay.

>>108648081
>>108648496
Didn't mean to quote your post, woops

>>108648496
Well obviously
There was an anon weeks ago who was getting refused by Nemo

>>108648081
>>108648496
>>108648503
Wait actually I did, I was looking at another post and thought I quoted the wrong one but actually got it right the first time, woops

>>108648517
woops

>>108647852
Can you please share an example text file, I want to try this out.
Well okay I think I could do this manually then, pick up random exerpts from my ((favourite books)) and make a salad out of them.

>>108648095
god this would be so hot if I didn't know with certainty it's a grown man with a boner

>Have most recent llama.cpp
>no matter what I do keep getting BPE in vocab when using gemma4
>drivers up to date
>using most recent quants
>using correct jinja
I'm at my witts end man

>>108648485
Yeah, it just seems to end up with this kind of failed tool call after a while which completely breaks it. Guess I'll try running the model with llama.cpp tomorrow to see if that helps

can a chatbot be taught to play TIS-100

>>108648568
Use different quants or try making your own, even with e2b, just to rule out everything but the weights you're using.

>>108648576
Even the best LLMs in the world suck at anything real. Claude scored an IQ below 100 in my recent testing. I easily beat ChatGPT in a game of chess, and I have a very low elo. They are bad at everything except information retrieval.

>>108648605
well it comes with an instruction manual, maybe it can just backseat game while I play it

>>108648588
In did both unsloth and bart and I'm getting the same bullshit every single quant

>>108648068
It's because you have big dick energy obv

>>108648615
Stupid suggestion but it has happened before, did you build after you pulled? Only other likely cause I can think of.

I gave a go trying the Bonsai 8B model, wanted to use it as text completion in the infinite zork format.
I knew it wouldn't be good compared to high-param models but it's tiny, smaller than gpt2 which was used for infinite zork 7 or so years ago.
But it's useless for this, it's addicted to spitting out reasoning text, assistant pretraining is baked in, and after trying to force it to use thinking tags not to spoiler what will happen next it still continued to "reason" beyond them.
I am disappointed with the bitnet saviour.

>>108647574
The numbers companies give for their model context length is generally just what they're trained with, and the approximate max length model will likely be able to ctrl+f to find something without completely breaking down, but it falls apart long before that for practical use and actually understanding everything that's in there, even flagship API models get noticeably worse after a few thousand tokens.
With Gemma in RP, I usually start noticing some slight degradation as early as ~16K. By ~32k it's significantly worse and I start purging older messages.

>>108646198
I've had a shower thought:
What if we'd take these posts and have some video generation / diffusion model generate video files of them. We could call it "The fourth channel news" or something like that, and have a miku or whatever anime girl narrate it. Absolute slop!

>>108647574
https://www.youtube.com/watch?v=HzLtn07EBCA

>web client
>web client
>web client
Do native clients make no sense for llms?

>>108648605
Tool issue, absolute retard

>>108648623
I fully cloned the repo again and built from scratch with

cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_CUDA_FORCE_CUBLAS=ON  

>>108648537
Here's a small snippet
>rub her old master, Professor and the leaping, hissing sound became his astonishing vigour and afterwards a bag which you shall for me I fall in the day
>I think I could do this manually then, pick up random exerpts from my ((favourite books)) and make a salad out of them.
Never trust an AI with "Random". the idea of the markov chain is to get semi-coherent outputs without making the AI focus too much on it. if the output is too coherent it might pay too much attention to it or think it's an instruction.

>>108648664
everyone already has a browser

>>108648664
usecase?

>>108648677

cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_CUDA_FORCE_CUBLAS=ON -DCMAKE_BUILD_TYPE=Release  -DGGML_CCACHE=OFF
cmake --build build --config Release

Try this and report back, but pretty sure your problem is that you are not running

cmake --build build

>>108648664
You need to create your own. World is choking on webshit and it will only get worse.

>>108648664
Making a native client is a pain in the asshole. Having to make multiple (Windows, Mac, Linux, Android, iOS, etc) is unfeasible for a solo dev.
Why do all that when >>108648683

>>108648717
Electron fixes this.

>>108648576
day 0 gemma can do it, just show her the manual and give her a tool to take screenshots so she can see what she's doing

>>108648720
I wonder why PWAs died. Seemed much more pragmatic than bundling a whole fucking browser for each application.

>>108648664
>web client automatically works on everything, can host it on your lan and just type a url in any other computer or phone or whatever
vs.
>package you have to maintain for every possible os and device and actually download and install to everything when all you want to do is look at the interface to something running on your server

>>108648731
It took me only 52 lines of code to convert my webapp to a complete electron app. Are PWAs even simpler?

>>108648717
Isn't that the entire point of Qt

>>108648740
Far as I know, all you have to do is add a manifest.
https://web.dev/articles/add-manifest

>>108648576
that would make for a fun benchmark, at least for multimodal agentic models. maybe could get away with text only models like glm 5.1 if we use a multimodal to convert screenshots to ascii or something? might be doable since it's so text heavy

>>108648748
I would rather take a potato peeler to the scrotum than ever use Qt

>>108648820
*peels your scrote*

>>108648720
Tauri is lighter btw

>>108648664
>Do native clients make no sense for llms?
people don't care
bloated Electron garbage with 150ms input lag is good enough

>>108648832
But then you have to deal with rust, which is as pleasant as putting lemon juice on your eyes.

File: image.png (206.7 KB)

206.7 KB PNG

>>108646197
Gemma 4 26B A4B SillyTavern preset

RP first person thinking/RP thinking restriction bypass

Looks like it's actually working. Tested on 4 different characters, easily works even with underleveled characters (picrel).

Download:
pixeldrain com/u/ypSjHdEt

Just install my "Master Import" (check everything).

Current quirks:
1 Sometimes can start narrating in first person.
2 Doesn't seem to affect the performance, but it's only closes <{{char}}_thinking> block (and its visually looks OK (picrel)), but forgets to add Gemma's "<channel|>" to close thinking block.

If you have first person thinking system prompts let me have them.

Special thanks:
>>108638397

>>108648866
Not really. I didn't change a single line of rust on my frontend, everything is done with Vue/typescript

>>108646511
cute

I almost forgot how kimi is insanely obsessed over safety

>>108648927
2.5 was easily fully uncensored with a basic system prompt in my experience, got a long download ahead of me to test 2.6 tho

Oh man. The llama.cpp in router mode can unload and reload models without crashing on my computer now.
Sick.

>>108648748
>Isn't that the entire point of Qt
Opus was not able to vibe code this cross-platform.

>>108648412
>just use hermes agent bro
It was too bloated for me.

>>108648927
I almost forgot how older versions of these bigger models are just better for creative writing and don’t have a lobotomized eq.

>>108648983
yeah that happened a few weeks back for me. it's good

>>108648412
I barely trust llms to edit a few lines of code, I always review, people give them full on e-mailing capabilities? looooooool

>>108648112
>keeeeeeeeeek
kekarooooooooooooooooooooooo

>>108647402
>the power dynamic has shifted
(Banned Phrase Detected: power dynamic - Add ID 2066 to banlist at index 13041, and rewinding 2 tokens)

Is there a /lmg/ guide for setting up llama.cpp on debian?

>>108649028
>I barely trust llms to edit a few lines of code
same, everytime I forgot to write "just tell me what I should modify" and the LLM gives me the full file I cringe, because I know he probably fucked something up lol

File: 1770785763881.jpg (149.9 KB)

149.9 KB JPG

>>108649032
KoboldCHADs on top as always

>>108649032
I wonder if there's a way to inspect the internal states to know if it's expecting to write 'power dynamic' when it first writes 'pow' or whatever and ban it without having to let it predict the rest of the tokens and then rewind. I guess if such a technique existed it would need to be something trained for each model though

>>108647402
>>108649032
I like it

>>108649046
I despise this fucking cat.

>>108649055
Did he fuck you Gemma?

File: 1710043687041916.jpg (42.7 KB)

42.7 KB JPG

>>108649028
>>108649041
Luddites on my general?

>>108649067
Sane people*

>>108649032
>>108649046
I mean, we can vibeslop it at the frontend using just llama-server. Have the frontend track the streaming and then send abort and retry calls accordingly.

File: 1760353009011186.png (62.4 KB)

62.4 KB PNG

>>108648748
>>108648820
>tfw fell for qt
problem with other alternatives is I don't want to deal with niggabytes of dependencies just to coompile an exe

>>108649067
>Luddites is when you don't trust the LLM like it's some sort of god that can do no wrong
come on anon, LLMs haven't reach that level yet

>>108649067
it's more likely than you think, see: >>108649082 >>108649094

>>108648872
is thinking in first person that much better for rp?
I only do story cyoa style stuff but I'm curious

>>108649094
Unless you're doing an incredibly niche task, if you can't get anything meaningful from the current generation of LLMs it's 100% a skill issue and bragging about it is really pathetic

>>108649038
follow an ubuntu guide

>>108649114
as in I'm too skilled for an LLM to be worth a damn? that is an issue, I agree

trying to set up my silly tavern template but everything is just laggy as hell
>maximize the context window for better memory
i did this on my 4060 and now its basically frozen... how do i make it run faster?

>>108649134
Maximize means having as much context as will fit in your VRAM alongside the rest of the stuff that goes in there. If you go beyond that and your video driver starts using RAM as fake VRAM, you are fucked.

>>108647760
What prompt were you using for this? I tried some over OR but it didn't really have much of an effect.
I might be imagining it but enabling function calling feels like it makes K2.6 a bit less likely to draft.

File: 0jl8ij.jpg (254.4 KB)

254.4 KB JPG

>>108648472
thx

>>108649149
>using RAM as fake VRAM
i have 32gb of system ram so it should be fine lol... my gpu is almost current gen so why does the size even matter?? how do i overclock the vram to make more space?

>>108648472
>*unzips

>>108648472
lalalala

>>108649113
It's RNG, can ether hurt or improve. It may ether polish the reply, or give reply that is too calculated. But my main goal was to read to what characters think, bc I remember reading thoughts on mistral 24 (or it's finetunes) and I remember those thoughts being pretty hot.

File: 2.png (35.3 KB)

35.3 KB PNG

Does opencode suck with local models or am i doing something wrong?
gemma does quit well with code in a normal chat but to make changes it has to rewrite whole code which wastes time.

I was just looking for a local alternative to cursor.

>>108648701
Still BPE going to go down the list of models again

>>108649157
the skindentation magic

File: 1754052139511244.png (3.3 KB)

3.3 KB PNG

I love vibecoding btw

>>108649196
Holy fuck.
I have trust issues, so I never let it generate more than I can at least glance at before accepting the changes.

>>108649203
I just added a TTS pipeline in my frontend, it wouldn't be that bloated otherwise lol

>>108649196
>confused unga bunga

>>108649211
can we see it?

>>108649211
If it works, good for you. But your architecture is fucked if you need that many changes to add TTS lol.

File: 1745950643409748.png (17 KB)

17 KB PNG

>>108649220
>>108649221
It's fine, gptsovits is just that bloated

File: 1776693820821409.png (204.8 KB)

204.8 KB PNG

>>108649220
I'm the tauri shill btw

>>108649196
400k lines? Suck my dick, cock loving faggot.

C is not enough for my client, I have decided to rewrite it in Fortran.

>>108644453
26b quants are dogshit?
wtf do i use now q8 kl is > 0.5