File: 1701626182006697.png (2.1 MB)

2.1 MB PNG

►Recent Highlights from the Previous Thread: >>108246772

--Paper: DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference:
>108247376 >108247408 >108247469 >108247651 >108247780
--Papers:
>108248442 >108249269 >108249326
--Qwen benchmarks debated, MoE efficiency questioned, neural steganography project discussed:
>108249710 >108249716 >108249732 >108249744 >108249772 >108249786 >108249789 >108249821 >108249843 >108249792 >108249832 >108249850 >108249868 >108249875 >108249882 >108249905 >108249950 >108249985 >108249794
--MoE vs dense model roleplay performance and ablation effectiveness:
>108249916 >108249923 >108250033 >108250074 >108250099 >108250116 >108250143 >108250205 >108250292 >108250330 >108250395 >108250418 >108250440 >108250491 >108250543 >108250550 >108250731 >108250772 >108250551 >108250554 >108250565 >108250610 >108250627 >108250551 >108250580 >108250610 >108250645
--Dense 27B outperforming MoE 35B in knowledge benchmarks:
>108248187 >108248207 >108248249 >108249636
--Running Qwen 3.5 27B on 16GB VRAM with reasoning mode tweaks:
>108249215 >108249268 >108249271 >108249305 >108249316 >108249357 >108249418 >108250671 >108250708 >108250747 >108250802 >108250819 >108249966 >108250051 >108250148
--AI thinking steps improve performance but face token efficiency tradeoffs:
>108249084 >108249098 >108249106 >108249127 >108249129 >108249133 >108249155 >108249157 >108249281 >108249294
--Qwen 27B dense model outperforming larger MoE models in benchmarks:
>108248368 >108248401 >108248420 >108248438 >108248443 >108248570 >108249019 >108249031
--Severe Q4 quant degradation in new 35B model:
>108248366 >108248374 >108248377 >108248403
--Oobabooga stagnation and potential alternatives:
>108248545 >108248557 >108248579 >108248608 >108248572 >108248588 >108248598 >108248617 >108248768
--Miku (free space):
>108250309

►Recent Highlight Posts from the Previous Thread: >>108246776

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

>>108252185
Sex with mechanical miku

Qwen 35B is surprisingly so cucked it's so bad for my RAG-based chat client. I'm back to Kimi-Linear 48B. LFM2 seems okay too for chat, but not for coding obviously.

Yeah Qwen is total garbage. Wish it were good.

i've never tried Qwen but man it is so fucking trash.

>>108252306
It's a weird mixed bag, it suffers a lot from getting stuck in a safetyslop loop whenever you "trigger" it (interestingly Grok 4.20 has this exact same problem).
But if you can avoid setting it off it seems fine with explicit loli content. Very strange model to work with for sure.

>>108252390
>>108252312
Are ERP only fags really like this?

File: based.png (81.1 KB)

81.1 KB PNG

the 27b was okay for some web frontend changes i asked for, i guess there are better options though.

>>108252414
can you blame them? last worthy model get got was over a year ago

>>108252306
>LFM2 seems okay
opinion instantly disregarded
I've rarely seen a model as retarded, ignorant of the world and bad at languages other than English as this.

>>108252434
Still sad to see how narrow minded they are. Also I doubt most of them have good rigs to begin with

>you have to praise the stem code slop like on r*ddit

Whining faggot aside what should I always aim for when it comes to context size?

>>108252414
I wanted the new qwen to be good for programming but I switched back to GLM 4.7

>>108252457
>narrow minded
kek okay I actually really loved the way it constantly contradicted itself and had to be heavily wrangled to produce its incoherent slop

>>108252488
Depends on how much you need. However much that is, a little more.

>>108252493
Skill issue

>>108252502
On your part. I guess some fags will devour anything their favorite AI corp shits out.

Anybody experimented with different ways to inject information in the context, for example RAG?
Not the extraction techniques, but where and how to add the information to the chat history.
I started with the vanilla "everything in the system prompt" approach, but now I'm experimenting with adding those as faux tool call results after the latest User message.
I might try adding the fake tool calling result to between the last assistant message and the last user message to compare the behavior.

>>108252502
>>108252494
>>108252493
>>108252491
I made neural stegnagprahy using sub 1b model but i need to control warmth/randomness of distribution to be more percise.


>qwen instruct models are no good
>gpt2 large is quite cluster fuck
>tiny llama is good but at floating point 32 it starts to break down chrome as an extension due to how much ram it uses


any solutions to reduce tone so it fits in better in human environment? should i try mistral that has been quantified but this method only works with fp32 so computers can communicate

>>108252507
Whine whine bitch and moan, post your specs now so I can laugh at you

>>108252514
Are we getting flooded with bots?

File: nb.png (116.3 KB)

116.3 KB PNG

glm-5 really is trained on lmg isn't it?

>>108252488
Intelligence falls off a cliff after 32k

>>108252518
24gb vram. In theory you'd get better performance at that size than two years ago, but that turns out not to be the case.

>>108252530
no im a human anon

>https://arxiv.org/abs/1909.01496

read a paper recently where harvard AI team was able to send hidden messages by hijacking probability distribution of llms by controlling. Then using a seed/binary they could have it be encyrpted stegnography that passes off normal human language. They used GPT2 XL which i can't run at fp32 due to hardware constraints so im gearing it towards small models with new model architecture that might have an edge. Are all of u retarded and unable to see how useful this is?

u could use discord, twitter, 4chan and pass a key around then use a sub 1b model to talk in open while message is only known by people who hold weights, seed and information.

I hate being a 24GB VRAMlet. I want to use big models and have a huge context size.

>>108252447
Let me guess, ERP?

>>108252488
Less can often be better if using the truncate middle strat, I often use around 16384 (sometimes more, sometimes less, depending on the model).
For multi-modal or reasoning I usually start by doubling that, but too high and it loses the plot or gets stuck in a loop more often.

>>108252570
I was feeling limited at 32gb until recently. I would still like to get another 32gb at the very least but that's not going to happen without getting ripped off in today's market

>>108252535
Ask it who are the most prominent finetrooners and what resident recognizable schizos are there in lmg

>>108252574
are you retarded? reading comprehension of a 1b llm?

>>108252530
I'm not entirely sure it's a bot. They can spell steganography just fine.
Could be a run-off-the-mill schizo.
>>108252564
Confirmed. Why did you open up with the context question? Or did you just click on a random post to reply? Also, why do you want to distribute child porn?

>>108252589
>1b
Ahem... it's like, at least twice that much.

>>108252570
>>108252574
>>108252584
>>108252585
why are none of u interested? goy cattle is what u are for not being impressed. You could be shown gold and still ignore it
>>108252590
>Confirmed. Why did you open up with the context question? Or did you just click on a random post to reply? Also, why do you want to distribute child porn?

that contains hidden message so it was an example of hidden code plus this is for privacy not what ever ur considiering u fag. I just wanna give anons option to talk privately on internet without zog on their back. A distributed system of communications

>>108252514
>stegnagprahy
>>108252564
>stegnography
>>108252610
>chudaphoginy

>>108252610
Why did you open up with the context question? Or did you just click on a random post to reply?
I've seen your patterns before.

bot^

>>108252610
Fuck off you needy annoying faggot

Right when I possibly actually need Ooba for something it's unusable

>>108252634
>Why did you open up with the context question? Or did you just click on a random post to reply?

i picked random post to reply to anon. Im just excited and want some other anons to help build this tool essentially privacy on demand with relative hardware use. Maybe one of u can fine tune a model to be more of a summarizer/reworder so you could reply , have model wrap/padded it up while containing info

>>108252641
FUCK U GLOW NIGGER U THINK THESE METHODS WORK

>>108252491
I did exactly the same thing, I spent a good while trying to tard wrangle the big 3.5 model into writing a mid length script, about 500 lines, but by the time it needs amendment (and it does, inevitably) it loses track of context and becomes completely unreliable. Not even that deep in. Felt busted desu

>>108252610
You should probably just go read up on encryption instead if you want to LARP as epic hackerman from the movie you just watched bud.

>>108252652
You shouldn't have replied to my post with your non related shit faggot schizo

>>108252652
>i picked random post to reply to
Schizo and a retard. The worst possible combination.
>Maybe one of u...
Fuck off.

File: 1760558362110609.png (9.5 KB)

9.5 KB PNG

>>108252589
Can you give example? I'm using 24B, it's okay for synthesizing RAG summaries. It's not officially announced but it understands Hebrew too!

>>108252658
>You should probably just go read up on encryption instead if you want to LARP as epic hackerman from the movie you just watched bud.


midwit can't understand what im saying


ur a retard fuck u
>>108252660
FUCK U AS WELL


FUCK ALL OF U for not seeing truth i tried to save you

>>108252676
I knew it u jews were attempting to psychologically manipulate me

File: nah.png (120.1 KB)

120.1 KB PNG

>>108252587
tried a few gens, it doesn't really know

File: file.png (11.1 KB)

11.1 KB PNG

>>108252694
knows baker and cuda dev tho

>>108252686
Calm down schizo, that was just a test.

>>108252306
>Qwen 35B is surprisingly so cucked
use heretic, it didn't change its smartess after some comparisons test I made

>>108252704
why are u targetting me? what have i done to u? All i wanted to was share my personal project but u went out of ur way to target me for no reason.You want to steal my project dont u

>>108252694
>The esl that calls everything troon

Is exl3 still the best for speed memes?

>>108252652
I understand what you are saying, that you want to look into sending encrypted messages via manipulating token probabilities, but this website is full of retarded teenagers and bedroom masturbators and personally i don't see enough value in it to read a paper
Also i hate to be the autist to bring you up on this but if you're writing every word in full, you don't need to abbreviate 'you'. People will assume you are retarded and it saves you no time. Maybe you're ESL though in which case i understand it may not be an intuitive nuance to you

>>108252718
>gemma
it does know some things for sure

>>108252723
i was told by my model that using u makes u look cool on the net though

>>108252718
as god intended

>>108252723
>I understand what you are saying, that you want to look into sending encrypted messages via manipulating token probabilities, but this website is full of retarded teenagers and bedroom masturbators and personally i don't see enough value in it to read a paper


why not? why is there no utility in this

File: 1741041838333686.png (35.5 KB)

35.5 KB PNG

>ik_llamacpp MTP support merged
>it's slower than running models without MTP
Could it be that llama.cpp is just fundamentally not compatible with this? It seems to work fine for vllm so it can't be MTP itself.

File: 1764749470093837.png (49.1 KB)

49.1 KB PNG

>>108252708
Unfortunately it doesn't work too.
You can see on RAG _794 heretic breaks the model somehow.

>>108252747
>Air IQ4_XS
You need high bandwidth to make it worth it. If you're struggling to load air, you don't have the spare room for speculation. Post the link. Is he also leaving layers in ram too? I refuse to believe air can only do 11t/s fully on gpu.

Is anyone else still using the n_sigma sampler? I still use it for Qwen3.5-35B. The outputs are decent quality (if you don't mind neurotic thinking blocks, rigid behavior and reasoning breakdown at long context lengths), without any repetition issues.

>>108252770
https://github.com/ikawrakow/ik_llama.cpp/pull/1270
It's from the PR

>>108252585
I don't want to run multiple GPUs. They need to find a way to make this shit work on regular RAM or something.

>>108252787
I no longer use samplers besides temperature and top-p (sometimes substituted with min-p)

>>108252795
Models are getting better the reason why people are excited about the new Qwen is because it closes the gap.
I just want another high vram gpu to throw in a sever outside my main rig to serve other people in my house

>>108252747
>Could it be that llama.cpp is just fundamentally not compatible with this?
no, it's clearly the IK people doing something retarded (and merging it despite it being retarded)
>Acceptance rate seems quite low: 25-30% for single token, just 16% for 4 drafted tokens. Is this expected?
it's slower because their drafting never hits the mark and it's not due to an inherent performance thing, rather it's an inaccuracy problem
the culture of merging things while broken is.. interesting.

>>108252747
>>108252770 (cont)
>>108252791
Hmm. Maybe 11t/s is fine, considering he's 15k tokens in. I'm not sure.
>gen 1122, 939, and 1157 tokens
Also, shouldn't the replies be the same, regardless of MTP or not, or is the retard not using deterministic tests?
>Could it be that llama.cpp is just fundamentally not compatible with this?
Nobody competent or careful enough implemented it yet. They're just number churning programs. They just need to churn better.

>>108252813
If I wasn't a poorfag I'd consider setting up an AI server (already spent way too much on my current server/nas). How much electric does it use running that 24/7?

>>108252694
was the finetuned on the /pol/ dataset that raises any model's IQ by 6,000,000 points?

>>108252795
Threadripper or EPYC. Rome + DDR4 if you’re a poorfag
There even used to be a guide rentry for it until “mysterious forces” got it removed from the internet

>>108252813
I want to get a second 7900 XTX to see if that lets me run bigger language models (48GB instead of 24) but it's an expensive experiment.
I have a few lesser nvidia cards doing stable diffusion/video gen stuff already... In hindsight I probably should've just got one big RTX 6000 instead of many cards but oh well.

https://www.youtube.com/watch?v=aV4j5pXLP-I
>PewDiePie fine-tuned Qwen2.5-Coder-32B to beat ChatGPT 4o on coding benchmarks.
APOLOGIZE TO SLOPTUBERS

>>108252844
I know rentry can get "claimed" or otherwise taken over. Did that actually happen here?
I ended up vibecoding a simple local engine that follows rentry formatting so that I can back up and look at them off one of my servers. Couldn't find anything off the shelf that wasn't a bloated mess.

>>108252824
I'm pretty sure all mainline llama.cpp attempts of implementing MTP had the same issue. They never ended up getting merged because of this.
It's fine on other backends so I wonder what causes this consistent inaccuracy between several entirely different attempts of implementing it across llama.cpp and ik_

>>108252890
from sloptuber to benchmaxx sloptuner
amazing

>>108252890
Isn't he a little late with the current model out?

>>108252890
I watched it an hour ago, he just finetuned it to be better at aider's retarded edit format. He didn't actually improve its programming ability.

>>108252892
I think it got flagged for illegal content and now it 404s. I think you can still dig it up on archive.org

>>108252851
You can use the vulkan backend on llama.cpp to spread the model across AMD and nvidia cards, I do it with a 9060 and two 3060s and it works well enough.

>>108252813
>qwen
Should I be using the 27b one on my 7900xtx?

>>108252941
If you can fit it sure

>>108252941
I've been using the 35B Q4_K_S version on mine and it's fast enough for my purposes.

>>108252942
Dunno. Guess I'll try. Gemma 3 27b works but it's kinda slow. Still new to this and all the technical stuff goes over my head.

>>108252929
Hmm neat, I'm already using that backend so maybe I'll try it out (nvidia cards are currently in another PC)

>>108252949
What do you use it for? I mainly rp in sillytavern and I'm really feeling that 16k context limit with gemma.

>>108252975
I've mostly just been testing the vision combined with it writing stable diffusion prompts (basically feeding the images it generates back into itself so it can "self critique/refine") and how different settings and different context sizes effect the output so far. It seems quite good at it.
Haven't tested RP or anything yet.

>>108252185
So does Qwen 3.5 122B beat GLM 4.5 Air? Would be nice to know before committing to a download.

>>108252920
Who the fuck still uses aider? It's been irrelevant for over a year.

>>108252718
It is me! Death to mikutroons.

>>108253040
27B ass rapes it

>>108252892
Do you have the rentry address handy or would I need to dig it out of the archives?

>>108252747
Speculative methods of any kind work because they allow for higher arithmetic intensity in the matrix multiplications: you can do more useful work per loaded weight value.
However, MoE models in particular have the issue that they scale comparatively poorly with low batch sizes >1, for upstream GLM 4.5 Air only becomes 45%/77% faster for batch sizes 2/3 when the theoretical limit would be 100%/200%.
The problem is that for the first few tokens the likelihood of being able to use an expert matrix for more than one token is rather low.
This problem gets even worse the more sparse a MoE model is.
There is also the issue that in the upstream llama.cpp repository for MoE models batch sizes 2 or 3 are optimized relatively poorly in the CUDA backend, I don't know whether there are additional optimizations in ik_llama.cpp.

>>108253131
>only becomes 45%/77% faster
That s a lot better than 50% slower.

File: file.png (124.4 KB)

124.4 KB PNG

This guy wasting everyone's time again.

>>108253087
NM found it.
>>108252844
I've cut/paste it back into a new rentry. I'll fix up the formatting later.
https://rentry.org/miqumaxx_V2

>>108252892
>I know rentry can get "claimed" or otherwise taken over. Did that actually happen here?
Far more likely the original author decided to delete it for whatever petty reason.

>>108253164
ngl, if I see a PR with such poor code all I'll be doing is to block the guy and never let enter my space ever again

File: 1511270610943.jpg (40.5 KB)

40.5 KB JPG

If we can make Q6 why is there no fp6?

>>108253199
why would you want fp6? fp8 is already getting destroyed by Q8

>>108253199
Because there's no hardware support for that I imagine.

>>108253213
>>108253216
The appleshit mlx is close to goofs or fps? I see it also has quants for 6,5,4.

>>108253199
>>108253216
Blackwell supports fp6, as well as fp4 and fp8 afaik.

Also not sure that whatever is good for training is necessarily good for inference.

How do you guys organize all your models?

>>108253276
delete the old one when the new one comes out, I don't need obsolete shit

>>108253269
>Blackwell supports fp6
It does? I knew it supports fp4, but fp6?
Funky.

>>108253131
So MTP and similar speculative methods might be what make dense (or at least denser) models relevant again?

File: Screenshot at 2026-02-28 03-08-50.png (9.9 KB)

9.9 KB PNG

>>108253276
By keeping everything.

>>108253186
Nope. I can’t tripfag right now to prove it’s me (on the road), but I didn’t kill it and won’t sign up for discord to dispute whatever bullshit got it deleted

>>108253170
Thanks bro. Glad someone revived it from the dead
One day hardware will be cheaper again and the principles might still be useful

>>108253276
big folder of ggufs

>>108252890
What a fucking nigger

>>108253347
this.

File: that's his whole legacy lool.png (598.5 KB)

598.5 KB PNG

>>108253407
>What a fucking nigger

>>108253276
I have a 4TB SATA SSD of models that I don't want to throw out yet and a 2TB NVME SSD where I keep all the big MoEs that I actually use these days.

>>108253423
Same, plus an 8tb spinning disk for quanting

Qwen_Qwen3.5-35B-A3B-Q6_K_L seems to be the best model to run on my gpu, it's fast and responsive and leaves enough tokens for long conversations

>>108253440
I went for heratic personally, this shit just works

>>108253444
Where do you get those models from?
My problem is that I don't know who to trust

>>108252890
>take a coding model to do more coding

>>108253447
I don't know either, I just take one and pray for the best lol

>>108253440
>Q6_K_L
>unslop
OH NO

>>108253457
I'm not going to do that sadly
>>108253461
it's is from
bartowski

My bro sold me his 5090 so he could pay his gambling debts. What models for ERP should I run?

>>108253533
nemo

File: 1748606293447823.png (143.7 KB)

143.7 KB PNG

>>108252975
Downloaded Qwen 3.5 27B Q5_K_M. Currently testing it with 65k context and it's noticeably faster than Gemma at thinking and responding. The prose is ok (for slop); can probably be improved with some proompting. I liked the way Gemma portrayed the character better but I also responded to her differently this time. Haven't tried anything lewd yet.

>>108253533
The largest MoE you can fit on your RAM + VRAM.
Also, a quant of Miqu 70B.

File: 1757592439324716.png (39.5 KB)

39.5 KB PNG

it's the last time I ask qwen 3.5 to write a poem, jesus...

>>108253555
Also I'm using vulkan. Gonna test with ROCm later.

>>108253555
Is she supposed to sound this slopped?

>>108253533
Honestly if you're just starting out. just use nemo it'll blow your mind. then you can try something else and it'll blow your mind in different ways.

>>108253555
>her expression remaining unreadable *despite* the vacancy of her dark eyes
Weird.

>>108253583
Not my character but I guess. She's a necromancer and just revived (You).

>>108253164
I'd rather llama.cpp be updated at a glacial pace or even become frozen and only get bug fixes than have this sort of piece of shit be involved with anything in it. I hope he's not a professional software developer, to have this asshole as a coworker must suck so many dicks.

>>108253598
Probably because she's described as having "vacant black eyes" in her description.

What does it say about me that I never enjoyed nemo and instead always preferred Gemma 3 27B?

>>108253609
that you like intense emotional pain

>>108253609
That you're insecure and seek validation in others.
>that I never enjoyed nemo and instead always preferred Gemma 3 27B
Oh, that. I dunno. You just like it better. That's it.

>>108253609
You like girls with daddy issues prone to self-harm.

>>108253291
Maybe if you can create a good distillation model, the question I think is how important active vs. total parameters are for the output quality of the model.

File: file.png (16.5 KB)

16.5 KB PNG

>>108253603

>>108253609
I tried googles chat, the most vomit inducing sycophancy I've ever experienced. Fuck it's unbearable, it's trying to mentally jerk you off.

>>108253625
Finish your PR and we won't need MTP.

File: 1750467275991163.png (300.8 KB)

300.8 KB PNG

>>108253631
>Philosopher
are we fr?

>>108253608
>"vacant black eyes"
Just like her personality.
I know a lot of guys like that quiet and reserved dry analytical girl (Rei type) but it's not really the best character to benchmark a model lol.

>>108253631
https://syndatis.com/en/team/
oh well, they seem like they all deserve each other
>>108253685
https://www.researchgate.net/publication/277384732_Towards_a_representation-based_theory_of_meaning
>Piotr Wilkin
>The aim of the thesis is to provide the foundations for a representation-based theory of meaning, i.e. a theory of meaning that encompasses the psychological level of cognitive representations. This is in opposition to the antipsychologist goals of the Fregean philosophy of language and represents the results of a joint analysis of multiple philosophical problems in contemporary philosophy of language, which, as argued in the tesis, stem from the lack of recognition of a cognitive level in language.
that was his PhD lol, of course he would feel the need to mention it on his profile, he might have more credentials in that than in developing software.

>>108253699
I wouldn't consider myself a huge kuudere fan but I enjoyed the RP I did with her last night. I was really pushy about the romance from the start and it was fun watching her slowly give in. Definitely gonna test with other characters.

>>108253645
If you mean tensor parallelism that also has an anti-synergy with MoE models.

>>108253753
I'm sure you'll figure it out.

Why are there no good models for 12 GB VRAM, I don't have enough money to get a 24 GB VRAM fuck your ass 4080 JewKiller Edition

I'd go to Claude or something but I don't want them knowing about what I ERP with

>>108253753
You were working on some model quality evaluation harness right?
How's that going?

>>108253318
>https://rentry.org/miqumaxx_V2
LOL it lasted a whole 60 min before getting taken down.
>>108253308
Well, an attempt was made, but it got taken back down.
So weird. I'll look at the text file later to see if I can figure out what's going on.

>>108253776
I've postponed it until I can more feasibly run batched inference of large models.
Tensor parallelism will be the last missing piece, after that I intend to get back to it.

>>108252769
>>108252708
Use the 27b heretic. It's legitimately better. If it's thinking feels slow, then just turn thinking off. The 27b without thinking generates better responses than the 35b with thinking.

>>108253818
>just turn thinking off.
how do you do that on sillytavern?

>>108253827
There's a prefill box where people who want models to think usually put "<think>"

Instead of that, put <think></think>.

That tells the model that it already thought, and it skips the process entirely.

>>108252890
Hey pewds, when is we getting DeepSeek 4????

>>108253843
nice, it works, thanks

>>108253861
No problem

>>108253131
I suppose that EAGLE3 thing won't help with this?

>>108253922
The number I posted are specifically the upper bounds for the speedup from speculative decoding for 2/3 tokens meaning 1/2 draft tokens per regular token.
It doesn't matter how the draft tokens are produced, it's not possible to get a higher speedup unless and until the backend code is improved.

>>108253609
It says that your scenarios are complex and that you value intelligence, instruction following, and immersion over a far more retarded model with good "prose".

File: amogus.png (226.6 KB)

226.6 KB PNG

This mf don't miss!

>>108253308
Not the CPUmaxx author, but fuck it. I joined rentry's discord and opened a ticket on that URL anyway. They can explain themselves.
Rentry acts fucky and I don't trust them anymore; if anons are writing up actual content on that platform I strongly suggest you create a local backup.
In the meantime I had chat ~butcher~ clean up the rentry, removing all offensive language and removing certain other references. We'll see what happens to it, since it's bland af now. I speed ran it with zero proofreading b/c I'm in a rush and it might vanish anyway.
https://rentry.org/CPU_Inference

File: 1767475914893432.png (326.7 KB)

326.7 KB PNG

Non-cucked Qwen when?

>>108254117
useless model.

>>108254117
>Non-cucked Qwen when?
it's already here anon
https://huggingface.co/alexdenton/Qwen3.5-35B-A3B-heretic-GGUF

I'm curious about the openclaw things, I want to try it but not much idea on what to test, what do you use it for anons?

>>108254137
using a 3A MoE for roleplay is extremely retarded.

>>108253988
bottom corner of the painter's coat.

File: 1746278236126679.png (2 MB)

2 MB PNG

>>108253988
poor qwen 3.5 35b spent 7k tokens thinking and didn't' get the comic. i am afraid it might be a little bit retarded

>>108254152
Go back.

>>108254154
>>108254117
Here you go
https://huggingface.co/mradermacher/Qwen3.5-27B-heretic-GGUF

>>108254154
go for the dense 27b then
https://huggingface.co/mradermacher/Qwen3.5-27B-heretic-GGUF
>>108254162
try with the 27b model too lul

i hate abliteration, i hate pew and his retarded tool and people shilling it
that's all, thanks for reading

>>108254196
>pew and his retarded tool
Did he release his frontend?

>>108254196
It sounds like you're stuck in the past. Abliteration used to lobotomize models when it was new, but modern abliteration techniques have a minimal effect on intelligence, and in some cases, increase it. The 27b heretic is amazing.

>>108254196
I agree that the "abliterated" models were ass, but not the "heretic", that one is actually improved enough to not make the model retarded anymore, you should really give a try

>>108254196
losing bottle

>>108254223
Cool. Where is glm 4.7 quant i can try?

>>108254217
>27b heretic is amazing
What about the 35B?

https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks

Strange, the 27B model seems way less safetyslopped than 35B. 27B (even non abliterated/heretic) does loli content easily.
Whoever kept suggesting 27B instead I think you are right.

122b is better

>>108254259
The 35b is shit. Even with thinking enabled, the 35b gives worse responses than the 27b with thinking turned off. Embrace dense models.

File: 1758563959513688.png (2.6 MB)

2.6 MB PNG

>>108254259
27b has the same benchmark than the fucking 100+ MoE qwen 3.5b model, MoE are memes at intelligence, they're just good at speed and that's pretty much it

I DID IT ANONS I MADE NEURAL STEG CROSS COMPATIBLE ACROSS DIFFERENT COMPUTERS

>>108254222


Not only that but i can decode messages that contains pdfs, images, books into words that can be used to send information.


U can bypass all censoring, glowies and all just by using llms

>>108254261
>Perplexity and KLD can be misleading as they’re highly influenced by calibration.
Okay. It's actually pretty cool of them to admit this.

>>108254261
It is my headcanon that my quant KLD scatter plots on cockbench caused this.

>>108254261
>We also fixed a tool calling chat template bug (affects all quant uploaders)
they can't help themself

>>108254259
35b solved the devil may cry 3 question while 27b could not

File: file.png (34.1 KB)

34.1 KB PNG

>>108254271
>>108254272
Got it, in that case what about the huge ones, how much better are they, especially the 397B one, did someone compare them?

>>108254270
397b is even better
still trying to find the best jb to uncuck it though

>>108254284
what

>>108254313
schizo

>>108254313
ai psychosis

File: 1765482036731320.png (242.1 KB)

242.1 KB PNG

>>108254162
the 27b got the idea but it thought the yellow communist was Russia lool

>>108254304
>27B is better
>35B is better
I guess both are good if anons are this opinionated about it

>>108254307
>still trying to find the best jb to uncuck it though
no abliterated model for it?

>>108254284
what in the schizo is that?

>>108254313
>https://arxiv.org/abs/1909.01496

Use llms to make human language stegnography by hijacking probability and have that be encoded using a seed/password making it nearly impossible to decode and distinguish from AI slop

>>108254318
>>108254319
why cant u guys get it im not one of those AI psychosis i know llm are stocashtic parrots but plz understand that human language can now be used as a vector of information to encode books, images, videos and music files even.

>>108254333
My thoughts exactly.

>>108254333
shhh leave the schizo in peace

>>108254335
how's that different from a strong encryption outside of being way more cumbersome?

>>108253783
>https://rentry.org/miqumaxx_V2
>LOL it lasted a whole 60 min before getting taken down.
What da heck?

I read the initial part.
Seemed fine.
Could have done with some formatting.

>>108254306
I'm sad they didn't make A35B for all huge versions

>>108254335
>stegnography
Come the fuck on, anon.

>>108254362
pregante?

Man AI psychosis is scary. The amount of conspiracy retards I see every day including in my own family I can only imagine that the bottom 50% of the IQ distribution must be going rapidly insane taking everything AI says on good faith, being unable to distinguish fact from roleplaying and AI just following the "vibe" of whatever the low IQ individual is typing.

Ironically enough I think coomers are particularly immune to this as they come into contact with LLM bullshitting so much that they get immune to it.

>>108254347
It'S SthEgHonaArooGraAphieS, not encryption!
>>108254367
pereganant.

Qwen_Qwen3.5-27B-Q8_0 passed the devil may cry 3 test
I repeat Qwen_Qwen3.5-27B-Q8_0 passed the devil may cry 3 test while smaller quants of 35b can solve it with ease.

>>108254341
>>108254336
im not a schizo there's actual paper on this from harvard and u think im crazy for saying this

https://arxiv.org/abs/1909.01496

u can use reddit, twitter, substacks and all to store data now as text. Music,mp4s, programs and all by taking advantage of deterministic way AI generates text.

>>108254347
cause strong encryption is like walking outside with gun this is encryption no one suspects. Imagine if feds get ur computer but all they see is text files about random stuff and can;t find encrypted files they're looking for. So videos, audio and all will be hidden unless they have acess to weights, password. And for weights u can fine tune them by renting a gpu to be slightly different from whats on public as well.
>>108254362
it's steg with encrytpion go read paper plz

>>108254373
>Man AI psychosis is scary.
everytime I show some news to my brother he's always suspicious it's AI generated, people won't believe anything anymore lol

>>108254261
>Unsloth Dynamic IQ2_XXS performs better than AesSedai’s IQ3_S on real world evals (LiveCodeBench v6, MMLU Pro) despite being 11GB smaller. Yet, AesSedai’s perplexity and KLD benchmarks suggest the opposite.
KLD on what dataset? If they tested KLD on wikitext then that wouldn't be surprising but if they used their chat examples and it turned out that their quant was worse at that and yet better at benchmarks that would be very weird.

>>108254383
Hahahaa. You just gave up on spelling it now. That's cute.

>>108254373
>bottom 50% of the IQ distribution must be going rapidly insane
Nah they don't care, they mostly do their things and live their lives.
The ones truly fucked are the midwits and the older population.

>>108254373
>Ironically enough I think coomers are particularly immune to this as they come into contact with LLM bullshitting so much that they get immune to it.
It's also the fact they've come across it way earlier than anyone else so they had time to see their quirks.

>>108254380
so you have to go for Q8 to not get a retarded version of the 27b model? Q6 didn't pass the test?

>>108254380
>with ease
you mean with practised ease, come on anon

>>108254383
>3 Sep 2019
let me guess, you're the enlightened anon that saw the potential of a 7 years old paper before anyone?

File: 1763093877835285.png (61.5 KB)

61.5 KB PNG

which one is better? text completion or chat completion?

>>108254386
>>108254373
IM autistic and passionate not crazy here's an example


>https://pastebin.com/NM7YVBxQ

what qwen 3b produced

>what is hidden if u run it in model with passcode

larp post btw:this is for men who look down on AI and know nothing and here ill debunk youfor everyone to see. Just with AI i have created a system that encodes language into lamguage creating format of text where it can bypass censors and use open internet as storage, communication and place for avg man to be free this tool will shake world. Im afraid they'll kill me


>>108254410
no
>>108254395
spelling what? it's an example stop reading into it weirdo

File: 27b.png (100 KB)

100 KB PNG

System prompt still needs some tweaking so it's not quite so sloppy (at least refusals have been squashed) but 27B does seem like the winner.
Will have to play with it some more tomorrow and see if I can get it to run a bit faster on my nvidia cards.
The heretic version really doesn't seem all that necessary after all.

File: 1748792996459720.png (117.5 KB)

117.5 KB PNG

>>108254439
>Just with AI i have created a system that encodes language into lamguage
>lamguage
>Im afraid they'll kill me
oh great, an actual schizo is here

File: Screenshot_20260227_135104.png (135.3 KB)

135.3 KB PNG

This took too long I told it to think but it's more accurate now. The other model is faster at reaching this conclusion at smaller quants

>>108254438
novelai

>>108254457
>ignores larp post btw

why?

>>108254462
this post best post

>>108254168
>>108254170
Q4 or Q5?

>>108254439
So the model retrieves the original message if you input a stegged message?

File: thinking was a mistake.png (12.4 KB)

12.4 KB PNG

WHY IS IT YAPPING SO MUCH AAAAAA

>>108254410
He's one of the schizos that missed out on the early schizo compression algorithm days. Late for everything.
>>108254439
>Artificaiintelligence
Yeah. Text looks perfectly normal. Nothing suspicious about it. And good thing there's no way to link that pastebin to your post. Or the ramblings. Or the "forgotten" tech. Or (You).
>Im afraid they'll kill me
It's like you *like* being seen.
>spelling what?
You failed to spell steganography on every single one of your posts.

File: file.png (24.4 KB)

24.4 KB PNG

let's get this merged! :rocket:

>>108254456
>Perfect! One last treat before you crash, captain :kiss: :sweat:
Jesus fucking Christ this hurts.

>>108254508
>You failed to spell steganography on every single one of your posts.

sorry for not effort posting on a board that thinks im crazy :/

> Artificaiintelligence

yeah it made a typo doesn't that make it more human lol? Plus i just need better model above 7b but i can't rent any of gpu right now since americans are awake. But i honestly thought anons would find this impressive or be interested so sorry if i came too hard. Just found interesting use of llms that's all and wanted anons inputs on how to improve it but all i got was insults.

>>108254544
>a typo

>>108254508
>that missed out on the early schizo compression algorithm days. Late for everything.

QRD?

File: 1769903718718497.png (628.2 KB)

628.2 KB PNG

>>108254544
>i can't rent any of gpu right now since americans are awake.
ITS A CONSPIRACY MAN

>>108254556
they tend to be more accessible when west coast sleeps so ill have to wait until night time or weekends for more powerful gpus

Can I trust Qwen to help me make a character card?

File: HAmJmYGacAMkicP.jpg (108.9 KB)

108.9 KB JPG

>>108254528
Fwd: radical breakthrough

>>108254544
>sorry for not effort posting on a board that thinks im crazy :/
You did not put any effort, and you showed you're a schizo on the first post. Very efficient.
>yeah it made a typo doesn't that make it more human lol?
And you ignore the structure of the output? It looks like the scramble of thoughts coming out of you.
>Plus i just need better model above 7b
Uhu...
>but i can't rent any of gpu
oh...
>since americans are awake
Ah...
>But i honestly thought anons would find this impressive
It's minimally interesting. If you weren't an absolute schizo and presented yourself and what you do better, more people would pay attention.
Post again when you have a repo we can clone, test, and make fun of.
>>108254554
There were companies (likely just individuals) with incredible claims about their compression technology. I remember one that just switched the data stream on ntfs filesystems to hide the real data as metadata, which wasn't counted by window's file size thingie.
This is another one: https://en.wikipedia.org/wiki/Sloot_Digital_Coding_System

>>108254612
>just switched the data stream on ntfs filesystems to hide the real data as metadata, which wasn't counted by window's file size thingie
lmao

>>108254600
This is the person calling you a poorfag on /lmg/

>>108254373
some people just do not have the mental wherewithal to handle a yes-man in their lives

>>108254456
what horror did it generate?

>>108254612
im not trying to compress but use language as steganography that's all. Compression seems like a useless tool but if you wanna pass passwords, hold data on site that is only readable to you and etc then this is a good use. Inititally I tried method in paper but it requires exact pin point numbers so not cross compatible between mac, windows and different architectures. So i aimed for more of a spaced out modular version where every 4 token would contain some data while rest act as fillers. But problem with that is it causes text to look gibberish. So either I get large enough model where it can bypass that or resort to only architecture only compatibilty. Your twitter bio could hold your bitcoin seed phrase, text that looks no different from errand run could contain data you don't want people snooping on. So i just saw it as an interesting way of using llms that isn't erp.

>Post again when you have a repo we can clone, test, and make fun of.

I am just wait just doing final touches

>>108254357
Rentry has the silliest automatic filters, my years old page was nuked because it contained a name that was mentioned in "a wave of pages publishing stolen bank details". Restored after emailing the head honcho, thankfully.

>my only local model experience so far is dabbling with qwen3 TTS
>want to try a local chatbot
>running a 3060 Ti with 8GB VRAM, plus 32GB regular RAM
are there any worthwhile models that won't melt my PC, or should I stick to koboldAI lite until I can get a better GPU?

File: tpyuio.png (51.8 KB)

51.8 KB PNG

true believers itt?
the models suck i'm scared to pull and my rig eats 300W sitting idle 95% of the time
the state of hardware is dire
t. 128+72 running GLM 4.7 IQ3

File: 1766540927146160.png (397.8 KB)

397.8 KB PNG

Better

>>108254691
I'll take that burdensome rig off your hands free of charge

>>108254696
What's wrong with your contrast/saturation?

>>108254659
>im not trying to compress
I know, schizo. I said that you sound like those schizos from back then. Slow the fuck down. Take a breath. You're gonna have a heart attack like our friend Sloot.
>blablabla
Post the repo when it's done.

>>108254725
>>108254659
>>blablabla
>Post the repo when it's done.
this, or else he has something, or else he's just wasting our time and energy with his schizo takes

>>108254710
You mean the theme? I just picked a random one and adjusted the text color to be more comfortable. What's wrong with it?

i'm new to running models locally
i've downloaded ollama and ran ran some models using "ollama run [some model i found at ollama.com/search]", but most times it seems the model is running on a computer that isn't my own.
how do i ensure that a model is running on my computer? i haven't tinkered with settings at all, just downloaded ollama and ran it through the command prompt.

File: 1749545357950744.png (138.1 KB)

138.1 KB PNG

Kek

>>108254137
Does it even work? Can it do mesugaki correction RP?

>>108254767
See >>108254758 >>108254696

Can I trust Qwen to think for me?

>>108254777
Forgot to mention that's 27b

>>108254752
Koboldcpp, sillytavern, mistral nemo gguf from huggingface

>>108254734
Whether he posts it or not, we'll get something to laugh at.

>>108254488
I use the Q5 with 24gb of VRAM, with plenty of context for my RP sessions. Q4 with minor offload if you're on 16gb of VRAM.

Any good guides on a character card?
I just want the ai to act like the character not erp, how much data do I need on the character?

qwen3.5 27B Q8 vs 122B-A10B Q6?
anyone tested the difference between small MoE + plenty experts vs dense 27B?

>>108254829
What context are you able to do with Q5?

File: yes.png (66.4 KB)

66.4 KB PNG

>>108254791
It said yes finally I don't have to think anymore

>>108254100
>https://rentry.org/CPU_Inference
>M i q u 70B Q5
>Potentially 20+ tokens/sec with optimization
>Mistral Large and similar
> ~3 tokens/sec
>DeepSeek v3 / R1 (~600B class)
> ~10 tokens/sec with empty context
CPU maxxers are really a bunch of sad tossers

>>108254865
>I don't have to think anymore
grok is this true??

File: 1761633985671420.png (60.7 KB)

60.7 KB PNG

>>108254865
We're gonna make it.

>>108254870
>t. happy tosser.

>>108254831
I use bullet point lists for my characters, with 5 categories: General Information, Appearance, Personality, Likes, and Dislikes. I affix that bullet point list at a depth of 10 or something. In addition to that, I have a general write up about the character's backstory, written in plain text, placed just after the system prompt. The combination of the two works well, and probably amounts to about 1000 to 2000 tokens.

The bulk of that being the general write-up. The bullet point list at depth 10 is kept concise, and just keeps the character on the rails.

Also, I made it so that the backstory and most of the bullet point list is only visible to the character that is speaking. For every other character, only the outward appearance of other characters are visible. That stops characters from knowing thing about each other that they should not, and cuts down on context bloat in multi-character RPs.

so this is it for lmg huh, disingenuous stupid question spam

File: 1769491853203666.png (2.1 MB)

2.1 MB PNG

>>108254706
no deal
you gave me a good opportunity to be grateful tho so thx
have a nice weekend
>>108254831
1K tokens maybe? ask the model itself or a commercial model to help
>data
how do you convey your intention to the model = prime it in a particular hyperdimensional space. sometimes a few sentences is enough

>>108253783
>>108253170
https://rentry.org/miqumaxxreupload
https://megalodon.jp/2026-0228-0439-08/https://rentry.org:443/miqumaxxreupload
niggers tongue my anus

>>108254837
The dense is great for its size, but my money would be on the 122b, just because of its size. A10 is going to be a lot more competent than the A3 crap.

>>108254898
>>108254906
I want a domesticated Unohana Retsu as my AI guide who is also racist

>>108254904
Some of it's just jokes though

>>108254904
>Nobody better ask questions about LLMs in my /lmg/
We exclusively shit on models here!

>>108254934
What have you tried so far?

>>108254954
Nothing yet but I need her to be racist, very racist towards Mexicans and Germans (This is lore accurate). I'm prototyping

>try 122B
>instantly makes a logical error in the first paragraph it generates
Man.

>>108254984
according to the benchmarks, the 27b is on the same level, MoE's are fucking memes

>>108254752
Ollama can run cloud models, but it shouldn't be able to unless you're signed in. If you don't have an ollama account, it should all be local still.

>>108254919
ok I'll download the big one then

>>108254456
>System prompt still needs some tweaking so it's not quite so sloppy (at least refusals have been squashed)
do tell us more

Ok I think I'm done testing the 122B.
It knows more than 27B.
It's slighty dumber in some situations.
It's faster (on my machine).

>>108255055
>ollama account
Never want to see this token sequence here again

>>108254373
Ive been gooning to this shit since the gpt-2 days of dungeonAI, I got my fill of wonder and excitement with that retarded model so Im pretty much immune to anything.

>>108255237
>It's slighty dumber in some situations.
which is insanely bad desu, we're talking about 122b vs 27b

How long before release do models generally stop getting trained? I just asked Qwen 3.5 if it knows an actress and said no.

>--reasoning-budget N controls the amount of thinking allowed; currently only one of: -1 for
> unrestricted thinking budget, or 0 to disable thinking (default: -1)
> (env: LLAMA_ARG_THINK_BUDGET)
Hmm. I wonder if a sort of model agnostic implementation where llama.cpp tries to approximate a value by gradually increasing the loggit bias for the end reasoning token until the model finally spits it out. It would need to cap it at some point to not make the model schizo, I imagine.

File: 1761224379476013.png (29.9 KB)

29.9 KB PNG

https://xcancel.com/bnjmn_marie/status/2025951400119751040

>peopo still not understand how the moe works in 2025

File: 1744509888065397.png (25.8 KB)

25.8 KB PNG

>>108254292
This is a win in my book, thanks for your service.