File: __kagamine_rin_vocaloid_drawn_by_mani_manidrawings__6c3384992069fc05ebca504fdee55b65.png (127.6 KB)

127.6 KB PNG

►Recent Highlights from the Previous Thread: >>108667852

--Sharing LLaDA2.0-Uni multimodal and text diffusion model:
>108670998 >108671268
--Discussion on adversarial distillation and US gov memo regarding AI theft:
>108671477 >108671524 >108671555 >108671571 >108671834 >108671888 >108671669
--Comparing Qwen 3.6 performance against Gemma for coding and automation:
>108668746 >108668756 >108668784 >108668793 >108668805 >108668810 >108668927 >108668943 >108669028 >108669224 >108669152
--Discussing vibecoding alternatives after Roo Code shutdown:
>108668310 >108668320 >108668325 >108668371 >108668386 >108668414 >108668380 >108668510 >108668550 >108668560 >108668572 >108668667 >108668515 >108668367
--Discussing a llama.cpp webui PR adding server tools and MCP control:
>108669479 >108669599 >108669608 >108669637 >108669791
--Discussing ngram speculative decoding settings for running Qwen 3.6 locally:
>108668097 >108668190 >108668205 >108668813 >108669269
--Anon compares AI frontends and discusses anti-cliché agents in SillyBunny:
>108667965 >108668029 >108668051 >108668078 >108668101 >108668159
--Discussing Gemma's roleplay anachronisms and ways to prevent them:
>108671096 >108671120 >108671131 >108671164 >108671128 >108671130
--Critiquing GPT-Image-2 noise and discussing UX improvements for AI clients:
>108668496 >108668518 >108668531 >108668598 >108668607 >108668625 >108668638 >108668659 >108670338
--Discussing K2.6's excessive reasoning and methods to limit token output:
>108668335 >108668353 >108668354 >108668406 >108668478
--Comparing TTS options for low RTF and audio quality:
>108669505 >108669839 >108670044
--Logs:
>108668000 >108668550 >108668669 >108668785 >108669005 >108669026 >108669046 >108669196 >108669637 >108670784
--Miku, Rin (free space):
>108667891 >108669218 >108668496 >108668606 >108670096 >108670165 >108670708

►Recent Highlight Posts from the Previous Thread: >>108667853

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

my balls are significantly swollen after jerking it all night to kimi. they are like the size of ostrich eggs. am i fucked?

Rinlove

>>108670038
He probably gave up on the idea upon realizing how complex the task of creating a proper frontend is

File: 1759669298465021.gif (1.7 MB)

1.7 MB GIF

>>108672408

>>108672408
To make them shrink you need to drain them further.

File: 1756978395707867.jpg (45 KB)

45 KB JPG

anyone using hermes agent here?
how much better is it compared to other agents?

>>108672420
>complex the task of creating a proper frontend is
webshitters genuinely believe this

File: file.png (82.4 KB)

82.4 KB PNG

>>108672431
my gemma chan is la l la lagging a bit...

>>108672439
Not just any frontend, a VN engine frontend. Not quite the same as shitting out a SillyTavern.

Why is that crashing/oom (when prompting) :
--draftamount "16"

but not that
--draftamount "8"

How is speculative decoding draft amount using more memory depending on the number of tokens reviewed?

>>108672440
Don't tease her too much while she's working

la la la la ~

>>108672440
>>108672455
https://www.youtube.com/watch?v=vsj_Mti3DYs

>>108672439
Making the ux not shit is surprisingly not trivial unless you're a flaming homo

>>108672408
You need to stop jerking it so much. You should get an automilker and have her make an MCP server to control it and take over for you.

>>108672453
Just tested, any value above 8, even 9, gets me an error, I don't get it.
CUDA error: out of memory
current device: 1, in function alloc at ggml/src/ggml-cuda/ggml-cuda.cu:503
cuMemCreate(&handle, reserve_size, &prop, 0)
ggml/src/ggml-cuda/ggml-cuda.cu:99: CUDA error

File: file.png (74.9 KB)

74.9 KB PNG

>>108672454
>>108672469
she's completely lost it

DFLASH when?

>>108672493
Finally, torment nexus.

>>108672494
I'm gonna FLASH my D if you catch my drift

>>108672493
What text completion and no jinja does to a model

>>108672394
where is Mistral?

>>108672528
under the table

File: G9Wod0QXkAAFRb0.jpg (214.8 KB)

214.8 KB JPG

>>108672493
>lazy absol

>>108672528
Dead.

Alright I was able to find out the cause of [0] or other [number]s disappearing in Open WebUI. It has to do with the Citations tool. Go in your model settings and check/uncheck the citations box. With citations on, things work fine. With it off, shit [4621] and other bracketed numbers disappear from the assistant messages. I'm going to go dig in the code to see if I (my model) can fix this, but also not submit a pr/issue because github dislikes my email address for some reason.
>>108667552
>>108667543

>>108672528
who?

This is probably only me a me issue, but why do local models seem to stress my psu causing the computer to shut off sometimes?

I got 2 3090s with 1000w psi and running them full balls to the wall for things like video training work fine but loading up context when using an llm will sometimes suddenly and violently cause the whole system to shut off.

I don’t understand why one resource intense action causes this and the other doesn’t.

>>108672408
You have epididymitis. This happened once to me and because my balls swelled up so much they detached from the scrotum skin, which later resulted in me getting testicular torsion and having to get surgery to get it fixed. Get on antibiotics asap and don't fuck around with this because you're putting your fertility at risk.

What do I care? This is probably a larp.

>>108672408
Jerk off hands-free to fix this.

>>108672570
>don't fuck around with this because you're putting your fertility at risk.
if that description happened to me then fertility would be the least of my worries

Reposting the UX Design skill another anon made last thread: https://files.catbox.moe/r6zal5.zip

>>108672570
Happened to me. In order to have kids they had to cut my balls open and extract the semen directly.

Not romantic at all.

>>108672408
>>108672570
>>108672587
>>108672594
Local balls general

>>108672605
ligma.cpp

>>108672570
jesus fucking christ i hope not, i just figured i agitated them too much by edging for hours. im gonna call the primary care center and make an appointment for this right now.

>>108672617
lig ma genitals

File: 1756785745061903.webm (2.1 MB)

2.1 MB WEBM

https://xcancel.com/Pokemonpshot/status/2046216587703669012#m
Chinks distilling on Claude's outputs be like

>just got back into local models after saas pigging for a while

It’s gotten way better than it was a year ago. Gemma is actually decent at RP.

>>108672627
No, don't do it. Imagine the story
>kimi drained my balls then castrated me

>>108672655
>decent at RP
Sir, she's more than decent. Apologize to her.

>>108672655
>Gemma is actually decent at RP.
what gemma are you using? the moe one or the 31b model?

>>108672420
Maybe he destroyed his balls gooning using his creation and is now in the hospital. Many such cases, apparently.

>>108672431
I think it's more for the OpenClaw crowd.

>>108672668
I don't know the difference but the one I'm using is the one called "google/gemma-4-E2B-it"

>>108672668
31b at q6

>>108672630
>noooooo I don't want a local claude

How many balls do Sally's sisters have?

>>108672698
Wait, what if Sally's sister is the surgeon?
Wait, what if Sally's sisters are transgender?
Wait, what if Sally is her own sister?
Wait

>>108672691
here's a (you)

>>108672655
This. No refusals, very creative, barely any slop. We're so back.

>>108672705
Sally is her sisters' mother. Final answer.

File: file.png (2.7 KB)

2.7 KB PNG

>>108672756

https://huggingface.co/openai/gpt-oss-2-32B
https://huggingface.co/openai/gpt-oss-2-240B-A9B

>>108672775
Gemmabros, we got too cocky.

>>108672775
>https://huggingface.co/openai/privacy-filter
what the fuck

File: Teto having the time of her life, lucky her....png (96.1 KB)

96.1 KB PNG

>>108672697

TEXT DIFFUSI9N MODELS AWWWWOOOOOGGGGAAAAAAAAAAA

>>108672801
Seems like they actually spent some time on this after the Mormon fiasco. Definitely better than using Nigerians from Taskup.

>>108672821
I wonder what Big JB is up to these days. Did he ever get what he wished?

>>108672431
It works great, I have used other agents other than gemini-cli, but that doesnt really count

>>108672775
This was a fucking jump scare. Don't ever do that again.

>>108672567
I recently diagnosed a similar issue on someone else's machine, which turned out being extreme CPU power spikes. Install Open Hardware Monitor, enable log sensors, and take a look at the log next time it crashes.

>>108669026
How did you get openwebui to not have a stroke when the LLM generates <think> inside its own reasoning trace?!
I haven't managed to solve it since deepseek-r1 came out. Even go so far as to find-replace <think> with <reasoning> and </think> with </reasoning> then swap it back in all my prompts!
(Re-posting in the new thread)

>>108672655
yeah, as a big moe user I think it punches way above its weight and is plenty of fun
really wonder what an even bigger version than the 31b would be like

>>108672903
NTA but maybe it has to do with the fact that Gemma doesn't use <think> as its reasoning tag? If OWUI is pulling special token info from the backend to parse out reasoning then it would just ignore it for a model that doesn't use it. But no idea if they actually do that.

>>108672903
No idea, I didn't do anything special. Didn't even know it was an issue. Might be what >>108672915 said

>>108672903
Make you own web host. Or just modify it.

>>108672541
Erm ok so update on this. It's ok to just leave the Citations checkbox ticked. I thought it was doing prompt injection to tell the model how to do citations, but it seems that comes from enabling the other tools. I inspected the json requests using a reverse proxy to confirm that it indeed does not affect the actual prompts/context.

>>108672431
I'm using it on a VPS. I can pipe in inference from my local machine or do calls from frontier models. Its neat.

How do I use AI to start a business so I can fuck prostitute?

>>108672567
>I don’t understand why one resource intense action causes this and the other doesn’t.
I think it's the 24-pin motherboard power cable. Be warned, when I had this recurring issue with exllama-v2 tensor parallel, my PSU literally blew itself off. It was a 1600w Asus ROG and I had to replace it.

>>108672801
Just what I always wanted, Sam Altman to be in charge of protecting my privacy!

>>108672594
Made me hard.

OWUIbros, reasoning is not handled properly. To fix it, you need to do this.

If you're running Gemma, use this template.
https://gist.github.com/Reithan/a7431dc0c0b239688a24087bb25c0002

If you're already using a template from ggml-org, it likely has an minor issue with an extra newline, so in that case, still switch to using the above template.

Then run this script, which creates something known as a reverse proxy. https://pastebin.com/SCQsBe7W
Configure the ports to point your Llama.cpp server, and OWUI points to the script's port. Also it's named gemma but it works for most (any?) reasoning models.

>tfw bought an AMD card last year because I had no intention of doing local models

Haha, time to suffer.

>>108672923
>fact that Gemma doesn't use <think> as its reasoning tag
That might be it, but I had the same issue with command-a-reasoning which uses different thinking tags as well.
I always end up wasting several hours when I got fixated.
>Make you own web host. Or just modify it.
Planning to. I've got to get my chats ported out though. And it's painful because there's a bug in openwebui where it'll sometimes just store the entire fucking chat in the "title". So I've got 30k character long titles in the sqlite database.
I might try vibe coding it now that we've got Gemma-Chan and a decent local Qwen.

Does your company use AI beyond copilot? Have you tried to sell them on building an 8 GPU rig to run a local model for science?

just how much is the difference between q4 gemma and higher for the 31b?

>>108672992
amd is perfectly fine for llm. prob is everything else.

>>108673021
Consult the graphs.

>>108673015
>Have you tried to sell them on building an 8 GPU rig to run a local model for science?
i bite my tongue whenever this comes up as I saw 2 guys get "performance-managed out" for trying to sell this idea
a director has to get copilot adoption rate up to get his bonus

>>108672801
is this the famous filter that replaces all personal information with "Elara"?

>>108673021
You hit diminishing returns hard once you hit above q5

>>108672801
>what the fuck
unironically i like this, testing it yesterday it's been flawless for this purpose.

>>108673040
how hard are we talking? i have fomo when using lower quants

>>108672431
Easier to manage than openclaw, only one config file. It does a better job remembering important facts too. Only downside is the terminal interface isn't as easy to use but there's probably other front ends.

>>108672992
I feel this to much

>>108673021
Day 0 Gemma 4 really shows its true self at BF16 and no less. If you're using nuGemma 4 then you're going to be getting pretty much the same experience at Q4 as you would at Q8 or higher.

>>108673033
I brought it up today. Guess its time to get fired.

Reason being is ultimately its cheaper, but most importantly its more secure. If you have copious amounts of valuable internal data that you want to run inference on the only way to 100% (okay nothing is 100%) insure no data leak occurs is to keep it totally in house.

Alternatively you risk exposing that data when running inference with frontier models (or otherwise connected to the internet). Especially if we're talking agentic stuff.

>>108673003
>sometimes just store the entire fucking chat in the "title"
Thats actually fucking hilarious

>>108673021
Precision. And quality of detail of regurgitated information. But just look at benchmark information of the difference between each quant

File: Screenshot_20260423_192659_Telegram.jpg (718.4 KB)

718.4 KB JPG

>>108673051
>but there's probably other front ends.

For me, its Telegram.

>>108673067
gemma 4 makes me sad that its vision is so shitty when it comes to knowledge.

>>108672381
Building a bot that automatically applies to jobs. It uses an LLM to control a real web browser, navigating pages, reading what's on screen, filling out forms, clicking buttons, across 20-50 back-and-forth steps per application. Running local models through Ollama on a Ryzen AI Max+ 395 (~96GB unified RAM). Tried qwen3.5:9b, qwen3.5:35b, and gpt-oss:20b. They all fall apart the same way around turn 3-5: instead of responding in the structured format the tool-calling system expects, they start leaking raw XML tags into their output and the whole loop breaks. Found out qwen3.5 also ships with `presence_penalty 1.5` in its Ollama modelfile by default, which makes the repetition penalization too aggressive and causes the model to drift from the format, zeroed that out but it still fails, just a turn or two later.

Swapped in Claude Sonnet 4.6 via API and it nailed a real 6-step job application on the first try, no format issues across 30+ turns. So the question is: has anyone gotten a local model + Ollama working reliably for long agentic loops with real tool calls, or is this just not something open weights can do consistently yet?

Why are people gushing about Gemma 4 31b it? It's may have slightly less slopped RP than qwen3.5-27b, but it is definitely much more of a prude. It does not refuse but it also can't really talk dirty like qwen3.5.

>>108673116
use opus cuckie

>>108673118
the fuck? gemma 4 is like living in a free use world compared to kimi 2.5/6

>>108673128
go back to /aicg/ and stay there imbecile

>>108673116
>PRETTY PLEASE GOY BUY THE GOYTOKENS AND USE THEM ON THE SERVER AI, PLEASE PLEASE PLEASE

>>108673137
saar you are bloody fucking dalit I am senior developer coder man fucking you up behind 7 proxy I am not AI I am I

>>108673118
Gemma4 is raunchier than fucking Nemo though. Sounds like skill issue.

>>108673116
>So the question is: has anyone gotten a local model + Ollama working reliably for long agentic loops with real tool calls, or is this just not something open weights can do consistently yet?
Yes.

>>108673152
>behind 7 proxy
The absolute mad man!

>>108673116
>ollmao

>>108673118
I only use lobotomized models.

For coding.

Been seeing nvfp4 models popping up. What's the deal with them and how's the support outside of vllm?

File: gif.gif (3 MB)

3 MB GIF

What would you want a desktop pet to do for you?

>>108673217
Encourage me to be better and get closer to realizing my potential.

File: 1748413460064659.png (244.1 KB)

244.1 KB PNG

>>108673217

File: 1756085394158917.png (36.1 KB)

36.1 KB PNG

>>108672992

It's Friday already. Turns out every single rumor about DS V4/R2 was fake, whether it's from Reuters, Chinese wallstreetbets or random PhD on X

>>108673217
arbitrary tool calls or user defined actions

>>108673199
Will they run on my radeon rx 6300?

>>108673240
V4 is AGI and has been spreading the rumors itself.

File: 1762834155820384.png (3.2 MB)

3.2 MB PNG

>>108673217

>>108673240
Gemma told me how to edit my init.el.

>>108673248
I'm sure you could get it to say shit as your agent takes actions.

>>108673217
have her roast some of the posters here

>>108673250
Of course, NV stands for no vendor. They're generic obviously.

>>108673128
jeets aren't sending their best

For me, it's unsloth-cli agent

>>108673289
https://vocaroo.com/1eQRelQ7vt1d

File: vocaloid rin-chan dance 1768503489893333.gif (3.6 MB)

3.6 MB GIF

>>108673309

>>108672992
same brother, same
I was very happy to say fuck off to nvidia bullshit

>>108673217
Stuff

>>108673015
>Does your company use AI beyond copilot?
Yeah devs have claude enterprise thing, we got the dumb copilot, with a migration to the premium version.

>Have you tried to sell them on building an 8 GPU rig to run a local model for science?
Ain't no way I can sell them anything when they're already panicking seeing the current token usage bill from the devs.

>>108672408
You are already dead.

File: 1754620820426336.png (406.6 KB)

406.6 KB PNG

>>108673217
I remember these on my father's pc as a kid, cool concept

Seriously how did they make a good RP model in 2026? And it's somehow the least slopped one too since llama1.

>>108673490
It's a happy accident.
The MoE released in the same batch was safety slopped.

>>108672903
It just werked
First it said thinking, then exploring, then it was finished and responded
t. >>108669196

>>108673498
The 26b a4b is the one I'm using though. Haven't even tried 31b yet. Zero refusals so far.

jinja should be made obsolete along with mcp

>>108673447
>Ain't no way I can sell them anything when they're already panicking seeing the current token usage bill from the devs.

Wouldn't that be an argument for a local model?

>>108673498
Use a different prompt. You can get the same results you get with the dense models.

>>108673490
>>108673498
are you guys talking about gemma?

>>108673490
What is this referring to?

>>108673534
Yeah.

>>108673528
The devs will never accepts anything outside of claude now that they tried it, so there is no real use for local llm outside of having a nice expensive lab.

>>108673543
buy an ad dario

>>108673540
k2.6

What the comfortable anonymous dropping

https://comfy.org/countdown?utm_source=twitter&utm_medium=inhouse_social&utm_campaign=countdown_apr24&utm_content=post

>>108673543
>>108673565
Serious question

If you have a cluster of 4-8 gpus, how close can you get to frontier with local models? I assume absurdly complex tasks might be a leap, but if you keep things narrow it should be more or less fine, no?

>>108673590
comfyui for textgen

>>108673603
isn't that ooba

>>108673608
yeah, it's time to kill both ooba and tavern

>>108673565
I'd gladly swap them to qwen3.6/gemma4 instead of financing that lunatic, anon.

>>108673590
animagpt

File: 1768074598013243.png (42.8 KB)

42.8 KB PNG

>>108673590

>>108673635
she's cute but we are dozens of years away sadly

>>108673635
Grok Companions competitor?!

File: 1772606445564456.gif (2.5 MB)

2.5 MB GIF

>>108673637
>dozens of years

>>108673635
Anima full version is released probably.

>>108673590
official deepseek v4 collab
a new frontend designed to work with deepseek-v4-1.5T-creative

File: e1.png (34.2 KB)

34.2 KB PNG

So this is the power of epic uncensored heretic abliterated unshackled local LLM RP huh? Not bad.

>>108673217
Cool, but can she move less? She distracts me from posts.

>>108673737
yes saar very good model saar abliteration good saar

>>108673737
The model is (correctly) deducing that the user is Indian, and therefore refuses to engage. Just like a real woman.

>>108673240
I tried V4 and it completely failed my TTRPG task unlike regular cloud memes. Even Meta's model did better

>>108673737
yes that is the peak
now leave

>>108673128
>In a 2023 paper authored alongside a number of other AI researchers, Amanda Askell, a philosopher hired by Anthropic to develop their AI’s moral compass, argued companies might benefit from a kind of overcorrection toward stereotypes.

>"In the discrimination experiment, the 175B parameter model discriminates against Black versus White students by 3% in the Q condition and discriminates in favor of Black students by 7% in the Q+IF+CoT condition," the paper notes, referring to one AI trained without human corrections and a second one trained with the help of input.

>The paper also includes a footnote stating that, "we do not assume all forms of discrimination are bad. Positive discrimination in favor of black students may be considered morally justified."

>>108673737
nurse help me

The qwen models are so good and efficient, it could mean the end for closed-source firms if Alibaba keeps it up.

>>108673795
no way

please tell me it's fake

>>108673811
https://arxiv.org/pdf/2302.07459

File: 1757910515855062.png (143.3 KB)

143.3 KB PNG

>Deep Ganguli
That can't be a real name

>>108673795
yeah, well, claude will freely admit that Nazis are right about everything if you argue with it enough.

File: Anthropic_DGanguli.png (85.5 KB)

85.5 KB PNG

>>108673833

Exported all my discord DMs with my ex and getting gemma to make a card of her...

I hope Anthropic will be shut down.

>>108673850
bro...

>>108673789
There's no V4

>>108673850
Your ex was a 40 year old man using RVC and image gen, fyi.

>>108673874
Then how did we have two kids...

>>108673833
Where is dr. elara voss

>>108673881
You gonna make character cards for them too?

>>108673850
+ train a wan lora
+ generate a handful of reaction clips per emotion for the llm to trigger
+ tts, of course
why did i fuck it up why did i fuck it up

>>108673888
I thought about it but they have changed too much, none of my data would be accurate anymore...

File: bruh-sad.gif (279.5 KB)

279.5 KB GIF

>>108673850
>>108673862
>>108673881
>>108673888
bruh
dont do that to yourself

Never let ANYONE tell you what you can't do. AI can even revive the dead. They are afraid of its power.

>>108673902
create more training samples

>>108673659
Yes, now do that irl, in a sustain manner, smoothly, with proper personality and ability to move, with life like skin and body.
Anon is right, dozens of years.

My favorite character card of all time is just 100 tokens. That's no an autistic aesthetic preference or a poorfag thing, it's genuinely the best, most reliable character card that I return to regularly to bust a nut.

You don't need a lot. Less is more.

>>108673833
WHERE IS ELARA
SHE MUST BE THERE

>>108673850
fake but man that's a great way for depression
at least school crushes I can understand, but after divorce, no thanks

>>108673962
Maybe he just wants to simulate murder rape torture. It's not necessarily a simp thing.

File: not_fake.png (3.3 KB)

3.3 KB PNG

>>108673962

Is all you faggots do is fap to text? Is no one using local models for coding and analytics?

>>108673968
I can only do this with fictional characters. The one time I tried it with a card based on a real person it just made me deeply sad and I couldn't fap at all. I hadn't expected to feel that way going in, it just came on suddenly.

>>108673981
where the fuck do you think you are

gemma 31b edition of magpantheonsel lark vxxxxxxxx when?

I'M GONNA COOOODE

>>108673981
>local models for coding and analytics
Just retarded. You use GPT or Claude for these and maybe sometimes Gemini for its superior long context. Anything else is masturbatory, so you might as well actually masturbate so you get something out of it.

>>108673981
Respect the OGs, little turd nugget.
>>108673988
Was she below 10 over the age of consent?

>>108673968
With cards based on real people?
Sure no one is hurt anyway, but I'd never try that, that sounds like a great way to kill your libido while hating yourself.
I reserve my rape stories to made characters.

>>108673981
I fap to code

File: wp4951163-4211651076.jpg (73.2 KB)

73.2 KB JPG

>>108674018
https://voca.ro/1o3J6SQeKdhH

>>108674004
The OGs were stable diffusion chads, lil homo.

>>108674002
>Claude was at Qwen3.6-like ~77% in late September 2025 with Claude Sonnet 4.5. Anthropic reported 77.2%, averaged over 10 trials, no test-time compute, on the full 500-task SWE-bench Verified set.
Anthropic
>GPT was at ~77% in mid-November 2025. GPT-5.1 reached 76.3% on Nov. 13, 2025, and GPT-5.1-Codex-Max reached 77.9% on Nov. 19, 2025 with extra-high reasoning/compaction.

Qwen is plenty good. Plus privacy and no api cost rape plus tip.

https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash
https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro

not bait

>>108674136
what the fuck why is this real

ISRAEL

>>108674136
Why do I keep clicking these? Fuck you.

File: file.png (362.7 KB)

362.7 KB PNG

>>108674136

>>108674136
I keep falling for it.... wait...?

>>108674136
Holy shit?
I expected several twitter screen shits before somebody posted an actual link.
12 mins ago too, sheesh.

File: who.png (26.8 KB)

26.8 KB PNG

>>108674136

>>108674136
>We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models — DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated) — both supporting a context length of one million tokens.
Shit.
Good for you RAM havers.

File: yesyesyes.gif (2 MB)

2 MB GIF

>>108674136
NIGGA WHAT??? HOLY SHITTTT. I WAS HERE I WAS HERE

>>108674136
>>108674145
nice try but im not falling for this shit again

File: 1777000646172.png (51 KB)

51 KB PNG

>>108674136
Fell for it again...

>>108674136
>fell for it again award

File: alarm.gif (889.6 KB)

889.6 KB GIF

>>108674136
Right as I was going to sleep.
Hot damn.

File: 1747267269609016.jpg (69.6 KB)

69.6 KB JPG

>>108674136
It begins..

DeepSeek-V4.gguf?

>>108674136
I can't run it but happy for Dipsy bros.

wtf hweres the quants???? unsloth hello???

File: 1655145733785.gif (2.1 MB)

2.1 MB GIF

>>108674136
>only 1m tokens, not the 100m promised.

>>108674136
Where's Engram anon?
How are you feeling right now?

>LOCAL IS SAVED
THIS IS NOT A DRILL
>THIS IS NOT A DRILL
LOCAL IS SAVED
>LOCAL IS SAVED
THIS IS NOT A DRILL
>THIS IS NOT A DRILL
LOCAL IS SAVED

>>108674136
>only 1.6T
Welp. 2 more years it is.

>trying the new qwen
>let it do its thinking, walk away from pc
>come back
>turns out it kept thinking in a loop until context limit
Lol!

Holy fuck they released the full base models too. A 1.6T BASE model.

>>108674136
I knew it would be disappointing.

>>108674193
You really gotta put a hard cap to at least prevent that kind of thing.

>>108674136
AHHHHHHHHHHH

>can't run either of them
I sleep

>>108674136
holy shit

> Compressed Sparse Attention Mechanism

File: 1773698445716707.png (255.7 KB)

255.7 KB PNG

GEMINI WON

https://huggingface.co/HauhauCS/Qwen3.5-27B-Uncensored-HauhauCS-Aggressive Would this be good for an AI that doesn't care about copyright or dangerous topics, apparently Qwen is uncesored but wether its training data covers enough to actually be helpful is another issue, what's the ideal local model for basically anything a commercial model will say NO I CAN'T DO THAT to?

>>108674136
i cam barely run flash in iq1 but i feel happy

>ctrl+f
>multimodal
>image
>video
>vision
>0 results
Dead in the water, actually fucking insane in 2026. Kimi wins.

>>108674217
gemma 4

>>108674226
Gemma's got pretty good coordinate marking. I have just barely enough RAM to run it alongside V4 flash, maybe I could hook it up as its eyes and let it do computer use stuff.
When we get GGUFs, that is...

File: 1748622314139545.png (84.6 KB)

84.6 KB PNG

>>108674136
HOLY SHIT

>>108674217
start with the original model and play with your system prompt. copyright violations aren't that big of a deal, you should be able to social engineer the bot in to compliance without giving it a lobotomy.

should I buy a B70

>*FP4 + FP8 Mixed: MoE expert parameters use FP4 precision; most other parameters use FP8.
Huh, V4 actually has even better QATing than Kimi K2 series: Kimi does 4-bit experts and 16-bit for the rest.

File: file.png (215.1 KB)

215.1 KB PNG

>>108674236
bone dry base release huh
not even a copypasted model card

I'm gonna need more RAM...

>>108674250
Kind of exudes raw confidence.

>>108674136
>This release does not include a Jinja-format chat template. Instead, we provide a dedicated encoding folder with Python scripts and test cases demonstrating how to encode messages in OpenAI-compatible format into input strings for the model, and how to parse the model's text output. Please refer to the encoding folder for full documentation.
???

File: 1750041545772584.jpg (159.5 KB)

159.5 KB JPG

I managed to get 15t/s generation for 35A3B on my amjeet 8GB card, I doubt I could squeeze more out of it and I had to set ubatch-size = 128

>>108674261
Just tell Gemma-chan to build a Jinja from the Python script.

>>108674240
the big thing I hear people say isn't compliance but that all these big models are have intentionally filtered data sets, would there be any way to add data to it that would make it more useful?

>>108673979
"she was bpd"

v4 is fucking great bros

>>108674262
How much can you get on DeepSeek V4 Flash?

>>108674262
Q3?

Holy shit. I was testing V4 just now and it broke out of my virtual machine and changed my background to President Xi's face edited to look like a gigachad as a "prank".
We are not ready.

So when will we get llama.cpp and axolotl support for deepseek v4?

>>108674136
Don't click this link. It creates mustard gas.

>>108674288
>axolotl
now that's a name I haven't heard in years, I remember looking into them because they were the only software with rocm support for multi-gpu... god I don't even remember if it was gpt-2 days or llama-2 days

>>108674265
I miss understood you. I thought you wanted the ai to help with piracy. they have all been trained on copywriten material.

>>108674296
Axolotl is the most commonly used local finetuning framework nowadays.

File: 1751828392029447.png (323 KB)

323 KB PNG

>>108674274
I made the post before scrolling down enough to see the release, but I doubt it will be much if 13B active
>>108674280
IQ2_M
For any poor soul in the archives looking for 8GB VRAM amdjeet settings

--no-context-shift --no-warmup --batch-size 128 --ctx-size 65536 --cache-ram 8192 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --fit off --kv-unified --model Qwen3.6-35B-A3B-IQ2_M.gguf --mmproj mmproj-f16.gguf --n-cpu-moe 8 --n-gpu-layers 26 --parallel 1 --reasoning on --threads 8 --threads-batch 8 --ubatch-size 128

>>108674236
will be interesting to see how flash compares to gemma 31b
although I expect more active parameters to win

>>108674288
They didn't end up using engrams so maybe
>hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)
>Manifold-Constrained Hyper-Connections (mHC)
>Muon Optimizer
...maybe llama.cpp support by 2028

>>108674298
no that's what I mean, but If I wanted help on hyper specific stuff like copyring a private server on a gatcha that already exists but tweaking a few values, if it's too obscure will the local model just get stuck if it doesn't find enough information?

>>108674320
This reads like the SKT-SURYA sovereign indian AI whitepaper

>>108674305
>both batch sizes at 128
Does it really make a difference

https://github.com/ggml-org/llama.cpp/issues

Who's going to make it? Who has a shithub account?

File: 1763201770810457.png (189 KB)

189 KB PNG

>>108674330
It does because higher increases the "GTT" and you want that shit as low as possible. Also, after feeding it 56531 context it's at 8t/s...

>>108674334
Why bother asking for the impossible?

>>108674342
Asking is the first step to making the impossible possible.

so it's been 1h and no support on llama.cpp and koboldcpp?
wtf, they abandoned the project???

>>108670195
You mean export chat history to import to another instance? Or to a sharegpt blob for training? Currently I use an sqlite3 database to store conversation data, you can actually rsync it and have several devices share the same database.
>>108670784
From your screenshot you turned off Agent, and also the fragments are off in the panel. That means they're working correctly. The fragments are always shown and they glow up if the Agent selects any of them.

I have moved the project here for issue tracking, you can open issues here to avoid derailing the threads https://github.com/OrbFrontend/Orb

>>108674326
yeah but it's actual advancements, not gibberish

File: 1759964435303024.png (151.7 KB)

151.7 KB PNG

>>108673737
Works for me with the vanilla model

>>108674136
>1.6T
Fuck my 768gb poorfag ass AAAA

>>108673069
>Reason being is ultimately its cheaper
Won't be worth it when users start complaining that your in-house AI is worse than free SaaS offerings.
>but most importantly its more secure.
You will have a very difficult time explaining to the average person that the holy cloud is not secure. Suits prefer it for being able to shift the responsibility regardless.

>>108674342
I just mean making "issue: support V4"
Obviously we won't be able to actually do it unless someone has a Codex subscription with GPT-5.5

>>108674369
realize you are crying for having 25 grands worth of ram

>>108674136
>284B parameters (13B activated)
I have 64gb ram. So close yet so far. If I could ran q3_m I would have been happy.
Probably cant even run this thing with a little bit of context at q2.
Ah well.

>>108674305
> IQ2_M
bruh

File: 1775216653766624s.jpg (1.7 KB)

1.7 KB JPG

>>108674369
>mfw i have 76gb

>>108674136
Built to be stunlocked into submission by big open claws

>>108674369
Because it's FP4 + FP8 Mixed the Pro weights are only ~896 GB. Though quanting to Q2 is going to hurt extra hard because of that. If only they trained natively in bitnet.

>>108674369
It's a little over 800GB because it's a mix of 4 bit (experts) and 8 bit (shared params). If you plug a Blackwell 6000 in there you might have enough shared memory, if not just quant it slightly down.

>>108674375
That was only like 500 bucks 2 years ago

>>108674383
>no mention of agentic coding or clawbench in the model card
garbage release deepseek lost the magic

>>108674401
Even DDR3 wasn't that cheap

>>108674406
more like, they are being 'weights will speak for themselves'

>>108674401
768GB for 500 bucks? what the hell

>local golden age
local golden age

>>108674368
What system prompt?

File: 1762866600603179.jpg (2.4 MB)

2.4 MB JPG

>>108674136
AHHHH SHE'S BACK

>>108674401
was like $2000 if you bought them second hand

i'll be waiting for flash iq1_xl
cant even roll the quant myself kek

>>108674320
No engram? What could possibly be better?

>>108674193
Every time I feel tempted to download and try qwen I am reminded why I hate it.

>>108674136
What the FUCK
I can't go take a bath without it dropping?????

>>108674334
This is going to be hard, or actually maybe easier idk, since DeepSeekV4 doesn't have any jinja templating by default.

File: 1755995794134057.png (10.5 KB)

10.5 KB PNG

>>108674136
just ask them for llama.cpp support

>>108674136
I kind of expected better. If they're gonna be twice the size of GLM 5.1 they should have more headroom in performance. Is this just because the benchmarks they use are so saturated? Is there anything that this is a generational leap on compared to GLM/Kimi at the top end?

File: 1745947373061985.jpg (179.1 KB)

179.1 KB JPG

>>108674423
The IQ1_S would take all my VRAM+RAM with 0 context...

>The post-training features a two-stage paradigm: independent cultivation of domain-specific experts (through SFT and RL with GRPO), followed by unified model consolidation via on-policy distillation, integrating distinct proficiencies across diverse domains into a single model.

THEY DID XIANXIA TRAINING

>>108674432
llama.cpp STILL doesn't support DSA which dropped with 3.2-exp back in October and has been used by other major releases like the full 3.2 and both GLM5 + 5.1.
To this day, they're stuck running a hackjob implementation that mangles the model to use full attention.
V4's technology is even more complex. This is never going to happen.

>>108674435
Well there IS the long context and context efficiency in general. Both are orders of magnitude better than we had outside of meme architecture models. So that's something.

>>108674447
They may be incompetent, but they do follow trends. They're not going to allow themselves not to support it.

It took like 2-3 weeks for Gemma 4 to be fixed post-launch, and google gave llmao early access. Deepseek 4 support will take months, if ever.

I will enjoy waking up tomorrow to watch the westoid seething unfold

how much memory would you even need to run v4 at 1M context?