https://youtu.be/8AtqQWORLvM

Use open weight models either small enough to run yourself or via your diverse online host of choice.

The size and token rates mentioned in this thread for local inference are all going to be exceptionally limited compared to the frontier

Forget about local inference with your hardware, it’s too far behind. 3 years behind in this technology is basically stone age

What you’re looking for is “ZDR” (zero data retention) policies from online providers. There are many options. One powerful model with ZDR you can use is gpt-6 Luna on duck.ai

Even if you’re trying to code, you’ll get better mileage copy pasting to and from duck than ollama with your hardware

I like deepseek v4flash, I use opencode go and commandcode, somehow feels not as bad since its cheaper and technically can be ran at home if you can afford it. I don’t care for regular ai chat, feels useless after using agents. Agents are what you expect when you think of ai at the computer, hermes is a solid free one. Those two subs rotating as one runs out should last you a while at 20$ a month. These smaller companies/datacenters aren’t the issue imo, the larger ones like Anthropic, google, openai, etc. are.

The answer to your question is, it depends on your expectations. The tradeoff of a typical no VRAM machine is latency and quality.

My progression:

• I decided to try ollama and ran a small model on 8G RAM on a VPS. Slow but it worked. Ran an 3B parameter models – slowly. Struggled with 8B models. Problems with hallucinations from these small models. This was my induction into self hosting llms.
• Installed similar setup on a SFF desktop with 16G RAM. Ran 8B models slowly. More capability, but still below the threshold I was looking for. Open WebUI + ollama worked but still too slow.
• Acquired a 16G VRAM card and installed in an old desktop with 16G RAM. This runs 12B models and crosses the threshold, in my opinion, of usability as an assistant. This setup runs Open WebUI, again reinforcing usability.
• Am I satisfied? No. I’d like bigger, faster meaning more VRAM, on a newer platform, but content enough now to continue experimenting with what I have.

In sum: Don’t expect low laitency multi-step reasoning/planning with a low resource / no VRAM machine. Anything close to what you can get online at reasonable speeds requires ample VRAM.

For a reality check … I did a quick run on those three machines as they are still active. What follows has been synthesized through a cloud based model from real data:

same prompt, three machines

Prompt: “Explain how a hash table works, including collisions, in about 250 words.”
Run once on each machine with ollama run <model> --verbose, all Q4_K_M quantization.

What stood out

• The GPU box generated about 4-5x faster than the SFF while running a model roughly 3x bigger.
• Both CPU-only machines landed at 5-7 t/s. Usable, but slow for interactive chat.
• Turning thinking off on the 12B cut total time from 38s to 11.5s with no change in speed (~31 t/s) and no drop in answer quality on this prompt. Thinking would likely matter more for math, code, or multi-step problems, which this test doesn’t cover.
• The 12B had no factual slips. The smaller models made small conceptual errors (blurring hash code vs. index, calling the whole array a “bucket”) and one skipped lookup entirely.
• All of them left out worst-case O(n) behavior and resizing.

Caveats

• Different models on each machine, so this shows what each box can run well, not identical hardware running identical software.
• The thinking-on run generated about 1,180 tokens versus about 350 with thinking off, so compare tokens/sec rather than total time across those two rows.
• Quality scores are my rough 1-5 ratings on accuracy, length, completeness and cleanliness, with an LLM’s help, from one run each at temperature 1. Treat them as anecdotal, not a benchmark.
• gemma3n is listed by Ollama as 6.9B, but it uses an “effective parameter” design (Google calls it E4B, roughly 4B in practice), so it’s lighter than a dense 7B. That’s why it ran on the 8 GB VPS where a standard 8B struggled.

Yes, very much so.

Take a look at my comment here.

PS: you might also be interested in this

https://en.wikipedia.org/wiki/Apertus_(LLM)

I don’t think I’ve seen this recommended yet, but maybe you can check out CoCore https://www.cocore.dev/. I saw a demo of this some time ago and it’s still pretty much an experimental project. Essentially this is more of a community shared way to run local LLMs, especially for those without necessary compute. You are basically using an LLM hosted on another person’s computer in an encrypted manner. There’s a lot of technical detail about it that goes over my head. It is slow, and of course, still requires trust, or some auditing to verify it. But I think it’s kinda a cool way to crowdsource resource to run. The network of users is on fediverse btw.

Srsly, unless you grow all your food and make all your possessions, you’re constantly supporting billionaires. I mean sure, run your own LLM, but don’t let it create a delusion of ethical purity.

No, and why do you want to rot your brain and/or produce more slop anyway?

99% of AI models available are based on stolen work, so even running it locally is immoral…

!localllama@sh.itjust.works

Which model GPU and how much VRAM? 16GB RAM limits your options a bit but maybe LFM 2.5 24B-A2B

If not, Gemma 4 e4b or Ling 3.0 Tiny are about as small and fast as you can get while still being decently smart.

Sure, there are open-weight models and many of them are very usable on quite limited hardware, some can even run on phones which is ridiculous. Don’t expect paid-service-level speed or abilities or even free-tier of those same services where it’s running on datacenter hardware, but running locally on reasonable consumer hardware it’s certainly possible to get close to that, at least sometimes, in some situations. YMMV depending on your configuration and your needs. Most of the good open-weight models are in fact originally released by billionaire asshole companies who have the resources to actually put down these foundations (but a lot of the best open-weight releases are coming from China, which is its own sort of concern), but how and why they get released is not really the important part for us because once they’re released, they’re released forever and people can take them and adjust them and re-train them for all kinds of different purposes. You just have to be aware of where they came from. On their own, they do not read your disk or phone home (they are not capable of it without an external client enabling and allowing them to use tools at all). They are just text input, and text output (or images/audio input using mmproj techniques, or stable diffusion for image/audio generation, those are totally different cans of worms and outside the scope of this, but it can be done) They’re ultimately just text generators, but some of them are trained to general particular formats of text that tell an external software when it should try to use tools.

Personally, I found ollama the easiest way to get started, but I’m sort of a minimalist and started out very casual (and in some ways still am). Another place that looked nice to start out with was the desktop application “Jan”. OpenWebUI was relentlessly recommended to me but I found it a bit buggy and unnecessarily resource intensive and ultimately dropped it. I tried Librechat also and it was a little better but ultimately I had pretty similar problems with it. (A lot of stuff in the AI field is vibe-coded slopware, and it really shows sometimes, be warned and be aware) Capek is a very minimal alternative web chat frontend that I’ve kind of settled on personally. I find minimalism is the best way I can think of to avoid all the feature-bloated insecure AI slop. I don’t know if it’s actually helping or a good strategy, but it’s the strategy I’ve chosen.

Can I just slow it down and get it to work locally?

Technically yes, but pushed too far it slows down dramatically to the point of not really being useful anymore, unless you’re careful about how you do it. Key things to look for:

• File size/download size, and memory usage are pretty much the same number. Everything in the file has to be loaded into preferably VRAM. Most of the time you can go a little bit over and the leftovers will be loaded into system RAM for your CPU to process instead and this can be manageable, but often slows things down dramatically.
• Quantization (most widespread is Q4_K_M) trades a tiny bit of model quality/precision for vastly smaller sizes and Imatrixes (IMAT, IQ_*) have been developed to make even smaller sizes viable, but with somewhat more significant damage to the quality of the model’s processing and outputs.
• Uncensoring (common terminology and keywords include abliterated/abliteration, heretic, unchained/unleashed, etc) is often included in unofficial open-weight models and reduces or removes the refusal behaviors of the model (telling you it can’t help you with that because blah blah blah) which can be annoying when it’s simply misunderstanding the request or false-triggering or if you just don’t want the model’s training dictating its own morality to you. Done poorly it can make the model stupid and broken, especially on brand-new models or that haven’t been heavily experimented with yet, but most of the time the community gets the models uncensored pretty smoothly after some time.
• Some models are trained to try to use external tools when they’re told they’re available but it’s usually not their default mode and they typically won’t even try unless you’re specifically using client software that specifically tells the models it is designed for and supports tool use (most often called “harnesses”) similar to claude code. Opencode is a popular claude-like one, personally I prefer ‘pi’ (pi-mono) which is very minimal. Many GUI applications also support some kind of tool use but I’m not super familiar with those personally.
• Some models are very highly specialized, and may require very specific configuration, prompts and inputs to work as intended, they may not all be a traditional “chatbot” that you can just write text and converse with, but most of them are, and the popular ones pretty much all are intended for general use like this.
• Parameter count sort of represents the model’s knowledge base and breadth of knowledge and kind of also how smart or stupid it is, this is usually measured in billions (for example 9B or 27B) and usually provides a good hint towards the approximate expected file size (A Q4 is usually less but in the same ballpark of GB
• MoE (mixture of experts) is usually noted by “A<???>B” notation (for example, 31B A4B tells you it’s a 31 billion parameter model with 4 billion of those parameters active during the processing) MoE models are still big, but MoE is a technique that makes the model run much faster than it’s file size would suggest by ranking out only the parameters that “seem” like they will be most relevant for what you’re asking about, which can help a lot when you’re trying to cram a larger model than you can reasonably run with parts of it in much slower CPU/system memory. It’s not a magic bullet, but it is a popular strategy that works pretty well. You still need to have the file loaded into some kind of memory, and if you’re using system memory/CPU inference at all it’s still not going to be as fast as a model fitting in pure GPU is, but MoE it can make it practical to run much larger models than you would otherwise be able to tolerate. On the other hand, because only, for example, 4B parameters are active (and they may not be all that great at deciding which 4B they should use) they can sometimes be stupid. Generally it’s a good thing though depending on your needs. The default models use all parameters all the time. This is the opposite of MoE and is called “dense”. There’s no right or wrong answer, try both and see what you like. They’re all flaky sometimes but in my experience a dense model is more consistent, whereas MoE can have some very good results and some very bad results depending on how the dice roll inside its brainbox.
• Context is important when you’re trying to have longer conversations or process a lot of local data at once. Context uses VRAM memory too. If your context is limited the model may have poor “memory” and become forgetful or have to keep compressing its context. You can set context to use Q4 quantization too which lets you fit more context into memory, but there’s always a limit, the model itself will have a maximum and the software you’re running it with will likely have its own limit. It’s not super important for short questions but becomes important for more advanced use, especially coding.

Hope that helps, have fun, be safe!

For those who wish to avoid big tech, and don’t mind potentially being on a data center: openrouter lets you use w/e model you wish (even free), paying per token, from whatever provider you wish (in particular, smaller companies, folks self hosting, etc.)

Some datacenters are built in terrible places, but not all.

Man you can absolutely run something on ollama yourself. Right now I have ollama running qwen 3:4b-instruct on 6vCPU and no GPU and for very simple conversations it’s quick enough. TTFT is slow when prompt is uncached but otherwise TTFT is like 2s and I get maybe 5 words/s out.

If I gave it my 1660 I could probably get ~100tps but my card won’t pass through to my VM properly…

You can, but it’s not worth the effort. The technology itself is set up for people of power, such as governments and corporations. That’s why they steal our hardware and make it unaffordable. The only genuine use is accessibility, but it’s purely audio/text recognition. Everything else is for targeting people with drones and surveillance state.

Edit: how could I forget deepfakes? You practically can’t live without them nowadays /s