I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?
The size and token rates mentioned in this thread for local inference are all going to be exceptionally limited compared to the frontier
Forget about local inference with your hardware, it’s too far behind. 3 years behind in this technology is basically stone age
What you’re looking for is “ZDR” (zero data retention) policies from online providers. There are many options. One powerful model with ZDR you can use is gpt-6 Luna on duck.ai
Even if you’re trying to code, you’ll get better mileage copy pasting to and from duck than ollama with your hardware
That’s, uh… The worst way to do any of this. And GPT-6 Luna is the weakest GPT-6, if you’re going to run something that’s on someone else’s hardware anyway, you can just use openrouter and choose ZDR hosters of open-weight models that are somewhere between GPT-6.1 Sol and Terra in accuracy.