I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?
I like deepseek v4flash, I use opencode go and commandcode, somehow feels not as bad since its cheaper and technically can be ran at home if you can afford it. I don’t care for regular ai chat, feels useless after using agents. Agents are what you expect when you think of ai at the computer, hermes is a solid free one. Those two subs rotating as one runs out should last you a while at 20$ a month. These smaller companies/datacenters aren’t the issue imo, the larger ones like Anthropic, google, openai, etc. are.
It just functions so much better to use low cost api subs over trying to run anything locally. Let them subsidize this shit for you, don’t abuse your own hardware while it’s expensive af to replace, can spend 100s a year on subs and it’ll be less than the hardware costs for that amount of time.
No. Because knowledge is power, and LLMs teach you not to teach yourself to acquire knowledge.
I don’t think I’ve seen this recommended yet, but maybe you can check out CoCore https://www.cocore.dev/. I saw a demo of this some time ago and it’s still pretty much an experimental project. Essentially this is more of a community shared way to run local LLMs, especially for those without necessary compute. You are basically using an LLM hosted on another person’s computer in an encrypted manner. There’s a lot of technical detail about it that goes over my head. It is slow, and of course, still requires trust, or some auditing to verify it. But I think it’s kinda a cool way to crowdsource resource to run. The network of users is on fediverse btw.
Sounds like golem. Interesting
For those who wish to avoid big tech, and don’t mind potentially being on a data center: openrouter lets you use w/e model you wish (even free), paying per token, from whatever provider you wish (in particular, smaller companies, folks self hosting, etc.)
Some datacenters are built in terrible places, but not all.
OmniRoute is also pretty nice for that.
Srsly, unless you grow all your food and make all your possessions, you’re constantly supporting billionaires. I mean sure, run your own LLM, but don’t let it create a delusion of ethical purity.
Most of the world’s food comes from small local farms. Maybe join a CSA if you live in an industrialized capitalist country without a proper farmer’s market nearby
There are 3 or 4 billionaires in my country. Founders of Wise and Bolt. None of them own farms or grocery stores.
Matter of fact, the most popular grocery store chain is coop so most of the “owners” (members) of your local cooperative aren’t even millionaires.
you can use smaller open weight models, but they aren’t very good
The size and token rates mentioned in this thread for local inference are all going to be exceptionally limited compared to the frontier
Forget about local inference with your hardware, it’s too far behind. 3 years behind in this technology is basically stone age
What you’re looking for is “ZDR” (zero data retention) policies from online providers. There are many options. One powerful model with ZDR you can use is gpt-6 Luna on duck.ai
Even if you’re trying to code, you’ll get better mileage copy pasting to and from duck than ollama with your hardware
trying to code
copy pasting to and from duck
That’s, uh… The worst way to do any of this. And GPT-6 Luna is the weakest GPT-6, if you’re going to run something that’s on someone else’s hardware anyway, you can just use openrouter and choose ZDR hosters of open-weight models that are somewhere between GPT-6.1 Sol and Terra in accuracy.
OP said they didn’t want to support billionaires and data centers
The money used to make zdr requests on openrouter to good models like GLM 5.3 flows into large inference providers like fireworks and baseten
Yes, copy pasting is a terrible way to vibe code. But my point is it’s still going to be better than trying to ollama with a 1060 and 16 GB RAM
Yes, I don’t have a very beefy graphics card but I can run LLMs locally. I use them for things like proof-reading. Look into llama.cpp. If you have an Nvidia card then look into using CUDA so your LLMs don’t run slow as fuck using only your CPU.
Check how much VRAM you have and then do some research into the open-weight models. The best models require quite a bit of VRAM, but there are models that work well on cards with less VRAM.
It’s energy-intensive, but on a level that most people already find acceptable. I’d say it’s comparable to playing a AAA video game for a few minutes to generate an answer.
Local LLMs aren’t slower if you pick a model well-suited for your GPU, but they are dumber. But that doesn’t make a difference for me as I wouldn’t use LLMs for any particularly cerebral tasks e.g. programming. They can work fine for things like bouncing off ideas, proof-reading, making suggestions for things like writing and debugging, etc, the kind of thing where you can read their response and just discard it if it’s stupid. Whether or not that’s “useful enough” for you is down to you. You can just give it a go and uninstall things if you don’t find them useful.
No, and why do you want to rot your brain and/or produce more slop anyway?
Because LLMs now work better for searches than commercial search engines.
At producing misinformation.
Why would you want misinformation? It’s going to lower your productivity, because you then have to fact check everything it tells you
Ha! No they don’t! It gives you a hallucinated answer that’s not even on the webpage that it claims it got the answer from.
The answer to your question is, it depends on your expectations. The tradeoff of a typical no VRAM machine is latency and quality.
My progression:
- I decided to try ollama and ran a small model on 8G RAM on a VPS. Slow but it worked. Ran an 3B parameter models – slowly. Struggled with 8B models. Problems with hallucinations from these small models. This was my induction into self hosting llms.
- Installed similar setup on a SFF desktop with 16G RAM. Ran 8B models slowly. More capability, but still below the threshold I was looking for. Open WebUI + ollama worked but still too slow.
- Acquired a 16G VRAM card and installed in an old desktop with 16G RAM. This runs 12B models and crosses the threshold, in my opinion, of usability as an assistant. This setup runs Open WebUI, again reinforcing usability.
- Am I satisfied? No. I’d like bigger, faster meaning more VRAM, on a newer platform, but content enough now to continue experimenting with what I have.
In sum: Don’t expect low laitency multi-step reasoning/planning with a low resource / no VRAM machine. Anything close to what you can get online at reasonable speeds requires ample VRAM.
For a reality check … I did a quick run on those three machines as they are still active. What follows has been synthesized through a cloud based model from real data:
same prompt, three machines
Prompt: “Explain how a hash table works, including collisions, in about 250 words.”
Run once on each machine withollama run <model> --verbose, all Q4_K_M quantization.Machine Model Processor Speed (tokens/s) Total time Quality (/20) VPS, 8 GB RAM gemma3n:latest (6.9B) CPU only 5.3 1m 11s 16 SFF desktop, 16 GB RAM gemma3:4b (3.9B) CPU only 6.8 51s 13 Desktop, 16 GB VRAM gemma4:12b (11.9B), thinking off GPU 31.2 11.5s 18 Desktop, 16 GB VRAM gemma4:12b (11.9B), thinking on GPU 30.9 38.5s 18 What stood out
- The GPU box generated about 4-5x faster than the SFF while running a model roughly 3x bigger.
- Both CPU-only machines landed at 5-7 t/s. Usable, but slow for interactive chat.
- Turning thinking off on the 12B cut total time from 38s to 11.5s with no change in speed (~31 t/s) and no drop in answer quality on this prompt. Thinking would likely matter more for math, code, or multi-step problems, which this test doesn’t cover.
- The 12B had no factual slips. The smaller models made small conceptual errors (blurring hash code vs. index, calling the whole array a “bucket”) and one skipped lookup entirely.
- All of them left out worst-case O(n) behavior and resizing.
Caveats
- Different models on each machine, so this shows what each box can run well, not identical hardware running identical software.
- The thinking-on run generated about 1,180 tokens versus about 350 with thinking off, so compare tokens/sec rather than total time across those two rows.
- Quality scores are my rough 1-5 ratings on accuracy, length, completeness and cleanliness, with an LLM’s help, from one run each at temperature 1. Treat them as anecdotal, not a benchmark.
- gemma3n is listed by Ollama as 6.9B, but it uses an “effective parameter” design (Google calls it E4B, roughly 4B in practice), so it’s lighter than a dense 7B. That’s why it ran on the 8 GB VPS where a standard 8B struggled.
99% of AI models available are based on stolen work, so even running it locally is immoral…
“Copyright” is immoral and exists to protect capitalists. Intellectual property as a concept needs to die, and knowledge needs to be accessable to everybody.
This is what a person who has never developed anything would say.
Like, Stallman?
I’ve developed and written loads. Everything licensed CC BY-SA.
I will now reuse your work without attribution and claim it is the AI’s fault. Easy.
Everything you’ve made is now CC BY-SA too, awesome. Thanks for contributing.
No problem, you’re welcome. I also paid a small car’s worth of money to a fascist oligarch while borrowing the fruits of your work. I, too, am very happy with this.
But look at the bright side, you too can now pay a small car’s worth of money to a fascist oligarch and then get some source code that may or may not be close to my proprietary stuff. You know, the one made from your stuff without attributing. This is fine.
I think you may have lost the plot there.
Which model GPU and how much VRAM? 16GB RAM limits your options a bit but maybe LFM 2.5 24B-A2B
If not, Gemma 4 e4b or Ling 3.0 Tiny are about as small and fast as you can get while still being decently smart.
Man you can absolutely run something on ollama yourself. Right now I have ollama running qwen 3:4b-instruct on 6vCPU and no GPU and for very simple conversations it’s quick enough. TTFT is slow when prompt is uncached but otherwise TTFT is like 2s and I get maybe 5 words/s out.
If I gave it my 1660 I could probably get ~100tps but my card won’t pass through to my VM properly…
I’ll just leave this here. https://sleepingrobots.com/dreams/stop-using-ollama/
So… I’m with you/them about the vendor lock in but I think they’re slightly underestimating what ollama brings to the table. For example, ollama bundles CUDA libraries so I don’t need to worry about that as I wrangle with my GPU. It’s also well supported by other ecosystems like home assistant in a way that raw openai compatible API server isn’t.
Overall I don’t know if I find these arguments compelling at this time.
ollama
Shittiest local inference engine. Use llama.cpp instead.
qwen 3:4b-instruct
Old version 3 model from Alibaba. 4 billion parameters total. It will struggle with anything other than natural language generation.
on 6vCPU and no GPU
Running on a 6 vcore VM with no GPU pass through.
TTFT
Time.To.First.Token. How long it takes the model to start responding after you hit enter. Dependent on not only processing speed but also context length.
prompt is uncached
Cold start = no cache = slow. But faster when mid conversation and the key-value cache is hot.
100tps
100 tokens per second. 1 token is roughly 1/4 word.
You can, but it’s not worth the effort. The technology itself is set up for people of power, such as governments and corporations. That’s why they steal our hardware and make it unaffordable. The only genuine use is accessibility, but it’s purely audio/text recognition. Everything else is for targeting people with drones and surveillance state
Wild conclusion
While I respect your cynicism and it’s fully justified, in fact they can do useful and interesting things for individuals too. They must be used cautiously, and with understanding of their limitations and a realistic appreciation for their capabilities. Large neural networks like this are in a fundamental sense statistical models. I have pointed out in the past that many of the large general-purpose “assistant” models that have been trained on what amounts to very close to the entire corpus of knowledge and information contained in libraries and on the internet. Regardless of why they were trained, and whether that was done ethically (it wasn’t) the fact is these models now exist, and we have access to many of them, and they contain statistical models of what I will refer to as the vast majority of human knowledge to-date.
If you can’t imagine that there are ways that statistical information could be valuable, you are either incredibly lacking in creativity or simply being ignorant and myopic as far as I’m concerned.
As an analogy, national and sociological demographics and census data is abbreviated and summarized statistics much like neural networks. It cannot in many cases correctly and confidently predict any particular outcome. But that doesn’t mean it’s not useful or meaningful. It is not a complete and accurate listing of every human’s thoughts, feelings, and preferences, and most honest people would not imagine it is. If you ask it for that, you are doing it wrong, or you are selling something. Same with generative AI models. The fact that some people (including huge companies) misuse it does not mean that such statistics are evil and we should throw it away.
Using it for intelligent, thoughtful purposes is a form of rebellion and I believe it is important that we continue to have access to these technologies. They are not intelligent, and they are not even artificial. They are loosely curated distillations of human knowledge. That is very powerful, it is a very fascinating and practically useful tool. It is not magic, it is not a real boy, it does not have feelings or purpose, it is a tool. It is being used for purposes, but you have the power to use it for purposes too, and those should be your own purposes, not what purposes companies dictate to you. Engage your brain and learn how and when to use it, don’t use it to replace your brain, it’s not what it’s being sold as, but it is a very clever and interesting technology.
It is so refreshing to see a well-formed opinion on the character, quality, and appropriate use of AI. I would like to thank you for your service.
I appreciate your support. Trying to be nuanced and realistic and find common ground on a topic that is so polarizing, and frankly potentially existential, is difficult. We live in a scary world right now for a lot of different reasons, and people are scared (of being left behind, of having their work stolen, of having their humanity stolen), and I don’t blame them. These are valid and justifiable concerns that deserve serious consideration and serious discussion. Which, in a lot of cases, they’re unfortunately not getting.
If I were running the show, I would do things a lot differently. But I’m not running the show, and I don’t think the people running this show have even the vaguest interest in listening to me or probably pretty much anyone, so all I can do is try to educate and provide advice, which is free, while I too worry if AI is going to take my job.
I think I’m (unfortunately) starting to really understand what Frank Herbert meant when he wrote, “Fear is the mind killer.” It’s very disheartening to see legitimate concerns snowball into lemming-like behavior.
When in the throws of despair or grieving for a society in decline I’d typically appeal to the classic MiB quote:
A person is smart. People are dumb, panicky dangerous animals and you know it.
But as the years go by it’s becoming increasingly difficult to find solace there — mostly because my experience is showing me that it’s simply no longer true. I feel like I’m living in a world where everyone read Bonhoeffer’s Theory of Stupidity and most people responded with, “Where do I sign?”
I’ve observed this behavior from both doomers and AI evangelists.
However, in the effort of full-disclosure, I must admit that I am a staunch accelerationist. That’s because, when viewed from a cosmological perspective, the only rational way to describe the last few hundred years of human development would be as an explosion, and I firmly believe the only path out is forward. Standing against the tide of technological progress at this point would be akin to standing against a blast wave.
That’s really damn scary.
The real sticking point seems to be that increased knowledge does not alleviate the existential dread. On the contrary, in many ways it exacerbates the mortal fear. Knowledge without power or prescience is impotent. Not that it would make a difference anyway.
If we had developed the concept of The Great Filter before the steam engine, would the world have responded to Watt’s machine in the same way it’s responding to Dario’s now? And would that have even made a lick of difference?
Very interesting commentary. I get what you’re saying, and I agree. I think there’s something similarly paradoxical to Dunning-Kruger going on with these kind of existential concerns, too, and I think that is probably reflected in your accelerationist position. We can also see it in climate change for example, the people who actually understand it and study it are less likely to find or see anything that can actually stop or reverse it, besides the one thing that it’s increasingly clear we won’t do: drastically cut emissions. And the deeper you understand the actual threat and the realistic psychology around it the more overwhelming and hopeless it seems, and the less it feels worthwhile to even try to stop it.
By the time all the less-understanding, less-pessimistic people start to actually see the unavoidable long-term consequences and trends of climate change, we’re going to be so far beyond having any sort of handle on it to control it, the only remaining hope is for a miracle, no matter how improbable, and if you have no hope of stopping it, then pinning your hopes on the miracle and trying to get as far ahead as you can until and unless that happens is (sadly) actually the rational choice.
The only genuine use is accessibility, but it’s purely audio/text recognition.
On that note:
LocalTranslate is an offline translation application that uses Firefox’s neural translation models (from mozilla/firefox-translations-models) to perform high-quality translations locally on your device.
