In reply to @arvidkahl · May 16, 2025For people hacking on their own local LLMs: both ollama and llama.cpp now support vision models. That's VERY cool. You can now privately and securely run LLM inference on documents & images.
If that doesn't spawn new business ideas, you haven't thought about this enough.
Hey Arvid! My wife and ran a local LLM experiment inspired by your post last weekend ("Local LLMs" are a must-consider option for every SaaS that wants to deal with private customer data...").
After comparing results, I'm actually conflicted about the whole thing.
For context:
My wife does leadership coaching and recently used vanilla GPT-4o via ChatGPT to summarize a text-transcript of an hour-long conversation.
Over the weekend, I read your post and thought "Hey, let's test local LLMs for more privacy control. The open source models must be pretty good in 2025."
So I installed Ollama + Open WebUI plus five models on a 128GB MacBook Pro.
I am genuinely dumbfounded about the actual results we got today of comparing ChatGPT/GPT-4o vs. Llama4, Llama3.3, Llama3.2, DeepSeekR1 and Gemma.
In short: Compared to our reference GPT-4o output, none (as in NONE, zero, zilch, nil) of the open source models were able to create even a basic summary based on the exact same prompt + text.
In my opinion, the open source summaries were offensively bad. It felt like reading the most bland, generic and idiotic SEO slop I've read since I last used Google. None of the obvious topics mentioned in the text were part of the summary. Just blah. I tested this with 5 models to boot! I ran the same test twice for each.
I'm not an OpenAI fan per se, but if this is truly OS/SOTA then, we shouldn't even mention Llama4 or the others in the same breath as the newer OpenAI models.
What do you think?