points by ActorNightly 4 days ago | 0 comments
Local models requires investment of about $2500 into hardware, so that you can run models at decent tok/sec. You then need to basically write custom harnesses. All of my stuff runs on a llama.cpp backend on a dedicated inference machine, and a client runs custom code built with OpenAPI library to make inference calls to accomplish things.
[–] dr_hooo 4 days ago
[–] ActorNightly 4 days ago
The latent space of models, which basically steers the token generation into a particular direction is much coarser with the local models compared to big models. Small models known fundamentals well, but do not have the breadth to realize the particularities of combination of fundamental task given certain criteria.
As such, if you want to replicate big model behaviour, you need to basically do context augumentation. How you do that is up to you for your own needs.
[–] dr_hooo 4 days ago
But my question remains: isn't something like pi.dev good enough for this? Or are you using custom harness to describe the scripts/skills/etc that you create yourself?
[–] nickspacek 4 days ago
Yes, this is a straightforward extension for Pi to have it continually steer the model at every turn. You could write your own with Pi itself or use an existing extension.
[–] ActorNightly 4 days ago
No. All agents out there basically operate on skills, mempry, and tools. There are almost none that do context injection with every prompt. You basically need an external system that modifies every submitted prompt with additional context
[–] dr_hooo 3 days ago
Okay, just started to read up an Agentic Context Engineering (ACE) & Co,: really interesting. Thank you for the explanation!
[–] trog 4 days ago
I am cheerfully using Opencode and pi.dev on a local model without thinking about any of that stuff, just running it straight out of the box and it's working great. I have access to an RTX6000 on a remote machine so using it for 'free' and it's replaced Claude for my personal projects. It gets stuck occasionally and gets things wrong from time to time, but so does everything.
[–] bityard 4 days ago
> You then need to basically write custom harnesses.
Why? Most existing harnesses let you point them at whatever API you want. I'm currently have a great time with Juggler.
[–] dcrazy 4 days ago
I find myself having to write a lot more skills for Qwen3.8-27b to effectively use tools it hasn’t seen before, versus virtually no skill-writing needed for Claude anymore. It seems to understand novel tools and when to use them pretty intuitively.
[–] ActorNightly 4 days ago
For basic tasks, yea. For more advanced tasks, where the agents running on big llms will do more actions autonomously, the smaller models will sometimes stop execution or go off in wrong directions.
[–] irthomasthomas 4 days ago
Why use llamacpp instead of something faster like exllama?
Curious, why/how does that help? Or do you have some niche needs?