Hacker News
points by nfw2 2 days ago | 0 comments
I've also found it very hard to leave AI agents up to their own devices for long periods and get much productive benefit from it.

I suppose it's possible that frontier labs have internal models that are so far beyond that autonomous research becomes possible, but I'm not sure that's it.

It seems to me like these models are like a very competent, important part of some hypothetical digital brain, but alone they don't have all the facilities they need to be useful, like a genius with severe ADHD and short-term memory loss.

For anything where quality isn't objective in a clear-cut way, very intentional context management is needed to get good outputs. As context size expands, performance degrades. As the number of tasks increases, performance degrees. As the number of independent decisions based on subjective criteria increases, performance degrees.

If you can get a harness that makes sure every decision is made one at a time, focused with only the context it needs, things work great. But making those harnesses is still hard, a lot of work, and something that agents seem to have no clue how to help you with.

[–] pona-a 2 days ago
That's a kind of bitter lesson thinking, that we take this generalist model, which already mastered everything from English to Javascript with unsupervised training, and stuff it inside an expert system so it can do long division for it. That's much similar to the last AI winter, when the researchers realized their logics couldn't cope with certain logical scenarios, but still thought they could salvage their flawed reasoners as dispatchers to specific algorithms.

LLMs will either solve issues like this at the architecture level, or declare them out-of-scope entirely.

[–] skew-aberration 2 days ago
You can try adapting to this reality in your methodology though. Divide up your project into clear cut and ambitious modules, with well-defined objectives to evaluate each part.

Codebase porting, implementing sophisticated algorithms, performance tuning, model training, reverse engineering, fuzz testing / benchmarking suites, refactor to support a new interface, etc, scale so much better than 'fix a then b. don't break c' style prompts.

[–] nfw2 1 day ago
Exactly, I agree. I have been working on a ux harness that creates a deep product model with personas, user journeys, etc as well as designs. Originally I was hoping it that it could work just some skills and a data export format. It was burning 300M tokens just to get nowhere.

Now I have this DAG with something like 40 specialized subagents. It's sort of a monstrosity, but it works much better for 100x fewer tokens. I've found that basically every single judgment call (which is characteristic of ux) really wants an agent solely focused on it with plenty of concrete examples.

[–] nathias 2 days ago
I don't think its the frontier models, my opus 5.5 has no problem working on a project fairly autonomously, for example if I prompt it in the evening it will work the whole night without issues...