Hacker News
points by the1024 2 days ago | 0 comments
This is definitely something the companies most focused on agent experience are thinking about:

- https://supabase.com/evals - https://nextjs.org/evals

What is a better analogy for how you think developer tools should think about agent experience: an RL environment (hillclimbing a task like you mention), or a test suite (to be used in CI/CD, etc) as referenced by Supabase/Vercel?

[–] 0x63_Problems 2 days ago
Active work should mostly be hillclimbing, or at least more exploratory. The search space is very large and you typically find a lot more low-hanging friction than you'd think by just running a broad battery of sessions. And I'd pair active hillclimbing work with feature launches/product work - essentially part of releasing new things is QA-ing them against coding agent workflows.

The test suite eval style is useful and mostly a complement. You do want regression tests on critical features.

[–] the1024 2 days ago
The search space being large feels like a fundamentally hard and costly problem to solve for. Where do you recommend folks start given such a high number of possible scenarios?