Hacker News

points by yfontana 22 hours ago | 0 comments

So far I've turned off automatic memory creation in every harness I've used. Claude Code sometimes creates some anyway, and I probably delete 95% of them. The memories it creates are almost always relevant only to the specific task that it is currently working on. If it's a task I'll need to do again, then yeah, maybe that's useful. But that isn't the case most of the time, so I delete them to avoid cluttering context.

So the main improvement to memories I'd like to see is better judgment on which ones to create.

The parts about "dreaming" make me think of how agent transcripts contain a ton of information, which I think is currently under-utilized. I'll repeat advice I've seen here a few times: ingest your own transcripts into duckdb (claude code will build that pipeline in about 10 minutes) and do some data analysis on them. Some ideas if you don't know where to start: look at tool call failures to see if you can prevent common ones, check out if you have large cache misses and what causes them, look at various efficiency metrics over time (tokens per $, lines of code per tokens, etc.) with main thread vs sub-agents, etc. I started with this sort of things, and ended up following some threads to interesting insights. For example, I found indications that I tended to make Opus 5.5 / Sol 6.1 divide their work into units that were too small. They have the capacity to work on more stuff at once than older models, and I wasn't really taking advantage of that.

[–] sunaurus 18 hours ago

I give the same advice constantly. Turning off memory has been one of the best ways I've found to reduce "WTF per minute" when using agents.

[–] yakkomajuri 21 hours ago

I've also disabled memory in Claude for finding it more annoying than helpful.

However, I have it on for ChatGPT (use it mostly for on the go queries via mobile) and their memory experience is significantly better. I haven't enabled Claude's memory in a long time so things may have changed though.

[–] agentdev001 18 hours ago

And somewhat open source.

[–] stingraycharles 22 hours ago

I just periodically ask it to curate all memory, and ask me questions to do so.

And in the Claude Desktop UI I manually curate the memories.

Works well enough, I don’t like the concept of dreaming as it quickly leads to memory bloat and/or outdated stuff.

[–] rav 19 hours ago

> look at tool call failures to see if you can prevent common ones

I often see tool call failures because the agent invokes the tool with wrong parameter names, e.g. for file creation it provides "file_content", but the tool expects "file_text" instead. My colleague told me to put this in a pre-prompt file, but I don't understand why the models make these kinds of mistakes - the model should have the available tools with their parameter names in context, should it not?

[–] ford 16 hours ago

They are in context, but models are probabilistic. Sometimes they bias to the tools they were trained with, or sometimes they're just wrong. More references in the context window makes them more likely to output the right things.

[–] solarkraft 18 hours ago

> If it's a task I'll need to do again, then yeah, maybe that's useful. But that isn't the case most of the time, so I delete them to avoid cluttering context.

It would be useful if they could scope them properly. Then it would be useful to keep around. But I haven’t seen memory systems be able to do that.

[–] jv22222 17 hours ago

Yep I turned off memory and then I made this. It interactively suggests what to store when it sees you make headway or solve problems. I found it to be very good but ymmv.

https://innerloop.works/breadcrumb

[–] andai 19 hours ago

>I found indications that I tended to make Opus 5.5 / Sol 6.1 divide their work into units that were too small.

How did you determine this?

Do you mean that you were giving them tasks that were too small or you were telling them to split the tasks up or something?

[–] yfontana 17 hours ago

For most dev work I ask claude code / codex to plan larger work as epics and stories (after ideation / brainstorm if needed). I find that this works well to let the AI, my coworkers and me track what has been done so far and what's left in the backlog. Sometimes I direct it on how to split the work, sometimes I don't.

When analyzing my session logs, I tried to look at how story size relates to how much work the AI had to do to fully implement it (made easier by my sessions always having epic#.story# - like 2.6 - in their name). Neither "story size" nor "AI work" have fully obvious metrics, so I looked at a variety of things, like length of story plan, number of lines of codes written for the story, number of different files read/written (and for AI work: number of tool calls, number of tokens for main thread / implementer / reviewer subagents, etc.).

The pattern is always pretty much the same: smaller stories are less token-efficient. One major factor for this is that in bigger stories, the average number of lines of code written per tool call is significantly higher (and not just a little bit, if I remember correctly it's a factor of 1 to 3 between my smallest and largest stories).

None of this is scientifically rigorous, but it's consistent over a fairly significant sample (about 60 stories up to this point). I'm also only measuring amount of work and not quality, but when I find things wrong with a story, I generally ask the AI to fix the issues in the same session, so the two should be correlated (sometimes the issues are large enough that they require new, separate stories to fix, but this is normally caused by problems in ideation and/or planning, not implementation). This is enough to encourages me to ask for more rather than less per story. It might also not apply to everyone, I'm sure this is dependent on everyone's workflow and way to work with the AI. But yeah, that's the sort of stuff you can explore when analyzing your session logs.

[–] reubenmorais 15 hours ago

I have a similar experience, I think integration overhead is also a big part of it. Running tests, linting, verifying commits are matching repo conventions, rebasing before pushing, are all tasks that are roughly constant per story, so smaller stories pay a bigger chunk of overhead.

[–] plaguuuuuu 14 hours ago

You made me realise that this might be a function of planning granularity, since I have the opposite problem and never (ever) run full plans from the top down any more, as I started getting better results without it since Fable dropped. I'm cutting features into high level slices instead and having subagents work on the slice and hand back to an orchestrator.

The slices are usually a bit too wide. I terminate subagents at 200k and get them to handoff to lead. It works pretty well but again, it's hard to get the orchestrator to size the slices a bit smaller.

[–] epihelix 21 hours ago

> So far I've turned off automatic memory creation in every harness I've used.

I also never use memories. I don't see the usefulness above project-specific prompting and a good knowledge of your own codebase. Am I missing something?

[–] hypfer 22 hours ago

> Claude Code sometimes creates some anyway, and I probably delete 95% of them.

The thing you're asking it for might be intent, which is something the technology fundamentally lacks.

Which.. is something that has been discussed very extensively, which makes one wonder why Anthropic still keeps trying to push that square peg through a round hole.

[–] O_H_E 14 hours ago

> is something that has been discussed very extensively

Curios to learn more, you do have some reference discussion or posts about that.

[–] hridyaagrawal 13 hours ago

same, turned it off. i just keep codebase stuff in AGENTS.md + a progress.md in the repo. survives switching between claude code and codex, and you can actually see it in a diff

[–] troupo 7 hours ago

> So the main improvement to memories I'd like to see is better judgment on which ones to create.

The main improvement to memories is to actually use the memories, and not ignore them wholeseale. Directly from the article:

--- start quote ---

Injected memories use up valuable tokens, and an incorrect note can be carried into every session. Loading a note also doesn’t guarantee that Claude acts on it: our documentation on Claude Code warns that longer instruction files reduce adherence

...

It loads only the first 200 lines or 25 KB of MEMORY.md

--- end quote ---

The reason it can't create coherent or good "memories" (aka random .md files) is that it cannot recall or use any of them.

[–] bbor 19 hours ago

  So the main improvement to memories I'd like to see is better judgment on which ones to create.

Well you have good instincts, then! It's probably not obvious on the first read-through, but Figure 4 shows the qualitative strides they've made on exactly this goal.

   I started with [proactive post-hoc analysis], and ended up following some threads to interesting insights.

Can definitely echo that experience, glad I'm not alone. I think this actually speaks indirectly to your initial complaint/observation, too: Anthropic designs the [memory/tendency-to-reflect] systems to cover a huge breadth of usecases and preferences and they help a lot compared to [statelessness/never reflecting], but are distractingly inferior to alternatives that the user takes an active part in shaping and using.

I partly say this because I also finally pulled the trigger on deprecating the last ClaudeCode memories on my system in favor of a personal system[1] recently, and the issues I noticed were things a casual user/'vibecoder' might appreciate, but any engineer working fulltime on multiple projects via multiple harnesses would find counterintuitive. E.g.:

- The memories were hidden away and clearly not written for human eyes; useful for low-stakes ledgers, frustrating when the right kind of errant note can derail a longrun 2 weeks later.

- The memories seemed to err towards the side of remembering everything and recording incidental details; useful for remembering the user is gluten-free, but not for tracking spec changes over time.

- The memories could be specific to a subagent or the project or the whole system; an obvious & elegant system for toy examples, but a terribly unweildy one for anyone working at scale over the span of months.

[1]: if curious, a somewhat-outdated version of it is here: https://gitlab.com/doering-ai/libs/fleet TL;DR: Friendship ended with ClaudeCode memory files; my new best friends are `BULLETIN.md` for transitory alerts, `campaigns/[slug]/LEDGER.md` for basic episodic memory, and `reports/[date]-[slug].md` for writeups polished specifically for posterity.