Why we withdrew our memory benchmark number

Withdrawn, 17 September 2026. An earlier version of this note reported Anjo's LongMemEval-S retrieval scores, led by 0.821 recall@5. Those scores are withdrawn from the site. The corrected harness has since re-run them and they held; this note explains what went wrong and why the figure is staying in the repository.

What we measured

LongMemEval (Wu et al., ICLR 2025) is a public benchmark for long-term memory in chat assistants. Its S variant gives each of 500 questions a haystack of roughly fifty prior chat sessions. Only one to six of those sessions hold the answer. The retrieval task is to rank the sessions so the right ones come first.

The harness, anjo/evals/external/longmemeval_s.py, drives the production read path rather than a copy of it. Memories are written and queried with the same functions the app uses.

What went wrong

To score a ranking, the harness has to know which session each result came from. It worked that out by matching the result's text and taking the first session in the haystack with that text.

That breaks when a haystack holds two identical sessions with different dates. Anjo's ranking includes recency, so it can put the newer copy, a distractor, first. The harness would then credit the older copy, the answer, and score a hit that was really a miss.

We had believed this error could only lower the score. That was wrong. It can raise it too.

What we don't know

We have not measured how much the error moved the score. In the 58 questions we could re-check, no answer session had an identical twin. All 58 were one question type, though. The two types built around near-duplicate sessions with different dates, knowledge update and temporal reasoning, were not checked.

So the honest statement is not "the real number is a bit lower." It is that we do not know the real number yet.

What happens next

The harness now identifies each result by its memory id rather than its text, and tests/test_longmemeval_scoring.py pins the duplicate-session case. It has since re-run all 500 questions, and the result held.

We are still not putting the number back on the site. A retrieval score is not something you can act on as a user, and we would rather be judged on whether Anjo actually remembers you than on a figure in a table. It lives in the repository now, where it catches regressions. The site's own test suite fails if any page cites the withdrawn figures.

What has not changed

The memory did not change; the claim did. When retrieval misses something, the grounding guard still keeps that miss from turning into an invented recollection.