How we measure Renor: LongMemEval, LoCoMo and BEAM
Which reader answered, which grader scored, and which settings produced each Renor figure, so you can tell what a percentage does and does not show.
A memory benchmark percentage means little without three more facts: which model answered, which model graded, and how much text the memory handed over. This post gives all three for every Renor figure we publish, says what each run is not, and says where a competitor is ahead.
The results table below is one run per row, not held out from development, answered by Claude Opus 5.5 at xhigh effort on Renor's deep-search page. Deep search means --top-k 50 and a 4,000-token ranked-expansion budget (RENOR_TOPK_SCALES_BUDGET=1). Deep search is off by default. The evaluation code is in the open-source engine: renor-engine/scripts/eval.
The results
| Benchmark | Questions | Score | Grader |
|---|---|---|---|
| LongMemEval-S | 500 | 98.4% (492/500) | gpt-4o-2024-08-06, official LongMemEval prompt |
| LoCoMo, categories 1 to 4 | 1,540 | 96.7% (1,489/1,540) | the Mem0-paper J judge, run on gpt-4o-mini |
| BEAM ~100K | 400 | 76.2 | BEAM nugget scorer, judge DeepSeek V4.1 Flash |
| BEAM 1M | 700 | 74.7 | same |
| BEAM 10M | 200 | 60.2 | same |
BEAM scores are nugget scores out of 100. The judge is a stand-in, not BEAM's official evaluator, so the BEAM numbers say how Renor does under that judge and do not transfer to another one. The 76.2 counts the one ~100K question the scorer could not parse as zero (0.764 over the 399 it scored).
LongMemEval-S: 98.4% with deep search, 94.4% by default
LongMemEval-S is 500 questions about long chat histories of about 115,000 tokens each. The 98.4% run was on 2026-10-27, one run, Wilson interval 96.9 to 99.2.
With Renor's default settings and the same Opus reader, the score is 94.4% (472/500). Deep search moved Opus from 472 to 492 correct: 23 questions gained, 3 lost. The settings were developed on these 500 questions. On the 400 outside the 100-question development set, deep search scored 394/400 and the default settings 380/400 (95.0%).
The reader matters as much as the memory. On identical default pages, Claude Haiku 5.5 scored 83.9% and Opus 5.5 scored 94.4%.
LoCoMo: 96.7% now, 78.8% before
The 96.7% (Wilson interval 95.7 to 97.5) is Opus 5.5 reading the deep-search page, graded by the J judge from the Mem0 paper run on gpt-4o-mini. By category: single-hop 98.0, multi-hop 96.5, open-domain 94.8, temporal 94.1.
Our earlier LoCoMo figure was 78.8% (1,213/1,540). That run used a low-cost reader, DeepSeek V4.1 Flash, and gemma4:31b as the grader, with the shipped default page. The two runs differ in the reader, the grader and the page together, so the 18-point gap is not a retrieval improvement alone, and it is not a fair before-and-after of one change.
BEAM: where Renor is behind
BEAM tests memory at growing scale. Renor's nugget scores fall as the history grows: 76.2 at about 100K tokens, 74.7 at 1M and 60.2 at 10M (bootstrap interval at the 95 percent level 54.5 to 65.9 at 10M).
At 10M tokens, Hindsight publishes a higher BEAM score than Renor's 60.2 (its post, April 2, 2026), so Renor is behind there. We do not set competitors' BEAM figures beside ours: each vendor uses its own judge, and we have not recorded a source that names the answer model and judge. Renor's weakest BEAM abilities at 10M are event ordering, multi-session reasoning and summarization.
An earlier BEAM screening with a DeepSeek V4.1 Flash reader and other judges gave 0.524 at 128K, 0.468 at 1M and 0.292 at 10M (a lower bound, with 68 of 200 answers empty). Those rows are listed in research-details.txt and are a different experiment.
Tokens read per question
On LongMemEval-S, the mean text sent to the answering model is:
| Source | Mean tokens |
|---|---|
| Renor, deep-search page of the 98.4% run | 5,537 (default page: 2,508) |
| Mem0, its own figure at a top-200 budget (docs.mem0.ai) | 6,787 |
| The whole conversation | about 115,000 |
This is the one row where Renor and a competitor are measured in the same unit.
The coding-agent test
Separate from the benchmarks: Claude Code (Claude Haiku 5.5) does 50 small coding tasks three times each, where the right answer was settled in an earlier session, scored exactly with no model grader. With Renor hosted (hybrid search) it got 97%, with keyword-only search 81%, with no memory 45%, with hand-written notes 96% and with everything in CLAUDE.md 99%. These come from separate runs, not one paired table, so they do not show Renor beating notes or CLAUDE.md. Details are in research-details.txt.
How to read anyone's scores
Before you treat two percentages as a race, ask:
- Same benchmark and the same questions?
- Same answer model?
- Same judge and prompt?
- Same retrieval depth or token budget?
If any answer is no, they are different experiments. For a side-by-side that lists published numbers with those caveats, see Renor vs Mem0 vs Zep.
Try the product these numbers describe
Create a free account, follow the quickstart, or read the research page. Free: 3,000 memory calls a month, no card. Pro: US$10 a month (pricing).