rema keeps a team's notes in a sqlite file on the engineer's own machine. storing them was the easy part. a coding agent only knows what lands in its prompt, so what matters is whether the right passage gets there, under a budget, when someone asks.
on october 6 i measured that with no model in the loop.
the test set
i used 139 questions over 327 passages. 99 of them have a known answer passage. the other 40 have no answer on purpose, because i wanted to see what rema does when it should stay quiet. the questions come in types: direct, multihop, negation, paraphrase, typo and wrong premise. i ran two checks with no model and no network. is the answer passage in the top five, and does it actually reach the agent's context. two full runs on the same machine came out byte-identical.
| type | questions | recall@5 |
|---|---|---|
| direct | 32 | 0.875 |
| negation | 10 | 0.900 |
| wrong premise | 10 | 0.800 |
| multihop | 12 | 0.750 |
| typo | 15 | 0.733 |
| paraphrase | 20 | 0.500 |
| all | 99 | 0.758 |
0.758 isn't what the agent gets
0.758 sounds fine. the hook doesn't hand the agent the top five though. it only passes along passages that clear a relevance bar, five at most, inside a token budget. measured that way, the answer passage reached the agent's context on 58.6% of answerable questions.
| arm | answer in context | tokens per question |
|---|---|---|
| no rema | 0.000 | 0 |
| rema as it ships | 0.586 | 1,386 |
| relevance bar off | 0.798 | 1,963 |
| answer handed in | 1.000 | 304 |
the gap splits about evenly. 20 points are ranking, where the answer isn't even in the top six. the other 21 points are the bar turning away answerable questions whose answer was right there. i can't just turn the bar off, because it's there for the other 40 questions. even with it on, 13 of 40 unanswerable questions still got passages. about 49,000 of the 163,000 estimated tokens in the run went to passages that didn't have the answer.
rewriting the query made it worse
paraphrase was the weakest type at 0.500, so i tried rewriting the query. i fused three deterministic rewrites with the original. on the 20 paraphrase questions it dropped to 0.450. one question gained and two lost. across all 99 it went from 0.758 to 0.717, so i didn't ship it. i wrote it down as a dead end with its numbers, so the next time it looks obvious the numbers show up first.
what a long prompt does
the hook searches with whatever you typed. in real use that's almost never a bare question. it's a pasted log, two paragraphs of context and then the ask. so i padded each question with filler paragraphs of about 500 characters and put the question last, first or in the middle.
| what the hook searched with | whole prompt | last paragraph | last sentence ending in ? |
|---|---|---|---|
| the question alone | 0.758 | 0.758 | 0.758 |
| 4 paragraphs of chat, question last | 0.091 | 0.758 | 0.646 |
| 19 paragraphs of chat, question last | 0.010 | 0.758 | 0.646 |
| 4 paragraphs of chat, question first | 0.091 | 0.000 | 0.333 |
| 4 paragraphs from the same docs, question last | 0.000 | 0.758 | 0.646 |
four paragraphs of unrelated chat took recall from 0.76 to 0.09. pasting text from the same docs was worse at 0.00, because the pasted text's own passages outrank the answer. you can see why from a single question. the search adds up a score for every word in the query, and a stray word that shows up in only one stored passage counts about as much as the words of the actual question. the stray words end up picking the passages.
rule-based fixes only work if they know where the ask is. keeping the last paragraph gets back to 0.758 when the question is last and gets about zero when it isn't. keeping the 12 rarest words doesn't assume anything about position, and it stays at or below 0.11. so part of this is a null result. i couldn't find a general rule-based fix.
lost in the middle showed that models use the start and end of a long context better than the middle. this is the same pressure one step earlier. a long prompt decides which passages the model gets to read before the model reads anything.
if you hit this
take ten questions where you know which doc has the answer. paste four paragraphs of real chat above each one and check if your retriever still returns that doc in its top five. if it doesn't, your agents are searching with the noise.
what this changed in rema
most of the hard part of memory is retrieval. rema measures what reaches the agent instead of what sits in the store, and it doesn't rewrite queries because that made recall worse. long prompts are still open. right now the hook searches with your prompt as typed, up to its length cap, so a long one gets about what the table shows.
Sources
Revised .
