· Diar Shakimov · 7 min

storing it was the easy part. four paragraphs of chat took recall from 0.76 to 0.09.

an agent only knows what lands in its prompt, so i measured what actually gets there. the right passage made it on 58.6% of answerable questions. rewriting the query made it worse, and a long prompt almost wiped it out.

case 04

rema keeps a team's notes in a sqlite file on the engineer's own machine. storing them was the easy part. a coding agent only knows what lands in its prompt, so what matters is whether the right passage gets there, under a budget, when someone asks.

on october 6 i measured that with no model in the loop.

the test set

i used 139 questions over 327 passages. 99 of them have a known answer passage. the other 40 have no answer on purpose, because i wanted to see what rema does when it should stay quiet. the questions come in types: direct, multihop, negation, paraphrase, typo and wrong premise. i ran two checks with no model and no network. is the answer passage in the top five, and does it actually reach the agent's context. two full runs on the same machine came out byte-identical.

typequestionsrecall@5
direct320.875
negation100.900
wrong premise100.800
multihop120.750
typo150.733
paraphrase200.500
all990.758
Table 1. recall@5 by question type: how often the answer passage is in the top five. 99 answerable questions.

0.758 isn't what the agent gets

0.758 sounds fine. the hook doesn't hand the agent the top five though. it only passes along passages that clear a relevance bar, five at most, inside a token budget. measured that way, the answer passage reached the agent's context on 58.6% of answerable questions.

armanswer in contexttokens per question
no rema0.0000
rema as it ships0.5861,386
relevance bar off0.7981,963
answer handed in1.000304
Table 2. where the answer passage ends up, 99 answerable questions. tokens are an estimate (characters divided by 4). the first and last rows are fixed by construction.

the gap splits about evenly. 20 points are ranking, where the answer isn't even in the top six. the other 21 points are the bar turning away answerable questions whose answer was right there. i can't just turn the bar off, because it's there for the other 40 questions. even with it on, 13 of 40 unanswerable questions still got passages. about 49,000 of the 163,000 estimated tokens in the run went to passages that didn't have the answer.

rewriting the query made it worse

paraphrase was the weakest type at 0.500, so i tried rewriting the query. i fused three deterministic rewrites with the original. on the 20 paraphrase questions it dropped to 0.450. one question gained and two lost. across all 99 it went from 0.758 to 0.717, so i didn't ship it. i wrote it down as a dead end with its numbers, so the next time it looks obvious the numbers show up first.

what a long prompt does

the hook searches with whatever you typed. in real use that's almost never a bare question. it's a pasted log, two paragraphs of context and then the ask. so i padded each question with filler paragraphs of about 500 characters and put the question last, first or in the middle.

what the hook searched withwhole promptlast paragraphlast sentence ending in ?
the question alone0.7580.7580.758
4 paragraphs of chat, question last0.0910.7580.646
19 paragraphs of chat, question last0.0100.7580.646
4 paragraphs of chat, question first0.0910.0000.333
4 paragraphs from the same docs, question last0.0000.7580.646
Table 3. recall@5, 99 questions per cell. the last two columns keep only part of the prompt, by a fixed rule. none of them sees the answer labels.

four paragraphs of unrelated chat took recall from 0.76 to 0.09. pasting text from the same docs was worse at 0.00, because the pasted text's own passages outrank the answer. you can see why from a single question. the search adds up a score for every word in the query, and a stray word that shows up in only one stored passage counts about as much as the words of the actual question. the stray words end up picking the passages.

rule-based fixes only work if they know where the ask is. keeping the last paragraph gets back to 0.758 when the question is last and gets about zero when it isn't. keeping the 12 rarest words doesn't assume anything about position, and it stays at or below 0.11. so part of this is a null result. i couldn't find a general rule-based fix.

lost in the middle showed that models use the start and end of a long context better than the middle. this is the same pressure one step earlier. a long prompt decides which passages the model gets to read before the model reads anything.

if you hit this

take ten questions where you know which doc has the answer. paste four paragraphs of real chat above each one and check if your retriever still returns that doc in its top five. if it doesn't, your agents are searching with the noise.

what this changed in rema

most of the hard part of memory is retrieval. rema measures what reaches the agent instead of what sits in the store, and it doesn't rewrite queries because that made recall worse. long prompts are still open. right now the hook searches with your prompt as typed, up to its length cap, so a long one gets about what the table shows.

Sources

Revised .