in september i went back to the IntelliAudit research pipeline, the one behind our paper, to look at one control, the asset inventory. the pipeline is four agents on one model: search, auditor, defender and judge. the evaluation runs 12 controls against 4 simulated organizations, 48 cells, and the evidence for this control is a big spreadsheet. the findings read like the system had seen all of it.
what didn't work: three wrong answers
it hadn't, and it took three wrong answers to see why. the roadmap blamed a 150-line cap that doesn't exist. the paper draft blamed the model for being lazy and bet that a newer model would fix it, and the rerun on the newer model produced a parsing error instead. a later commit blamed a 120-row cap, which lives in the survey app and never touched the pipeline.
why it was hard to see: the caps were stacked
the real one was mine. back in march i had given the spreadsheet loader a cap of 500 rows and 50 columns. it kept those and dropped the rest, no warning. and it wasn't the only cap. retrieval returned a bounded number of chunks, the planner only saw the first 50 evidence items, and the judge saw 200-character excerpts. the sufficiency check that should have asked for more never ran, because its counter counted queries.
the fix: counting what got in
| sheet | in the file | loaded |
|---|---|---|
| asset inventory | 1,500 x 65 | 500 x 50 |
| lifecycle events | 2,973 rows | 500 rows |
the auditor agent ended up seeing 131 to 140 of 2,395 chunks, about 6%, and two sheets were never returned at all. lifting the caps showed how much was missing.
| organization | capped | cap-free |
|---|---|---|
| first | 2,395 | 7,239 |
| second | 1,504 | 2,104 |
| fourth | 1,503 | 2,403 |
the fixes went in over a few days. attestation first, so every chunk that goes in is counted and the system can say how much of a file it read. then a gate, so an incomplete ingest gets its own label and withholds the verdict instead of guessing. that one mattered beyond spreadsheets: a judge timeout had been coming back as INSUFFICIENT_EVIDENCE, and a planner parse failure as PARTIAL, and both read exactly like real findings. then runs with the caps lifted.
those cost more than i expected. attesting the smallest organization took 235k input tokens and 233 seconds. the first organization's cap-free run hit credit balance too low 64 times and finished 5,960 of 7,239 chunks. the one fully attested cap-free run, 981 of 981 chunks, was overwritten by a second run that attested 0, and now survives only in my notes.
the case: did the auditors notice?
we had real auditor ratings on these findings from the survey, so i checked whether they pushed back more on the truncated ones.
| findings | concerns | rate |
|---|---|---|
| built on truncated files | 5 of 25 | 20.0% |
| built on complete files | 25 of 138 | 18.1% |
some did notice. 2 of the 7 auditors who rated one truncated finding named the omission exactly, one of them writing that they could only see the hardware assets. but the aggregate barely moved, so across careful, experienced reviewers, a finding built on a third of the file mostly passed. it's a small sample, and it measures perceived quality, not ground truth.
hamel husain tells people starting on evals to examine as much data as possible and read the traces from every test case. i agree, and this is the case for reading one level lower. every trace here looked fine. the problem was in what the loader let into the trace.
so the system has to say it, every time, in the finding itself.
if you hit this
for every file you ingest, log the rows and columns in the file next to the rows and columns you loaded. if they differ, say so in the output a person reads, not only in the log.
what this changed in rema
a truncated ingest is recorded as truncated. rema's spreadsheet extractor reads the whole workbook, coverage is never called complete if any file was cut, and a finding says so when no population-wide conclusion is available.