- case 03
i capped my own evidence at 500 rows. most auditors didn't notice.
a loader cap i had added in march kept 500 of 1,500 rows and 50 of 65 columns, and said nothing. 6 of 48 evaluation cells ran on truncated files, and the auditors' concern rate barely moved.
· 7 min read - case 02
we blamed the api key for 42 hours. it was 768 tokens.
the live demo served a months-old cached answer for about 42 hours. three different causes produced the same response, and the one that mattered only showed up in a narrow log window.
· 5 min read - case 01
my benchmark was grading its own homework
i grew the benchmark from 26 cases to 77 in a day. then found that the gold labels for 51 of them had been picked by the retriever the benchmark grades.
· 6 min read