· Diar Shakimov · 6 min

21 of 21 tests passed. the uninstall deleted ~/Documents.

the installer passed its own 21 tests. then a separate tester started from an empty home folder and found that uninstall deleted whatever folder one variable pointed at. the tests had set that variable to the one place the bug couldn't show up.

case 03

on september 30 rema's installer passed its own 21 automated tests. i went back and reran them on the same commit and got 21 of 21 again.

i didn't want to take my own tests' word for it. so before anyone outside could install it, i set up a separate QA pass. it was another agent session with its own package registry and its own scripts. every install ran in a brand new empty home folder under env -i, and the interactive runs got a real terminal. its brief had one rule. the builder's test scripts don't count as evidence.

twelve passes

none of the twelve passes came back clean.

passversionverdictwhat decided it
10.2.0faila read-only settings folder crashed with a raw stack trace, and a private 0600 settings file came back 0644
20.2.1failuninstall deleted whatever folder REMA_HOME named
30.2.2faila store's trust check could be bypassed by reordering its columns
40.2.3faila cloned repo could link .rema to a sibling folder, and indexing wrote into it
50.2.4-rc2failno public install path worked yet, plus two medium and five low defects
6 to 110.2.4fail5, 5, 7, 20, 26 and 79 defects, by pass 10 mostly redaction and sign-in
120.2.4 plus fixesfailthe 11 safety items from pass 11: the exact repro fixed for 8, closed for 0
Table 1. every independent pass so far. each verdict names the exact tarball it applies to, by its sha256. later passes covered more surface (windows, receipts, scanning), so the defect counts are not a trend.

the second one still bugs me.

the uninstall

pass 2 found it with one line in a fresh home folder: mkdir ~/Documents; REMA_HOME=~/Documents npx -y askrema@latest uninstall. rema had never been installed there. it deleted ~/Documents anyway and printed Rema is removed. with REMA_HOME=$HOME it deleted the whole home folder.

the bug was one branch near the end of uninstall. it ran if (fs.existsSync(REMA_HOME)) and then fs.rmSync(REMA_HOME, { recursive: true }). it never checked whether rema made that folder, and there was no receipt to check against. if the folder existed, it got deleted.

why the tests never caught it

both test files start with process.env.REMA_HOME = fs.mkdtempSync(...). every run gets a fresh temp folder, so the tests never touch a real install. that part is a good habit. the problem is that every time uninstall ran in a test, the folder it deleted was a throwaway the test had just made. my test setup was the one environment where this bug couldn't happen.

Program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence.

dijkstra wrote that in 1972, in the humble programmer. i keep relearning a smaller version of it. a green run means the tests passed and that's all it means. the cases nobody wrote down never ran, and for an installer that edits someone's home folder those are the ones that hurt people. pass 3 confirmed the fix from a fresh home folder the same day.

what i do with failures now

pass 12 changed how i count. the fix round before it repaired the exact repro for 8 of the 11 safety items from pass 11. the QA rule is that an item only closes when its whole family closes. if the fix misses a sibling spelling, file type or launcher line, the item stays open. by that rule 0 of 11 closed. the redaction fix is a good example. it caught the key shape from the report, but 2 of the 26 key shapes the docs said we caught still leaked.

so i write failures down as their own entries now. each one has the situation where you'd hit it, what it rules out and the evidence. the next session that lands in that situation finds the wall before it pays for it again. a fix only tells me one input works. the written-down failure tells me which family to test next. google's sre book calls a postmortem a learning opportunity for the entire company, and a dead end is basically the same thing one step earlier, before anything reached a user.

if you hit this

find every place your tests point a path or an environment variable at a temp folder. then ask what the code does when a real user points it at something real, like their home folder. anything that deletes should check that it created the thing before it deletes it.

what this changed in rema

a green test run doesn't mean the release is done. every rema release candidate so far has gone through an independent pass from an empty home folder, and the verdict names the exact tarball it applies to. when something fails it gets filed with its whole family, and the next fix gets checked against the family and the repro both.

Sources

Revised .