It’s Friday, 12 January 2024 in Portland, Oregon. Tomorrow’s weather ranges from a low of -9 Celsius to -5 Celsius, with which I disagree.
\nI wrote about horrific UK Post Office Horizon scandal back in s09e21, nearly 3 years ago[1].
\nThe Horizon scandal is back in the news because ITV, a UK terrestrial channel, recently aired a dramatization of the shitty turn of events in Mr Bates vs the Post Office[2]. BBC News has a good summary[3], and I’d be remiss if I didn’t mention the dogged reporting by Nick Wallis, a freelance journalist[4].
\nA lot of the gnashing and wailing of teeth in the UK right now includes indignation at the repeated procurement of IT systems from a small number of suppliers, in this case, Fujitsu. If you’ve been reading this for a while, you’ll know why this is the case; in any event, The Register has a good summary[5].
\nI am going to say this: yes, the software was terrible. And the software was developed and managed at the direction of people. Managers and leadership at the Post Office made horrendous, fatal decisions with, so far, negligible consequences. I’ll excerpt what I said last time[1:1]:
\n\n\nThe sister of the former post office worker who died by suicide said: “a bloody faulty computer system killed my brother”. [The Evening Standard, April 12 2021]
\nMy heart goes out to Jayne Caveen. And I hate to do this: a computer system did not kill her brother. Horrible people in management killed her brother, and it’s easiest to blame it on a computer system.
\nPeople, people in management, people in positions of trust people running one of the most trusted institutions in England made those decisions to double down and to persecute and prosecute people knowing that the evidence wasn’t reliable and concealing that evidence wasn’t reliable. Intentionally not investigating reports for fear of what might be found, because it might affect public perception of trust, or because it might be discoverable and admissable in court. Cowardice, fear and a lack of integrity is what happened. Pride and boastfulness in a system that could never do what it could. Using technology was and is an excuse. Not taking responsibility is what happened.
\nTechnology is for people and made by people and this is what happens when the people running it don’t realize that. [1:2]
\n
Aside from the behavior of the Post Office, the legal system of England and Wales behaved in an institutionally egregious way, one that must change. Much of this is based on a great post from Bentham’s Gaze, The legal rule that computers are presumed to be operating correctly – unforeseen and unjust consequences[6].
\n\n\nOnce computers began to be used in everyday life, it was necessary to consider how evidence in electronic form was to be presented in legal proceedings. A document produced by a computer is ‘hearsay’ evidence, the kind of evidence that the courts treat with caution because a person relying upon it has no direct personal knowledge. While such evidence was admissible, courts needed to decide how reliable it was and what weight could be placed upon it.
\nA solution was provided by section 69 of the Police and Criminal Evidence Act 1984 (PACE 1984), that required the prosecution to prove that a computer was operating properly at the relevant time before a document produced by such a computer could be admitted as evidence. As the volumes of computer evidence increased, this requirement became burdensome and inconvenient.[6:1]
\n
Burdensome and inconvenient.
\nSo in 1997, the Law Commission proposed that, essentially “courts will presume that [computers] were in order at the material time]. In 1999, the Law Commission’s recommendation was put into law.
\nIt’s hard not to get angry at the Law Commission about its recommendation. Again, Bentham’s Gaze:
\n\n\na presumption that a computer ‘works correctly’ will appear wholly unrealistic for anyone with expertise in computer science or software engineering. That is because s69 demands a ‘yes or no’ – that is, a binary – answer to the question of whether a computer is working correctly or not and assumes that the answer is trivially easy to provide.[6:2]
\n
It betrays -- in 1997, for goddamn’s sake -- a curious belief on the part of the Law Commission (disclosure: I haven’t read the recommendation in full, but believe me, I was a baby fresher law student in 1998 and precocious me certainly had opinions about how law and policy understood technology) that somehow something like a bureaucracy implemented in software code is presumed to be working as designed when clearly human-operated bureaucracies aren’t. The description of mechanical devices is a dead giveaway. And I’m not even getting into complexity.
\nIn 2004, the BBC sketch comedy Little Britain introduced a sketch with the catchphrase “Computer says no”[7]. By this point, criticism of bureaucracies implemented in software had become so embedded in popular culture that the phrase became incredibly popular and recognizable. Even more, that it reflected the restraint of human agency in following encoded bureaucracies. It doesn’t matter if it’s clearly wrong, the system says what it says.
\nWe understand, now, that computers can be wrong.
\nIn 2020, the ACLU filed a complaint against Detroit police “over what it says is the first known example of a wrongful arrest caused by faulty facial recognition technology.” (The Verge)[8]
\nIn 2017, a federal judge unsealed the source code “for a software program developed by New York City’s crime lab, exposing to public scrutiny a disputed technique for analyzing complex DNA evidence” (ProPublica)[9]
\nThese days, there’s more visible concern about bias in machine learning algorithms leading to misclassification (ugh, what a euphemism), but principle remains.
\nSo. Again, today, the U.K. government is being urged to update the law on this presumption: “Ministers need to “immediately” update the law to acknowledge that computers are fallible or risk a repeat of the Horizon scandal, legal experts say.” [10]
\nThe presumption is untenable. Nobody in their right mind would support such a blanket presumption of correctness these days. The evidence is widespread, incontrovertible and in your face all the time.
\nBut with this would come a stupendous reckoning. Get this:
\n\n\nJames Christie, a software consultant who co-authored the recommendations for an update to the law, said the changes could come in two stages. “The first would require providers of evidence to show the court that they have developed and managed their systems responsibly, and to disclose their record of known bugs.[10:1]
\n
I’ve written before, lots, about how many, and what kind of Therac-25-style deaths would need to happen before the approach to software and liability changes. Boeing’s MCAS is a close example of a combination of software and hardware -- the result of a business decision, made by people -- that was a factor in the deaths of 346 people, that led to a $2.5 billion agreement but also immunity for Boeing from criminal prosecution.
\nAnd remember the VW diesel emissions scandal, with the software-based defeat devices? The one that led to $33.3 billion worth of fines, penalties, financial settlements and buyback costs?[11]
\nAnd if you’re the kind of person who subscribes to belief that there’s no, uh, particulate emissions without fire [sic], then you wouldn’t be surprised to see that Cummins, who make engines, just agreed to pay a $1.67 billion fine to settle claims of bypassing emissions tests, too[12].
\nSoftware ate the world. Some people make it lie. Some software has genuine unintended mistakes. Some people lie about that. And any presumption that puts the burden of proof unduly on those accused is a shitty, shitty law that reflects a misunderstanding of how our world works now.
\nLook, the deal is that if it’s really, really cold, then we should get snow. It just being really, really cold is not on. If it’s really, really cold and it rains, then that’s just not on either.
\nHope you have a good weekend,
\nBest,
\nDan
\nThings That Caught My Attention is a free newsletter, and if you like it and find it useful, please [consider becoming a paid supporter]({{ upgrade_url }}).
\nDo you have an expense account or a training/research materials budget? Let your boss pay, at $25/month, or $270/year, $35/month, or $380/year, or $50/month, or $500/year.
\nPaid supporters get a free copy of Things That Caught My Attention, Volume 1, collecting the best essays from the first 50 episodes, and free subscribers get a 20% discount.
\ns09e21: Seeing Past The Horizon (archive.is), me, this newsletter, 27 April, 2021 ↩︎ ↩︎ ↩︎
\nPost Office scandal explained: What the Horizon saga is all about (archive.is) ↩︎
\nHow governments become addicted to suppliers like Fujitsu • The Register (archive.is), Lindsay Clark, The Register, 11 January, 2024 ↩︎
\nThe legal rule that computers are presumed to be operating correctly – unforeseen and unjust consequences – Bentham’s Gaze (archive.is), Information Security Research & Education, University College London (UCL), 30 June, 2022 ↩︎ ↩︎ ↩︎
\nA black man was wrongfully arrested because of facial recognition - The Verge (archive.is), Jon Porter, The Verge, 24 June, 2020 ↩︎
\nFederal Judge Unseals New York Crime Lab’s Software for Analyzing DNA Evidence — ProPublica (archive.is) ↩︎
\nUpdate law on computer evidence to avoid Horizon repeat, ministers urged | Post Office Horizon scandal | The Guardian (archive.is), Alex Hern, The Guardian, 12 January, 2024 ↩︎ ↩︎
\nEngine maker Cummins agrees to pay $1.67 billion to settle claims it bypassed emissions tests | AP News (archive.is) ↩︎
\nPentiment | Lettermatic (archive.is), Riley Cran ↩︎
\n