On the BBC now - Gareth Jenkins is being crucified by the barrister representing the SPMs. It's excruciating - if it wasn't for the suffering inflicted on 100s of SPMs that his woeful ignorance, buck passing and perjury made a significant contribution towards you might even feel sorry for him.
Do you have a time stamp please so I can go straight to the good bits online rather than watch the whole day.
I just dipped my toe in 30 mins ago and was straightaway transfixed. So, 11:40 would be a good start - but I suspect anytime will do equally well.
If really is exquisitely painful.
> I just dipped my toe in 30 mins ago and was straightaway transfixed. So, 11:40 would be a good start - but I suspect anytime will do equally well.
> If really is exquisitely painful.
Thanks. I'll go back to 11am then.
Ouch I had been glancing on and off for the last couple of days but it is really gloves off now. Am I missing something or is he still trying to defend the system.
I am amazed Seema Misra is able to sit there so calmly.
I was once asked to be an expert witness for a climbing wall accident claim. Probably, fortunately, it never got to court.
Despite having never been involved before, it was pretty obvious to me that being an expert witness would involve telling the truth, the whole truth and nothing but the truth. Any opinions expressed would be subject to analysis and scrutiny. Sadly, this does not appear to have happened in the Post Office saga.
Edit, I was also gobsmacked how much I was expected to invoice for my time and expenses.
A bit of a gravy train, not that that would influence my judgement. Pah.
Did you miss the bit where the Post Office investigator tried to weasel out of admitting that he'd told Jenkins to change his witness statement which was going to be used as part of a Post Office prosecution case against a sub-postmaster?
https://www.postofficescandal.uk/post/graham-ward-a-man-in-trouble/
Or the bit where the Post Office's General Counsel issued a veiled threat to bankrupt Second Sight if they wrote anything in their independent report that the Post Office didn't like:
https://www.postofficescandal.uk/post/second-sights-ron-warmington-and-ian-...
Or the bit where one of the lawyers acting for some the sub-postmasters described the Post Office's actions as "fraud" rather than [just] "the biggest miscarriage of justice in history" as it is usually described (unfortunately I can't track down the link for that - I think I read it on the BBC news site).
It is interesting that they inflict that sort of questioning on those lower down the chain . But when faced with tougher skins and those higher up I do think the aggressive questioning from the solicitors acting for the SPM does ease off and is softer. I have noticed the reverence to others in the legal profession who are quizzed and are more polished performers.
it’s easy to pick on somebody like Jenkins but not the real culprits in the cover up. Only 2 days for PV for example compared with 3 for this guy .
Both deserve to be crucified, PV for corporate responsibility, and Jenkins for technical responsibility, around the massive ethical failings/ outright fraud which the Post Office engaged in.
Society needs to be able to trust engineers to act in an ethical and competent fashion. Jenkins has failed in one, or both, of those areas.
That really doesn't make for pretty reading - it's all just repeated doubling-down, again & again & again - absolutely phenomenal.
There are some people out there who really should be prosecuted.
Jenkins was no minion. He had a title - 'respected engineer' or some such. How much money do you suppose he was trousering - £100 k perhaps? More?
He was the sort of sh*t software engineer that we're still lumbered with. There's no frigging excuse for cashing up and bank reconciliation systems to not be auditable - none. Listening to Jenkins wittering on about 'timing issues' - every cr*p engineers fallback excuse - took me straight back to the 70s, when most of us didn't know any better. Which coincidentally is when Jenkins left university, entered the industry and turned his brain off. People died, people went to prison, because of his incompetence.
> Jenkins was no minion. He had a title - 'respected engineer' or some such.
distinguished engineer. He was someone who was allegedly good enough that he got offered the nonmanagerial path in a software firm. So yeah I doubt he was doing badly for cash.
> He was the sort of sh*t software engineer that we're still lumbered with.
Yup. He seemed to be one of those who takes the line "well thats what the requirements say" as an excuse for not doing a proper job and asking do they make sense. For me thats when someone switches from being a junior to being someone who can be trusted to work alone. Dont follow the requirements or my design notes blindly. If they look stupid explain why.
I am still somewhat confused as to what exactly his role was. He seems to have had bugger all knowledge of the system he was supposedly in charge of and was happy to go to court to defend.
That he still defends it is also baffling.
> Jenkins was no minion. He had a title - 'respected engineer' or some such. How much money do you suppose he was trousering - £100 k perhaps? More?
> He was the sort of sh*t software engineer that we're still lumbered with. There's no frigging excuse for cashing up and bank reconciliation systems to not be auditable - none. Listening to Jenkins wittering on about 'timing issues' - every cr*p engineers fallback excuse - took me straight back to the 70s, when most of us didn't know any better. Which coincidentally is when Jenkins left university, entered the industry and turned his brain off. People died, people went to prison, because of his incompetence.
Well said
> I was once asked to be an expert witness for a climbing wall accident claim. Probably, fortunately, it never got to court.
> Despite having never been involved before, it was pretty obvious to me that being an expert witness would involve telling the truth, the whole truth and nothing but the truth. Any opinions expressed would be subject to analysis and scrutiny. Sadly, this does not appear to have happened in the Post Office saga.
> Edit, I was also gobsmacked how much I was expected to invoice for my time and expenses.
> A bit of a gravy train, not that that would influence my judgement. Pah.
Can anyone explain why I have only got dislikes for this post?
Thanks
> There's no frigging excuse for cashing up and bank reconciliation systems to not be auditable - none.
I still find this staggering.
He was hardly an independent expert witness though…
the guys from second sight summed the whole thing up very easily. It’s was an issue of materiality in accounting.
> I still find this staggering.
Horizon is/was a financial system. Why would you possibly want it to be auditable? 😁
Have a look at Xero, popular, fairly easy to use, relatively recent accounting system. Then tell me where the "obvious" audit trail is.
"Timing issues" might be a real problem, but that's a bug to be fixed, not just shoved under the carpet and then 'conveniently' forgotten about and then denied.
'Audit trails' per se were a necessity for legacy systems where for reasons of performance (and not having any better ideas) reports and so on were based on 'total' figures accumulated within the system: 'gross sales to date', 'VAT Output tax' , '# of widgets in stock' would be held as a single figure added to and decremented programmatically. That then begged the question 'where do those figures come from', which you needed an audit trail to answer. Unfortunately that approach is intrinsically corrupting - you are storing the same data items (e.g. 'sales ytd') twice - once as a total, once as sum total of the transactions stored in the audit trail. That can - and will - go wrong.
That all changed with the development of decent databases, (or if it didn't, that's because a lot of developers and system designers never read up on the theory, or applied it in practice. I suspect the Horizon botchers were among their number.)
In the case of Xero, if you need to look at, say, a sales figure (with consequent VAT liability etc), you can drill down into ALL the individual transactions that make up that figure; if you want to do a bank rec, you can compare all the transactions that make up the closing balance with the figure on the bank statement. And if there is a discrepancy you can drill down into each of the transactions. ('Ah look, xxx has put the decimal place in the wrong place on the transaction entered on dd/mm/yy hh:mm.')
The fact that the Horizon 'engineers' were adjusting total figures while the transactions that should have supported that were untouched sounds like a botched design and fundamental rookie errors. But one, sadly, that is reproduced on other systems daily that have similar design flaws. I could name names, but I won't.
This was covered by second sight in their submissions etc to the Enquiy. They covered this very neatly in their verbal statements the other week. Most systems have a level of materiality and allow for errors etc. The big issue was that the SPM picked this risk up and the Po needed to give the SPM about 60 days to resolve accord to second sight. But the PO did not do that and prosecuted straight away because of the contract.
I know nothing about accounting software, or any software, but if a bug was moving decimal points or doubling amounts wouldn’t it be 50/50 that it would work the other way and credit the spm with these huge amounts?
That's the point. A process might debit the data item called 'cash received' with £x, but because of a bug, 'timing issue' or whatever, fail to update the corresponding data item called 'liability' with the same amount.
It's pretty clear this is what was happening here.
> I know nothing about accounting software, or any software, but if a bug was moving decimal points or doubling amounts wouldn’t it be 50/50 that it would work the other way and credit the spm with these huge amounts?
The focus on accounting systems does seem somewhat faulty to me given the key issue was in the synchronisation of local vs central systems. That said a properly written system would register a deduction from the central system vs an addition to the locl.
Most of the problems (there were multiple) seem to have been around the system telling the user the payment had failed whilst not properly backed out on the central system.
Hence they then retry the payment and so end up with the central system saying "today your PO recorded 2 payments of 100" and you only having taken a £100.
They were throwing stuff into event log but not giving the user enough info.
If you have an error then it should be put into the actual system the user sees even if its "no idea whats happened here. good luck" and then stick the more detailed info into the event log for someone with the right privileges to review. Its normal to hide details about the failure from the end user and use the event log (or similar) for it but you should tell them it went bang. Since if you have that then its rather useful in a court case if you are accused of nicking a 100k.
Your reply clears the air somewhat but I’m still baffled that the discrepancies seem to be very easy to check.
if I sell 100 stamps but the system says 200 then I would still have 100 left. Likewise monies being deposited and credited would be easy to backcheck via the depositors.
Perhaps this is too logical or hard work for the Post Office.
> if I sell 100 stamps but the system says 200 then I would still have 100 left. Likewise monies being deposited and credited would be easy to backcheck via the depositors.
The central computer was right.
End of discussion.
Sadly that was pretty much their position. They ignored any local ledgers etc but just announced the computer said x and so you owed that
> The central computer was right.
> End of discussion.
> Sadly that was pretty much their position. They ignored any local ledgers etc but just announced the computer said x and so you owed that
What I cannot get my head round is why nobody in court asked
1. How can a system be perfect and without possible fault.
2. If any checking mechanisms were built into the system. Basic accounting,engineering, surveying , mathematical and probably lots of other field's principals to try to ensure a correct result or answer.
... or why the PO suddenly discovered that literally hundreds of its' SPMs that it had carefully recruited were in fact criminals.
> ... or why the PO suddenly discovered that literally hundreds of its' SPMs that it had carefully recruited were in fact criminals.
And suddenly stopped being criminals when there was a software update later (fixing the problem that did not exist).
> What I cannot get my head round is why nobody in court asked
> 1. How can a system be perfect and without possible fault.
There's something to do with a legal presumption that computer systems are considered to be correct unless shown otherwise; this came in I think not long after the millennium.
Hopefully this debacle will consign that presumption to the bin where it belongs.
Any computer system that is non-trivial in size will have bugs when it is initially released; not a chance of producing a totally bug free system. The bugs might be fixable and they might not be critical (unlike the Horizon ones) but they'll be there.
> There's something to do with a legal presumption that computer systems are considered to be correct unless shown otherwise; this came in I think not long after the millennium.
I think that presumption may have gone into law in response to international (or PO?) pressure about Horizon.
https://www.computerweekly.com/feature/Fujitsu-put-pressure-on-UK-governmen...
Nope, quick google, preceded Horizon scandal, 1999, see...
https://evidencecritical.systems/2022/06/30/briefing-presumption-that-compu....
From a very quick skim, if I've understood correctly came about because of problems enforcing the PACE act without that presumption. As an ex software engineer I think that presumption is absolutely ridiculous.
There are accredited processes for high integrity software development, but even they don't necessarily prevent the presence of bugs, but they do provide an information trail if something goes wrong and make it easier to review the code and any design decisions.
I want to say it is related to Safety Integrity Levels but I think it's adjacent to them:
https://en.m.wikipedia.org/wiki/Safety_integrity_level
> ... or why the PO suddenly discovered that literally hundreds of its' SPMs that it had carefully recruited were in fact criminals.
Nobody in court asked this because each SPM was told "You're the only one experiencing these issues with Horizon"!
With any accounting package you always get errors ( they can be bugs or input errors for example). It is whether they are material or not that is the big issue. My accountants allow about a 1.5% error based on our turnover. Otherwise they have to reconcile every single item and that is not really practical or cost effective.
That is fine for my business that error rate is not really material to the numbers. it’s the same for 99% of businesses
But for the SPM it was very material , because under the contract they were liable. Nobody at the PO understood this “gap”. Just about the only people who really grasped it were Sir Alan and then Second Sight.
Second Sight said it was the worst example of corporate bullying they had ever seen(considering their cv’s and background that is a pretty far reaching statement) . Most companies they work with are open and want to know what is going wrong. The PO did not.
Things like this did get asked in court. Some of the falsely accused had good records and good lawyers and their prosecution failed. Fundamentally any focus on the software is a sideshow (outside of the ridiculous legal trust placed on accuracy in court), when the PO corporate response was clearly to perjure and to pervert the course of justice.
> 1. How can a system be perfect and without possible fault.
It did occasionally happen. In one of the early cases they got in a proper expert witness who did read the paperwork about what being one meant and so gave evidence that he didnt think much of the system and could readily believe it was a system error.
After that though they used the Fujitsu expert witnesses who announced it had all been checked and was ultra safe. I suspect that would convince most people although admittedly since I am a developer I would immediately distrust anything they said afterwards.
> 2. If any checking mechanisms were built into the system.
It does depend on where the problem occurs but yes this seems drastially lacking. It does seem they have a message store based system but didnt have any error checking on it. So they could have a phone line issue and lose a bunch of messages which then get resent.
>Nobody at the PO understood this “gap”. Just about the only people who really grasped it were Sir Alan and then Second Sight.
I really don't see how you can watch this sorry tale and think that. Enough in the PO and Fujitsu clearly understood there were real problems but they simply didn't give a shit and covered it up, and ruined peoples lives in the process; they arrogantly felt confident their actions would never be uncovered.
> I really don't see how you can watch this sorry tale and think that. Enough in the PO and Fujitsu clearly understood there were real problems but they simply didn't give a shit and covered it up
I guess from 2000-2010ish there was a bit of an excuse although even then you would need to ignoring basic job requirements and not asking questions like is our HR team confused about the vetting process and selecting only people who fail?
From 2010 onwards though it was falling rapidly into full cover up as the computer weekly and subsequently PE really started and the MPs got involved asking asking awkward questions.
I prefer to listen to Second Sight who are far more knowledgeable about what went wrong.
As far as I remember it came about earlier than that as part of a Law Commission Review. A systemic failure which I think must have resulted from the Law Commission not having any or sufficient embedded computing industry expertise to employ on assessing the quality of evidence it considered in reaching the conclusion that this principle should be embodied in a presumption. A critical lack resulting in a critical failure by an otherwise useful instrument for non-politically sponsored Law Reform.
As a slight aside, I don't think anyone's commented on the thread title (unless I've missed it).
Should be inquiry not enquiry.
Trivial I know, just me being pedantic 😁
What are you on about? Second Sight (who indeed did their job properly) we're sacked: so how does that possibly explain the much more serious behaviour of the PO or Fujitsu since that work.
https://www.postofficetrial.com/2019/12/second-sights-ron-warmington-breaks...
> A systemic failure which I think must have resulted from the Law Commission not having any or sufficient embedded computing industry expertise to employ on assessing the quality of evidence it considered in reaching the conclusion that this principle should be embodied in a presumption
Did they ask for advice from relevant industry or academia...?
'Formal methods' was a research thing in the late 80s, hoping to create 'provably correct' software, but most software practitioners considered that ... shall we say ... optimistic, and proved correct in the end (though there has been a bit of a resurgence in the idea more recently).
I doubt any practitioner in that entire period would claim their software was 100% free of bugs; it's why reporting systems exist.
My rough understanding was that the presumption of correctness was to prevent e.g. lawyers trying to get someone off on a speeding fine by demanding access to the full proprietary firmware of a radar gun, and things like that. But the concept of putting a stop to that sort of malicious tactic then seems to have been stretched way beyond what is appropriate and fair.
Yes, I can see that tactic being exploited. Access to bug report logs etc. might have been a reasonable compromise. And encourage the rollout of bug fixes. Or maybe discourage the logging of bugs... Unintended consequences and all that.
I'm going through something similar with a client at the moment, nothing on this scale obviously, and the risk is all on them and there's no-one similar to an SPM involved however....
Risk and Issues log, client doesn't want anything on the Risk log that hasn't been confirmed as a Risk, my approach is to have a backlog of things that have the potential to become a risk and mitigate accordingly. They don't sit on the log as Risks as such but are recorded in ADO/JIRA and discussed in the R&I call. Client doesn't want these surfaced in the call or on ADO/JIRA as the R&I log and backlog is seen by the Governance board and questions will be asked (and this is what the GB is there for, to ask questions and smooth the way). Thusly, we look great - amberish/green for RAG and yet have an undercurrent of crap that we need to be mindful of and which can bite us on the bum at a moments notice.
Clients and their PM's a just f*&kers at times!
> My rough understanding was that the presumption of correctness was to prevent e.g. lawyers trying to get someone off on a speeding fine by demanding access to the full proprietary firmware of a radar gun, and things like that.
Interesting that you should bring up that example. AIUI it is possible to contest a speeding fine if the radar gun or whatever other device hasn't been tested according to the schedule & process stipulated in the certification issued by whichever Government depart or agency is responsible for that task (a quick Google suggests that this falls to the Vehicle Certification Agency, at least in England: https://www.vehicle-certification-agency.gov.uk/other-certification/civil-t... Not a body I recall having heard of previously. I thought it was the DVSA that handled vehicle certification. Just goes to show how much I know.)
That would appear to follow the law as set out in PACE (1984) Section 69 - but that is supposed to have been repealed by the Youth Justice and Criminal Evidence Act 1999. Just because a device hasn't been tested doesn't mean that it's not working correctly, and the repeal of PACE(1989) Section 69, which the Law Commission said would have the effect that "In the absence of evidence to the contrary, the courts will presume that mechanical instruments were in order at the material time" would seem to put the onus back on the defence to prove that the speed gun was actually defective.
What is beyond doubt is that the presumption in law that a computer system should be regarded as working correctly unless there is evidence to the contrary is fundamentally flawed. It appears to have been based on the idea that a computer system is a "mechanical device", which is simplistic to the point of being useless. There's an old adage in computing: "Hardware goes wrong. Software is wrong." It seems clear that the Law Commission, in this instance, was including computers "by default" in its proposal for the repeal of PACE(1988) Section 69 out of pure ignorance.
https://www.benthamsgaze.org/2022/06/30/the-legal-rule-that-computers-are-p...
Can't help wondering if we have the same client...
Or more likely it's just a depressingly common situation.
I find myself citing the McNamara fallacy all too often. I've completely given up trying to use Jira, anything that upsets the metrics instantly draws ire and it's now just a tool for the client to maintain their delusions. I think a lot of time and effort could be saved by just drawing a burn down graph in MS Paint instead of having legions of "PMs" making sure the lines look suitably pleasing.
The software has safety implications and some of the outcomes of the PO inquiry have given me pause for thought about how my own actions might appear if anything went wrong down the line.
Without giving anything away that's not already known - we were pushed out of the PO Horizon thing in 2015 and I like to think that the diligence we apply (at staff level anyway) means that this would not have happened under our watch. As a Principal Prod Man I'm very much minded that every product I define and deliver is riddled with issues until it's not, and launch date is merely the end of one sprint and the start of continuous improvement with the next sprint.
A thing that does confuse me about the Horizon platform appears to be little or no continuity/smoke testing (I love a smoke test). The newly released code/feature is tested alongside a suite of common test cases to prove you didn't screw anything up with the latest release, also allows for an easy roll-back to a known point of stability. Working this way gives a linear view of progress and stability and allows the same to be applied to branches (not the PO Branch).
The Smoke Test also tends to be very usability focused too in that it can be run by the test team/product team, and friendly users to give multiple views on how stable something is as opposed to a point view - I don't trust the test team until I have tested and I don't trust my results until the user has tested. My personal view is that with Horizon it could have been a fundamental flaw in the delivery process, the test team says it works in the test environment therefore it works. What happens in the live environment is user error - or not as the case may be. A good test team obviously runs the test cases designed for the feature being released and then some, a brilliant test team spends as much time as possible trying to break the release in ways that the test cases are not designed to do.
The po did not understand the materiality issue…or are you another person who does not get what second sight were saying at the enquiry. Complete corporate failure. Anyway I am sure the police will be recommending prosecutions to the cps.
Regardless of how much effort you put in to testing, as I'm sure you know, the real test happens when actual users are let lose on the system. They tend to do things in ways that nobody's anticipated.
This fundamental reality is what led to the concept of beta testing.
It sounds like the Horizon development was in a different reality.
> Without giving anything away that's not already known - we were pushed out of the PO Horizon thing in 2015
2015 was the last year that the PO took someone to court. At that point even they had admitted the system was flawed. After that date it was pure arse covering.
> A thing that does confuse me about the Horizon platform appears to be little or no continuity/smoke testing (I love a smoke test).
You seem to be using a nonstandard definition of smoke testing. Normally a smoke test is a quick play to see if its actually worth bothering to test properly. A regression test is running the set of standard use cases.
Neither would have been much use in this case though since it seems to have been a mix of:
Unexpected user actions: A good team should pick this up by testing beyond the requirements but then you need the devs to take it seriously and then see if it can be blocked. One example was if the system froze and someone pressed enter again it would just rerun the transaction vs putting a lock on the front end.
Network/transmission issues: Unless you have an abnormally skilled test team unlikely to be identified. You only really get that when you go for chaos monkey testing.
Aside from that its what keeping reports of prod issues and log analysis are for. A clear problem for the PO/Horizon was just fixing stuff quietly rather than keeping a proper log and cross referencing to try and see any interesting patterns for investigation. Jenkins came out over and over about not knowing basically anything about the system he was responsible for.
> A regression test is running the set of standard use cases.
Yeah; sounded like regression testing to me, too.
> Unexpected user actions:
Naive users are very good for this; the same reason you don't get the dev team to write the user manual; they know too much about how the system works, and leave out the 'obvious' steps...
> Jenkins came out over and over about not knowing basically anything about the system he was responsible for.
Ironically, Jenkins is a tool commonly used for automating builds and regression testing...
As an aside - developers simply cannot test their own software. They (and I include myself in this) are literally, physically incapable of submitting that keystroke that they suspect will break the system
Genuine users, on the other hand, can usually manage that within their first five seconds of interaction.
In my experience users are not naive. They are simply working to user objectives and with a mindest different from the development and system test teams. They do things differently not because they are naive but because they have a different body of experience and knowledge, including knowledge that those who designed, created and implemented the system just had not picked up.
They are often aware of "dimensions" of use that developers are not. At least the level of users I had to deal with with decades of experience of using computer systems and satisfying clients and customers of their side of the business despite IT limitations.
> In my experience users are not naive.
I was using 'naive' in the sense of 'not being expert, unfamiliar'. In the same way as biologists use 'naive species'.
So, in terms of user familiarity with your software, it goes 'naive users', 'familiar users', 'expert users'.
(interestingly, this forms a useful example of the enshitification of Google search: it is so arrogant that it doesn't even offer you the option to search for the thing you asked for; it just replaces your search with 'native species'. You have to doubly force it to give you the results you're actually looking for ("naive species" -native).
> As an aside - developers simply cannot test their own software. They (and I include myself in this) are literally, physically incapable of submitting that keystroke that they suspect will break the system
Exactly; they know how the system is supposed to work, and the responses they are supposed to make. Naive users have no such foreknowledge, and will do all sorts of things the developers will never have dreamed of as they attempt to figure out how to make the system do what they want (as they learn how the system 'thinks').
> I'm going through something similar with a client at the moment, nothing on this scale obviously, and the risk is all on them and there's no-one similar to an SPM involved however....
> Risk and Issues log, client doesn't want anything on the Risk log that hasn't been confirmed as a Risk, my approach is to have a backlog of things that have the potential to become a risk and mitigate accordingly. They don't sit on the log as Risks as such but are recorded in ADO/JIRA and discussed in the R&I call. Client doesn't want these surfaced in the call or on ADO/JIRA as the R&I log and backlog is seen by the Governance board and questions will be asked (and this is what the GB is there for, to ask questions and smooth the way). Thusly, we look great - amberish/green for RAG and yet have an undercurrent of crap that we need to be mindful of and which can bite us on the bum at a moments notice.
> Clients and their PM's a just f*&kers at times!
I sympathise, and this is a distillation of the PO scandal in one reply, which can be transferred to the blood scandal, Grenfell, phone hacking any many others. I've been involved in Software testing in the past. Testing is a difficult and time consuming process, often outsourced, often manual, often overlooked, often ignored. Scripts badly written, and insufficient automation - but all this ignores the real issue.
In this whole PO episode, it is easy for onlookers to blame the 'software'. The software is not and never was the issue; it may have been badly written with more holes than a fine alpine milk product but it's the culture of blame, cover up, ignorance, arse covering that is the issue. Nobody can blame Horizon, Horizon wasnt the issue, it's what the people in positions of resposibility did with the knowledge and they didnt just make a mistake, they closed ranks, they were corrupt, cruel, driven by a weird cultish culture, and backed up by a legal system and bent lawyers which, when this is all finished, needs to be criticised as much as the culture at the Post Office.
> As an aside - developers simply cannot test their own software. They (and I include myself in this) are literally, physically incapable of submitting that keystroke that they suspect will break the system
I am currently working with a larger development house, not all their developers are quite so skilled. We had a problem recently where some code would go into an infinite loop, and eventually bring everything to it's knees. Took us about a month to find the common link, the developer who raised it as a blocker on. We then looked at the payload they sent an it included an emoji which their own code was unable to handle. When we had asked for a sample payload to test ourselves they had removed the character. The development company then tried to charge us for the environment being down!
> I sympathise, and this is a distillation of the PO scandal in one reply, which can be transferred to the blood scandal, Grenfell, phone hacking any many others. I've been involved in Software testing in the past. Testing is a difficult and time consuming process, often outsourced
We have recently outsourced our testing to a new specialist testing company. I have been having some great conversations along the lines of them demanding "dev should carry out a handover call and tell us what we should be testing and what every impact area is" and me sighing.
> In this whole PO episode, it is easy for onlookers to blame the 'software'. The software is not and never was the issue
It isnt and it isnt. It was appallingly bad and given a lot of the early info was in computer weekly and later the register plus PE its not surprising a lot of emphasis is on the system especially given it was Jenkins this week who gives us all a bad name.
> We have recently outsourced our testing to a new specialist testing company. I have been having some great conversations along the lines of them demanding "dev should carry out a handover call and tell us what we should be testing and what every impact area is" and me sighing.
> It isnt and it isnt. It was appallingly bad and given a lot of the early info was in computer weekly and later the register plus PE its not surprising a lot of emphasis is on the system especially given it was Jenkins this week who gives us all a bad name.
Oh I agree, the Software appears to have be very shit* but when this was known, the very second, it should have been raised and all court cases suspended and if anyone had already been prosecuted** then the convictions quashed until there was a very detailed analysis. We know the exact opposite happened.
*Fujitsu are still operating and bidding on large public contracts to this very day. Why are they not prevented from doing business in the UK, or at least the UK public sector, until they have paid their dues for the part they played.
**Im not entirely sure of the timelines
I am guilty of this, but not to the extent that I hid the error.
Internal product & dev team, built an app for a client in the office rental space, in the search for a location I added a couple of Easter Eggs. You could search for Paradise for example and it would take you to the island of Paradise in the Bahamas where the client had offices. I also added a Chuck Norris and you could input said actor in to the search field and be directed to a splash screen that said something like 'you don't find Chuck, Chuck finds you' Went live and all good, a while later someone tested in the live environment using the Easter Eggs, immediately the app/site were shut down by our hosting service as it assumed we were spamming/being spammed as Chuck Norris was at the time a well known and well used clickbait to redirect users to spam sites. Chuck and Norris were obviously key words on the naughty list.
I got a bollocking!
>The software is not and never was the issue; it may have been badly written with more holes than a fine alpine milk product but it's the culture of blame, cover up, ignorance, arse covering that is the issue. Nobody can blame Horizon, Horizon wasnt the issue, it's what the people in positions of resposibility did with the knowledge and they didnt just make a mistake, they closed ranks, they were corrupt, cruel, driven by a weird cultish culture, and backed up by a legal system and bent lawyers which, when this is all finished, needs to be criticised as much as the culture at the Post Office.
Exactly