United States: References Class Forecasting and Realistic Tech Contracts

Originally published in Society of Computers and the Law on March 2, 2017.

Michael Bywell suggests one route to overcoming cognitive delusion, deception and strategic manipulation in technology projects. He suggests that reference class forecasting may provide an answer and considers the legal implications.

* * * * *

In this article I revisit the pervasive problem of time and cost overruns in ICT projects – but from a different perspective.

I look at the work of leading scientists, economists and management consultants in trying to understand why these areas remain problematic and establish what can be done to ensure better outcomes. In particular, why is it that project planners routinely fail to produce realistic estimates of how long a project will take and how much it will cost even though they know, or should know, that similar projects have overrun in the past?

I look at reference class forecasting (RCF) and whether asking planners to compare their working estimates to similar past projects is likely to improve outcomes.

Finally, I consider the potential challenges associated with applying RCF in practice together with some of the related legal issues that could arise in the context of ICT projects.

I am a lawyer (who knows his limits) and not a scientist, economist or management consultant. Considerable reliance has therefore been placed on the published work of leading scholars such as Professor Bent Flyvbjerg of Oxford University's Saïd Business School and his colleague, Alexander Budzier. This description of RCF and how it works in practice is fairly rudimentary but I hope that it is true to the underlying theory and serves as a catalyst for discussion on whether RCF should be adopted by the ICT industry.

ICT projects – Overruns and Disputes

One does not need to look too far to find compelling evidence that time and cost overruns remain highly problematic in ICT projects.

Take, for example, the often cited study by Flyvbjerg and Budzier which is summarised in the September 2011 issue of the Harvard Business Review. Flyvbjerg and Budzier looked at 1,471 IT projects from around the world, comparing budgets and estimated performance benefits with actual costs and results. They found that, whilst the average cost overrun was 27%, one in six projects was a 'Black Swan' with a cost overrun of 200% and scheduling overrun (or delay) of 70%.

In a separate article on 'Megaprojects'¹Flyvbjerg gave several examples of high-profile projects with substantial cost overruns, including the NHS IT system where the overrun was a staggering 400-700%.

And ICT projects are not alone. Cost overruns for the Channel Tunnel and Sydney Opera House (going way back) were 80% and 1400% respectively.

Other work undertaken by Flyvbjerg (in relation to construction projects) found that, on average, a one-year delay correlates with an increase in percentage cost overruns of 4.64%. Thus a one-year delay on London's $26bn cross-rail project would result in cost overruns of $3.3m per day.²

In a more recent study, by the Queen Mary School of International Arbitration in London and Pinsent Masons,³ 49% of respondents said they had encountered 'IT systems development/implementation/integration' or 'outsourcing' disputes in the past five years. And 90% of respondents said that 'delay' was a common reason for 'IT systems disputes.'

Not all such projects are problematic and fail. Many do achieve technological success and make it across the finish line - but often at a significantly higher cost, later than originally planned and in a form different to that originally agreed.

Why is this and how do we explain the tendency of planners to routinely underestimate costs and over-estimate benefits when planning ICT projects? Why do projects get the go-ahead when it is (or at least should be) obvious that they are bound to overrun?

Cognitive Predisposition and the Inside View

According to a number of well-respected scholars (including Flyvberg) one explanation is that key participants involved in the planning of such projects are deluded (but genuinely mistaken) when they prepare their estimates. According to the experts, this deluded state of mind (or optimism bias) is caused by a cognitive predisposition found in most people to judge future events in a more positive light than is warranted by actual experience. This can give rise to a tendency to underestimate the time and cost to complete a project, even where people may know that similar projects have overrun in the past.

Indeed optimism bias pervades our everyday life and affects around 80% of the population. Thus 80% of us will likely predict deriving greater pleasure from a holiday than we subsequently do and although divorce rates in the Western World are nearing 50% most newlyweds estimate their own likelihood of divorce as negligible. Even experts show high levels of optimism bias: financial analysts expect improbably high profits and doctors overestimate the effectiveness of their treatment. Indeed the way in which our brains code positive information (it is good at that) and actual errors in prediction (not so good at that) means that we are programmed this way.⁴ On the plus side it has also been shown that optimistic people live longer.

Where optimism bias occurs, project planners commonly adopt an 'inside view' when preparing estimates for a new project – a view based on their own past experiences and the task in front of them. They do not pay sufficient attention to actual project outcomes or the 'outside view' which (according to Flyvberg and others) is more likely to produce realistic estimates and outcomes.

In an ICT context this could apply to a planner working for a customer or someone on the bid team of a vendor.

Strategic Manipulators

However, some mistakes can be deliberate (ie rather than genuine or honest) and involve deception or strategic manipulation (in the sense those terms are used by economists and management consultants). This behaviour can arise when planners deliberately and in a strategic manner understate cost estimates and exaggerate benefits in order to obtain approval and funding for a project or, on the contractor/supplier side, to (try to) win the contract.

Such behaviour of individuals is most likely due to political and organisational pressures placed on them in their roles as planners or bid managers. Where such individuals are also 'project champions' and involved throughout they may find it hard to turn back or re-set the project even when it runs into difficulties and they know it is wrong to carry on.

The Outside View and Reference Class Forecasting

One method of overcoming these challenges is RCF.

RCF compares the planned project with a reference class of similar and completed projects. Adjustments to estimates can be made depending on the outcome of the analysis undertaken. RCF relies on hard, statistical data rather than the planners own, ground-up assessment (or inside view) of how the project should be estimated. The detail of the proposed project is largely ignored. According to Flyvbjerg:⁵

Here is a simple illustration:

Indeed that is exactly what the UK Department of Transport did back in 2004 (with help from Flyvberg).⁸

RCF: Not a New Concept

RCF is not a new concept. It has its origins in theories developed by Daniel Kahneman and Amos Tversky which, in due course, helped them win the Nobel prize for economics in 2002.

As we have just seen, RCF was adopted by the UK Department of Transport back in 2004 and, according to Flyvbjerg, similar steps were taken by the Swiss Association of Road and Transportation Experts and the Danish Ministry for Transport – in 2006 and 2008 respectively.⁹

However, despite these (and other) examples of where RCF has been deployed the overall take-up appears fairly modest – certainly based on research done for the purposes of this article. This may reflect some of the potential challenges involved in applying RCF in practice.

Potential Challenges

Where do willing parties get the RCF data from?

This appears to be a key challenge. The theory is sound but how can willing parties routinely get hold of the data they need to formulate an 'outside view?' Not everyone will have access to the information they need for this purpose.

In the ICT area, governments and large corporate organisations that have undertaken multiple projects may have data sets they can use - provided the data is well-organised (by project type, location, size, time taken and actual cost incurred) and accessible.

That said, even these larger organisations may struggle to find ten or more similar projects (ie a valid reference class) each and every time they want to undertake RCF.

Others with useful data sets might include Flyvbjerg and Budzier themselves. Their study of 1,471 IT projects sounds like a very useful starting point – plus any subsequent data they may have gathered since that study was undertaken.

Organisations such as Gartner may also hold useful information - so too Standish Group and other international organisations and consultancies who, from time to time, gather and/or publish statistics relating to the ICT industry. However, smaller organisations may not have enough or any data to rely on.

Master Data Set?

Notwithstanding the probable existence of ad hoc data sets (such as those just described), there does not appear to be a global or master data set currently available to the ICT industry. In other words, a giant data set that all parties wishing to undertake RCF could access and utilise.

Is the creation of a shared database worthy of further consideration given the potential benefits for the industry as a whole? And could this provide an opportunity for any organisation prepared to take it on?

In my view, unless and until something like this is done, attempts by potential users to deploy RCF are likely to be constrained by the limited availability of reference class data.

The basis upon which data could be shared from a commercial standpoint is beyond the scope of this article. That said, any concerns over confidentiality should be capable of resolution through, for example, the anonymization of data and the appropriate confidentiality agreements.

Overcoming Resistance from Strategic Manipulators

As already noted, some planning mistakes will be genuine. Planners may not realise they are experiencing optimism bias. Where this occurs the chances of getting buy-in to using RCF should be high. Why would an honest planner want to achieve anything other than a better and improved forecast?

On the other hand, where deception or strategic manipulation occurs, there may be resistance to using RCF on a project. If a planner's inaccuracy is deliberate (to win funding or a new piece of business) they may not welcome a process such as RCF if it is likely to reveal the truth (ie that the working estimates are wrong and the project may fail as a result).

Wider adoption of RCF, and making it mandatory where appropriate, should help overcome this challenge. Strategic manipulators would then have no choice but to participate and abide by the outcomes.

Even without blanket adoption there is nothing to prevent senior executives and other decision-makers from insisting that RCF be used on a case-by-case basis.

Risk of Abuse?

There may be a risk that people become complacent in situations where a project budget is increased or time-scales extended as a result of RCF. If there is plenty of added contingency (to put it simply), project participants may simply play to those upper limits rather than maintaining efficiencies and keeping things tight. Is there a chance that the project will still overrun but against higher thresholds or upper limits?

This is where the project managers and lawyers (can and should) earn their keep. Good project management and the inclusion of realistic and enforceable legal sanctions in contracts between the parties to these projects should (in theory at least) help mitigate this sort of situation.

Place More Risk on Contractors?

In a separate article published in 2014, Flyvbjerg and his co-authors¹⁰ suggested that 'to ensure responsibility' the financial risk of delay and cost overruns should be placed with contractors who bid on the project – in order to mitigate situations where the winning bidder submits a low price in the expectation that they can make up the difference as the scope increases.¹¹ They appear to suggest that this risk allocation will incentivise contractors to be more honest at the outset.

I agree with this - although the probable reality is that a contractor (or supplier) will fight back and say that the reason for the scope increase was changing requirements on the part of the customer and not under-pricing when the deal was done. This is so often the battleground in ICT legal disputes.

However, if it can be shown that the original pricing and/or timescales (which tend to inter-relate) were always wrong (based perhaps on RCF undertaken 'after the event' to see what the original estimate should have been) then the contractor may be liable, particularly if the original estimates were hopelessly understated and there is limited or no evidence that a robust estimating process was conducted by the contractor when preparing its bid. In that instance, the contractor could face a claim based on fraudulent misrepresentation.

But what happens in a misrepresentation claim where the customer has undertaken RCF? Could this let the contractor off the hook? Could it be said that the customer relied on its own forensic estimates and not those of the contractor?

Is the position further complicated if both parties performed RCF (independently or on a collaborative basis) and agreed pricing and time-scales based on that information?

In these situations does it become harder for customer organisations to argue for compensation (and any other legal remedies sought) based on misrepresentation? If so, should customers de-risk the situation by leaving it entirely up to bidders to undertake RCF? Or would that risk undermining the original purpose of the exercise, namely for the customer, as the project initiator and gate-keeper, to form its own views on time and cost to complete?

Conclusions

RCF is not (nor is it intended to be) a 'silver bullet' solution to the problem of ICT project failure and there are practical challenges associated with it. However the theory does make a lot of sense and one can see how it might help improve outcomes, particularly when coupled with other (well-documented) good practices such as:

(i) breaking up larger projects into smaller, staged projects;

(ii) treating the project as a business endeavour (not a technical one);

(iii) recruiting and keeping good people to perform the work;

(iv) making people and organisations accountable (including those engaged to perform RCF if they get it wrong); and

(v) incentivising project participants to be realistic and honest.

In summary, potential benefits include:

• preventing parties from embarking on unachievable projects – or at least giving them a better sense of the actual risks involved so that adjustments can be made at the outset;
• lending more science to the art of estimating;
• creating more certainty on scope (ie the ability to set realistic requirements based on past similar projects);
• a transparent approach that levels the playing field by, for example, making it harder for bidders to under-price to win business; and
• fewer project failures.

Finally, in this age of big data, surely there must be scope to collect, collate and organise historical ICT project data from around the world and make it accessible to project planners across the industry? In other words, if the ad hoc collections of project data that currently exist can be pooled, shared and added to over time, RCF will become more accessible and the benefits for the industry as a whole could be spectacular.

1. 'What you should know about megaprojects and why: an overview – Draft 9.2, 2014, p 9'
2. ibid, p 10
3. 'Pre-empting and resolving Technology, Media & Telecommunications disputes: International Dispute Resolution Survey, November 2016'
4. These statistics and outcomes are from an article by Tali Sharot of UCL London dated 6 December 2011 and published by Cell Press
5. 'Quality control and due diligence in project management: getting decisions right by taking the outside view – November 2012,' version 5.2, page 6
6. The potential challenges associated with getting access to enough reliable reference class data are discussed below.
7. According to Flyvbjerg ten or more projects should normally make up a reference class. This statement appears in 'The British Department of Transport – Procedures for dealing with optimism bias in transport planning – Guidance Document – June 2004.' That document was authored by Flyvbjerg.

8. See 'The British Department of Transport – Procedures for dealing with optimism bias in transport planning – Guidance Document – June 2004.'

9. 'Quality control and due diligence in project management: getting decisions right by taking the outside view – November 2012,' version 5.2, page 23.'
10. 'Better forecasting for large capital projects – Flyvbjerg, Garbulo and Lovallo, December 2014 published by McKinsey & Co.'
11. I have also heard this described as 'land and expand'.

The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.