DERA - Ministry of Deference Grant Reference:CU003 - 0000000860
PROJECT: Consultancy Report on 'Investment Evaluation of IT/IS: Research and Application' for DERA/Ministry of Defence

Executive Summary
The literature shows that the increasing percentage of capital investments in Information Technology (IT) and Information Systems (IS) without commensurate returns is forcing a reconsideration of how to justify expenditure on technology-driven business process change. However, such organisations are reporting an inability to adequately evaluate these investments.

A better understanding of the impact IT/IS has on organisational performance can help an organisation better utilise resources and improve its position vis-à-vis its competitors. Viewed in simple system dynamics terms, evaluation provides the basic feedback function to managers as well as forming a fundamental component of the organisational learning process. Finally, evaluation provides the benchmarks of what is to be achieved by the information systems investment. These benchmarks can later be used to provide a measure of the actual implementation success of the information system.

Much of this difficulty centres around the range and complexity of managerial and technological knowledge needed to handle the multiplicity of justification and implementation paths. IT and IS investment yields a mixture of strategic, tactical and operational benefits and it is argued that there are a variety of project types each needing different approaches to their evaluation. In each case gains may be financially quantifiable, non-financially quantifiable and intangible. The full range of hidden costs, both direct and indirect, are also equally difficult to quantify. Appraisal processes are not "one stop decisions". From their sources, often at the roots of an organisation, proposals go through stages of development, filtering and aggregation before they become part of the capital budget.

There is no single simple technique that can reasonably be applied to IT/IS evaluation. Both ratio and discounted economically based assessment techniques loose their appeal of simplicity and objectivity as more and more factors are added to cover risks and intangible benefits. On the other hand strategic assessment techniques omit the essential element of financial viability necessary for corporate survival. The more comprehensive portfolio and integrated assessment techniques are complex to apply and remain heavily dependant subjective judgement.

Clearly, placing investment evaluation in a positive light is synonymous to best practice management however, much of the normative literature suggests that organisations often adopt a less than structured approach to the justification process. Many companies pursue one of the following strategies (i) Refusal to undertake IT/IS projects whose benefits and costs are not easily financially quantifiable; (ii) Invest in IT/IS projects as an act of faith; or, (iii) Use creative accounting as a means of passing the budgetary process.

There are numerous management concerns about investment appraisal. However, regardless of the complexity involved it is important to justify IT/IS investments before committing time and money to implementation and risking failures leading to loss and competitive disadvantage. Competition for scarce organisational resources often forces choices to be made between heterogeneous investment proposals without knowing the full impact each may have on organisational performance. Under pressure the decision processes are more often characterised by limited commitment of organisational resources, myopia surrounding technology related change and the need to show quick financial returns. This short term view can prejudice investment decisions against staged, or modular, system development - a practice which is almost totally ignored within the literature on appraisal mechanisms.

This analysis is supported by evidence from five case studies examining reports of real-life information systems investments. These show the variety of justification approaches that these organisations used in practice and highlight their weaknesses.

The first case study is of a local authority where middle management took a strategic decision to introduce an office automation system to integrate their disparate departmental systems, working environments and business cultures. Only a small-scale requirements specification study was undertaken, there was no monitoring of the implementation stage, and no post-implementation review was conducted. Associated costs such as training and support were also omitted from the original budget.

As a result the selected product that lacked some of the fundamental requirements and this problem was not detected or remedied either during or after implementation. The lack of adequate evaluation meant that the system lacked the expected impact, the true costs and benefits of the system were never fully understood and that lessons for future investments were missed.

The second case looks at a new pensions system in an insurance company. This development coincided with a restructuring of the IT department and throughout there was a lack of effective project management. Although the project arose from a user driven steering committee there appeared to be very little, if any, attempt at justifying the investment. Changing committee membership and a lack of consensus between individuals and departments lead to the repeated adoption of compromise solutions. Although the change factors (internal restructuring, new market opportunities and changing regulations) were realised, they were not managed to prevent excessive, incompatible and infeasible demands being made on and accepted by the IT department. The inevitable result was disillusionment, delivery of low quality of the systems with cost and time overruns.

The third case study was concerned with a system to provide Electronic Data Interchange (EDI) between two Canadian financial organisations. The investment was evaluated entirely on a financial basis, although softer benefits were mentioned to strengthened the arguments for adoption. The decision to proceed was based solely on an economic feasibility analysis and the ex ante analysis covered only hard costs and benefits. Despite a projected payback period of one year, the company made a significant loss in the first year of system operation. Although there were flaws in the calculations, a more critical oversight was the failure to assess the broader impact of the system. The low level of integration created new manual processing tasks and others, that it had been assumed would be eliminated, had to be retained.

The fourth case shows a successful project in a bank using IS to leverage the business opportunities in the 'home banking' market. The report deals with ex post evaluation to assess whether the planned benefits were actually realised and actively managed. This evaluation addressed three distinct levels: financial evaluation, quality assessment, and strategic appraisal. This emphasises the range of perspectives from which an information system can be assessed. Some lend themselves naturally to financially techniques, while others cannot legitimately be measured in this way. Since some perspectives may also show benefits that fall into both categories this case shows how a single evaluation technique would be inadequately to assess all the benefits and drawbacks of an investment.

The last study refers to the process of customer order fulfilment and two collaborating organisations aiming, by redesign and interconnection of their systems, to reduce the average time to process an order. An in-depth study of the processes was conducted using Business Process Modelling and Simulation. This was motivated by the need to generate the data that would feed a formal cost/benefit analysis in an objective manner. It also allowed exploration and extension of the initial assumptions about projected savings and costs. The application of simulation proved to be a valuable mechanism for realising the real business value of the proposed IT-based solution. Individuals were able to see and assess the costs and benefits associated with the proposed change giving them confidence in the technology without the risks, costs disruption of an implementation project. It also showed that the adoption of an IT-based system alone would not improve the order fulfilment process to the extent expect.

The methods used in this last case showed several beneficial side effects. Firstly, it brought participants from different levels in both companies to work together towards a solution that would be of mutual benefit. Second, facilitated communication between business and IS specialists. Thirdly, it structured the debate and speeded reaching a consensus. In many respects this illustrates the multi faceted nature of IT/IS evaluation and the strong relationships to the views of particular stakeholders in any particular project.

£ Value: £22, 243

Investigators: Professor Zahir Irani, Dr Tony Elliman, Professor Guy Fitzgerald