| 2.1 | The evaluation process can be broken down into 10 key parts outlined in figure 2.1 below. This framework should be flexible in recognition that circumstances differ within and between programmes. However, the items listed are the essential ingredients of policy or programme evaluation and will permit a consistency of approach across evaluations. |
| 
Fig 2.1: The Role of Evaluation in the Policy-Making Process |
(i) | Planning an Evaluation |
| 2.2 | An evaluation plan is a framework that can clarify the key elements of a proposed evaluation. As there are likely to be resource constraints choices will have to be made regarding which programmes should be evaluated first. Asking the following key questions will help answer this question (see figure 2.2). - How important is the programme in terms of its own characteristics and in its repercussions? for example, does the programme constitute a large proportion of the Department's budget or does it affect a significant group in the economy?
- Can the programme be influenced? This depends on the context or environment in which it operates. There may be specific commitments that severely constrain freedom to change the programme or there may exist strong lobby groups that are resistant to change.
Is there adequate information and/or data available to do a 'proper' evaluation? Data collection and analysis might be too costly in relation to the pay-off from the evaluation. |
| 2.3 | Having decided on which programmes to evaluate first, there are a number of questions to address: - When to plan the evaluation?
- What questions will the evaluation address?
- Who should carry out the evaluation and who it is for?
How much will the evaluation cost? |
| Criteria For Selecting Programmes For Evaluation
|
| Criteria for Selecting Programmes for Evaluation |
| Criteria How Important? (Characteristics and repercussions) Can it be Influenced? (Context) Possible to do? (Costs of action and analysis) When to plan an evaluation? |
| 2.4 | The best time to plan an evaluation is before the programme is introduced because: - Monitoring arrangements, the arrangements for collecting the relevant information, can be put in place.
A baseline, the position prior to the programme's implementation, can be identified. This enables a before and after methodology to be used. However, for existing programmes, where prior monitoring and evaluation procedures were not established at the outset, then the pure before and after approach will not be possible and other methods will have to be employed which may prove more costly. |
| What questions will the evaluation address? |
| 2.5 | The questions and issues to be considered in an evaluation can range from a comprehensive value for money assessment to questions about the effectiveness of the programme or questions about inputs and efficiency, for example, programme inputs compared with the inputs of related programmes, or cost per unit of output comparisons. These questions determine the scope and purpose of the evaluation. Those involved in commissioning an evaluation must decide what issues it will address and what will be the main focus. The focus of an evaluation might include the following: the extent to which the programme has been implemented as designed; the extent to which it has succeeded; the extent to which it can be improved (or indeed a combination of purposes). |
| Who should carry out evaluation? |
| 2.6 | Primarily the choice of who performs an evaluation will be between (i) insiders (i.e. staff who operate the programme), (ii) outsiders (e.g., outside consultants or academics) or (iii) a team comprising both insiders and outsiders. - Insiders: An advantage with insider evaluation is that staff concerned with the programme will have detailed knowledge of it. However outsiders can usually acquire a similar level of understanding provided they have the co-operation of programme staff and access to the relevant information. It is possible that insiders may lack some of the skills necessary for a proper evaluation although working on such exercises can enable expertise to be built up.
- Outsiders: Outsiders such can be commissioned to undertake evaluations. However, input will still be required from insider staff and good co-operation and information flow is essential. An advantage of evaluation performed by outsiders is the perceived objectivity that they can bring to the evaluation process.
Insiders & Outsiders: Another option is to form a joint working group of the programme providers and internal or external specialists, such as economists statisticians, social researchers, policy analysts, academics or management consultants respectively. The advantage of such a group is that it combines detailed insider knowledge with the objectivity and expertise that outside specialists can provide. However, a group of this kind might prove more costly and the duration of the evaluation might be prolonged. |
| How much will it cost? |
| 2.7 | Evaluation costs money and even if it is done as part of normal work there is an opportunity cost of other work foregone. The key question is whether the lessons to be learnt from the evaluation outweigh the costs. The importance of the programme (see figure 2.2) will help decide the amount of resources that it is sensible to allocate to the evaluation. Where possible, evaluations should use existing information and desk research. However, on occasions the absence of monitoring data, or the fact the evaluator may wish to gauge the views of programme recipients, may necessitate the use of surveys which can be expensive. Once the amount of resources has been set for the evaluation the evaluators should determine the best approach subject to the budget constraints |
(ii) | Establish the scope and purpose of the evaluation |
| 2.8 | Establishing the scope and purpose of an evaluation is an important part of the planning process. The questions outlined in the earlier section on planning are therefore relevant at this stage. |
| 2.9 | Consideration should be given to the type of conclusions which the evaluation study hopes to reach. This means being explicit about the boundaries and purpose of the proposed study. There are a number of approaches that may be pursued depending on the primary purpose of the evaluation and the answers being sought. The purpose of the evaluation could be concerned solely with discovering current weaknesses in the programme which need rectifying or identifying what may need to be remedied in the future (process evaluation). Other evaluations may concentrate more on determining the overall success, or otherwise, of a programme so that decisions can be taken on whether to continue, expand, or reduce the level of the operation (outcome evaluation). The purpose of the evaluation will determine the questions that need to be asked and the amount of detail required. The type of evaluation carried out will be unique since it should be tailored to the purpose of the study. |
| 2.10 | This guide is concerned mainly with the evaluation of an entire programme. However, when a programme is in its early stages, its full effects have yet to be realised. This makes it impossible to determine its overall effectiveness. Therefore it may be sensible only to look at say the administrative efficiency of the programme. Even when such a 'partial' evaluation is being undertaken, it is important to follow the evaluation process and to link the objectives of the particular aspect of the programme being evaluated with its overall aims and objectives. |
| 2.11 | As noted above the evaluator will not always be concerned with assessing the final outcomes or impact of a programme against pre-determined objectives. Departments may wish to carry out an efficiency scrutiny or a policy review. Efficiency scrutinies will usually focus on the implementation or administration of a policy. A policy review can be seen as a 're-appraisal' of a policy or programme. While the results of such scrutinies or reviews may usefully feed into a policy evaluation, they do not in themselves fulfil policy evaluation requirements. |
| 2.12 | All national or European Union (EU) policies impacting on Northern Ireland should be subject to evaluation. It is recognised that, as NI departments may be distanced from the formulation of some UK and EU policies and as political and legal considerations can limit their ability to diverge from such policies, their ability to modify programmes may be marginal. Notwithstanding these constraints it is important that National and EU policies are evaluated so that their specific impact on Northern Ireland can be assessed and thus feed through into future decision making. Close liaison with GB counterparts is important in order to explore the possibilities of linking into evaluations being carried out in GB. |
| 2.13 | Where there is a number of closely related policies or programmes with the same overall aim and similar objectives, they ought to be evaluated separately and together, for example, schemes aimed at training the long-term unemployed may include Enterprise Ulster, the Job Training Programme (JTP) and Action for Community Employment (ACE). These schemes could usefully be evaluated at the same time to assess any overlap or complementarity. |
(iii) | Establish Rationale, Aims, Objectives and Targets of the policy or Programme |
| 2.14 | The rationale, aims, objectives and targets of the programme under scrutiny should be clearly stated. Where these have not been established prior to programme implementation, the evaluator should attempt to determine them. Where a programme has a number of diverse aims and its objectives are unclear, an attempt should be made to rank the aims and objectives. |
| 2.15 | There is often confusion surrounding the distinction between rationale, aims, objectives and targets but they are best regarded as a hierarchy. Figure 2.3 provides an example of this hierarchy. |
| Rationale |
| 2.16 | Rationale is the justification for a particular programme. The rationale for public expenditure is to correct some form of market failure and/or to address social/equity issues. The rationale of the programme should be clearly stated in the evaluation report. Section A of the technical supplement explains the reasons for Government intervention in more detail. |
| 2.17 | Ideally, the rationale of an expenditure programme will have been determined prior to the programme implementation (i.e. in the baseline period). The programme will have been justified in terms of the need it is intended to address. This involves identifying indicators of need. |
| Aim |
| 2.18 | The aim of a programme is its ultimate intention, i.e. the purpose for which it exists. Aims are not necessarily achievable in a particular time period but shape the direction of the programme, for example, programmes may have aims relating to economic, security, health, safety and welfare matters. Where a programme has more than one aim these should be clearly distinguished and prioritised. |
| Objectives |
| 2.19 | These are derived from the aim(s) and are normally the intermediate representation of what the programme intends to achieve in concrete terms. Thus while the ultimate aim of the Industrial Development Programme is to support a durable growth in employment, the intermediate objective is to introduce and develop internationally competitive companies. Programmes are likely to have a number of objectives and again these should be prioritised. |
| Targets |
| 2.20 | Targets are more specific than objectives and should be included as a sub-layer to disaggregate the objectives and perhaps focus on specific target groups in the population and intended level of impact. Targets should be measurable, time-bounded and ideally linked to departmental Management Frameworks and performance indicators. The targets of the ID programme may be competitiveness or job creation targets, for example, to increase company profits by 10% in the next year, or to increase job creation by 5% in the next year. The importance of setting and monitoring targets should not be under-estimated as they help determine what data should be collected in the monitoring process. |
| 
Figure 2.3 Rationale, Aims, Need, Objectives and Targets |
(iv) | Specify Measures and Indicators |
| 2.21 | Having identified the objectives that the programme hopes to achieve, the evaluator should specify the indicators and performance measures that will be used to assess if the programme is achieving Value for Money, that is, how effectively, efficiently and economically did the programme meet its objectives. - Effectiveness measures should show the extent to which the aims, objectives and targets of the programme are being achieved. The effectiveness of the policy or programme is usually assessed by output measures.
- Efficiency measures (generally expressed as a ratio of inputs to outputs), are concerned with achieving the maximum output from a given set of inputs, for example, cost per job or output (£ or physical) per unit of staff. With efficiency measures, the same ratios can be compared on a year on year basis. In addition, ratios can be compared with those generated for alternative policies, those generated by the private sector and those generated in other areas of the UK or EU.
Economy measures are concerned with showing that the appropriate inputs have been obtained at least cost. |
| 2.22 | To determine effectiveness, efficiency and economy aspects of the programme the following indicators and measures should be examined. |
| (a) Input Indicators |
| 2.23 | Inputs are the resources used in carrying out the programme. They should be readily available and should be separately identified in both cost and volume terms, i.e. on a money and numbers basis. Common inputs include manpower, buildings, plant and machinery and general administrative expenditure. |
| (b) Activity Indicators |
| 2.24 | These are generally concerned with the frequency and extent with which resource inputs are applied. |
| (c) Output Indicators |
| 2.25 | Outputs are the goods/services produced by the programme under scrutiny through use of the identified inputs. These can be both quantitative and qualitative. For some programmes it may appear difficult to specify outputs. However, it should be possible to indicate those programme outputs which go some way to meet the programme's intermediate objectives and ultimate aims. |
| (d) Outcome/Impact Indicators |
| 2.26 | These are an extension of output indicators but are more concerned with the ultimate results of a programme. The impact of certain programmes may be difficult to quantify but some attempt should be made to assess if the impact has been positive or negative |
| 2.27 | An assessment of the value for money of the programme is made by looking at all of these indicators (see figure 2.4). Firstly, it should be shown that the inputs have been obtained at the least possible cost, and secondly the ratio of inputs to outputs should be calculated to give a measure of efficiency. These taken together with an assessment of the extent to which the outputs/outcomes of the programme are those which were originally intended will determine the value for money of the programme. |
| 
Figure 2.4 |
| Performance Indicators |
(v) | Establish a Base Case for Comparison with Actual Outcome |
| 2.28 | Evaluation of a programme necessarily involves consideration of what would have happened if the programme had not been implemented, i.e. the Base Case (or counterfactual position). This can be viewed as an attempt to predict an alternative outcome and is essential in assessing whether the activity has changed things for the better or would they have improved anyway. The difference between the base case and what actually happened is the 'net additionality' of the programme. |
| Types of Evaluation Design |
| 2.29 | The base case should be constructed within an evaluation design and quantified as far as possible. The main types of evaluation design considered here are based on a comparison of pre and post programme measures using control groups or the policy affected group only. The technical supplement provides more detail on the selection and construction of the base case |
| 2.30 | It may be possible to set up a control group. A control group is one whose members do not receive or benefit from a particular policy or programme even if they are eligible. The performance of this group can be compared with that of the group whose members did receive the programme. The control group acts as the base case with its performance, or output, showing what would have happened in the absence of the programme |
| 2.31 | If it is not feasible to set up control groups, an alternative approach is to compare pre and post programme measures of the policy affected group. If time series data is available, then it will be possible to extrapolate or project the trend in performance of the policy affected group from the 'policy off' period into the 'policy on' period and then to compare this projected trend with what actually happened. In this case the extrapolated trend acts as the base case. |
(vi) | Define Assumptions |
| 2.32 | It is often necessary during an evaluation exercise to take certain things for granted and to make assumptions about the programme under scrutiny. A good evaluation will clearly state these underlying assumptions and indicate whether they need to be regularly reviewed. This is necessary because quite often there will be an assumed causal relationship between the programme's ultimate aims, objectives, inputs and outputs, for example, it might be assumed that a certain vaccination programme produces health benefits for the whole population. This assumption may need to be reviewed as medical knowledge progresses or people change their behaviour. |
| 2.33 | Another type of assumption relates to the external environment. Action by the Government is seldom the only factor which determines whether the policy objectives are achieved or not. External factors are changing all the time, for example, the economic background when a support scheme was first decided upon may have changed during its period of operation. Similarly, what was once an innovative technology may now be a routine application in industry. Policies appropriate to a time of credit scarcity may persist into a time when borrowing is easy. It is important however not to think of the environment only in economic terms. Social priorities may have shifted. The demographic pattern may be different. Emerging new technology may be opening up possibilities that were previously closed. Both cause and effect and external environment assumptions will link back to the assumptions underlying the construction of the base case. |
(vii) | Identify Side Effects and Distribution Effects |
| 2.34 | A programme may have side effects (beneficial or otherwise) beyond the policy objectives originally envisaged. These spillovers are sometimes identified at the policy formulation stage but may emerge only as a consequence of the evaluation process. The most important side effects apart from the multiplier which is dealt with elsewhere include: |
| Externalities |
| 2.35 | These are benefits or costs falling on third parties, for example, the construction of a new road may bring windfall gains to a restaurant by generating extra custom (a pecuniary externality) while reducing the value of residential property because of noise and pollution (a production externality). An attempt should be made to quantify these social costs or benefits and this should be included in the cost-benefit equation. |
| Distribution effects |
| 2.36 | When carrying out an evaluation, account should be taken of the extent to which the policy or programme complied with the Government's aim of securing equality and equity. It is the Government's objective to take account of fair treatment aspects when reviewing existing policies and delivery of services and to monitor, as appropriate, the impact of its policies. When existing policies or services are reviewed, careful consideration should be given as to whether they give rise to direct or indirect discrimination or a differential impact which has not already been considered and justified. |
| 2.37 | The Government also wants to encourage effective voluntary activity and to promote volunteering. Government policies should not therefore place an unreasonable burden on, nor overlook the potential contributions of, the voluntary sector unless there are overriding policy reasons. When considering changes of policy or changes in procedures, an assessment should be made of the implication those changes may have for voluntary activity and the voluntary sector. |
(viii) | Analysis |
| 2.38 | The type of analysis adopted (stemming from the evaluation design) will depend upon the questions the evaluation is setting out to answer (process or outcome) and the approach adopted (quantitative or qualitative). It will also depend upon the information available, although this should not allow the evaluation to avoid addressing the important issues in a spending programme. Information can be obtained from several sources such as regular monitoring, desk research and surveys. |
| 2.39 | An outcome or summative evaluation is primarily concerned with assessing whether or not the programme is achieving value for money, i.e. achieving its objectives in the most efficient and economical way possible. Although the emphasis will be on output measures as overall indicators of what the policy or programme is buying, programme inputs also can be included to give a more complete measure of value for money. The key measure is that of net additional output which will take account of, inter alia, deadweight (i.e. the extent to which programme expenditure assists recipients to do something they would have done anyway), displacement (i.e. the extent to which a desirable programme output in one area leads to a loss of similar output in another area) and multiplier effects (i.e. programme expenditure creating a demand for locally produced goods and services which will stimulate additional employment. This additional local activity is likely to raise local income and this will generate additional expenditure adding further to local employment). |
| 2.40 | One method of analysis is to use a comparison between a control group and the policy affected group, the difference between the two groups being attributable to the effect of the particular policy or programme under consideration. This captures the net additional output of the programme in terms of the key output indicator(s) chosen. However it should be recognised that the programme may impact on a wider range of aspects not covered by the indicator(s) chosen, which although not considered to be of primary importance could affect the final level of net additional output. |
| 2.41 | If it is not feasible to set up control groups, another design is the comparison of 'before and after' policy measures of the single policy group. A variation of this, data permitting, is to use the projected trend outcome of the policy affected group as the base case and compare this with what actually happened in the 'policy on' period (trend extrapolation). Properly measured, these methods should show the net additional effect of the programme on the basis of the particular output indicator chosen. |
| 2.42 | A useful framework for analysing both process and outcome evaluations and accommodating both quantitative and qualitative approaches is Ex-post Cost Benefit Analysis (CBA). The CBA framework can be used for presenting the costs (inputs) and the benefits (outputs/outcomes) identified through the evaluation. It seeks to quantify as many of the costs and benefits as possible and it allows for the inclusion of items which cannot be easily valued in money terms. The technical supplement provides more detail on the CBA framework. |
(ix) | Outcome of Evaluation |
| 2.43 | Once the evaluation has been completed the next stage is to consider the implication for the programme in terms of the way it operates and recommend follow-up action. Various options can unfold at this stage: |
| Programme Continuation |
| 2.44 | The programme should continue in its present form; the evaluation result was clearly positive with benefits outweighing costs. However this should not mean that the programme continues indefinitely but rather that it is subject to monitoring and that it is reviewed at a specified future date. |
| Programme Modification |
| 2.45 | Changes should be made to the way the programme operates, for example, inputs could be obtained more cheaply but the objectives, content and coverage remain the same. |
| Programme Succession |
| 2.46 | More fundamental changes should be made to the programme in its content and coverage, for example, the target group may need to be redefined or adverse side effects are negating some of the programme benefits. In this case the old programme may need to be replaced by one which has different objectives or different coverage. |
| Programme Termination |
| 2.47 | The programme should be terminated as the logical outcome of the evaluation with no replacement. Programme termination in particular may be difficult to plan and implement and some form of exit strategy may be necessary to minimise termination difficulties. In devising an exit strategy the following questions should be considered: - Is the programme still needed?
- Will withdrawal cause loss of benefits already generated and loss of future economic benefits?
- Will particular population groups be adversely affected?
Should there be a transitional phasing out period or immediate termination? |
| 2.48 | The outcome of the evaluation may not be clear cut for a number of reasons, for example, crucial information on inputs, outputs and the impact of a programme may not be available due to a lack of monitoring information. Similarly, in the time available it may have only been possible to look at a narrow range of performance indicators. In addition, there may be uncertainty about the future and if and how the programme should continue. In such circumstances it is important to test the sensitivity of the outcomes to any particular assumptions that have been made. This will highlight the most important assumptions and allow more informed decisions to be taken. |
| 2.49 | Although evaluation is primarily a way of looking at the past, it leads on naturally to look forward into the future and may suggest alternatives to programmes in whole or in part and so lead on to appraisal of these options. Here evaluation merges into reappraisal and appraisal of new proposals and thus the cycle between evaluation, monitoring and appraisal is complete. |
| 2.50 | It is recognised that there is no single correct approach to all evaluation problems. There are many types of evaluation approaches, each with its own values and limitations and it is important to know which are suitable and which are not in different circumstances. Most evaluations will benefit and yield more useful results from a combination of approaches but each evaluation is unique and approaches should be modified to fit the particular exercise. |
(x) | Presentation and Dissemination of Results |
| 2.51 | The evaluation report should cover the following: 1. Description of the programme and its rationale, main objectives and expenditure 2. Evaluation design methodology used 3. The type of analysis and measures/indicators used 4. The main assumptions contained in the analysis 5. The results should indicate: - the costs and benefits of the programme (both quantifiable and unquantifiable)
- the net additionality of the programme
- an overall assessment of value for money.
<sty sys-align-left><sty Normal>6. Recommendations for the future of the programme. |
| 2.52 | The potential value of evaluation can only be realised when action is taken upon it. Evaluation reports must reach senior management and all results of substance should be widely disseminated following well defined lines of communication. Reports and/or their future recommendations, should be circulated widely to those staff concerned with future project design, planning, development and management. |
link to an external website
link (to an external website), opens in a new window
