Skip the NI Direct Bar
Skip navigation

The Weighting and Scoring Method

 

Introduction

15.1There are a number of approaches to the appraisal of costs and benefits that are difficult to value in money terms. These include, for example, listing and describing them, developing a matrix or impact statement, and applying the weighted scoring method. As indicated in section 2.7 above, these various approaches should be considered carefully before choosing the method most suited to the case in hand. Listing and describing is often adequate in simple cases. The impact statement approach is adaptable to most circumstances. The weighted scoring method, explained here, is a possible alternative approach.
15.2

Before explaining the weighted scoring method, some words of warning are appropriate.

  • Firstly, DFP is generally content with the appropriate use of either the 'list and describe' or impact statement approaches, and does not require the use of the weighted scoring method.
  • Secondly, where the weighted scoring method is employed, DFP expects the rationale for each weight and each score to be fully explained. Failure to do this can cause delays in the approval process.

What is the Weighted Scoring Method?

15.3The weighted scoring method, also known as 'weighting and scoring', is a form of multi-attribute or multi-criterion analysis. It involves identification of all the non-monetary factors (or "attributes") that are relevant to the project; the allocation of weights to each of them to reflect their relative importance; and the allocation of scores to each option to reflect how it performs in relation to each attribute. The result is a single weighted score for each option, which may be used to indicate and compare the overall performance of the options in non-monetary terms.
15.4

This process necessarily assigns numeric values to judgements. These judgements should not be arbitrary or subjective, but should reflect expert views, and should be supported by objective information. To achieve meaningful results which decision-makers can rely on, it is important that:

  1. the exercise is not left to the 'experts', but is undertaken by a group of people who represent all of the interested parties, including, for example, those who are directly affected by the project, and those who are responsible for its delivery;
  2. the group possesses the relevant knowledge and expertise required to make credible measurements and judgments of how the options will impact upon the attributes;
  3. the group is led by an independent chairman to steer the process, probe opinions, promote consensus and avoid prejudice; and
  4. the justification for the group's chosen weights and scores is fully explained.
15.5Appraisal reports should identify the personnel involved in the exercise, including an indication of their credentials, so that decision-makers are fully aware of whose views are represented. If there is a lack of consensus among members of the group regarding any of the weights or scores, the views of the dissenting individuals should be recorded.
15.6

The process of deriving weights and scores is explained below step by step, covering the following stages:

  1. Identify the relevant non-monetary attributes;
  2. Weight the attributes to reflect their relative importance;
  3. Score the options to reflect how each option performs against each attribute;
  4. Calculate the weighted scores;
  5. Test the results for robustness; and
  6. Interpret the results.

Step 1: Identification of Non-Monetary Attributes

15.7

Identifying the attributes may sound straightforward, but attributes must be clearly defined so that both appraisers and those reviewing appraisal reports have a clear understanding of them. To help in the scoring of options, attributes should be defined as far as possible in service or output-oriented terms, and they should generally relate closely to the service objectives and performance criteria established at the outset of the overall appraisal. Considerable care is also needed to ensure that:

  1. there is no double counting caused by an overlap in the attributes (e.g. aesthetic qualities and attractiveness);
  2. there is no double counting caused by attributes being covered by costs (e.g. including a 'reliability' attribute when reliability is already provided for by inclusion of maintenance costs); and
  3. all relevant attributes are included, even if they are common to all the options.
15.8Regarding point 3. above, it is important to include relevant attributes even when all the options appear to impact equally upon them. Omission of common attributes can distort scores and lead to an imbalanced comparison of the differences between the options. For example, Options X and Y may score 200 and 100 respectively, when common attributes are overlooked, giving the impression that X is twice as beneficial as Y. However, if common attributes are worth 300, the correct scores for X and Y should be 500 and 400 respectively, indicating that X has a significantly smaller advantage over Y when all the non-monetary factors are taken into account. Apart from distortion of scores, there is a general risk that the appraisal may focus on attributes that are relatively insignificant while overlooking the most important attributes.
15.9

Attributes are best defined so that the status quo or do minimum baseline option can be given a score other than zero. For example, if one of the project objectives is to improve access for the disabled, the attribute is better defined as 'accessibility for the disabled' rather than 'improvement in accessibility for the disabled'.

  • The first definition allows all of the options, including the baseline option, to be scored, and thus enables the options to be compared in proportion to the baseline.
  • The second definition necessitates a zero score for the baseline option, which means that the scores for the alternatives can not indicate how much better they perform than the baseline option.

(This is not to say that the baseline option should never be given a zero score. In the accessibility example, the baseline option will deserve a score of 0 if the current provision is completely inaccessible to the disabled. However, the more likely position is that the disabled can access it with a degree of difficulty, in which case a suitably positive score would be appropriate).

Example: In a certain health service appraisal, the relevant attributes are identified as:

  • number of cases treated;
  • waiting time;
  • patient access; and
  • disruption to services.

Step 2: Decide the Weights for Each Attribute

15.10The second stage is to decide on the weights to be attached to each of the attributes identified. This should reflect the group consensus about the relative importance of the attributes, which is a matter for judgement based on, for instance, relevant policy statements. The most common approach, and the one which is most readily comprehended, is to express the weights in percentage terms so that they sum to 100.
15.11

Justification for the weights ascribed should be recorded. Such an explicit approach helps to ensure that the basis of the weights is fully understood and accepted by all those participating in the exercise as well as those using its results.

Example: The group appraising our hypothetical health services project has decided that the following weights are appropriate:

  • number of cases treated - 40%
  • waiting time - 30%
  • patient access - 20%
  • disruption to services - 10%

Step 3: Scoring the Options

15.12The third stage is to score each option against each attribute on a suitable scale. The approach described here uses a cardinal scale. This means that if Option A is considered to perform three times as well as Option B, then Option A is given a score that is three times that of Option B. Simpler alternatives to cardinality are possible, for example an ordinal scale may be used. This provides a simple ranking of options against each attribute, which enables one to say that Option A is better than Option B, but it does not indicate how much better A is than B. Such an approach may be useful in some circumstances, but a cardinal approach, if sustainable, is more informative.
15.13Options are scored against the attributes by reference to a scale, say from 0 to +20. A score of 0 will indicate that the option offers no benefits at all in terms of the relevant attribute, while a score of +20 will indicate that it represents some "maximum" or "ideal" level of performance. Scores between 0 and +20 will indicate intermediate levels of performance. The scale used does not have to be from 0 to +20, but mathematical consistency demands that the same scale is used for all attributes. The meaning of the maximum and minimum score should always be clearly defined and the whole scoring system should be documented clearly in the appraisal report. Group members should have a common understanding of it.
15.14To achieve cardinality, the group needs to think carefully about the differences in the scores awarded to the options, and to provide meaningful justification for them. Suppose, for example, that the attribute 'waiting time' refers to the speed of delivery of a particular service, and that options are scored on a scale from 0 to +20. The group has decided that a score of 0 represents a waiting time that is completely unacceptable e.g. 12 months or more; while a score of 20 represents a waiting time at or close to zero. If Option C delivers in 3 months, while Option D delivers in 6 months, then, using the scale as defined, it would be reasonable to award Options C and D scores of 15 and 10 respectively. In another example, where the attribute is 'accessibility' it may be possible to justify different scores on the basis of objective information about differences in distances travelled.
15.15The weighted scoring method should not be used to avoid the effort of measuring differences between options in measurable non-monetary units. Nor should it be used to substitute vague subjective judgments of comparative performance for hard measurement. The credibility of the scores depends upon the provision of a rational justification to support them, including measurement where possible. In any case, project sponsors must be able to provide justification for each and every score that is awarded, and DFP will expect this to be recorded in full detail.
15.16Scores should be allocated to all of the options, including the baseline option (i.e. the status quo or 'do minimum'). A common error has been to overlook the baseline, but it is important to include it. However inadequate it may seem, the existing or 'do minimum' level of service will normally impact on the attributes to some extent, and scoring this helps to give a sense of proportion to the scores of the other options, and to compare their performance to that of the current or minimum level of provision.

Example: The health service group scores four options against the attributes as follows:

Option P
(Status Quo)

Option Q

Option R

Option S

No. of cases treated

5

10

12

15

Waiting Time

8

12

14

16

Patient access

10

10

15

15

Disruption to services

10

5

5

10

Step 4: Calculate the Weighted Scores

15.17This simply involves multiplying each score by the weight for the relevant attribute. Thus weighted, the scores are totalled to obtain an aggregate weighted score for each option.

Example: Combining the last two examples results in the following weighted scores:

Option P
(Status Quo)

Option Q

Option R

Option S

No. of cases treated

5x40 = 200

10x40 = 400

12x40 = 480

15x40 = 600

Waiting Time

8x30 = 240

12x30 = 360

14x30 = 420

16x30 = 480

Patient access

10x20 = 200

10x20 = 200

15x20 = 300

15x20 = 300

Disruption to services

15x10 = 150

5x10 = 50

5x10 = 50

10x10 = 100

Total
Weighted
Score:

790

1,010

1,250

1,480

Step 5: Test the Robustness of the Results

15.18It is important to examine how robust the results are to changes in the weights and scores used. This can be done with the aid of sensitivity analysis. For example, the weighted scores can be recalculated to demonstrate the effect upon them of changing the weights. Similarly, they can be recalculated to show the impact of different scores.
15.19Judgement should be used to select suitable variations in assumptions to subject to sensitivity analysis. For example, where there have been differences in opinion within the group about certain weights or scores, it will be helpful to explore the impact of using the different weights or scores advocated by different group members.
15.20

Details of the sensitivity analysis should be recorded, and the robustness of the results confirmed. Where appropriate, attention should be drawn to circumstances in which the ranking of options, or the differences in weighted scores, are particularly sensitive to plausible changes in certain weights or scores.

Step 6: Interpret the Results

15.21Non-monetary factors are generally important in public sector appraisals therefore weighted scores can have a crucial influence upon option selection. It is vital that they are compiled and interpreted carefully, and that the reasoning behind the figures is clearly presented in appraisal reports.
15.22The results will consist of a set of weighted scores, including one for each option. These should act as indices for comparing the options' overall performance on non-monetary factors, indicating not only how the options rank but also how great are the differences between them. Thus they should serve a similar purpose in respect to non-monetary factors as NPVs do in respect to monetary factors. For example, if Options E, F and G have weighted scores of 2000, 1000, and 950 respectively, this indicates that Option E is significantly better (about twice as good) as either Options F or G, while Option F is slightly better than Option G. This is more informative than the use of an ordinal scale, which can only indicate the rank order of E, F and G.
15.23Weighted scores can be directly compared with NPVs, to help assess trade-offs between costs and non-monetary performance. This is illustrated by the following example.

1.
Option

2.
Net Present Cost

3.
Weighted Score

4.
Total Cost per Unit of Weighted Score

5.
Marginal Increase in Weighted Score compared to Option P

6.
Marginal Cost per Extra Unit of Weighted Score compared to Option P

(£M)

(£)

(£)

P

3.0

790

3,797

Q

4.5

1,010

R

4.0

1,250

3,200

460

2,174

S

5.0

1,480

3,378

690

2,899

15.24Columns 2 & 3 show the Net Present Costs (NPCs) and Weighted Scores of Options P, Q, R and S. The information in these columns is sufficient to indicate that Option R dominates Option Q. In other words, Option Q is both more costly and less beneficial than Option R, and, other things being equal, can be dismissed from further consideration.
15.25The figures in columns 4 to 6 help to compare the cost-effectiveness of Options P, R and S. Column 4 implies that Option R is the most cost-effective in terms of total cost per unit of weighted score. Columns 5 & 6 help to indicate the differences between Options R and S and the least cost option, Option P. The figures suggest that Options R and S offers significant extra benefits than P, and that Option R does so at the lowest marginal cost.
15.26Such calculations need to be handled with care. It is important to bear in mind that weights and scores are based on judgements. They are not precise measurements against an interval scale, such as the measurement of temperature against the Fahrenheit or centigrade scales. The importance of explaining the weights and scores fully, and interpreting the results carefully, can not be over-stressed.
15.27The results of a weighted scoring exercise are specific to individual cases, and are not readily transferable to others. However, the attributes relevant to one project are likely to be relevant to other projects of a similar type. The weights given to these attributes may not be the same, but the principles for deciding the weights should show some consistency across similar projects. There should also be some consistency in the principles used for scoring options within similar categories of project.