Setting Individual Achievement Targets with DEA

by Emmanuel Thanassoulis and Gary Simpson

Data Envelopment Analysis can be used to set targets for individual attainments that are demanding but realistic. The targets are based on the best attainment levels realised by other individuals, after allowing for factors beyond an individual’s control. The approach will be illustrated using the example of school children but it has wider applicability in monitoring and improving the performance of individuals in any context where large numbers of individuals are engaged in a given activity but under varying conditions. For example salespersons may be set targets which reflect the varying sales environments in which they operate.

-oo0oo-

In many areas of life we attempt to meet targets or to set targets; whether as school children attempting to meet the expectations of parents and teachers, or as bank tellers attempting to reach a target level of transactions. In this paper we will describe a Data Envelopment Analysis (DEA) based technique introduced in Thanassoulis (forthcoming) to set attainment targets for individual school children, generalising it to other contexts. Targets at individual person level can play a very important role in helping individuals improve performance when the targets are suitably set to be challenging but not demotivating. For example, the Department for Education and Employment (DfEE 1996, p6) notes that effective target setting ‘helps schools to articulate clearly what is being aimed at, whether by schools as a whole, or by groups or individuals within the school’. Often our success in meeting targets depends not only on our own effort but also on factors that are beyond our control such as our aptitude for the task. The method discussed here endeavours to set realistic targets that are not unduly influenced by very exceptional performance by a small number of individuals or by random noise. Our belief is that this methodology can be readily applied in many contexts where the targets set for individuals should take into account extraneous factors and must be realistic to have the desired motivational effect.

Outline of the approach

Let us take a situation where we wish to set targets for each individual within a set, all individuals performing the same task. The targets will refer to levels to be attained on given variables over a specified period of time. For example, in the case of sales persons the targets might be in terms of the volumes of sales achieved while in the case of a tax officer the target might be in terms of the number of tax cases assessed.

In order to deploy the method we need recent data at individual person level on two sets of variables. The first set is that of attainment variables on which future targets are to be set. The second set is that of control variables. The latter relate to factors beyond the control of the individual but which nevertheless impact his/her ability to attain the targets set. For example, in the case of sales persons the attainment variables might be the volume and the value of sales of each type of product, while the control variables might be the corresponding total volume and value of sales in the geographical region covered by the sales person, the number of competitors s/he faced and so on.

The identification of the control and attainment variables is crucial if the targets are to have the desired motivational effect. The attainment variables must capture the essential aims of the activities of the individuals concerned, while the control variables must make allowances for key causal but uncontrollable factors influencing attainments by individuals. We do not wish to allow for causal but controllable factors in setting targets as we wish to eliminate controllable causes which might otherwise lead to targets which do not reflect the full potential for achievement. The approach assumes that attainment variables are continuouswhile control variables may be a mixture of continuous and categorical variables. Once attainment and control variables are decided upon, the approach proceeds in three steps as follows:

1. Partition the individuals into groups which share a common context;
2. Allow for the possibility of random noise inflating the estimated attainment targets;
3. Estimate attainment targets after having allowed for random noise.

These steps are carried out as follows:

Partition the individuals into groups which share a common context

This step is only used if not all the individuals are strictly comparable on attainment because they belong to different categories. Whether this is or not the case will be borne out by the existence or otherwise of categorical control variables. For example if we believe that all else being equal a salesperson covering a rural area would attain different sales levels on average than one covering an urban area then our control variables would include the categorical variables urban and rural market. In such a case the first step in the target setting procedure would be to separate salespersons into those covering rural and those covering urban areas so that targets can be estimated separately for the individuals of each group.

Allow for the possibility of random noise inflating the estimated attainment targets

In principle, once we have partitioned individuals into groups sharing a common context, we can proceed to set targets for each individual in a group at the best estimated feasible attainment levels which correspond to his/her values on the non categorical control variable(s). However, before estimating targets in this manner we must allow for the fact that the observed attainment levels we are using are subject to random noise. Such noise might have influenced observed attainment levels either upwards or downwards. As we set targets at the best observed attainment levels, we are only concerned not to permit random noise to lead to overestimated feasible attainment levels as this would be demotivating to individuals. We can allow for the possibility of an upward bias in observed attainment levels by discounting what we deem as exceptional attainments. This we can do using an index of attainment exceptionality defined as follows:

An individual’s index of attainment exceptionality is the percentage his/her attainment level(s) are of what we would expect for his/her non categorical control variable levels within the ‘incumbent’ group.

The incumbent group is initially the entire group, but subsequently it consists of the individuals left in the group after those deemed to have too high an exceptionality index are dropped. For any given referent group DEA software (eg Warwick DEA Software) can be used to compute an exceptionality index. The Appendix details how we can compute an individual’s index of attainment exceptionality. The flow chart in Figure 1 summarises how we use the index to progressively drop individuals from the incumbent group, stopping when some user – specified percentage p₁ of individuals of the initial group have either been dropped or have attainment exceptionality index to within 100 –p₂ %, p₂ being user – specified as explained below.

(The level of p₁ reflects the percentage of the original group which the user would deem to be large enough so that if that many individuals attain, get close to, or surpass a given level of attainment, then that level of attainment is genuinely feasible rather than the outcome of an upward bias, induced by random noise in the observed attainments. What constitutes a sufficiently large percentage in this context is subjective in the same manner as the choice of a level of significance is in hypothesis tests. For example, we will run a higher risk of incorrectly concluding that a certain attainment level is genuinely feasible if only 10% rather than 15% of the initial group reach it. On the other hand if we set p₁ too high we run the risk of making the targets less demanding. Similarly, the tolerance level p₂ for deeming an exceptionality index as being ‘close’ to 100% is decided subjectively. The lower the value the more likely it is that we will drop p₁% of the initial group as exceptional before the final incumbent group is arrived at on which attainment targets will be based. The more individuals we drop the less demanding the targets we define. On the other hand a large value of p₂ will run the risk of setting too demanding targets because before we drop any individuals we are likely to have p₁% of the group with index values of between (100- p₂)% and 100% inclusive. One way to see p₁ and p₂ is as parameters which help the user control how demanding the targets set at individual level are.

Estimate attainment targets after having allowed for random noise

Once the original group has been reduced to the final incumbent group in the manner outlined in Figure 1 attainment targets are estimated. For any given referent group DEA software (eg Warwick DEA software) can be used to compute the target levels that would render a unit DEA efficient. The model needed and the resulting targets can be found in the Appendix.

Efficient Peers

One important by product of the target setting procedure outlined is the identification of efficient peers which correspond to each individual whose targets require some improvement in performance their observed performance. The efficient peers are identified after solving the model for estimating that individual’s targets. DEA software will give the identity of the efficient peers of each inefficient unit. The key point is that an underperforming individual and his or her peer set share similar non – categorical as well as categorical control variable values, yet the efficient peers have better attainments. Knowing the efficient peers makes targets transparent and offers indirectly guidance as to how the underperforming individual might achieve better results. The body managing the individuals being set targets can attempt to identify the behaviours which enable the peer set to perform well and strive to induce the same behaviours in the corresponding underperfoming individual.

An illustrative use of the target setting method

We shall draw from Thanassoulis (forthcoming) to illustrate by graphical means the method outlined above. As an example we will consider data from approximately 1000 pupils in 10 London schools. We wish to set each pupil GCSE targets. We use a single attainment variable, the total GCSE score. This was computed by summing the pupil’s GCSE grades, treating a grade A as 8, a grade B as 7 and so on. (Although we use here real GCSE data, in practice schools would use mock GCSE grades to set targets for the GCSE grades proper.) We use four control variables, one non-categorical and the rest categorical. The non-categorical variable is the score the pupil had obtained in the London Reading Test (LRT) on entry to secondary education. This is used as a measure of the pupil’s innate ability. The three categorical control variables are the pupil’s gender, ethnicity and eligibility for free school meals. There is research evidence (see Thanassoulis (forthcoming) that the control variables impact pupil attainment though they are beyond his/her control. (It is not the aim of this paper to identify a definitive set of contextual variables influencing pupil attainment or indeed a definitive set of attainment measures but rather to offer a method for using such variables to the extent that research in the area can identify them.)

To initiate the target setting procedure we first partition the data according to the sex, ethnicity and eligibility for free school meals. Thus the pupils’ performance is compared with pupils within the same context. One of the groups created as a result of partitioning the pupils consisted of 146 boys, not eligible for free school meals and of the same ethnicity. We shall proceed with this group only.

We wish to set the pupils targets which we are reasonably confident that they are capable of achieving and to this end we begin the process of dropping from the incumbent set pupils whose achievements are deemed ‘exceptional’. This requires us first to compute the index of attainment exceptionality of each pupil. We can use for this purpose model [M1] of the Appendix but as we have a single attainment and a single non-categorical control variable we can illustrate graphically the computation of the attainment exceptionality index by that model. First we need to construct the space of feasible solutions to the model.

This is illustrated in Figure 2, for the case where no pupil has been dropped yet as exceptional. The pupils on the boundary A, B, C, D and E are DEA-efficient. There is no empirical evidence that their GCSE scores can be improved. (Pupils A and B had exactly the same LRT and GCSE scores). The pupils within the boundary, for example pupil F, are DEA-inefficient. There is evidence that their GCSE score can be improved. For such pupils the ratio of the observed score to the estimated maximum score for their LRT score is the DEA-efficiency rating

Let us now compute the attainment exceptionality index of pupil D. When D is excluded the efficient boundary moves to ABCFGE as shown in Figure 2 by the broken line. The ratio of the GCSE score of pupil D to the estimated maximum achievable GCSE score for the LRT score of pupil D, is the pupil’s attainment exceptionality index, the maximum being estimated within the set excluding D.

Thus, the exceptionality index for pupil D is given by

For DEA – inefficient pupils their DEA efficiency is their attainment exceptionality index. Thus we only need compute the attainment exceptionality index for DEA – efficient pupils. Once this has been done we proceed to see if we must drop any pupils before arriving at a final incumbent set to be used for estimating targets. To do this we check whether

p₁% of the initial set of pupils have an exceptionality index of 100-p₂% or higher. (For the definitions of p₁ and p₂ see Figure 1.) If so, then we will consider the attainment represented by the initial boundary to be achievable by all pupils and not drop any pupils as exceptional. Otherwise we will drop the pupil with the highest exceptionality index and repeat this same procedure with respect to the reduced set of pupils.

For this example we decided to use p₁ = p₂ =10 and using the procedure described in Figure 1 we arrived at the final incumbent efficient boundary shown in Figure 3, in solid line. After excluding 4 pupils whose results were considered exceptional, 16 pupils have attainment exceptionality index of 90% or more. Thus the incumbent DEA – boundary (A¢ -H¢ in Figure 3), representing 100% DEA efficiency, can be considered as providing a locus of realistic targets for all pupils.

The final step is to compute the target GCSE score on the final incumbent boundary for each pupil and to identify their efficient peers. For example, a pupil P with a LRT score of 30 would be set a target of 31 points in their GCSEs. Pupils D¢ and E¢ would be identified as the efficient peers of pupil P. The former are pupils who offer a LRT score about the same as pupil P but they offer much better attainment.

Conclusion

This paper has presented a general purpose procedure for setting targets at individual person level and for identifying role model individuals who can be emulated by an underperforming individual to improve his/her performance. The key advantages of the procedure are objective nature of the targets, the fact that they are demanding but attainable, and that they can be seen by the individual to take into account factors which are beyond his or her control. For example a salesperson can be set targets which reflect the market size in which he or she is operating and the competition they are facing at a local level. A further advantage of the procedure is that it identifies role model salespersons whom an under performing individual can emulate to improve his/her performance, the role models being especially selected by the procedure to share the same context as the under performing salesperson.

One disadvantage of the procedure is that the targets are set on the basis of relative efficiency. So, if all or nearly all of a sub-group of the population are inefficient in absolute terms, then the targets established for that sub-group will be undemanding in absolute terms. There is also the difficulty of setting targets for those individuals who are exceptional or who are relatively efficient within their sub - group. The procedure works better the more the individuals on who we have data.

The procedure can be seen as an important instrument in enabling agreement between the individual being set targets and those setting the targets in that it represents an objective estimation of realistic targets.

For the interested reader

• Anderson P and Petersen N C (1993) A Procedure for Ranking Efficient Units in Data Envelopment Anaysis. Management Science 39, 10, pp 1261-1264.
• Banker R D, Charnes A and Cooper W (1984), Some models for estimating technical and scale inefficiencies in Data Envelopment Analysis Management Science, 30, pp 1078-1092.
• Department for Education and Employment, (1996 Setting Targets to Raise Standards: A survey of good practice, ISBN 085522 486X. Issued under the ‘Improving Schools’ programme, DfEE, London, UK.
• Thanassoulis E, Setting Achievement Targets for School Children, Education Economics, (forthcoming)

EMMANUEL THANASSOULIS is Professor in Management Sciences at Aston Business School, University of Aston, Birmingham. He has researched and published extensively on the theory and application of Data Envelopment Analysis (DEA) as a method for performance measurement. He has used DEA in a wide range of areas including the provision of education and health services, in the regulation of UK water companies and in the provision of police and local authority services. His work on the use of DEA in education includes its use to guide schools to improved performance, to identify and alter differential school effectiveness, and to set targets for pupils.

GARY SIMPSON is a Lecturer in Operational Research and Applied Statistics at Aston Business School, University of Aston, Birmingham. He obtained his doctorate in Operational Research from Salford University in 1997. His research interests include performance measurements and knowledge management.

First published to members of the Operational Research Society in OR Insight April - June 1999