TECHNICAL ANNEX

How is a contextual value added score for a pupil calculated?

Our existing value added measures arise from a national 'median line', plotting the relationship observed between attainment at the start of the period over which progress is being assessed ('input point scores') and attainment at the end of that period ('output point scores'). The value added score for each student is the difference (positive or negative) between their own 'output' point score and the median (middle) output point score achieved by others with the same or similar 'input' point scores.

To take account of contextual factors as CVA seeks to do, we need a more complex model, but the principle remains the same as for the median line. We obtain a prediction for the pupil based on nationally observed patterns; their contextual value added score is then the difference (positive or negative) from this prediction. The particular technique used in this pilot is called multi-level modelling (MLM).

The Pupil Achievement Tracker will calculate these scores for you, however we have provided an additional resource (a ready reckoner) available from the Pupil Achievement Tracker website that will enable you to see how a pupil prediction is built up. Users familiar with regression equations may wish to use the coefficients provided with the ready reckoner.

The screenshots and charts included in this document are from the KS2-4 mainstream schools ready reckoner. Ready reckoners are also provided for KS1-2, KS2-3 and KS3-4.

We can break the calculation into four stages

1. We obtain a prediction of attainment based on the pupil's prior attainment.
2. We then adjust this prediction to take account of the pupil's set of characteristics.
3. For KS2-3, KS2-4 and KS3-4 we adjust further by taking account of school level prior attainment.
4. We obtain a value added score by measuring the difference (positive or negative) between the pupil's actual attainment and that predicted by the model.

1. Prior Attainment

Even when we include contextual factors we find that prior attainment is by far the strongest predictor of outcomes.

In the value added measure currently in Achievement and Attainment tables we take the average point score (based on levels in English, mathematics and science) and use it as our input. Similarly, in contextual value added we use an average point score. We also look at how a pupil's prior attainment in individual subjects differs from their overall average.

For point scores at Key Stage 2 and Key Stage 3 we use fine grades (see box below for further guidance.)

We use the model to obtain a prediction based on prior attainment. The ready reckoner will allow you to see the prediction based on a particular set of fine grade point scores. An example is given below.

2. Characteristics

We make adjustments to pupil predictions if the pupil has particular characteristics. The adjustment is the effect of that characteristic on attainment after taking account of all the other factors.

To see the adjustment made for each characteristic consult the ready reckoner.

3. School level prior attainment

When looking at KS2-3, KS2-4 or KS3-4 value added we observe that, even after allowing for pupil prior attainment and characteristics, the average level and spread of attainment on entry to a school will also affect the predicted outcome for a pupil.

When calculating contextual value added we take the straight average of pupil prior attainment average point scores (using fine grades). We include all those pupils who would be included in a non-contextual value added score.

The standard deviation measures the average "spread" of prior attainment around the average. It is calculated by taking the difference between each pupil's result and the school average and squaring it. The average of these squared differences is called the variance, and the square root of the variance is the standard deviation, or the "spread of prior attainment" used in the model.

The ready reckoner will show you typical values for these school level variables and how any value affects a pupil prediction.

4. Obtaining the pupil contextual value added score

A pupil's contextual value added score is the difference (positive or negative) between their predicted and actual attainment.

The ready reckoner can perform this calculation for you. Charts similar to the one above but for all pupils in your cohort are available through the Pupil Achievement Tracker or your PANDA.

Moving from pupil to school scores

The Shrinkage Factor

When calculating a school score for value added we take the average (mean) of all pupil scores within that school. With contextual value added we again take the average but we add an extra step known as the shrinkage factor. The shrinkage factor is determined by the number of pupils in your cohort (see chart below). It helps us to better estimate contextual value added for schools with small numbers in the calculation.

We multiply the average of all pupil scores by the shrinkage factor to obtain our final school contextual value added measure.

Confidence Intervals

We can use the contextual value added score as a measure of school effectiveness, but as with value added it is based on a given set of pupils' results for a particular test paper on a particular day.

The school could have been equally effective and yet the same set of pupils might have achieved different results on the day. And the school would almost certainly have shown slightly different results with a different set of pupils, even with the same levels of prior attainment. Hence, although the contextual value added score is based on all pupils in the school cohort (not just a sample of them), this degree of uncertainty should be taken into account if interpreting the figures as estimates of an 'underlying' measure of value added.

The uncertainty of a contextual value added score as a measure of school effectiveness can be presented as a confidence interval. This is a range of scores within which we can be statistically confident that the "true" school effectiveness will lie. Like the shrinkage factor, the size of the confidence interval is determined by the number of pupils in your calculation.

Smaller schools have larger confidence intervals, even after applying the shrinkage factor, since we are estimating the score on a smaller number of results.

The ready reckoner will enable you to see how different sizes of cohorts lead to different shrinkage factors and confidence intervals. You will also be able to see how the shrinkage factor affects a school score.

[ Home Page | Previous page | Next page ]