RAE Analysis
The University League table of RAE results that were produced in December by the media have been a source of frustration to Universities as they have been based upon what several universities believe to be a flawed approach. Harri Ap Rees, Head of Planning at the University of Surrey, explains why a different approach, illustrated here by a new League Table based upon Z-scores, is a better scientific basis for comparison.
Whilst individual subject scores are accurate, the mean point of the Grade Point Averages (GPA) varies considerably between subjects, as do the range of GPAs. The maximum GPA score varies from 3.60 (Communication, Cultural and Media Studies) down to 2.75 (Agriculture, Veterinary Science, Accounting and Finance, Sports-Related Studies, Asian Studies) and the average varies from 2.88 (Cancer Studies) to 2.04 (Allied Health Professions and Studies). There is considerable variation even within groups of subjects – for example in engineering subjects the maximum varies between 3.35 (Civil) to 3.05 (Electrical and Electronic Engineering and Metallurgy).
Given such variations it is impossible to provide any degree of meaningful comparison in the top scoring and median performance between very different subjects. This is in contrast to the previous system where the best ratings were all uniformly 5* and it was at least notionally possible to compare the best ranked institutions across all subjects. The impact of this variation in score achieved between Units of Assessment (UoAs) has had a profound effect on some institutions and their ranking in the RAE league tables. The mix of UoA’s in any institution had an important effect on the ranking. The methodology used is flawed in that it fails to take into account that there is a substantial variation in the difficulty involved in achieving high level performance in different UOAs.
Our view is that a more accurate way of ranking institutions is to employ the widely used Z-score methodology, as the Times does, combined with a full time equivalent FTE weighting. The Z score shows how far (and in what direction) an item deviates from its distribution’s mean, expressed in units of its distribution’s standard deviation, which overcomes the problem of UOAs having varying means, maxima and minima. The Planning Department with support from Peter Williams in the Maths Department have produced a Z-score ranking table. The League table we have constructed compares the Times Higher Education (THE) rank with the new Z-score ranking. The result, as you will see, tends to favour the 94 group compared with the Russell group. Some 1994 group institutions are ranked higher in the Z-score table and conversely some institutions are ranked lower. In the case of the University of Surrey, using the Z-score methodology results in the University being ranked 25th rather than 35th in the table constructed by the THE.
Download the RAE League Table in Excel format
Harri Ap Rees
Head of Planning
University of Surrey
