Consistent Marking
Andrew Hatt, LCM's Qualifications Officer, explains how it is ensured that RGT examiners award marks reliably and consistently…
One of our primary considerations as an examinations board has to be the reliability and consistency of marks, so that candidates can be assured that, presenting an identical performance to different examiners on different days, they would attain the same mark each time.
In performance exams, many factors combine to ensure comparability between examiners - examples include: robust training of examiners; periodic performance review of examiners; compulsory bi-annual retraining courses for all RGT examiners; the use of highly detailed assessment criteria to award marks.
STANDARDISATION PROCEDURES
An additional important factor in ensuring consistency is post-examination standardisation, the primary aim of which is to compare an examiner's results for a particular exam centre with expectations, based on historical statistical data.
After each set of results is inputted at the RGT office,
specialised exam-standardisation software calculates
whether the (mean) average mark for that results set
deviates from the 'universal average' by more than a
reasonably expected amount. (The 'universal average' is
the mean average of all RGT exams that have taken place
since the standardisation system was installed some years
ago.) Only if the average mark for the results set falls
into the 'acceptable' range (currently set at five marks
above or below the universal average) are the results
approved for release. If the average mark deviates from
the universal average by more than five marks, the results
are 'red-flagged' and referred to the RGT Examinations
Director. Further analyses of the results, using the
specialised exam-standardisation software, are then
undertaken. This analysis includes assessing whether the
examiner's average mark for the centre varies from the
examiner's own historical average by more than a
reasonably expected amount, as well as whether the
examiner's average mark for the centre varies from the
centre's historical average by more than a reasonably
expected amount.
This is not intended to suggest that we would expect the results of all centres to fall into the same pattern, and is certainly not meant to imply that marks would automatically be altered if they were not as statisically expected. Indeed there are many factors which might lead to a high or low average mark, including the size of the sample, a particularly experienced teacher at a centre, a preponderance of low or high grades, et al. Rather, it can alert us to query the results and the Examinations Director may lead an investigation with the relevant examiner, and teachers, if felt necessary. In many cases the results are approved following the extra information gathered from this procedure, but in very rare cases this process might lead to a moderation of the marks for that centre.
EXAMINER MARKING
Another significant use for the standardisation information is that we can monitor and evaluate an examiner's performance over time: a detailed profile of each examiner's marking emerges, and any tendency to deviation from standard marking patterns can be spotted and advised upon expeditiously.
To avoid any pre-existing associations with numbers (e.g. 6 out of 10 might imply 'quite good' to some people, but is actually below the pass mark in our exams) examiners do not initially award numerical marks for each section of the exams. Instead a very detailed description of what is expected at each level of attainment band (i.e. distinction, merit, pass, below pass upper level, below pass lower level) for every section of each exam is provided to the examiners and the examiner makes a note of which of these the candidate's performance matched. The examiner then uses a special conversion table to convert that level of attainment band into the appropriate mark for that section of that particular grade.
RELIABILITY
There is no such thing as a 'harsh' or 'lenient' RGT
examiner, as we go to great lengths to ensure that all
examiners award marks objectively using exactly the
same assessment criteria, and the marks awarded are
continually analysed and monitored via our exam
standardisation procedures described above. As a result
of all these procedures we can be sure that RGT
candidates will always be marked fairly and consistently
regardless of where they take their exam and no matter
which examiner examines them.
