6 Assessing productive and interactive skills Personal refl ection

Approaches to developing rating scales
Personal refl ection
If you were asked to produce a rating scale for scoring how well a 
learner can hold a casual conversation in English, what steps would 
you include in your development plan?
How would you judge whether or not use of your rating scale had led 
to accurate scores? 

---

 
Assessing productive and interactive skills 
153
Taking up a distinction made by North (2000), Fulcher (2003) 
categorised recent methods of scale development within two general 
approaches: 
intuitive
and 
empirical
. Intuitive rating scale development 
is carried out by appointed experts (language teachers, applied linguists 
or language testers), working either individually or in a group. These 
experts prepare rating scales according to their intuitions, established 
practice, a teaching syllabus, a needs analysis or some combination of 
these. The resulting scale may be refi ned over time according to 
experience or new developments in the fi eld. 
There is no direct correlation between the use of intuition or 
empirical data in scale development and the ability or behavioural 
orientation of the scale. Intuitively developed scales include both more 
ability-oriented scales, such as the scale for socio-linguistic ability 
proposed by Cohen and Olshtain (1993), and more behaviourally/real 
world-oriented scales such as the Foreign Service Institute scales or 
their successors, the Interagency Language Roundtable (ILR), 
American Council for the Teaching of Foreign Languages (ACTFL) 
and NATO STANAG 6001 scales.
Fulcher (2003) was critical of traditional intuitive approaches to 
development. He argued that they lack any persuasive evidence for the 
assertions they make about differences between assessees. He contrasted 
this with what he called 
empirical methods
, which