6 Assessing productive and interactive skills Personal refl ection

 
Assessing productive and interactive skills 
155
Such concerns have led other developers to take a third, more 
qualitative, empirical approach:
 
data-driven
 
scale development (Fulcher, 
1987). This involves the systematic analysis of performances by learners 
(from the population who will take the assessment) on the relevant tasks. 
Key features that help to discriminate between learners are identifi ed from 
these sample learner performances. These features are then incorporated 
into descriptions that can be used to build rating scales. Unfortunately, 
according to Fulcher 
et al.
(2011), a major shortcoming of this method is 
that it tends to generate very detailed level descriptors that raters fi nd too 
complex and time consuming to be practical for routine use.
Although there has certainly been a major shift towards more data-
driven scale construction and validation, the value of expert judgement 
and the wealth of knowledge which experts can bring to the 
development process should not be discounted. The best methods for 
rating scale development are said to take advantage of the strengths of 
a range of intuitive, quantitative and qualitative approaches. The 
CEFR (Council of Europe, 2001, p. 207) advocated ‘a complementary 
and cumulative process’ that brings together all three.
Rater training
A well-written scale helps to defi ne the test construct for the raters so 
that they are guided to features they should attend to in a performance. 
This can enhance levels of agreement. However, it is not enough 
simply to provide raters with scales. It is well established that 
interpretations of any linguistic description, however complete and 
clearly written, will vary. This means that raters have to be trained to 
use scales effectively and consistently. Quality control procedures are 
needed to ensure that this happens.
Before a rating session, or at fi xed times during a school year,