6 Assessing productive and interactive skills Personal refl ection

 
Assessing productive and interactive skills 
143
easy to understand and apply. Unfortunately, as Edgeworth 
demonstrated over 100 years ago (see 
Chapter 4
), individual raters 
can have worryingly incompatible ideas about the standards of 
performance represented by different scores. More rigorous 
alternatives are needed.
Error counts
In 
Chapter 4
, it was noted that it is not possible to have valid and 
useful assessments without a reasonable level of reliability. It would 
therefore seem better to replace fuzzy subjective judgements with 
something more concrete. At one time, error counting seemed to 
promise a more objective basis for scoring. Instead of giving scores to 
reward performance, this scoring process involves deducting points 
for each error or each sentence containing an error. However, this 
does not really solve the problem of reliability. First, some errors are 
obviously more serious than others, yet judgements about the relative 
seriousness of errors would reintroduce the subjectivity that error 
counting was intended to eliminate. Second, error counting is limited 
to mechanical accuracy: it takes no account of how well a text is 
constructed beyond the sentence level, or of how well an overall 
message is communicated. Third, error counting rewards very cautious 
language use. Responses made up of simple sentences could outscore 
those made up of more ambitious language. This was not the kind of 
washback that assessment designers wanted to promote.
Task 6.3
Look at these two sentences from student essays:
Eggs are one of the only foods contained natural occurring vitamin D. 
Eggs is naturally foods that vitamin D is contain.
Both sentences contain errors, although most judges would agree that 
the fi rst communicates more successfully. In some error counting 
systems, both would lead to deduction of one point. In others, errors 
would be judged for seriousness, reintroducing subjectivity.
Now try counting the errors in each. How many can you fi nd? Would 
other judges agree with you?
Checklists
The key advantage of using checklists to score spoken or written 
performance is that they direct attention to aspects of performance 
that are considered by the designers to be important. In other words, 
they communicate the assessment construct to the raters. They are