An Introduction to Applied Linguistics

6.1.1 Educational (including the psychometric) factors
ELTS was a proficiency test used in an educational setting, that is at entrance to a
British university. The educational questions that needed investigation were: to what
extent the test was being used at undergraduate and postgraduate levels, and for
temporary attachment by senior academics; and whether it was suitable at these
different levels; to what extent it was in use (and found suitable) in different faculties,
and whether it was (as had been intended) being used differentially so that test scores
deemed proficient in one faculty (or university) were rated not sufficiently proficient
in any others. Again, many universities had provision for remedial English tuition for
their foreign students; what was the extent of this provision and what use was made
by those involved of the ELTS results? Both sensitivity to the ways in which tertiary
institutions operated and an understanding of their role and complexity was called
for in the project.
In addition, there were other skills that were required. One of these was the
analysis of the ELTS test in terms of its relevance to the content of the specialist areas
it purported to test. For example, what connection, if any, was there between the test
materials in the life sciences module and the textbooks, lectures and so on in
university departments of biology, bio-chemistry, botany and so on? And how far
were these likely to be constant at the different levels, under- and postgraduate? For
76
An Introduction to Applied Linguistics
02 pages 001-202:Layout 1 31/5/07 09:31 Page 76

---

analyses of this kind it was necessary to draw on the expertise of those skilled in
teaching life sciences at the different levels. The applied linguist could not be
expected to carry out the content analysis but he/she would expect to interpret the
results.
Another essential skill, which appears to be unique to the testing area but in
practice is relevant everywhere in applied linguistics research, is that of familiarity
with measurement (the psychometric aspect of educational studies) and with
elementary statistics, to which must also be added the skill of data entry and of
statistical analysis by computer. Those applied linguists who are consulted about
testing projects (as with this ELTS project) are likely to have all three skills, the
psychometric, the statistical and computer literacy. If not, then they must call on
colleagues or external consultants. 
But for the purposes of understanding the working of any test these skills are all
essential since they provide the necessary evidence about the test’s workings, often
called its validity and reliability. Using these skills it was possible to examine the
extent to which ELTS gave its users the decisions they needed, in particular whether
it enabled them to make the right choices of candidates, those with adequate English
for their studies. 
It was also important to investigate whether there was even-handedness across
modules, so that one module (say the life sciences module) was no harder than any
other (say the social studies module). If it was harder, then it was possible it was right
that it should be, but this was a matter for empirical investigation: if the English of
life sciences is truly more difficult than the English of social studies, then it is
appropriate that the tests should mirror this distinction. If not, and one was in fact
more difficult than the other, then we had established a case of irrelevant bias which
should be corrected.
ELTS contained four parts, tests of the four skills: reading, writing, listening
and speaking. The results offered a profile score (skill by skill) and a global score,
combining the four skills. It was important to determine the extent to which the
profile scores provided useful (and usable) information to the test users (university
departments etc.). Here again what was needed was a statistical analysis across a wide
range of candidates of the various scores and a survey (through interview, question -
naire etc.) of the actual use of these different modes of reporting to the institutions.
Since ELTS information was already being used in a large number of tertiary
institutions, it was necessary to check how the information was being used. Were the
so-called profile scores (separate scores for the four skills of reading, writing, listening
and speaking) being used or were institutions interested only in the overall global
score?
By the time of the evaluation teachers of English as a foreign language (EFL) were
already beginning to consider the needs of those candidates intending to take the
test. Inevitably, courses in EFL, both the more general and those specifically designed
to provide pre-ELTS tuition, were taking account of what the test involved. The
question for the evaluators was whether such washback from the test was beneficial
in terms of its influence on EFL teaching or whether it was likely to exert a largely
Applied linguistics and language learning/teaching 77
02 pages 001-202:Layout 1 31/5/07 09:31 Page 77

---

negative influence, for example by overstressing the practice of multiple-choice test
questions.