O n f e r e n c e

the right balance between experimental control and 
ecological validity
. Ecological validity is concerned with the representativeness of the 
study and refers to how well a study reflects real-life learning situations. Ecological 
validity is also linked to external validity (Nation & Webb, 2011), which refers to the 
generalisability of the study’s findings. In order to safeguard ecological validity 
in our 
research, a series of elements were considered. For instance, we decided to work with 
authentic video materials (i.e. original TV reports that were created for a native language 
audience). However, this implies that a series of parameters cannot be controlled for, 
which may subsequently influence the results.
Using authentic clips makes it for instance difficult to have full control over the target 
words (i.e. the words included in the posttest to assess vocabulary learning 
effectiveness) since a s
eries of parameters such as the target words’ frequency of 
occurrence, length, concreteness, part of speech, etc. cannot be held constant. That is 
why studies on vocabulary learning from written input (reading) have often included non-
word selections, even though this might negatively affect the ecological validity. 
However, including non-words in authentic video does not seem appropriate as it would 
considerably harm the original soundtrack. Yet, in order to take into account parameters 
such as target word frequency of occurrence or word length, more complex statistical 
analyses can be considered (cf. 
infra
). There are inevitably a series of methodological 
challenges that apply to the specificity of the domain under investigation. When 
conducting experimental research on 
vocabulary acquisition
, it is crucial to administer 

---

-260- 
2014 CALL Conference 
LINGUAPOLIS
www.antwerpcall.be 
tests at different levels of sensitivity (Nation & Webb, 2011) in order to take into account 
the fact that acquiring a word consists of a series of interconnected partial knowledge 
components such as form, meaning, and use (Nation, 2001). Administering different 
tests requires carefully sequencing the test order (e.g. Webb, 2007) in order to avoid 
that test administration procedures produce a learning effect. In addition, while delayed 
posttesting may provide us with interesting insights into the durability of the memory 
trace, there are a number of elements which make it difficult to include delayed 
vocabulary posttesting. When using real target words, for instance, it is possible that 
students look up a word in the period between the learning session and the delayed 
posttest. In addition, it is also plausible that the administration of a test (e.g. a multiple 
choice meaning recognition test) helps students to establish or consolidate form-meaning 
connections. It is therefore difficult to determine whether the scores on the delayed 
posttest reflect long-
term effects of the treatment, students’ additional learning effort, or 
a learning effect produced by the immediate posttests. The inclusion of parallel groups 
could be considered in order to facilitate the use of delayed posttesting and take into 
account the potential impact of the testing procedures on learning outcomes. By carefully 
distributing the different tests over two or more parallel groups, we could thus make it 
possible to control for test administration and test order effects, even though this might 
make the research design more complex.
Other challenges are related to the type of