O n f e r e n c e

Sentence Classification and Evaluation 
Given the 
k
-gram sentence representation described so far, the next challenge was 
selecting an appropriate machine learning classifier that can handle a large dataset (over 
50,000 training points) in an efficient manner. In this paper, we report results using the 
Maximum Entropy method (Nigam et al., 1999) as implemented in the OpenNLP library 
(Baldridge et al., 2002), which performed better than Support Vector Machine and 
Multinomial Naive Bayes methods applied to our dataset. 

---

-120- 
2014 CALL Conference 
LINGUAPOLIS
www.antwerpcall.be 
During training, a separate Maximum Entropy classifier was learned for each value of 
k
(1 through 3) and for each type of sequence (stemmed words and part-of-speech 
tokens). Thus, when evaluating the algorithm, an ensemble of six classifiers was trained 
for each of the two classification tasks (i.e., moves and steps). To classify a new 
sentence, the outputs of the six classifiers were combined using a weighted combination 
in which the weights corresponded to prior estimates for the performance of each 
classifier (Lam & Suen, 1995).
The set of sentences in our dataset was highly unbalanced with respect to both moves 
and steps, which poses a special challenge to machine learning approaches as it often 
results in classifiers that are biased towards the majority class and away from the 
minority class. To tackle this problem, during the training stage, the sentences from 
steps that had less than 1,500 examples were oversampled so that for each step there 
are at least 1,500 training sentences.
To evaluate the proposed classification framework, we performed 10-fold cross 
validation. It must be noted here that the unbalanced nature of the dataset also 
complicates the evaluation; therefore, accuracy is not the most appropriate performance 
metric, as simply voting for the majority class can lead to high accuracy but is not better 
than randomly guessing. Therefore, we report performance in terms of Cohen’s kappa 
coefficient (Cohen, 1960), which is a statistic that compares the classifier’s accuracy 
against chance accuracy. A kappa of 0.0 would indicate a classifier that is no better than 
chance while a kappa of 1.0 would correspond to perfect classification.