2.1 Essay set selection

The corpus is in the form of 13,000 essays, totaling 2.9 million words—more than twice the length of Proust’s In Search of Lost Time. The length, however, was not as immediate an obstacle as the composition, shown in tbl. 1. The eight essay sets were not only responding to different prompts, but were of different lengths and genres, written by students of different grade levels, and, most importantly, scored using incommensurate rubrics and scoring protocols.

Limiting myself to a single essay set would have produced a somewhat feeble model, as words idiosyncratic to the topic in question became artificially elevated in importance. In the end, I combined sets 3 and 4, which both consisted of expository essays written by tenth graders, graded on a scale from 0 (worst) to 3 (best). These scores are holistic, i.e., not broken down into categories representing grammar and mechanics, relevance, organization, etc., which makes them easier for a model to predict.

2.2 Data cleaning

The scores are broken down, for each essay set, into “domain scores” representing the valuations of the individual scorers. In the interest of having a single number to try to predict, I combined these scores by taking the mean:

# If only one score exists, use that. Otherwise, take the mean of both scores.
essays["score"] = list(map(np.nanmean, zip(essays["domain1_score"], essays["domain2_score"])))

We can then look at the way scores are distributed among the essays in our chosen subset.

In fig. 1, we see that the scorers of the fourth essay set were somewhat less lenient than those grading the third, the latter of whom awarded the highest score to a full quarter of the papers, and the lowest score of 0 to only 39 unhappy test-takers. Putting these together, we have a roughly normal-looking distribution, with many ones and twos, and fewer zeroes and threes. This gives us a baseline to use for the modeling below: a dumb model, which assigned every essay to the plurality score class, giving every essay a score 1, would have an accuracy of 35%. This is the number our models must beat.

The essays themselves are in little need of cleaning: they are hand-transcribed from the originals, and have been anonymized by replacing named entities, including names, dates, addresses, and numbers, with enumerated placeholders.

2.3 Data exploration⁵

A basic exploration of the essays shows some striking patters. For example, as fig. 2 illustrates, score is highly correlated with length at $R^2 = 0.51$, meaning that over half the variation in score can be explained by variation in length. In other words, all else held equal, adding 82 words corresponds with a point increase in score.

One interesting thing we see is that, despite the correlation, there are many essays earning top marks that are almost impossibly short. The following are recorded in the dataset as having earned a top score (both prompts instructed students to use examples from the texts):

The features of the setting affect the cyclist in many ways. It made him tired thirsty and he was near exaustion [sic].⁶

Because she saying when the @CAPS1 grow back she will be @CAPS2 to take the test again.⁷

Reserved need to check keenly⁸

That that gnomic last “essay” (yes, that’s the whole text!) earned a coveted score of 3 is almost certainly an error, though the source of the error (the recording of the scores, the compilation of the dataset, or the scoring process itself) is as mysterious as the cryptic phrase’s meaning. However, there doesn’t seem to be an objective way of pruning these aberrant rows from the dataset, necessitating my leaving them in.

Other measures are telling as well. For instance, we can look at the rate of misspelled words, by tokenizing with spaCy, and counting each token that is not in a given wordlist.⁹

import spacy
nlp = spacy.load("en")

# Generate wordlist
with open("/usr/share/dict/words" , "r") as infile:
    wordlist = set(infile.read().lower().strip().split("\n"))

# Number of words that are misspelled
essays["misspellings"] = len([word for word in nlp(essays["essay"])
     if not word.is_space
     and not word.is_punct
     and not word.text.startswith("@") # named entities
     and not word.text.startswith("'") # contractions
     and word.text.lower() not in wordlist])

# Percentage of words misspelled
essays["misspellings"] /= essays["tokens"]

The results, in fig. 3, are curiously complementary to those in fig. 2: the rate of misspellings is practically the same across score classes ($R^2 = 0.002$), but those at the extremes, with 10% or more of their words misspelled, are overwhelmingly likely to be low scorers.

The question of assessing prompt-relevance is trickier. One way of tackling it is to calculate the document vector of the story to which the students are responding, and calculate its cosine similarity with the document vector of each essay. We can see the results in fig. 4.

The results aren’t bad ($R^2 = 0.28$), especially considering the outliers for score 3 are the same bizarrely short essays we saw above, including our Delphic “reserved need to check keenly”.

At this point, we must ask what the value of this metadata is. The ETS claims that its e-rater accounts for prompt-relevance, as well as:

• errors in grammar (e.g., subject–verb agreement)
• usage (e.g., preposition selection)
• mechanics (e.g., capitalization)
• style (e.g., repetitious word use)
• discourse structure (e.g., presence of a thesis statement, main points)
• vocabulary usage (e.g., relative sophistication of vocabulary)
• sentence variety
• source use
• discourse coherence quality

While it would take a sophisticated natural language parser to incorporate these details into our model, we may be able to approximate these things using metadata as proxies. Type–token ratio, for instance, could stand in for “repetitious word use”, and vector similarity to the prompt for relevance. As an alternative to parsing for narrative structure, I included a count of “linking words” that would likely signal a transition between paragraphs,¹⁰ but this bore little relationship to the human scorers’ judgments ($R^2 = 0.0004$). Finally, as a proxy for sentence complexity, I used spaCy to parse the syntactic trees of each sentence, and took the longest branch, thus rewarding complex sentences with prepositional phrases and dependent clauses.

# Depth of longest branch in dependency tree
essays["max_depth"] = [max([len(list(token.children))
                       for token in nlp(essay)])
                       for essay in essays["essay"]]

This fared somewhat better as a metric: $R^2 = 0.13$. Finally, I tried to measure “relative sophistication of vocabulary” by quantifying the uncommonness of the words used. I did this by building a word frequency list from the 14-million-word American National Corpus, the details of which are in Appendix B. This correlated well with score ($R^2 = 0.40$), although it was no doubt standing in somewhat for length.

3.1 Classical models¹¹

As hinted at by the high $R^2$ scores above, we can get fair prediction scores by modeling on metadata alone. The first step, after splitting our essays into train and test sets, is to standardize the data by scaling to $z$-score. I then ran principal component analysis (PCA) on the data, because many of the columns (e.g., type count and token count) encoded essentially the same information in parallel. The PCA transformation extracts those components which encode the greatest variance; together, the ten components extracted accounted for 98% of the variance within the metadata.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y)

# Standardize to z-score
ss = StandardScaler()
X_train_sc = ss.fit_transform(X_train))
X_test_sc = ss.transform(X_test))

# PCA-transform
pca = PCA(n_components=10)
Z_train = pca.fit_transform(X_train_sc)
Z_test = pca.transform(X_test_sc)

The modeling itself is fairly straightforward. I modeled the data both with and without the PCA transform, and found the latter to have a slight edge, although all models achieved similar test scores (tbl. 2).

from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.ensemble import AdaBoostClassifier, ExtraTreesClassifier
from sklearn.metrics import cohen_kappa_score

gnb = GaussianNB().fit(Z_train, y_train)
svm = SVC(kernel="rbf", C=1).fit(Z_train, y_train)
ext = ExtraTreesClassifier().fit(Z_train, y_train)
ada = AdaBoostClassifier().fit(Z_train, y_train)

for model in [gnb, svm, ext, ada]:
    print("Test score:", model.score(X_test_sc, y_test))
    print("Test kappa:", cohen_kappa_score(model.predict(X_test_sc), y_test),
                                           weighting="quadratic")

I also included the weighted Cohen’s kappa,¹² which was the metric used for the original competition, although Cohen’s kappa is typically used to compare model results to each other, not to a gold standard.

The support vector machine and ExtraTrees models performed slightly better than their rivals, and in fact made similar predictions to each other ($\kappa = 0.81$). We should also take into account that on essay sets 3 and 4, human graders agreed only about 75% of the time, with a weighted Cohen’s kappa of 0.77 and 0.85, respectively.¹³

3.2 Recurrent Neural Network¹⁴

One of the state of the art tools in text processing is the recurrent neural network, into which ordered data is fed in series, and the model is retrained on prior data, in order to learn things about the sequence. The first step to doing this with word data is to convert the words to numerical indices (so “a” becomes 1, “aardvark” becomes 2, “Aaron” becomes 3, etc.), then padding them to be of equal length.

from tensorflow.keras.preprocessing.sequence import pad_sequences

# Define vocabulary
vocab = set(token.text for essay in essays["essay"] for token in nlp.tokenizer(essay))

# Convert words to numerical indices <https://www.tensorflow.org/tutorials/text/text_generation>
word2idx = {u: i for i, u in enumerate(vocab)}

# Convert essays to vectors of indices
X_vector = [[word2idx[token.text]
            for token in nlp.tokenizer(essay)]
            for essay in essays["essay"]]

# Create padded sequences
X_vector = pad_sequences(X_vector)

# Split into train and test sets
X_vector_train, X_vector_test = train_test_split(X_vector);

This then goes into an embedding layer, which condenses it into a dense vector.

With neural networks, it is possible to include both the vectorized document and the metadata, by processing the former in a GRU or LSTM layer, concatenating the latter to its output neurons, and processing both in a regular perceptron structure.¹⁵ Following the example in this blog post, I implemented the code below:¹⁶

from tensorflow.keras.layers import Dense, GRU, Embedding, Input, Bidirectional, Concatenate
from tensorflow.keras.models import Model

# Define inputs
vector_input = Input(shape=(X_vector.shape[1],)) # Text vectors, in series of length 1,000
meta_input = Input(shape=(X_meta.shape[1],)) # Scaled metadata (types, tokens, etc.)

# Embedding layer turns lists of word indices into dense vectors
rnn = Embedding(
        input_dim = len(vocab),
        output_dim = 128,
        input_length = X_vector.shape[1],
    )(vector_input)

# GRU layers for RNN
rnn = Bidirectional(GRU(128, return_sequences=True, kernel_regularizer=l2(0.01)))(rnn)
rnn = Bidirectional(GRU(128, return_sequences=False, kernel_regularizer=l2(0.01)))(rnn)

# Incorporate metadata
rnn = Concatenate()([rnn, meta_input])

# Define hidden and output layers
rnn = Dense(128, activation="relu", kernel_regularizer=l2(0.01))(rnn)
rnn = Dense(128, activation="relu", kernel_regularizer=l2(0.01))(rnn)
rnn = Dense(4, activation="softmax")(rnn)

# Define model
model = Model(inputs=[vector_input, meta_input], outputs=[rnn])

# Fit model
model.fit([X_vector_train, X_meta_train_sc], y_train,
          validation_data=([X_vector_test, X_meta_test_sc], y_test))

The results are surprisingly close to the models in sec. 3.1 above. Amending our previous table:

It seems that the metadata was more valuable in predicting test scores than the vectorized documents—or else, that the RNN couldn’t make better use of the two than a support vector machine could of the one. Nevertheless, I have shown that using a few key linguistic metrics, we can train a simple model to predict essay scores in fairly good agreement with human scorers.