Examining the Effectiveness of Automated Essay Scoring Systems for Nigerian Secondary School Students

Automated essay scoring systems offer the potential to address the assessment bottleneck created by teacher shortages and large class sizes in Nigerian secondary schools, yet their accuracy and fairness for essays written by Nigerian students in Nigerian English have not been examined. This study empirically examines the effectiveness of automated essay scoring systems for Nigerian secondary school student writing. A corpus of 4,200 Nigerian WAEC-style essays from SS2 and SS3 students across six states was collected through collaboration with schools in the South East, South West, and North Central zones, with expert human scores (6-point rubric) assigned by two trained raters per essay. Three AES approaches were evaluated: a feature-engineered model (SVR with linguistic features), a BERT-based neural scorer fine-tuned on the Nigerian corpus, and a zero-shot GPT-4 scoring approach. Inter-rater reliability between human scorers was Cohen's kappa of 0.76. BERT fine-tuned on Nigerian essays achieved a quadratic weighted kappa of 0.71 against human scores, compared to 0.64 for the feature-engineered model. GPT-4 zero-shot scoring achieved 0.68. Error analysis revealed systematic underscoring of essays with Nigerian English idioms and cultural content. The study fills a documented gap in AES evaluation for African student populations and recommends the fine-tuned BERT model as a teacher aid tool, cautioning against high-stakes automated scoring until demographic bias testing is completed.

Keywords: automated essay scoring, Nigerian secondary school, BERT, WAEC, natural language processing