Understanding the autoscoring process
When a student response is Finished it is checked in our NZCER Assist platform before being sent for autoscoring. The first step in this process is checking if the written response is Too Short for scoring.
Our current definition of Too Short (for scoring) is one or more of the following conditions.
- Fewer than three complete sentences
- Fewer than 50 characters
- Fewer than 3 elements of punctuation
If a written response is considered Too Short for scoring it will not be sent for autoscoring, and the response will be tagged for review on the Rubric List report.
Under the Actions screen you can reopen a token for a Too Short response. The student can re-enter their token and return to the Writing and Editing screen to continue with their writing. There is also the opportunity to set the response as Confirm Unable to be marked if you do not want the student to revisit their writing task.
If a piece of student writing can be sent for scoring, there are two possible outcomes.
1. Successful Autoscored result
2. Needs Review result
In both outcomes scores are provided for each rubric element. However, a Needs Review result requires action by a School Admin or Teacher user in Assist. This sees the School Admin or Teacher accessing the Manage Marking screen and manually marking the response or setting it as Confirm Unable to Mark.
Any Needs Review outcome will have a Comment to tag the issue with the written response (as detected by our autoscoring process). Please note that an autoscore will be displayed for these responses. Assist users can choose to accept the autoscore when reviewing the response.
The possible Needs Review comments are:
- Off-Topic – writing does not align with the writing task
- Repetitious – writing is repetitious
- Unknown words – writing contains too many unknown words
- Insufficient development – writing has inadequate development
- Bad syntax – writing contains major syntax problems
- Copied question – writing substantially copied the writing task
We will be monitoring autoscoring and unsuccessful autoscoring events and working with our scoring provider to identify issues. As we gather more student data and analyse where and how schools have manually marked, or updated autoscoring marks, we will be able to improve autoscoring performance.