This open-source tool automatically scores orthographic listener transcripts for words correctly identified. This is useful in speech perception research, such as studies that examine listener understanding of disordered speech or speech in background noise. The program uses a flexible number of rules that determine whether a response set of words (i.e., listener transcripts) match a target set of words (i.e., speech corpus). At the most basic level, Autoscore scores words in the listener transcript as correct if they match the words in the target phrase exactly (regardless of word order). Additional individual scoring rules can be applied or removed, depending on the needs of researcher and the scoring rules of the research lab.

Step 1: Select/Remove Scoring Rules

Decide on which rules you will use. A list of rules are found on the side panel to the left. These are considered either Spelling Rules or Grammar Rules.

Spelling Rules:

• Acceptable Spell Rule: Response word counted correct if it is a homophone or common misspelling of the target word, according to a preloaded default acceptable spelling list (contains over 300 common acceptable spellings).
• Acceptable Spell Rule+: User can download the default acceptable spelling list, add/remove items, and upload for automation. Response word counted correct if it is on the acceptable spelling list.
• Root Word Rule: Response word counted correct if the target word (e.g. ‘day’) is embedded at the beginning (e.g. ‘daybreak’) of the target word.
• Double Letter Rule: Response word counted correct if it omitted a double letter within a word (e.g. ‘atack’ matches ‘attack’) or added an unnecessary double letter (e.g. ‘occassion’ matches ‘occasion’).
• Number to Text Rule: Response word counted correct if arabic numeral (e.g., 8, 10, 100) provided instead of the word (e.g., 'eight', 'ten', 'one hundred').
• Compound Rule: Response word counted correct if response used alternate spelling of compound (e.g., target was 'junkyard' and response was 'junk yard'). Default list of compounds used. Additional ones can be added to the Acceptable Spelling Rule.
• Contraction Rule: Response word counted correct if response used contraction of response (or vice versa). This rule has not been validated formally.

Grammar Rules:

• A/The Rule: Substitutions of ‘a’ and ‘the’ are scored as matches
• Tense Rule: Response word counted correct if it differs from the target word by the addition or omission of ‘d’ or ‘ed’ (e.g. ‘assumed’ matched ‘assume’ and ‘jump’ matches ‘jumped’)
• Tense+ Rule: Response word counted correct if differs by the target word by the addition (not omission) an ‘d’ or ‘ed’ (e.g. ‘jumped’ matches ‘jump’ but ‘jump’ does not match ‘jumped’)
• Plural Rule: Response word counted correct if it differs from the target word by the addition or omission of ‘s’ or ‘es’ (e.g. ‘cats’ matches ‘cat’ and ‘echo’ matches ‘echoes’)
• Plural+ Rule: Response word counted correct if differs from the target word by the addition (not omission) ‘s’ or ‘es’ of the target word (e.g. ‘cats’ matches ‘cat’ but ‘cat’ does not match ‘cats’)

Step 2: Format and Upload Data File(s)

Format the listener transcript data file(s) so that you have columns titled id, target, response, and (optionally) human, according to the downloadable template below. Additional columns will not pose a problem.

The 'id' column should contain a subject identifier (any format is acceptable); the 'target' column should contain the target words/phrases that are presented to the listener. The 'response' column should contain the listener’s response (i.e., what they thought they heard). You may also include an optional column titled 'human', if you have already had your transcripts scored by a human being.

Upload your data file(s) at the top left of this screen. Autoscore will immediately analyze the file(s), applying the rules that you have selected.

Step 3: Download Analyzed Data File

The auto-scored data will show up below. The output will contain the main variables of 'id', 'target', 'response', and 'autoscore'. The 'autoscore' variable contains the counts of the matches for that target/response row. If your data contains a 'human' variable, the comparison between 'human' and 'autoscore' called 'equal' will also be provided. The other variables in the original file will also be in this output (on the right side of the main variables).

Download your data by clicking on CSV or Excel buttons above the table. This will download the analyzed data file in the selected format containing the autoscored information included with the original data.