This open-source tool automatically scores orthographic listener transcripts for words correctly identified. This is useful in speech perception research (e.g., studies that examine listener understanding of disordered speech or speech in background noise) or for generating listener intelligibility measures in clinical disciplines such as speech-language pathology or audiology. The program uses a flexible number of rules that determine whether a response set of words (i.e., listener transcripts) match a target set of words (i.e., speech corpus). At the most basic level, Autoscore scores words in the listener transcript as correct if they match the words in the target phrase exactly (regardless of word order). Additional individual scoring rules can be applied or removed, depending on the needs of researcher and the scoring rules of the research lab.
Decide on which rules you will use. A list of rules are found on the side panel to the left. These are considered either Spelling Rules or Grammar Rules.
Note about the Acceptable Spell Rule+
To apply the Acceptable Spell Rule+, click on the 'Download Default Acceptable Spell List' in the left-side panel to download the CSV file. This file is the current default acceptable spelling list. You can add or remove words or upload the file exactly as is. Uploading exactly as is, will be the same as selecting the Acceptable Spell Rule. If you add to it, in the target column, list any target words that have any additional acceptable responses that will not be captured by the other scoring rules. In the acceptable column, list any additional responses that you would like counted as correct.
Format the listener transcript data file(s) so that you have columns titled id, target, response, and (optionally) human, according to the downloadable template below. Additional columns will not pose a problem.
The 'id' column should contain a subject identifier (any format is acceptable); the 'target' column should contain the target words/phrases that are presented to the listener. The 'response' column should contain the listener’s response (i.e., what they thought they heard). You may also include an optional column titled 'human', if you have already had your transcripts scored by a human being.
Upload your data file(s) at the top left of this screen. Autoscore will immediately analyze the file(s), applying the rules that you have selected.
The auto-scored data will show up below. The output will contain the main variables of 'id', 'target', 'response', and 'autoscore'. The 'autoscore' variable contains the counts of the matches for that target/response row. If your data contains a 'human' variable, the comparison between 'human' and 'autoscore' called 'equal' will also be provided. The other variables in the original file will also be in this output (on the right side of the main variables).
Download your data by clicking on CSV or Excel buttons above the table. This will download the analyzed data file in the selected format containing the autoscored information included with the original data.