Instructions
This open-source tool automatically scores orthographic listener transcripts for words correctly identified. This is useful in speech perception research (e.g., studies that examine listener understanding of disordered speech or speech in background noise) or for generating listener intelligibility measures in clinical disciplines such as speech-language pathology or audiology. The program uses a flexible number of rules that determine whether a response set of words (i.e., listener transcripts) match a target set of words (i.e., speech corpus). At the most basic level, Autoscore scores words in the listener transcript as correct if they match the words in the target phrase exactly (regardless of word order). Additional individual scoring rules can be applied or removed, depending on the needs of researcher and the scoring rules of the research lab.
Step 1: Select/Remove Scoring Rules
Decide on which rules you will use. A list of rules are found on the side panel to the left. These are considered either Spelling Rules or Grammar Rules.
Spelling Rules:
- Acceptable Spell Rule: Response word counted correct if it is a homophone or common misspelling of the target word, according to a preloaded default acceptable spelling list (contains over 300 common acceptable spellings).
- Acceptable Spell Rule+: User can download the default acceptable spelling list, add/remove items, and upload for automation. Response word counted correct if it is on the acceptable spelling list.
- Root Word Rule: Response word counted correct if the target word (e.g. ‘day’) is embedded at the beginning (e.g. ‘daybreak’) or the end (e.g., “monday”) of the target word.
- Double Letter Rule: Response word counted correct if it omitted a double letter within a word (e.g. ‘atack’ matches ‘attack’) or added an unnecessary double letter (e.g. ‘occassion’ matches ‘occasion’).
Note about the Acceptable Spell Rule+
To apply the Acceptable Spell Rule+, click on the 'Download Default Acceptable Spell List' in the left-side panel
to download the CSV file. This file is the current default acceptable spelling list.
You can add or remove words or upload the file exactly as is. Uploading exactly as is, will be the same as
selecting the Acceptable Spell Rule.
If you add to it, in the target column, list any target words that have any
additional acceptable responses that will not be captured by the other scoring rules.
In the acceptable column, list any additional responses that you would like counted as correct.
Grammar Rules:
- A/The Rule: Substitutions of ‘a’ and ‘the’ are scored as matches
- Tense Rule: Response word counted correct if it differs from the target word by the addition or omission of ‘d’ or ‘ed’ (e.g. ‘assumed’ matched ‘assume’ and ‘jump’ matches ‘jumped’)
- Tense+ Rule: Response word counted correct if differs by the target word by the addition (not omission) an ‘d’ or ‘ed’ (e.g. ‘jumped’ matches ‘jump’ but ‘jump’ does not match ‘jumped’)
- Plural Rule: Response word counted correct if it differs from the target word by the addition or omission of ‘s’ or ‘es’ (e.g. ‘cats’ matches ‘cat’ and ‘echo’ matches ‘echoes’)
- Plural+ Rule: Response word counted correct if differs from the target word by the addition (not omission) ‘s’ or ‘es’ of the target word (e.g. ‘cats’ matches ‘cat’ but ‘cat’ does not match ‘cats’)
Step 2: Format and Upload Data File(s)
Format the listener transcript data file(s) so that you have columns titled id, target, response, and (optionally) human, according to the downloadable template below. Additional columns will not pose a problem.
The 'id' column should contain a subject identifier (any format is acceptable); the 'target' column should contain the target words/phrases that are presented to the listener. The 'response' column should contain the listener’s response (i.e., what they thought they heard). You may also include an optional column titled 'human', if you have already had your transcripts scored by a human being.
Upload your data file(s) at the top left of this screen. Autoscore will immediately analyze the file(s), applying the rules that you have selected.
If you have > 1 files to analyze:
- You may select any/all files to be analyzed simultaneously.
- The process may take up to a minute to analyze if there are many files.
- The output will be a single combined data file containing all the observations/cases from all the files uploaded. This is one of the reasons the 'id' column is required.
Step 3: Download Analyzed Data File
The auto-scored data will show up below. The output will contain the main variables of 'id', 'target', 'response', and 'autoscore'. The 'autoscore' variable contains the counts of the matches for that target/response row. If your data contains a 'human' variable, the comparison between 'human' and 'autoscore' called 'equal' will also be provided. The other variables in the original file will also be in this output (on the right side of the main variables).
Download your data by clicking on CSV or Excel buttons above the table. This will download the analyzed data file in the selected format containing the autoscored information included with the original data.
Selecting or deselecting rules or changing the data file(s) uploaded will result in a re-analysis and will replace any previous analyses.