An evaluation of Korean students’ pronunciation of an English passage by a speech recognition application and two human raters

Thirty-one college and graduate students participated in the recording of a short English passage. They were divided into nineteen undergraduate and twelve graduate students. They majored in English education and took a course in applied phonetics before the data recording. Their English proficiency varied, but a majority of them were assumed to be at an intermediate or higher level. Two native English raters were recruited to evaluate the students’ recordings. They were Canadians currently teaching college English, and they had much experience evaluating Korean students’ pronunciations. One had a teaching career with students for more than eleven years in Korea and the other fifteen years.

The recording passage was obtained from a website (Rochester Institute of Technology, 2020). The rainbow passage consisted of 98 words and six sentences as follows.

To use the string comparison “equal” function in Microsoft Excel, each sentence was parsed into fifteen numbered rows within which the same word did not occur. According to two major groups of parts of speech, forty-seven content words were written in bold face, while fifty-one function words, as well as plural morphemes (-s, raindrops, colors, ends, friends) and inflectional morphemes (-s, strikes, finds), were denoted in plain type (see Benner, 2020; Nordquist, 2020; O’Shea, 2013 for group classification).

Data for analysis were collected in three steps: recording of the rainbow passage by the participants at home using their mobile phones; recognition of the recording by a speech recognition application, Speechnotes (https://speechnotes.co/); and evaluation of the recording by two native Canadian English-speaking raters. The application uses Google’s speech-recognition engines with expected accuracy levels higher than 90%, which was proved by the two Canadians’ recordings. Statistical analyses of the recognition rate and human raters were conducted using Microsoft Excel and R software (R Core Team, 2020).

To avoid the reader’s parsing errors without knowing the context of the passage, the author asked the participants to prepare for the task by listening to the native pronunciation of the passage using Papago (Naver, 2020) and by practicing it sufficiently in advance (Levis, 2011). As mentioned in the introduction, there might have been some evaluation problems regarding authenticity in read speech. However, we decided to use the passage to ensure comparable recordings in clear speech mode. If we used free speech, recognition rates and intelligibility levels on speech recordings of different sizes, word choices, or syntactic structures might not be comparable among various participants. Further studies of both free and read speeches are desirable to pursue any findings related to the comparability issue.

The original recordings of the participants were converted to the same file format, i.e., 16 bit mono at a 44.1-kHz sampling rate, and normalized to the peak of the sound files. Some speech segments with low intensity were boosted by SoundStudio on a Macintosh computer to render the sounds sufficiently loud for the speech recognition application and the human raters. Then, the sound files were played on the computer, and the recognized output texts were transcribed by the speech recognition application on a Samsung Galaxy Note 10 in a quiet room. The recognition procedure was applied twice per each speaker in a row. There were slight differences in the transcribed output texts. The recognition rate was calculated by determining the percentage of correctly recognized words, which occurred in either the first or second output texts. The recognized words were also divided into forty-seven content words and fifty-one function words to examine any group differences in the word types. Both the content words with and without the morphemes were counted as correctly recognized words in this paper. The “table” function in R was used to collect the categorical frequency distribution of unrecognized words or general patterns of incorrect recognition.

The two native raters listened to the recordings and evaluated the read speech of the thirty-one participants using International English Language Testing System (IELTS) band criteria for pronunciation (IDP Education Canada, 2020). The speaking section in IELTS assesses the use of free spoken English through an examiner’s individual face-to-face interview on test takers’ homes, families, and interests in Part 1, and the test takers may write down their own thoughts on an assigned topic for a minute and answer them in two minutes in Part 2 and discuss them further in Part 3. The level is determined by the band descriptors from 0 to 9 for fluency and coherence, lexical resource, grammatical range, accuracy and pronunciation. The scoring description for a very good user of band 8 reads a wide range of pronunciation features and has an easy understanding throughout, as well as minimal effect of the L1 accent on intelligibility. The raters were instructed to evaluate the participants’ recordings using the full range focusing only on pronunciation intelligibility of the read speech. The raters watched a 15-minute sample video interview of five Korean IELTS test takers in five bands: 5.5, 6.5, 7.5, 8.5, and 9. In this way, they could avoid confusion regarding the intelligibility band reported in Kang & Ahn (2013).

Table 1 lists basic statistics of speech recognition rates by the speech recognition application. To facilitate the readers’ understanding, both the number of correctly recognized words and their percentages of the total number of words in the rainbow passage are reported in the table. We use mostly percentages in the discussion. As described in the previous section, the total number of words in the passage was 98: 47 content words and 51 function words. The grand average percentage of correctly recognized words was 77.7%, i.e., 76.2 of 98 words. Had we used a different recording passage, the result would have varied. The correctly recognized percentage of all of the words ranged from 63.3% to 87.8%. The moderate rate generally indicates the proficiency level of the participants as intermediate or higher. It also suggested the usefulness of the recognition application as an evaluation tool. Interestingly, the correctly recognized percentage of the function words was greater than that of the content words by approximately 8% points.

The percentage of the standard deviation of all correctly recognized words was approximately 6%. There was a 3.5% difference in the deviation between the content and function word groups. The content word group showed lower recognition and wider deviation, which might be related to some pronunciation errors of specific consonants or vowels by the participants. Conversely, the range of correctly recognized function words extended much wider than the content words. We examine frequently unrecognized words and pronunciation errors in the section below. The higher recognition rate of the function words might be related to hyperarticulation of the participants without applying such general phonological processes as resyllabification or vowel weakening to their read speech.

The participants were divided into high- and low-level groups greater and less than 79% of the recognition rate: fifteen high-level and sixteen low-level participants. Table 2 shows the group statistics of speech recognition rates of the content and function words.

The table indicates that the two groups are different in basic statistics. The average percentage of the high-level group is 82.8%, while that of the low-level group is 73.0%. The difference between the two groups amounts to 9.8%. From the maximum and minimum rates we determine the range of each group from 8.2% to 15.3%, respectively. Both the high- and low-level groups demonstrate higher recognition rates of the function words by 5.1% and 11.2%, respectively. The result might be related to the reliability of the recognition algorithm and recording devices. The recordings were made by the participants with their own mobile phones in various settings. Further studies might find it interesting to compare the results of sound-proof booths or individual recording settings. The standard deviation of the high-level group was almost half of the deviation of the low-level group. The difference might be related to the pool of the participants and the selection of the passage in this study. Different pools of each group and level of the passage would lead to various patterns of difficulties with the content and function words.

Table 3 illustrates the percentage of correctly recognized words by sentence type. One can note that the recognition rates vary by the size and order of the sentence. The best recognition occurs in the last sentence, while the third sentence lists the worst recognition rate. The last phrase in Row 15 records 100% recognition. Among the rows in the third sentence, we found 53.5% recognition in Row 8. The word “path” lists many unrecognized cases, and in Row 4, for the first sentence, the phrase “form a” listed an almost comparable low rate of 58.1%, in which the two words were recognized as one word like the English word “former”. We discuss a few cases of one-word recognition for a pair of words in the following section. In contrast, the first sentence records 72.8% recognition. The low recognition rate at the beginning might be related to abrupt adaptation of the speech recognition application itself. There used to be a short silence period at the beginning of the recording so that the application had sufficient sound data to immediately begin the recognition process. From these results, we could say that the recognition could vary depending on the composition of the content and function words in each given sentence. To perform a meaningful evaluation of the participants’ speech, we might have to examine controlled or balanced components of the vocabulary from both free and read speech (Levis, 2011).

An examination of the unrecognized words might be necessary to diagnose and remedy the talker’s pronunciation errors. Table 4 lists the frequency counts of unrecognized content and function words.

The content word “looks” ranks as the most frequently unrecognized instance, followed by the word “is”. The two words consist of one syllable, and they are frequently connected to adjacent words. In addition, typical pronunciation problems of the Korean participants seemed to be related to the lateral [l] sound and the lax vowel in the word “look”. The lateral must be produced like a Korean flap in casual production of the English word “water”. The lax vowel [ʊ] is also quite difficult for Koreans to realize sufficient vowel quality (Yang & Whalen, 2015). The words “form” and “path” ranked third and fourth. A few instances of the word “form” were wrongly recognized as “from”. Several instances of the word “path” were recognized as “pet, pat, pass, past, pets”. Both the vowel quality and the fricative features could account for the variants. The second to fourth words on the list have either a fricative onset or coda, which must have contributed to the low recognition rate. Yang (2017) reported the same recognition problem in a paragraph with words with fricative codas, such as “slabs” and “these”. He attributed the low recognition rate of casual speech to problems of individual pronunciation errors and Google soundwriter errors with the fricative sounds. Similar fricative problems could be found with the function words on the list. Yang also noted that the main error sources might be traced to the phonotactic probability (Vitevitch & Luce, 2004) and lexical neighborhood density (Luce & Pisoni, 1998) of the words in the paragraph.

The vowel in the word “ends” was recognized as “and, ann’s, as, hands,” and others. The quality of the vowel might have to be considered for better recognition. A study of the recognition rate of a separate list of minimal pairs of vowels for the same participants might be useful to explain partial causes of the total recognition rate of a passage, including either one or the other minimal pair of words. A single sound production might not provide a full explanation of the holistic evaluation of the passage, as in Kang & Ahn (2013). These authors attempted to compare the accuracy of the English [r] sound and the intelligibility of the thirty Korean students to find no significant relationship. In the table, the word “prism” lists fourteen instances of variant forms gathered from both groups. It was frequently recognized as “prison” and other rare forms, such as “present, pleasing, prize”. The nasal consonant coda has low energy; thus, even on the human perception test, it is not very well recognized (Yang, 2005). Yang reported in his perception study that the codas [m, n] were perceived correctly by approximately 35% of the 130 participants, while the onsets [m, n] recorded 83% (see Table 2, Yang, 2005). House (1957) and Ohala (1990) noted that nasals were easily distinguishable from other consonants, but they also were easily confused among nasal sounds. The word “apparently” was recognized as six variant forms like “of parenting, parent Lee, of hair only”. In addition, seven instances of “raindrops” were recognized as “range, ranger, rangers”. A more sophisticated algorithm to sift through possible candidates for recognition might be necessary to provide valid and reliable transcription.

Within the function words, the definite article “the” was the most frequently unrecognized case, followed by the indefinite article “a”. The definite article was wrongly recognized as twelve instances of “does”. Many indefinite articles were combined to form words, like four instances of “formal” and twelve instances of “former” for the phrase “form a”. In contrast, the word “above” was recognized as two words, like “a boat” or “a bow”. The three words “these”, “its” and “when” occurred more than twenty times. The word “these” was recognized as four instances of “this” and three instances of “please”.

One could note that the voiced fricative sound [ð] in the function words accounted for a higher rank on the list of the words “the, these, there, with, they”. The fricative sound has very low energy because of the short resonance cavity formed by the tongue blade and the upper teeth on the upper and lower lips. It might add more confusion when the function word consists of only one syllable. The word “arch” was recognized as six instances of “art”, along with two “oranges”. Thus, we can say that the recognition rate depends on either segmentals or the syllable positions.

Table 5 lists the frequency counts of unrecognized words according to the group level divided by the mean correct recognition rate. In the table, the word frequencies of the high-level group generally were lower than those of the low-level group by approximately two to five instances for content words. However, the difference between the two groups becomes much greater in the upper rows of the function words. Specifically the definite article showed a difference of fifteen instances. The indefinite article and the others have differences of zero (“many”) to six (“a”) instances. Thus, we can state that the function words and specifically the definite article account for the major differences at the group level.

However, we might have to bear in mind that the results could have been biased by simply having more occurrences of the definite articles in the passage. The rainbow passage consists of nine definite articles and six indefinite articles. Researchers should be careful not to draw hasty conclusions based on simple frequency counts of unrecognized words. Perhaps a lexically balanced passage might lead to a better evaluation in this case. The word “when” was mostly identified as the words “in” or “and”. The approximant onset and sometimes the low energy of the following aspiration seem not to sound sufficiently clear to be recognized correctly. Similarly, a few instances occurred in that the word “and” was transcribed as “in” or “end”.

Table 6 shows statistics of the IELTS bands by two Canadian raters for the Korean participants’ pronunciation of the rainbow passage.

On average, both raters assigned 6.4 band point for the read speech of the participants. Figure 1 illustrates the actual data points of the IELTS bands for the participants. Rater B applied more lenient criteria than Rater A, which can be seen from the line of equity. The ranges of the two raters were almost the same by 0.5 band points. The bands of the participants ranged from 5.0 to 8.0. The relationship showed a moderately high correlation with a standard deviation less than band point 1. The absolute difference between the two raters’ bands ranged from 1.0 to 1.5 band points. Correlation analysis was conducted between the IELTS band points of the two raters to obtain a correlation coefficient of r=0.77 (p<.05). The narrow range without the higher or lower bands might have caused the low coefficient. We could have recruited participants with lower levels of fluency; then, the low recognition would not have produced valid and reliable outputs. However, the correlation might be stronger if the recognition worked appropriately to yield distributed recognition data. An additional correlation analysis was conducted between the number of all of the correctly recognized words and the raters’ bands to obtain a correlation coefficient of r=0.467 (p<.05), indicating a significant but very weak correlation.

Figure 1. Distribution of the IELTS band points by two raters. A line of equity is drawn through the band points.

Download Original Figure

An analysis between the number of correctly recognized content words and the raters’ bands yielded a correlation coefficient of r=0.567 (p<.05), while that between the number of correctly recognized function words and the raters’ bands produced a correlation coefficient of r=0.208 (p>.05). The raters’ bands seem to depend more on the correct recognition of content words. We can state that the speech recognition application could provide partial assistance for the evaluation of the participants’ fluency level. Since the parts of the passage matter, we might have to use a lexically balanced passage with an appropriate number of content and function words to gain the greatest advantage from the application for evaluation purposes. Were the passage lexically unbalanced, one solution would be to focus more on the recognition of content words, as Loukina et al. (2015) did. In addition, the two raters must have chosen the band point after considering not only the clear and correct pronunciation of each given word but also impressions of prosodic aspects of the passage. Any perfect agreement of the subjective evaluation might not be possible, but modifying mismatching band points among the raters after a round of evaluation might be required to secure the validity and reliability of the final band points. Further studies on the relationship between subjective human evaluation and objective machine recognition with more participants and passages might reveal hidden aspects of the evaluation.