Non-word repetition may reveal different errors in naive listeners and second language learners

As this study reports on the perception of speech elicited in a repetition task, the recording of the stimuli involved two stages: recording the original stimuli by a single original talker, and then playing these stimuli to four groups of participants. These participants were both “listeners”, in that they listened to the target productions of the single original talker, but they were also “talkers”, as their repetitions of these stimuli were recorded and then subsequently used in the construction of stimuli in the perception experiment reported in the current study. Thus, for the remainder of this paper, “original talker” will refer to the single original talker who produced the target stimuli; “talkers” will refer to the participants who both listened to and repeated these original stimuli; and “listeners” will refer to the listeners who identified and rated these repetitions.

The original stimuli were recorded by the original talker, a phonetically trained female native speaker of Seoul Korean. The speaker was aware that the stimuli would be played to listeners who would be asked to repeat them, and so every effort was made to produce the intended non-words as accurately and clearly as possible. A total of 54 non-words were recorded.

The talkers were recruited from four populations: naïve talkers (n=17), novice L2 learners (n=15), advanced L2 learners (n=17), and native Korean talkers (n=15). The naïve talkers were native Mandarin speakers with no experience learning Korean as an L2. These listeners were undergraduate students at Indiana University, and were recruited and tested in Bloomington, Indiana, USA. The novice L2 learners were L1 Mandarin speakers who were enrolled in a beginner-level full-time intensive Korean language course at Korea University. These listeners had completed three to five weeks of classes by the time of testing, and had arrived in Korea less than a week before those classes began. The advanced L2 learners were L1 Mandarin speakers who had completed at least one year of a full-time intensive Korean language course in Korea and were enrolled in undergraduate degree programs at Korea University at the time of testing, and had been living in Korea for one to five years. Lastly, the native Korean speakers all spoke a non-Gyeongsang variety of Korean, and were undergraduate students at Korea University at the time of testing. More demographic information from the subset of these talkers whose productions were used to create the perception stimuli is provided in Table 1.

These talkers participated as listeners in the series of perception tasks reported in Holliday (2016). The recording of the stimuli used for the current experiment was the first task done in the experiment session, done before any of the other perception tasks, and not reported in Holliday (2016). The 54 non- word stimuli were presented auditorily in random order on a laptop computer running OpenSesame (ver. 2.8.1; Mathôt et al., 2012). In each trial, each stimulus was played twice, with a brief intervening pause, after which the talker was asked to repeat the word exactly as they heard it. The L2 learners and native Korean talkers were told that the stimuli were Korean words produced by a native speaker of Korean, but were words they had never heard before. This was said to ensure that the listeners did not think the words were English, or some other language, and would also not try to recognize them as real Korean words. The naïve talkers were not specifically told that the words were Korean, but just that they were from a language other than Chinese or English. They were not told that the stimuli were Korean in case they had preconceived ideas about how Korean should be pronounced. It was thus hoped that the talkers in each group would try to produce the words as closely as possible to how they were perceived.

12 talkers (8 female, 4 male) from each of the four talker groups described above were selected (total n=48), from whose productions CV stimuli were extracted. The balance between females and males was decided based on the number of talkers available in each group with useable productions (e.g. some talkers had skipped one or more words). The CV stimuli were extracted word-initially from non-words with CV.CV or CV. CVC syllabic structure, in which the second consonant was always a coronal obstruent. The word-initial CV consisted of one of the consonants /sʰ, s*, ʨ, ʨʰ, ʨ*/ followed by one of the vowels /a, i, u/. There were 2 non-words chosen for each CV combination, resulting in 30 target non-words (5 consonants×3 vowels×2 tokens) from which the CV stimuli were extracted for each talker. Each of these 30 target non-words from each of the 48 talkers had its intensity RMS normalized at 70 dB, the initial CV was extracted, and then the waveform of the CV was zeroed out over the final 35 ms of the vowel to reduce audible clipping.

This procedure resulted in 1,440 unique CV stimuli (30 CVs×48 talkers). These stimuli were divided into 6 lists, with each list containing 240 stimuli (15 CVs×16 talkers). The 16 talkers in each list consisted of 4 talkers from each of the 4 talker groups, and an individual talker’s stimuli were always equally divided across 2 different lists. 4 of the lists contained only stimuli from female talkers, and 2 of the lists contained only stimuli from male talkers, ensuring that talker gender would vary across but not within listeners.

24 native Korean listeners (14 females, 10 males) who had no experience with any variety of Chinese participated in the perception experiment. Their mean age was 27.3 years with a range of 20 to 48 years. They were living in Seoul at the time of the experiment but were from different regions of Korea. Although two of the listeners were from the Gyeongsang region, which carries a stereotype of not maintaining the phonological contrast between /sʰ/ and /s*/, studies of the production (Holliday, 2012; Lee & Jongman, 2016) and perception (Holliday, 2014a) of these fricatives by younger native Gyeongsang speakers revealed no significant differences with respect to native Seoul speakers.

All listeners had studied English in school. In addition, although a few reported studying an additional foreign language such as Japanese, German, or French, they reported their proficiency as low, having only minimal knowledge of the language or not remembering anything at all. Lastly, the listeners had limited to no instruction in phonetics or any other related fields such as teaching Korean as a foreign language. Listeners were paid their participation.

Four listeners were randomly assigned to each of the 6 stimuli lists, ensuring that each individual stimulus received 4 unique judgements, and because each talker’s stimuli were always spread across 2 different lists, each talker’s stimuli were presented to a total of 8 listeners.

Listeners were tested in a quiet room at Korea University, seated in front of a Samsung NT900X-3B laptop computer running OpenSesame (ver. 3.2.4; Mathôt et al., 2012). Listeners were told that they would be hearing Korean sounds produced by native speakers of Mandarin. All instructions were presented on the screen in Korean, and stimuli were presented over high- quality headphones.

Each trial consisted of two parts: phoneme identification, and goodness rating. For each trial, listeners heard one stimulus twice and were asked to identify the Korean phoneme it sounded like the talker was trying to produce. Five buttons were presented on the screen, with each button labeled in Hangul with one of the Korean fricative or affricate phonemes, and listeners pressed one of the buttons to make their selection. Then, listeners were instructed to rate the sound they had just heard in terms of how accurately it was pronounced with reference to the Korean sound they selected. The rating was done on an integer scale from 1 to 5, with 1 labeled as “very accurate” and 5 labeled “totally inaccurate”. The response was made by clicking the mouse on the number on the scale.

Throughout the discussion of the results, the term target will be used to refer to the consonant in the original non-word. There is no way to know what phonological target the talker had in mind when repeating the non-word, of course, and so in the discussion of accuracy we will always be referring to the original non-word that the talker was asked to repeat.

Identification accuracy was calculated using two different methods: overall segmental accuracy (the response was the same as the original target consonant), and manner accuracy (the response and the original target consonant were both fricatives or both affricates, but may have differed in terms of the phonation type). For example, if the target consonant was /ʨ*/ and the response was /ʨ/ it would be considered incorrect in terms of segmental accuracy (since the target and response are not the same), but correct in terms of manner accuracy (since the target and response are both affricates). Manner accuracy was calculated separately because the misperception and subsequent repetition of fricatives as affricates was one of the particular phenomena that this study aimed to investigate.

However, the primary result of interest is not simply how many stimuli were correctly perceived as the target, but rather what segment was perceived. For this reason, the bulk of the results section will focus on the interpretation of confusion matrices showing, for each talker group and vowel context, how frequently each target was perceived as each of the five consonants. Some error patterns turned out to be quite common across talker groups (e.g. /ʨ/ targets being perceived as aspirated /ʨ^h/), but we are specifically interested in error patterns that differ between naïve talkers and L2 learners.

Lastly, we also investigated the goodness ratings that listeners assigned to each production. There were three primary questions we aimed to answer with these data. First, are there differences across talker groups in terms of how good their productions were rated? Second, were the productions whose target consonant was correctly identified rated as better than the productions whose target was misidentified? For example, we will see that many /ʨ/ targets were perceived as /ʨ^h/, but was there any difference in the goodness ratings between correctly identified /ʨ^h/ targets and /ʨ/ targets that were misperceived as /ʨ^h/? Third, are there differences in mean goodness rating across response categories, regardless of the target consonant? It is possible that some phonological categories have looser perceptual criteria, and that, for example, producing “a good /s*/” is more easily achieved than “a good /s^h/”. This idea is speculative, but these results could shed light on the question and suggest directions for future research. For ease of computation, the goodness rating scale of 1 (“very accurate”) to 5 (“totally inaccurate”) was transformed by reversing and shifting it, resulting in a scale of 0 (“totally inaccurate”) to 4 (“very accurate”). All goodness ratings reported in this paper use the transformed scale.

All analyses were carried out in RStudio version 1.1.463 (RStudio Team, 2016) running R version 3.5.2 (R Core Team, 2018), and using the Tidyverse package version 1.2.1 (Wickham, 2017).

Overall identification accuracy rates are presented in Table 2. As expected, L1 Korean talkers’ productions were the most accurately identified, at 83.0% segmental accuracy, with naïve and novice L2 talkers’ productions being the least accurately identified, at 59.8% and 58.2%, respectively. Advanced L2 talkers’ productions fell in the middle, at 68.3%. Given that listeners had five responses from which to choose, these results are far above chance level, 20%. A look at manner accuracy rates shows that talkers’ productions were rarely perceived as having incorrect manner (i.e. a fricative- for-affricate or affricate-for-fricative error), however, and that the vast majority of errors were due to incorrect phonation type.

The specific hypothesis that drove this study was that naïve talkers should be more likely to produce /s^h/ targets as an affricate, especially in the /a/ environment. To assess this hypothesis we further calculated manner accuracy just for the /s^h/ targets, according to vowel context, for each of the four groups. The L1 Korean talkers and both L2 groups had manner accuracy rates above 96% for every vowel context, suggesting that manner errors were rare. For the naïve talkers, manner accuracy rates for /a/, /i/, and /u/ environments were 46.9%, 90.6%, and 87.5%, respectively, indicating that although manner errors were slightly more frequent in naïve talkers, they were far more frequent specifically in the /a/ environment.

Table 3 shows the segmental accuracy rates broken down by target consonant and talker group. The most consistent trend across talker groups is that /ʨ/ and /s^h/ targets are on the whole more likely to be perceived incorrectly than /s*/, and especially /ʨ^h/ or /ʨ*/. Given the overall high rate of manner accuracy, and the fact that listeners could choose from only two fricative categories and three affricate categories, the segmental accuracy of /ʨ/ and /s^h/ are probably not directly comparable: among fricatives, chance level would be 50%, whereas among affricates it would be 33.3%. Thus, although the accuracy rates for these targets are both low, they should not necessarily be viewed as equally low.

The particularly low segmental accuracy rates for /ʨ/ and /s^h/ are further broken down by vowel context in Table 4. An interesting trend appears, in which within talker group, vowel context does not seem to affect the segmental accuracy of /ʨ/ targets, but has a consistent effect on /s^h/ targets: naïve, novice L2, and advanced L2 talkers’ /s^ha/ productions were less accurately perceived than their /s^hi/ and /s^hu/ productions, whereas for the L1 Korean talkers the trend is reversed, with their /s^ha/ productions being perceived more accurately than their /s^hi/ and /s^hu/ productions.

Tables 5, 6, and 7 show how frequently the repetitions of each target consonant were perceived as each consonant in the /a/, /i/, and /u/ vowel contexts, respectively, along with the transformed mean goodness ratings assigned by the listeners. Several trends can be observed.

First, we believe the high accuracy rates for the L1 Korean talkers’ productions in the /a/ context demonstrate that the task is feasible. That is, when a native Korean speaker hears an unknown word that begins with a fricative or affricate followed by /a/, they are generally able to imitate it in a way that another listener would be able correctly identify the original intended consonant. The accuracy rates for four of the five consonants were above 90%, and for the remaining consonant target, /ʨ/, it seems that listeners sometimes simply interpreted the phonetic aspiration in /ʨ/ as a cue to /ʨʰ/, which may be due to the lack of a reliable f0 cue in an isolated CV. It is also clear that /ʨ/ targets were perceived as /ʨʰ/ far more often than /ʨʰ/ targets were perceived as /ʨ/, indicating that the confusion between /ʨ/ and /ʨʰ/ was not symmetric. In any case, even though accuracy rates are not as high in the high vowel contexts or for the non-native talkers, the overall high accuracy rates for the L1 Korean talkers’ productions in the /a/ context nevertheless demonstrate that the task is not inherently too difficult.

Second, manner errors are indeed rare. Repetitions of affricate targets were rarely perceived as fricatives, and repetitions of fricative targets were rarely perceived as affricates, with only two notable exceptions. One was the perception of /ʨa/ targets produced by non-native talkers as /s^h/, of which there was a total of 25 such responses. These responses represent 6.5% of non-native talker /ʨa/ target trials from 10 unique stimuli (3 naïve, 4 novice L2, and 3 advanced L2). Given that there were 24 unique /ʨa/ target stimuli per talker group, these /ʨa/ target stimuli perceived as /s^h/ do not seem to reflect a broad trend, but rather isolated cases. Also important to the current study is the fact that these cases were spread across both naïve and L2 talkers, and are thus not an example of a change in perception or production that accompanies L2 experience.

The other exception was the one that was predicted at the outset of the study: /s^ha/ targets produced by naïve talkers (and only naïve talkers) were frequently perceived as affricates, either /ʨ/ or /ʨʰ/. Combined, these affricate responses were more frequent (53.2%) than fricative responses (46.9%). This result was found to a much lesser degree in the /i/ and /u/ contexts: only 8.3% and 12.5% of /s^hi/ and /s^hu/ targets were perceived as /ʨʰ/. Crucially, however, this result was only ever found in the naïve talkers, and not for the talkers that actually had experience with Korean.

Third, it can be seen that the two fricatives /s^h/ and /s*/ are quite confusable in all vowel contexts and by all talkers, with the exception of L1 Korean talkers in the /a/ context. Because manner errors are so rare, the vast majority of incorrect fricative trials are due to the target being perceived as the other fricative (see the segmental accuracy rates in Table 3). This result is unsurprising, and is in line with previous studies showing that both L1 listeners and (especially) L2 listeners have difficulty accurately identifying Korean /s^h/ and /s*/ when followed by a high vowel (Holliday, 2014a). Table 6 shows the confusion matrix for the /i/ context, and that even L1 talkers’ /s^hi/ and /s*i/ targets were identified at rates not far from chance level, a result indicating that either the talkers who produced the stimuli misperceived the intended targets in the original stimuli in the non-word repetition task, or the listeners in the current study misperceived the intended targets produced by the talkers. Regardless, the results demonstrate again that native listeners struggle to accurately identify Korean fricatives in high vowel contexts.

First, while there were marginal differences in overall mean goodness rating across talker groups, as shown in Table 8, these differences did not reach statistical significance [F(3,44)= 2.41, p=.08]. This lack of a difference between the native and non-native talkers’ productions could be because not only were the stimuli short, isolated CVs, but the listeners did not actually know what the original intended target was – the goodness rating was made with respect to the category chosen by the listener. Thus, a production could have been misidentified with respect to the intended target of the original non-word, but still sound like a good production of whatever it was perceived as. Another possible factor is that the listeners were told that the stimuli were produced by non-native speakers, whereas in reality they included some productions of native speakers as well. This could have biased listeners to not rate productions too highly overall, which would include the stimuli produced by the L1 Korean talkers. It could also have the opposite effect, in that L1 Korean talkers’ productions could be considered very good for a non-native speaker. Given that the ratings could range from 0 to 4, with 4 being “excellent”, mean ratings of 2.8 to 3.2 are relatively high.

With respect to the second question of whether the goodness ratings varied according to whether or not the production was correctly perceived as the target consonant in the original non- word, the two right columns in Table 8 show the mean ratings for incorrect and correct responses by talker group. It can be seen that in all four talker groups, productions whose intended consonant was correctly identified received higher goodness ratings than those that were incorrectly identified. These differences yielded statistically significant post-hoc paired t-tests [Naïve: t(11)=2.97, p=.013; Novice L2: t(11)=3.09, p=.010; Advanced L2: t(11)=3.32, p=.007; L1 Korean: t(11)= 6.69, p < .001]. There exists a possible confound between talker group and target consonant, however, and so an additional post- hoc paired t-test was run, which confirmed that within each target consonant correctly identified productions were given higher goodness ratings than productions that were misidentified [t(4)= 3.48, p=.025], regardless of which talker group the production came from.

Lastly, the mean goodness ratings assigned by response category, regardless of what the original intended target was, are shown in Table 9. Across productions from all talker groups, productions perceived as /s*/ are generally given the highest ratings, even though productions perceived as /s*/ are not any more likely to have been perceived as the original intended target (see the second column from the right). In addition, /s*/ was the second most frequent response, which suggests that listeners are overall less picky about what a good /s*/ is.

On the other hand, /ʨʰ/ responses were given, on average, lower goodness ratings, even though they were just about equally as likely to have been perceived as the original intended target. As Tables 5, 6, and 7 indicate, incorrect /ʨʰ/ responses were most frequently given in response to /ʨ/ targets. It is thus possible that when listeners hear an aspirated affricate it gets matched to their /ʨʰ/ category, but because the f0 is not as high it does not sound as good as true /ʨʰ/ productions produced with a higher f0.