Michael McAuliffe

Dear Diarization: Using MFA 2.1 to assign speaker labels to utterances

2023-01-14T00:00:00-08:00

Some sort of intro

Methods ¶

Data ¶

English - Buckeye corpus ¶

Korean - Seoul corpus ¶

Playing around with PolyglotDB

2021-08-29T00:00:00-07:00

[TOC]

Overview ¶

The past few blog posts have focused on some of the updates I’ve done recently to the Montreal Forced Aligner , a tool I made while doing a postdoc at McGill with Morgan Sonderegger and Michael Wagner . The other primary tool is PolyglotDB (and the web interface ISCAN ), a tool for analyzing forced-aligned corpora and extracting automated measures, particularly of large corpora. I’ve recently gotten it working with the latest versions of its backend databases, Neo4j and InfluxDB , and thought it might be neat to do some super basic demonstrations. I also have a collection of headsets and microphones now, and so what I’ve done this time is record a couple of different passages with the different microphones to see if their quality affects some acoustic measurements.

Dataset ¶

The text materials come from two sources. The first is the North Wind and the Sun Aesop fable that has been used in some phonetic research (I remember Molly Babel and Phoebe Wong doing a project that used it), and the second is a reading passage from Fridland and Macrae (2008) .

The microphone setups that I have are:

With each of those I record both passages each in two recording conditions, one where there’s no background noise (though I live on the second floor, so there’s likely some traffic noise), and one with my AC on in the background. For the recording set up, I just used the Windows voice recorder app, or the stock Android voice recorder on my phone. So that’s 24 recordings all told (6 microphones, 2 stories, 2 noise conditions). The North Wind recordings are around 35 seconds, and the Fridland passage ones are around a minute and a half. If there was a major speech error, I would start over (mostly because I would accidentally add additional swear words), but minor speech errors or restarts were kept.

The recordings were forced aligned with MFA using the english_ipa acoustic model and corresponding dictionary . I didn’t touch the transcript or add words for any errors or restarts. I did some very basic spot-checking, and the TextGrids looked good enough to me. Anecdotally, the microphone in the Bose headset is the worst, as its primary purpose is as noise-cancelling headphones and not as a quality headset, and a number of friends have reacted negatively to its quality compared to the HyperX headsets. HyperX headsets tend to be on the quieter side for the microphone, so I have some preamp software set up to boost the gain.

You can download the sound files and transcriptions here if you want to play around with them.

Analysis ¶

The scripts I’m using are based on the SPADE scripts for analyzing dialects of English. I’m going to talk about vowel measures, a bit of sibilant measures, and pitch tracking measures.

Vowel space ¶

The vowel formant measurement script uses the refinement algorithm based on FAVE-extract and modified by Jeff Mielke during the SPADE project . The basic idea is that you get some distributional information about the vowel categories overall, and use these to make more informed selection of formant track candidates on a per-token basis, minimizing distance to the vowel prototypes. These prototypes don’t have to be generated on the fly, and there are a number available in the SPADE repository . For the initial analysis, I’m working with 4 versions of the algorithm:

base : generate prototypes on a first pass and use them to generate the final measures
spade-Buckeye : Use prototypes from the Buckeye Corpus (relabelling the Buckeye phoneset into IPA)
spade-SantaBarbara : Use prototypes from the Santa Barbara corpus (relabelling the phoneset into IPA)
spade-SOTC : Use prototypes from the Sounds of the City corpus (relabelling the phoneset into IPA)

The reason behind these selections is the range of dialectal similarity to my own. My dialect is Western American, early years split between Boulder, Colorado and Seattle, Washington, grad school in Vancouver, BC, and postdoc in Montreal, QC. So my dialect is West Coast, with some Canadian exemplars sprinkled in. Of the three external sources above, the Santa Barbara corpus should be the closest to my dialect (though not exactly, since California has its own peculiarities), with Buckeye as a little less similar, but still American English dialect. Sounds of the City corpus is made of Glaswegian speakers, so their dialect should be quite different from mine.

My initial pass for generating the formants had each microphone treated as an individual speaker, as shown below. This disadvantages the base condition the most, because the number of examples to form mean and standard deviation estimates is much lower. Particularly, it struggles with /u/ tokens in the bose headset, primarily due to consistent issues in first formant tracking. Note as well, that spade-SOTC struggles with the /ʊ/ tokens, often having them very fronted. The two American dialect sources show pretty good clustering.

Vowel spaces generated using per-microphone speakers in refinement

As a more fair comparison, I grouped all the microphones together and re-ran the analysis. They are all recorded by me, after all! The plot below shows much better performance for the base condition. Across the board, the phones cluster more tightly in these cases, since all the conditions benefit from having more data to create better per-speaker distributions following the initial fitting based on the prototypes.

Vowel spaces generated using a single speaker in refinement

Astute readers (looking at you, Kaylynn!) might have noticed that there are /ɔ/ vowels distinct from /ɑ/ , which doesn’t make sense since my dialect is a merged one. In general, these /ɔ/ vowels are from NORTH and FORCE words, where the distinction remains. MFA did select the /ɔ/ pronunciation variant for some words that would be merged, and you can see those words in the plot below mixed in with the /ɑ/ words.

Distribution of individual tokens with their word label from the "no noise" recording condition for both passages on the HyperX Cloud 2 wired headset

So generally speaking, formant tracking is not too bad here, and this is with a relatively low count of tokens (about 400-450 total vowel tokens per microphone setup). Even using completely different dialect prototypes to seed the initial formant estimation lets it recover somewhat from very sparse data.

Sibilants ¶

In addition to the vowel scripts from SPADE, I also ran the sibilant analysis. Somewhat surprisingly, the various microphones did pretty well on center of gravity, except for the Predator laptop that has the measure for /ʃ/ approaching that of /s/

Sibilant center of gravity estimation across recording setups

I was tempted to look into more sophisticated measures, like multitaper spectral analysis, for analyzing the sibilants, but it’s a bit outside the scope for the current post.

Pitch ¶

The pitch measures here aren’t in SPADE, but they come from some work on intonational contours with Michael Wagner. The set up for this section uses the speaker adjusted pitch algorithm with two sources, Praat and Reaper . The speaker adjustment algorithm is similar to the refined formants above, where we do an initial pass to estimate a good pitch range for a speaker using a baseline pitch range that will likely have some errors, and then generate the actual pitch tracks with the better pitch range. See the appendix for a more detailed look at the performance gains that speaker adaptation brings. As with the formant refinement algorithm above, the speaker adaptation benefits from larger amounts of data on the same speaker, so all the pitch analyses use the single speaker set up.

Pitch tracks across the recordings of the North Wind and the Sun passage

In general, pitch measurements from Reaper are more consistent than from Praat. Praat gives random high frequency pitch estimations, but the HyperX microphones perform pretty well for both pitch programs. This is consistent with anecdotal reports where friends have commented on poor quality from the Bose microphone compared to the HyperX ones. A similar story is present below for the Fridland passages.

Pitch tracks across the recordings of the Fridland passage

Taking a closer look at one of the recordings below, we can see generally pretty good results. The Reaper algorithm tends to be a bit more conservative in its voiced/unvoiced decision, so there are several places where Praat estimates pitch without a corresponding Reaper estimate. Conversely, there are also places where Reaper reports pitch and praat does not, particularly in creaky voice situations. For example, around second 23, where the glottal stop for “at” causes creaky voice. Reaper reports a dramatic dip in pitch, but Praat just gives up.

Pitch track detail with word alignments from a single North Wind and the Sun recording (hyperx_wired_noise)

In general, Reaper has better performance across the full dataset and recording setups. It’s a bit harder to set up and compile than Praat, but I don’t think the choice between the two will have a meaningful difference at larger scales of data.

Summing up ¶

I wanted to take a look at PolyglotDB for a couple of reasons. First was to motivate the update in database versions and make sure it’s still a functional tool and not abandonware. The second is to see what it brings to the table for smaller datasets. The recordings here are such a small fraction of the size of the data that Polyglot was built for in the SPADE project, and I did worry that the benefits that Polyglot brings wouldn’t be apparent. And using each microphone as a different speaker does reveal that to a certain extent. When there’s less than 10 minutes of speech per speaker and only 25 minutes of speech, we do run into some issues. The formant refinement and speaker adapted pitch algorithms do help combat this and give some very reasonable results for our tiny speech corpus.

Appendix ¶

The plot below shows one particularly bad recording (using the bose headset) where Praat has a lot of errors. The below plot doesn’t take advantage of the speaker adaptation algorithm as a way to showcase the performance gains from it. The default settings in PolyglotDB (similar to Praat itself) are pretty forgiving, with a minimum pitch of 50 Hz and maximum pitch of 500 Hz.

Pitch track for a single recording (bose_no_noise) using the base algorithm rather than the speaker adaptation

We can see some pretty horrendous pitch tracking errors (no way my voice is making it up to 500 Hz, let alone 300 Hz). Most likely there’s a fair bit of creaky voice that’s messing up the tracking. Using the speaker adaptation algorithm generates a first pass using the base settings above, calculates the mean pitch for the speaker, and then uses a two octave range around it to generate a more likely range for that speaker. The results for this run are below.

Pitch track for a single recording (bose_no_noise) using the speaker adaptation algorithm

You can see that we’ve cleaned up a lot of the big errors, and the ones that remain are smaller (I’m much more likely to get close to 250 Hz than to 500 Hz). In general, Reaper performs better than Praat just out of the box, but the speaker adaptation brings Praat pretty close.

How much data do you need for a good MFA alignment?

2021-08-24T00:00:00-07:00

[TOC]

Overview ¶

One of the most frequent questions I get is what is the threshold for getting a “good” alignment. My usual answer is just to throw the Montreal Forced Aligner at the corpus and see how well it does on spot check. My other glib answer is “more is better”. The more nuanced answer is that when there’s a pretrained model on 20+ hours of speech similar to the variety you’re trying to align, that will likely generate pretty good alignments. You can also try to use the recently released command for adapting pretrained models to new datasets . These solutions work for overresourced languages like English, but for fieldwork situations, they’re not going to be the best.

So say you’ve collected some audio data of some language, transcribed it with orthography, have a pronunciation dictionary, and want to generate alignments. At what point does it make more sense to automatically align it with MFA versus a more manual process? Let’s take a look.

Dataset ¶

The dataset that I’m using to evaluate quality of alignments based on the amount of training data is the Buckeye Corpus (as in my previous post about MFA performance ). So there is a large caveat here that this is North American English, collected in a sociolinguistic interview context, with generally good recording quality. Aspects like noise in the background, recording quality of the mic, etc are all going to factor into the necessary quantity of training data to generate good models. Additionally, the number of hours reported below represents the duration of speech, and not necessarily the total duration, as interview utterances and silences are removed. The total size of the Buckeye corpus is generally reported as about 40 hours, but when including just the speakers’ utterances, that drops to about 16.5 hours of speech.

The procedure I used created 195 “corpora”. Between 1 and 39 speakers were included in a given subset, with 5 subsets per number of speaker. The 5 subsets is to try to control for some of the variability introduced from particular speakers, since especially for the lower numbers of speakers, which speaker is included is going to affect the quality a lot. For each of these 195 corpora, the inverse subset was generated as held-out data to test how well the model could generalize to new speakers. The durations for each of these are below.

Durations of the various training runs (and the durations of the held out data)

Some speakers certainly have more data than others and including one over the other is going to cause some variation, but it’s not so much that is obscures the very clear relationship of increasing the number of speakers increasing the duration of the data. This might not be fully representative of your data, since the set up is based off of speakers, so if you have the same duration of data, you might have more or less speakers that could affect the quality of the alignments.

Some miscellaneous details about the training regime:

Used the same procedure of MFA English IPA train in the post about MFA performance
Used Linux installation of MFA to allow for easier use of symbolic links in creating the subset corpora
The timings below happened over a long period of time and often I was doing other stuff while it ran in the background, so take them with a grain of salt (shout out to Roguebook which provided me with many hours of entertainment during the training and alignment runs over the past few weeks)

Results ¶

In general, and unsurprisingly, the more speakers in the training set, the more time MFA takes to train. There are some deviations during points when I was using my computer for other purposes in parallel, but in general each training took under an hour (with 12 parallel cores). The parallelization in MFA is based on speakers, so for the smaller subsets, it can’t take as much advantage of multiprocessing.

Time to train

In addition to looking at how the alignments perform on the training data, I also used the held out data for each subset to give a sense of overall how good the model generated is, and to give a sense of how generalizable the acoustic models are to other speakers of the same variety.

Time to align held out data, note that the y-axis is log-scaled as the time to align held out data with lower amounts of training data can get ridiculously long

In general most alignments took less than 10 minutes, but for the runs with lower number of speakers in the training data, there was a pretty substantial explosion of time required to generate alignments for held out data. The reason for this can be seen below in the log-likelihoods. The log-likelihoods for training are not super useful for comparing quality because the dataset is completely different in each run, but they do give a sense of how overfitted a given model is. You can see below that log-likelihood of training peaks around 5-6 speakers, which correlates with the peak in training time above.

Log-likelihoods after training

When looking at the log-likelihood over the held out data, we don’t really see a trough in log-likelihood corresponding to the peak in overfitting. The alignments in the 5-6 speaker range aren’t suffering as much as we might first think. The alignment beam in MFA is a pretty forgiving 100 with 400 as a retry beam, whereas for training this beam is much more restrictive at 10 with 40 for retry, so alignment has the ability to keep a larger range of suboptimal candidates at the cost of larger lattices. The overfitted models take advantage of the larger beam and generate larger lattices, leading to the super long processing times. The Viterbi algorithm has a time complexity of $O(K^2T)$ where K is the number states (constrained by the beam width) and T the number of frames (if you’re curious about the beam search algorithm take a look here ).

Log-likelihoods after aligning held out data

The overfitting is largely an issue of temporal performance, and it looks like the log likelihood stabilizes around 10 ish speakers. Overfitting isn’t really an evil here that it normally is in model training, since the goal is usually just to generate the best alignments as possible on the training data. If the goal is to generate a model for ASR transcription or more widespread use, then this overfitting becomes a larger concern. To take a more in depth look, we’ll look at actual alignment errors in the training data and in held out data. These are largely distinct cases, as when you have a large (enough) set of target utterances to align, you care more about performance on training data, and when you distribute models you care more about the generalizability of the model.

Results on training data alignment ¶

First we’ll take a look at how the alignments stack up on the training data. This will be the usual use case for fieldwork data, where you just want to generate alignments for phonetic analysis. To begin with, let’s take a look at the word boundaries.

Word boundary errors in training data, colorboxes represent performance of


     MFA English IPA train

on all Buckeye speakers

Word boundaries show some variability towards the lower end of training speakers, but stabilizes around 20 speakers, but in general it oscillates close to the values for the full training set across all the subsets. Honestly though, the scale of errors is very comparable between reduced training speakers and the full corpus.

There’s always quite a bit of variation in selected CVC boundaries. The boundary errors stabilize fairly quickly, with only the smallest numbers of speakers showing increased errors. The VC transition does stabilize much later, around the 27 speaker mark.

Phone boundary error rate in selected CVC words in training data, colorboxes represent performance of


     MFA English IPA train

on all Buckeye speakers

For phone boundaries across the training data, we can see that errors are generally above the full run until about 27 speakers again. However, one thing to note is that the scale of this plot is much smaller than the above, so the increase in error rate isn’t too bad.

Phone error rates across training data, the colorbox represents performance of


     MFA English IPA train

on all Buckeye speakers

In general, the performance across all training runs is not too bad. For reference, the overall mean phone error for FAVE and MAUS on the Buckeye corpus is 22-23 ms (see the previous metrics post for more comparison to other aligners, I’ve only included the MFA English IPA train results in the above plots for visual clarity). So performance is generally better and pretty consistent once you hit the 5-10 speaker range, I would say, which corresponds to about 3-5 hours of speech. One outstanding question from this is whether increased speakers (and less data per speaker) might hurt performance, but I think 3-5 hours is probably a pretty good lower bound on good performance on the training data itself.

Results on held out data alignment ¶

Another use case is training a model (or having a pretrained model like the ones distributed on MFA’s pretrained model page ) and applying this model to new data. So the question here is basically how well do the models we trained above generalize to new speakers within the same dialect as the training data?

Starting with word boundary distances, we can see some pretty stark errors for the lower numbers of speakers. Around about 20 speakers we hit around the target of the full training data, or at least they’re at a similar level to what those models performed on the training data.

Word boundary errors in held out data, colorboxes represent performance of


     MFA English IPA train

on all Buckeye speakers

For the phone boundary errors in selected CVC words, it’s not super good results, particularly at the lower end of speakers, to be expected. Impressionistically, it again looks like 20 speakers is the point where it gets in the same ballpark as the full training run.

Phone boundary error rate in selected CVC words in held out data, colorboxes represent performance of


     MFA English IPA train

on all Buckeye speakers

For phone errors across all words, performance gets close to the best performance around that 20 speaker mark again. At that point we’re in the 19-21 ms range that we saw above for where the models generally fell on their training data.

Phone error rates across held out data, colorbox represents performance of


     MFA English IPA train

on all Buckeye speakers

So in general, I think the recommendation here is that around 20 speakers (corresponding to about 8-10 hours of training audio) generates a model that’s pretty good at generalizing to new speakers in the same variety. That caveat’s pretty important, as the farther away from the specific variety, the worse the generalization is going to be. Another note is that I didn’t do any balancing or control of speaker gender in the training data, so that would definitely affect generalization.

Summing up ¶

So hopefully this was a useful look at training data and how it impacts MFA alignments. Based on the above, I would recommend the following rules of thumb:

If you care only about alignments of the training data, 3-5 hours should be enough.
- Caveat: increasing the number of speakers/varieties in the training data will likely need more training data
If you care about generating models for more widespread use, 8-10 should be enough for generalizing to the same variety
- The more speakers the better, but also more speakers should need more data
- I usually recommend about 20 hours for a decently performant model

Speaker dictionaries and multilingual IPA

2021-08-15T00:00:00-07:00

[TOC]

Overview ¶

There’s a couple of new features in Montreal Forced Aligner 2.0 that I’d like to cover in a bit more depth to showcase how they might be useful for other people’s workflows, namely speaker-specific dictionaries and the multilingual IPA mode . I conceived of these in the context of a multilingual aligner that uses IPA pronunciation dictionaries to generate “pretty good” alignments for languages where you don’t have much annotated data yet, but want to get to reasonable boundaries for acoustic analysis.

There are already aspects of MFA that play into the idea of adaptation to new contexts, namely phone clustering and speaker adaptation. Phone clustering arose to combat the problem of sparsity in creating triphone models. In monophone models (i.e. those that do not take phonological context into account), you’re likely to see very good representation for each phone (with the exception perhaps of phones in loanwords). Once you move to triphone models and start to look at the preceding and following phones, you’ll hit problems with seeing all combinations. For a word like “Sri” in “Sri Lanka”, we’ll have many instances of /s/ , /ɹ/ , and /i/ occurring throughout a corpus, but very few instances of /sɹi/ as a triphone. To solve the issue of these low counts, phone clustering looks to combine similar model states across triphones. So in /sɹi/ , the /ɹ/ can leverage /ɹ/ in other more common phonological contexts. You can also think of it as similar to modelling allophonic variation, as the stops in triphones like /ɪti/ and /ɪdi/ can be modelled with the same state capturing a flap, even if it’s not explicitly coded in the pronunciation dictionary.

The speaker adaptation step transforms features from an initial alignment in a speaker independent feature space to a speaker normalized feature space. You can think about this a bit like vowel space normalization, where the vowel space for two people of the same dialect in raw acoustic formant space don’t correspond, but normalizing the formants yields a more consistent vowel space for a dialect.

We can apply this logic to the multilingual IPA mode. Say two languages both have a sound that’s transcribed /ɛ/ , speakers of these two languages will produce it slightly (or maybe even not so slightly) different. But with the phone clustering and speaker transforms, we can (probably) model both /ɛ/ phones pretty well. At least to the point that good forced alignments are possible, because forced alignment is a much more constrained problem space than something like large vocabulary ASR.

Introducing speaker dictionaries ¶

So that’s the MFA 1.0 feature set that could support multilingual models. However there’s a couple more issues to tackle to make it a more manageable problem. The first is speaker dictionaries. There is an assumption in ASR of one pronunciation dictionary to rule them all. That is, there is a singular lexicon that all possible pronunciations should be drawn from. This assumption kinda works okay with the above methods for dealing with variability, since it can cover a decent chunk of speaker variability (and some dialect variability), but once you get into the realm of deletions or very different pronunciations it starts to fall apart. Dialects in English that are r-ful versus those that are r-less are going to run into confusions for words like “center” ( /s ɛ n t ɚ/ vs /s ɛ n t ə/ ). It is possible that the system will learn and output the correct variant, but if we can constrain the system to only produce correct variants per dialect/speaker, then it’s just much easier for the system to learn the acoustic models overall.

The problem gets even larger when you consider two languages that have radically different phonologies but lots of shared vocabulary. A shared lexicon between English and French will have a decent overlap in orthographical forms, but the pronunciations for those shared words are going to be completely different. As a concrete example, compare “centre” between French ( /s ɑ̃ t ʁ ə/ ) and British English ( /s ɛ n t ə/ ). Mistakenly selecting the French pronunciation for an English speaker will result in no /n/ being modelled, and might lead to issues later on when you do have a sequence of an oral vowel and a nasal consonant. Just in general if we can remove (well, reduce) the noise and variability in the pronunciation dictionary, we can constrain the problem space and learn better models.

The solution in the current version of MFA 2.0 is to allow the user to specify dictionaries by speaker . So we can say given a speaker and transcript, here is a much smaller list of pronunciation variants that can be chosen. To do this, MFA accepts a .yaml file as an argument to the dictionary path which will be parsed as a mapping between speakers and dictionaries. So the command below

    ::bash
mfa train /path/to/corpus /path/to/dictionary.txt /path/to/output_directory

becomes

    ::bash
mfa train /path/to/corpus /path/to/speaker_dictionaries.yaml /path/to/output_directory

Where /path/to/speaker_dictionaries.yaml is a file that looks like:

    ::yaml
default: english_ipa

uk_speaker_one: /path/to/english_uk_ipa.txt
uk_speaker_two: /path/to/english_uk_ipa.txt

The only requirement is that one key specifies the default dictionary, which will be used for all speakers that do not have an explicit dictionary.

Multilingual IPA mode ¶

Multilingual IPA mode is a flag specified during training that transforms IPA characters when parsing the dictionary. The idea is to preserve quality contrasts and remove length markers (i.e., collapse long and short vowels) and to split digraphs into their component sounds (i.e., split /eɪ tʃ/ into /e ɪ t ʃ/ ). One consequence of splitting digraphs is the minimum duration for these sounds will be greater (60ms rather than 30ms), but given the dynamics and complexities of these sounds, I think that’s a reasonable condition on them. Triphthongs, like in Mandarin, will likewise be split into three separate vowels.

The list of diacritics that multilingual IPA mode will strip are:

Diacritic	Description	Example Input	Example Output
◌ː	Long	iː	i
◌ˑ	Half-long	iˑ	i
◌̆	Extra short	ĭ	i
◌̩	Syllabic	m̩	m
◌̯ ◌̑	Non-syllabic	ʊ̯	ʊ
◌͡◌	Linking, not necessary in MFA because phones are delimited by spaces	t͡ʃ	tʃ
◌‿◌	Linking, not necessary in MFA because phones are delimited by spaces	t‿ʃ	tʃ
◌͜◌	Linking, not necessary in MFA because phones are delimited by spaces	t͜ʃ	tʃ

The following are the default digraph patterns for English. Note that digraph splitting happens after removing the diacritics above, so in cases of affricates with linking bars, the bar will be removed and then the component characters will be split.

Digraph pattern	Description	Example Input	Example Output
[dt][szʒʃʐʑʂɕç]	Affricates	tʃ	t ʃ
[aoɔe][ʊɪ]	Diphthongs	ɔɪ	ɔ ɪ

The above settings are primarily useful for the IPA English dictionary that I’m working on . To modify the settings, simply create a train_config.yaml and add/modify the following settings as necessary:

    ::yaml

strip_diacritics:
  - "ː"  # long, i.e. /ɑː/
  - "ˑ"  # half long, i.e. /ɑˑ/
  - "̆"  # extra short, i.e. /ĭ/
  - "̯"   # non syllabic, i.e. /i̯/
  - "͡"  # linking, i.e. /d͡ʒ/
  - "‿"  # linking, i.e. /d‿ʒ/
  - "͜"  # linking, i.e. /d͜ʒ/
  - "̩"   # syllabic, i.e. /n̩/

digraphs:
  - "[dt][szʒʃʐʑʂɕç]" # affricates
  - "[aoɔe][ʊɪ]" # diphthongs

   

The digraphs will be parsed into regular expressions, so you can use Python’s re syntax to specify any additional patterns that should be split into separate characters.

One final thing important to note is that the modifications are all internal to the aligner. When the output TextGrids are generated, the original pronunciation is used to reconstruct the output, so split digraphs will be fused back together and remove the phone-internal boundaries and diacritics will be added back in.

Evaluating the multilingual mode ¶

Even though the Buckeye corpus is monolingual English corpus, we can use it to evaluate improvements that multilingual mode provide. The measures here are the same ones that I used in the updated MFA metrics blog post . The plot below shows the word boundary error across the Buckeye corpus. For comparison, I’ve included some default models from the metrics post.

Word boundary errors in the Buckeye corpus

The word boundaries are pretty consistent across the three training conditions. The consistency is likely the result of learning good representations of silence in the Buckeye corpus.

Phone boundary error rate in selected CVC words

Phone boundaries in the CVC words show pretty comparable rates, but in general nothing very conclusive. However, looking at phone errors across the whole of the corpus, we see improved phone boundary accuracy for the multilingual trained mode.

Phone error rates across the whole Buckeye corpus

Multilingual mode looks pretty good! It shows some additional improvements over just the IPA dictionary, but it’s pretty slight.

Combining multilingual IPA and speaker dictionaries ¶

One of my major side projects is to create a corpus of esports (professional video games) casting, which is a good test case for these new features. As an aside, a key feature of esports casting is that it is a broadcast style of spontaneous where speakers are reacting to events in game in real time.Vocabulary is much more restricted than general spontaneous speech, with in-game character and ability names repeated often, as well as player and team names. There are usually two casters that trade off turns, with a “play-by-play caster” that walks viewers through the low-level action and a “color caster” that does more analytical commentary about the bigger picture aspects of the match during any downtimes between high intensity moments. Relevant to the new features, each caster has their own accent, with representation across British, Australian, New Zealand, and American varieties, along with many non-native speakers.

Below is an example alignment in the corpus of speech by Sideshow, who is a British English speaker who does casting in Overwatch.

Example of UK speech in the esports corpus, note the non rhotic pronunciations in the highlighted region ( sound file and TextGrid if you want to take a look)

The above can be achieved with a single dictionary, of course, but what’s handy is being able to align a single file with speakers of different dialects (or languages). In the below example, there are two Overwatch casters, Nekkra and Tridd. Nekkra is a speaker of American English, whereas Tridd speaks a British dialect.

Example of US and UK speakers aligned in the same file, note the non rhotic pronunciation in the highlighted region versus the rhoticity earlier ( sound file and TextGrid if you want to take a look)

Summing up ¶

So that was a brief look into a couple of new features targeting datasets that use different speech varieties, either multiple dialects or multiple languages, with the same phone set. Multilingual IPA uses an internal representation of a simplified phone set to maximize similarities and reduce complexity while recombining and formatting the output to match the pronunciation dictionary representations. Speaker dictionaries allow for specifying pronunciation dictionaries to vary by speaker, while still sharing a single acoustic model, so the overall modelling should be more accurate and alignments better. The actual projects that I’m using these features for are still very much in progress, but hopefully these features will be helpful for you!

Update on Montreal Forced Aligner performance

2021-08-03T00:00:00-07:00

[TOC]

Overview ¶

The Montreal Forced Aligner version 1.0 released on May 18, 2017, with the Interspeech paper appearing that year as well. In the four years since, it has changed considerably and with the eventual release of 2.0 once it feels a bit more stable, I thought I’d take a moment to analyze some performance experiments that I’ve been playing around with.

In addition to features internal to MFA, I also want to take a look at how the performance compares to FAVE and MAUS , as the other two standards in the field. At some point I’d like to get LaBB-CAT ’s aligner set up, but that’s a bit more involved than the other two and leverages the same pretrained acoustic model as FAVE. If there are other aligner systems out there that I should benchmark, let me know.

Methodology ¶

Dataset ¶

The benchmark dataset is derived from the Buckeye corpus , the same as the 2017 Interspeech paper , where utterances are extracted with text transcriptions into small chunks of audio that are fed into the respective aligners.

33,370 extracted chunks
40 speakers
16.5 hours total duration

I’m expanding the benchmark of the Buckeye corpus . The word distance and CVC datasets are the same, but I’m also interested in the larger phone accuracies outside the more controlled CVC dataset. I’ll cover the measures pulled from the corpus more in depth as they come up in the results section.

Procedure ¶

Aligner ID	Phone set	Alignment procedure
FAVE	ARPABET	align
MAUS	X-Sampa	align
MFA English	ARPABET	align
MFA English adapted	ARPABET	adapt
MFA default train	ARPABET	train_and_align
MFA English IPA	IPA	align
MFA English IPA adapted	IPA	adapt
MFA English IPA train	IPA	train_and_align

The MFA runs represent a cross between training procedure and pronunciation dictionary. Training procedures are as follows:

align : Performs an initial alignment and then computes speaker-adapted features for a second pass alignment
adapt : Performs an initial alignment with an acoustic model followed by 3 rounds of speaker adaptation, alignment and re-estimation of the model
train_and_align : Performs the default training regime in MFA 2.0, which starts with a subset of utterances to train a monophone model, expands the subset to train a triphone model, performs LDA calculations and two rounds of speaker adaptation to refine the input features over increasing sizes of subsets.

For pronunciation dictionaries, the ARPABET dictionary is the standard one provided via MFA’s download command. The IPA dictionary was created recently (see the blog post about it for more details about its creation). Additionally, I used a newly introduced multilingual_ipa mode that does some normalizations to the IPA pronunciations to remove some unnecessary (for the aligner) variation (blog post coming soon! In the meantime refer to the docs ). In all cases of pretraining, the audio data is from LibriSpeech, just the pronunciation dictionaries differ.

Accuracy Results ¶

Let’s begin with the replication of Interspeech paper. First step is looking at the word boundary errors. So for every word boundary find the absolute distance between the manual annotation in the Buckeye corpus and the aligner generated annotations. The plot below shows these by the aligner and split into whether the boundary is next to a silence or not, as that heavily influences accuracy.

Word boundary errors in the Buckeye corpus

MFA generally looks to be performing well! One interesting thing to highlight is that aligners trained on the Buckeye corpus itself handle boundaries next to pauses better (though still not the same degree as within continuous speech). This benefit is likely due to the silence models being trained on the actual audio, whereas the silence models for the other MFA models are trained on LibriSpeech, which is overall less noisy.

Next up we have the CVC phone boundaries calculated in the Interspeech paper. To generate these, we took a list of 534 CVC word types in the Buckeye corpus (I believe originally from Yao Yao’s dissertation ), words like base , choose , gone , young , etc. For every token that matched, first we check whether the surface form from the manual annotation and the aligner annotation have three phones, and skip if they don’t. This avoids instances where a consonant is deleted. Following the check, we extract 4 boundaries:

beginning of the initial consonant
between the initial consonant and the vowel
between the vowel and the final consonant
at the end of the final consonant

Phone boundary error rate in selected CVC words

We can see that the word boundaries have the highest error (also note that we’re not controlling for pause boundaries here, so that’s going to make these a bit messier). General trends are also that CV transitions are more accurate than VC transitions, even across aligners. Somewhat surprising is that the pretrained IPA English model is doing as well/better than the adapted and trained versions.

For completeness, here are the same summary statistics from the above plots.

Aligner	Word boundaries		Phone boundaries
Aligner	mean (ms)	median (ms)	mean (ms)	median (ms)
FAVE	24.8	16.6	19.2	12
MAUS	29.6	15	20.5	10.4
MFA English	23.6	15.4	16.3	11
MFA English adapted	23.4	15	16.6	11.1
MFA default train	21.1	14.1	16.9	11.3
MFA English IPA	21.4	14.1	15.8	9.9
MFA English IPA adapted	21.6	14.3	16.4	10.3
MFA English IPA train	20.2	14	16.5	10.7

So that replicates the basic findings of the Interspeech paper. Note that the numbers for MFA english (in the Interspeech paper labelled as MFA-LS ) and MFA default train ( MFA-Retrained in the Interspeech paper) have actually improved very slightly from MFA 1.0 to 2.0. MFA english went from 24.1 ms mean word boundary distance to 23.6 ms, 17 ms mean phone boundary distance to 16.3 ms. MFA default train went from 22.6 ms mean distance for words to 21.1 and from 17.3 ms to 16.9 ms for phone distance.

So that’s all cool, but the main reason I really wanted to re-explore this space is to come up with some better measures for actually measuring alignment accuracy. The phone boundary measures after all cover just CVC words, are a very small subset of total words in the Buckeye corpus, and pointedly ignore pronunciation variation in order to make things as controlled as possible. I think we can do better.

So the algorithm that I’ve done here is still a work in progress, but I think it’s pretty promising. The basic idea is that we want to align intervals in different transcripts (haha, aligning the alignments) based on how close those intervals begin and end (i.e., if they occupy the same space in the audio file, they’re likely the same segment). We also want to incorporate some very basic phone-to-phone mapping between the different phone sets to make sure it’s not entirely based off the timing, because the identity of phones is important too for evaluating whether an alignment is good.

The Python code is something like this (I’ll describe it more in words after the code if you want to skip that):

   :::python
import re
from Bio import pairwise2
import functools

MATCH_SCORE = 0
MISMATCH_SCORE = 2
GAP_START_SCORE = -5
GAP_CONTINUE_SCORE = -5


def silence_check(phone):
    return phone in {'sp', '<p:>', '', None}


def compare_labels(ref, test, aligner_name):
    if 'ipa' in aligner_name:
        mapping = ipa_mapping
    elif 'maus' in aligner_name:
        mapping = maus_mapping
    else:
        mapping = arpa_mapping
        if re.match(r'[a-z]+[0-9]', test):
            if ref == test[:-1].lower():
                return MATCH_SCORE
            else:
                return MISMATCH_SCORE
    if ref == test:
        return MATCH_SCORE
    if test in mapping and mapping[test] == ref:
        return MATCH_SCORE
    ref = ref.lower()
    test = test.lower()
    if ref == test:
        return MATCH_SCORE
    return MISMATCH_SCORE


def overlap_scoring(firstElement, secondElement, aligner_name):
    begin_diff = abs(firstElement.minTime - secondElement.minTime)
    end_diff = abs(firstElement.maxTime - secondElement.maxTime)
    label_diff = compare_labels(firstElement.mark, secondElement.mark, aligner_name)
    return -1 * (begin_diff + end_diff + label_diff)


def align_phones(ref, test, aligner_name):
    ref = [x for x in ref]
    test = [x for x in test]
    score_func = functools.partial(overlap_scoring, aligner_name=aligner_name)
    alignments = pairwise2.align.globalcs(ref, test,
                                          score_func, GAP_START_SCORE, GAP_CONTINUE_SCORE,
                                          gap_char=['-'], one_alignment_only=True)
    overlap_count = 0
    overlap_sum = 0
    num_insertions = 0
    num_deletions = 0
    for a in alignments:
        for i, sa in enumerate(a.seqA):
            sb = a.seqB[i]
            if sa == '-':
                if not silence_check(sb.mark):
                    num_insertions += 1
                else:
                    continue
            elif sb == '-':
                if not silence_check(sa.mark):
                    num_deletions += 1
                else:
                    continue
            else:
                overlap_sum += abs(sa.minTime - sb.minTime) + abs(sa.maxTime - sb.maxTime)
                overlap_count += 1
    return overlap_sum / overlap_count, num_insertions, num_deletions

  

So what it does is use biopython ’s pairwise sequence alignment algorithm with a custom match/mismatch function. The score that we give to match/mismatch is not fully based on the label of the phone (though if it does not match, it gets a base score of -2 vs 0 for if it does match). There are three mappings listed above (but not defined just because they’re so long, I’ll provide tables in the appendix if you’re curious). These basically transform “known” or “ unambiguous” phones from X-Sampa, IPA and ARPABET into Buckeye’s specific phone set. In addition to the label match, we also calculate the distance from the reference boundaries to the corresponding aligner-generated boundaries, in order to locally constrain the alignment. The last parameter is the gap scores, which specify the penalty for inserting skips in one of the sequences, either for insertions or deletions. The value of -5 for both seems to work pretty well for this particular case. To illustrate the output from running with a FAVE alignment:

Example phone-to-phone alignment

One final thing to note is that we’re basically ignoring any mismatches due to silences, so not inserting silence is not penalized per se.

Onto the actual results! Well, ok, before that, a note about some removals. In analyzing the data, I found that MAUS for some reason phonemetized yknow as /i:keIkeIkeIkeI/ . I’m not really sure where this is coming from, but it’s fairly common in the Buckeye transcriptions and resulted in an inflated number of insertions. So for the following analysis, any utterance with yknow in it has been removed (2,575 utterances of 33,166 total utterances).

To begin with, let’s take a look at the phone boundary errors.

Phone error rates across the whole Buckeye corpus

Even removing the yknow utterances, MAUS is still having some issues here, which is a little weird because we weren’t seeing these disparities show up this much in the word boundary or CVC datasets. To dig a little deeper, if we look at this same data faceted by speaker, we can see a lot of these issues are driven by a few speakers.

Phone boundary error analysis by speaker

Particularly s01 , s13 , s15 , s20 , s23 , s24 , s27 , and s36 are contributing more so than others to the increased error rate. I don’t know for sure what’s causing the issues, it might be something about vocal quality, recording quality, or pronunciation generation that’s affecting those more so than others.

Looking at the other measures of quality, we can see that MAUS is performing the best when it comes to insertions ( when yknow isn’t counted…). Additionally, you can see that IPA versions of MFA are inserting less, which makes sense because the ARPABET versions have multiple characters for syllabic consonants (i.e. /l̩/ like in bottle is transcribed as AH0 L ).

Number of phone insertions

Finally, in terms of deletions, these are less common than insertions overall. I think the primary reason for this is that our reference is the surface form annotations in the Buckeye corpus. These surface forms are generally pretty reduced and close to the spoken form, so our the ground truth is not citation forms. Of note, FAVE tends to insert much more than it deletes, which is due to its reliance on CMU dictionary’s citation forms. MAUS on the flip side tends to delete more than it inserts, likely due to the post-lexical pronunciation modelling that it does. IPA versions of MFA tend to delete more than the ARPABET versions, I haven’t fully analyzed this, but my guess here is that some of these phones that are still in the Buckeye surface form transcriptions get deleted in the IPA version because they’re shorter than the minimum duration for a phone (30ms), so the cost of deleting is less for it.

Number of phone deletions

Summing up ¶

MFA’s not looking too shabby! It’s nice to see the improvements made in the transition to version 2.0 having concrete improvements. Not surprisingly, the IPA dictionary is showing some good performance specifically with the Buckeye dataset due to similarities in the syllabic consonants. It’s not as phonetic as the Buckeye phone set, but with the limitation of 30ms minimum on phones, it can’t really quite get there.

Now, of course, the huge caveat to all this is that it’s English. I’d be super curious to see how performance differs across languages, but I’m not aware of too many manually corrected datasets out there that could serve as additional benchmarks, but I’d be curious to see how the methodology here stacks up.

Appendix ¶

Benchmark performance metrics ¶

The accuracy metrics above are the primary point of this point, but I do want to touch on the timing benchmarks done as part of this, as speed can be a consideration as well. The below graph shows how long each of the MFA runs took. Note that I didn’t benchmark the FAVE or MAUS cases, since they were not really optimized for this use case, but they each took several hours to generate, FAVE about a day’s worth of run time, MAUS about 6 hours of manual upload and download via WebMAUS.

For the MFA runs, the following system was used:

12 cores
i9-10900KF at 3.7 GHz
32 GB of RAM
Windows installation of MFA

MFA timing results

The align runs are the quickest, coming in at under 15 minutes to generate all the TextGrids. The adapt runs are the next fastest, at a little over half an hour to do a couple of adaptation iterations and then align. Training unsurprisingly takes the longest at just around an hour and a half.

The next metrics are the final average per-frame log-likelihood on the final alignment. While I haven’t had success using this kind of metric on a per-utterance basis to figure out what files have issues like missing words, it does seem generally useful for gauging how well an alignment system is doing. Compare the below figure with the above phone boundary error rates across the full corpus. The log-likelihood for MFA English and MFA English adapted are much lower than the other runs, and these are exactly the runs that have higher phone boundary error rate above. So at least a sanity check that this can be used as a proxy measure for when human annotations aren’t available to compare against.

MFA log-likelihood results

Phone mappings ¶

    maus_mapping = {
    'dZ': 'jh',
    'Z': 'zh',
    '6': 'er',
    'tS': 'ch',
    'S': 'sh',
    'Q': 'aa',
    'A': 'aa',
    'A:': 'aa',
    'Q:': 'aa',
    '{': 'ae',
    '{:': 'ae',
    'E': 'eh',
    '@': 'ah',
    'V': 'ah',
    'V:': 'ah',
    'D': 'dh',
    'T': 'th',
    'e': 'ey',
    'e:': 'ey',
    'aU': 'aw',
    '@U': 'ow',
    'oU': 'ow',
    'eI': 'ey',
    'i:': 'iy',
    'u:': 'uw',
    'u': 'uw',
    'U': 'uh',
    'O:': 'ao',
    'O': 'ao',
    'o': 'ow',
    'o:': 'ow',
    'aI': 'ay',
    'OI': 'oy',
    'oI': 'oy',
    'n': 'en',
    'm': 'em',
    'l': 'el',
    'i': 'iy',
    'j': 'y',
    'I': 'ih',
    '?': 'tq',
    'N': 'ng',
    'h': 'hh',
}

ipa_mapping = {
    'ʔ': 'tq',
    'i': 'iy',
    'h': 'hh',
    'iː': 'iy',
    'ɡ': 'g',
    'ɚ': 'er',
    'ɝ': 'er',
    'ɝː': 'er',
    '3`': 'er',
    'dʒ': 'jh',
    'tʃ': 'ch',
    'ʒ': 'zh',
    'ɑ': 'aa',
    'ɑː': 'aa',
    'ʊ': 'uh',
    'ɛ': 'eh',
    'oʊ': 'ow',
    'aʊ': 'aw',
    'aɪ': 'ay',
    'ɔ': 'ao',
    'ɔː': 'ao',
    'ɔɪ': 'oy',
    'u': 'uw',
    'uː': 'uw',
    'æ': 'ae',
    'eɪ': 'ey',
    'ð': 'dh',
    'ʃ': 'sh',
    'ɹ': 'r',
    'j': 'y',
    'θ': 'th',
    'ə': 'ah',
    'ŋ': 'ng',
    'ʌ': 'ah',
    'n̩': 'en',
    'm̩': 'em',
    'l̩': 'el',
}

arpa_mapping = {
    'N': 'en',
    'M': 'em',
    'L': 'el',

}

   

Bootstrapping an IPA dictionary for English using Montreal Forced Aligner 2.0

2021-04-19T00:00:00-07:00

[TOC]

Overview ¶

One of my big tinkering projects that I’ve been thinking a lot about is a multilingual aligner that uses IPA to be able to share acoustic models across languages and maybe even help align data from languages without requiring an initial corpus of ~20 hours to get a good acoustic model. That whole project is outside the scope of the current post (but see the final thoughts section at the end for more on this). What I wanted to do with this blog series is document the end-to-end process of constructing this multilingual aligner with the hopes that these tutorials are useful for other people working with their specific data sets. With the WikiPron data sets, there’s a number of crowdsourced pronunciation dictionaries available (see here for a listing), this blog post is intended to serve as a guide for cleaning those up and bootstrapping larger pronunciation dictionaries.

What I’m currently working on is getting an IPA dictionary set up on the level of the Arpabet dictionary for LibriSpeech. The primary issue is in terms of coverage, where the English dictionary we use for aligning LibriSpeech has 206,518 lexical entries whereas the WikiPron English US dictionary only has 57,214 entries, leaving a large gap in the dictionary if we were to use it with supplementing.

The basic process for bootstrapping a larger dictionary with new entries was as follows:

Clean up and standardize initial WikiPron dictionary
Train a G2P model for generating IPA pronunciations for English
Generate Out of Vocabulary (OOV) list
Generate pronunciations using the G2P model for a subset of OOV words
Check pronunciations generated and add new ones to the dictionary
Retrain G2P model as necessary

Generating the initial dictionary ¶

The basic dictionary was downloaded from WikiPron’s scrape page . Given Wikitionary’s crowdsourced nature, I did some initial analysis of the graphemes and phones being used. There were several foreign graphemes included, so words containing these were ignored to simplify the grapheme set:

   % / @ ² à á â ä æ ç è é ê ë í î ï ñ ó ô õ ö ø ù ú ü ā ą č ē ę ğ ı ł ń ō ő œ ř ū ș ț ʼ ṭ ₂

  

All told, there were only 487 words with these characters, so removing these still left 56,727 total entries.

For processing the pronunciations, the initial phone set for the dictionary was:

   a aː b d d͡ʒ e eː f h i iː j k l l̩ m m̩ n n̩ o oː p r s t t͡ʃ u uː v w z æ æː ð ŋ ɑ ɑː ɒ ɔ ɔː ə ɚ ɛ ɛː ɜ ɜː ɝ ɝː ɡ ɪ ɪː ɪ̯ ɫ ɹ ʃ ʊ ʊ̯ ʌ ʍ ʒ ʔ θ

  

The first automated pass transforming the dictionary took out ligatures. With phones in pronunciation dictionaries separated by spaces, having the ligatures is not necessary. However, the WikiPron pronunciations had some inconsistencies for digraphs for affricates and diphthongs. The following sequences were normalized:

Original WikiPron phone	Phone for MFA dictionary
/t͡ʃ/, /t ʃ/	/tʃ/
/d͡ʒ/, /d ʒ/	/dʒ/
/e i/, /e ɪ/, /e ɪ̯/	/eɪ/
/a i/, /a ɪ/, /a j/, /a ɪ̯/	/aɪ/
/a u/, /a ʊ/, /a ʊ̯/	/aʊ/
/o u/, /o ʊ/, /o w/, /o ʊ̯/	/oʊ/
/o i/, /o ɪ/, /ɔ i/, /ɔ ɪ/, /ɔ ɪ̯/	/ɔɪ/

Even following this, there were some /e/, /o/ and /a/ phones left over. These were manually corrected into some combination of /eɪ/, /ɛ/, /oʊ/, /ɔ/, /ɑ/. Additionally, some orthographic characters were used in pronunciations, so these were replaced with the correct IPA character:

Original WikiPron phone	Phone for MFA dictionary
/ɒ/	/ɔ/
/r/	/ɹ/
/ʍ/, /h w/	/w/
/ɫ/	/l/

The voiceless /ʍ/ was merged with /w/, both because these sounds are merged in most American dialects, and /ʍ/ appeared in a very inconsistent subset of pronunciations. Similarly, /ɫ/ was merged with /l/ due to a similar inconsistency.

Vowels before rhotics were standardized as well to the following set based on the set listed on the English phonology wikipedia .

Lexical set	English US IPA
NURSE	/ɝ/
START	/ɑr/
NORTH	/ɔr/
FORCE	/ɔr/
NEAR	/ɪr/
SQUARE	/ɛr/
CURE	/ʊr/

To make the transcriptions in WikiPron adhere to the sets above, the following transformations were applied to standardize the vowel-rhotic sequences:

Original WikiPron phone	Phone for MFA dictionary
/æ ɹ/	/ɛ r/
/i ɹ/	/ɪ ɹ/

Finally some unstressed syllabic consonants were created to make them consistent with syllabic consonants elsewhere and to aid in alignment.

Original WikiPron phone	Phone for MFA dictionary
/ə n/	/n̩/
/ə l/	/l̩/
/ə m/	/m̩/
/ə ɹ/	/ɚ/

In unstressed syllables, Arpabet pronunciations usually have separate schwa-consonant sequences (i.e. AH0 L , AH0 N , AH0 M ), but not for syllabic r (i.e. ER0 ). This leads to alignment issues using these dictionaries, because the default settings lead to 30ms minimum for phones, so these unstressed syllables have minimum 60ms duration. In running spontaneous speech, we’re unlikely to see unstressed syllables that are that long, and the quality of these syllables are much more like their consonant articulations than a vowel-consonant sequence.

For reference, the final phone inventory is:

Consonants	Vowels
p	i
b	iː
t	u
d	uː
k	ɪ
ɡ	ʊ
ʔ	eɪ
m	oʊ
m̩	ɛ
n	ɛː
n̩	ɝ
ŋ	ɝː
f	ʌ
v	ə
θ	ɚ
ð	ɔ
s	ɔː
z	æ
ʃ	æː
ʒ	ɑ
tʃ	ɑː
dʒ	aɪ
h	aʊ
j	ɔɪ
w
l̩
l
ɹ

As a more general point, the specific purpose for this dictionary is for alignment and speech tech and does not strictly represent English phonology per se. However, it’s also not strictly phonetic either, as /t/ and /d/ flapping are not represented in pronunciations. As the above mentions, the most important part is number of segments, since allophonic variation can be capture through the use of clustered triphones in alignment systems.

Bootstrapping a new dictionary ¶

Following the transformations above, we have a reasonable starting point for bootstrapping a large scale dictionary.

Training a G2P model ¶

Following installation of MFA into a conda environment, the first step is train a G2P model:

    mfa train_g2p english_us.txt english_us_ipa_g2p.zip --num_jobs 13

   

And voila! Sort of. It takes me about 30 minutes to train a G2P model over the (as of this post) 60,756 words in english_us.txt. The command above uses 13 processes to perform training, so depending on your set up, you may need to adjust it (the default is 3). The training can be customized (see the MFA docs on training G2P models for more details), but the default configuration should be a reasonable set up for both the current English dictionary and other dictionaries you might want to train on.

Get list of unknown words ¶

Once the G2P model is trained, we can generate a list of OOV words by running MFA’s validator utility:

    mfa validate /mnt/d/Data/speech/LibriSpeech/librispeech_mfa english_us.txt -t /mnt/d/temp/validate_temp --ignore_acoustics

   

So what this command will do is run through all of the files in the LibriSpeech folder (the librispeech_mfa folder in the command above is a preprocessed version that uses the Prosodylab format for MFA ), check for various issues like OOV words, errors reading the sound files, missing transcriptions, etc. The validator will generally also do a pass of generating the MFCC features and doing a quick monophone alignment to check for issues, but having the --ignore_acoustics flag skips this sequence to save time. The output of this command should be something like:

    (aligner) user@hostname:/mnt/d/Data/speech/librispeech_ipa$ mfa validate /mnt/d/Data/speech/LibriSpeech/librispeech_mfa english_us.txt -t /mnt/d/temp/validate_temp --ignore_acoustics
/mnt/d/temp/validate_temp/librispeech_mfa/validate.log
INFO - Setting up corpus information...
INFO - Parsing dictionary without pronunciation probabilities without silence probabilities
INFO - Creating dictionary information...
INFO - Setting up training data...
Skipping acoustic feature generation
INFO -   ===============================Corpus===============================
292367 sound files
292367 sound files with .lab transcription files
0 sound files with TextGrids transcription files
0 additional sound files ignored (see below)
2484 speakers
292367 utterances
0.000 seconds total duration

DICTIONARY
----------
There were 55415 word types not found in the dictionary with a total of 271084 instances.
Please see/mnt/d/temp/validate_temp/librispeech_mfa/corpus_data/oovs_found.txtfor a full list of the word types and/mnt/d/temp/validate_temp/librispeech_mfa/corpus_data/utterance_oovs.txtfor a by-utterance breakdown of missing words.

SOUND FILE READ ERRORS
----------------------
There were no sound files that could not be read. There were no sound files that had unsupported bit depths.

FEATURE CALCULATION
-------------------
Acoustic feature generation was skipped.

FILES WITHOUT TRANSCRIPTIONS
----------------------------
There were no sound files missing transcriptions.

TRANSCRIPTIONS WITHOUT FILES
--------------------
There were no transcription files missing sound files.

TEXTGRID READ ERRORS
--------------------
There were no issues reading TextGrids.

UNREADABLE TEXT FILES
--------------------
There were no issues reading text files.Skipping test alignments.
INFO - All done!

   

In the temporary directory, MFA will output an oovs_found.txt file (in this case, /mnt/d/temp/validate_temp/librispeech_mfa/corpus_data/oovs_found.txt ). This file will be a list of all words in the corpus that were not found in the dictionary, sorted by frequency in the corpus, so adding pronunciations for the initial entries is more important than adding entries for the words at the end that might only occur once or twice. You can see the exact frequencies for OOV words in the oovs_count.txt file.

Perform G2P ¶

In the validation output above, there’s a lot of missing words (55,415 types and 271,084 tokens!). Generating pronunciations for all of these is going to take a long time, even with multiprocessing. In addition, at the start of bootstrapping, the data isn’t going to be as consistent as you might like. Some of the transformations I mentioned above were discovered only when I generated a pronunciation (and somehow got a trill /r/ as an output phone). The best process I’ve found is an iterative bootstrap approach, where I generate pronunciations for the most frequent 1,000 OOV words (this doesn’t take very long), add the vetted pronunciations to the dictionary and retrain as necessary.

The command I use to generate pronunciations for the subset file ( oovs_g2p.txt below) is:

    mfa g2p english_us_ipa_g2p.zip to_g2p.txt oovs_g2p.txt --num_jobs 13 --num_pronunciations 3 -t /mnt/d/temp/g2p_temp

   

As with the training, I’m using 13 cores for generating these pronunciations (took about 10 minutes to process 1,500 words in the latest batch). The additional flag for --num_pronunciations is set to 3, so for each word, the best 3 candidates will get returned. For transparent orthographies like Spanish, you probably don’t need to generate as many pronunciations, but for English it provides variable pronunciations in the output.

The output of the above command will be a file ( oovs_g2p.txt above) with contents like the following:

Orthography	Pronunciation from G2P
paupers	p ɔ p ɚ z
paupers	p ɑ p ɚ z
paupers	p ɔ p ɚ s
incarnate	ɪ n k ɑ ɹ n ɪ t
incarnate	ɪ n k ɑ ɹ n eɪ t
incarnate	ɪ n k ɑ ɹ n ə t
peerless	p ɪ ɹ l ə s
peerless	p ɪ ɹ l ɪ s
peerless	p ɪ ɹ l ɛ s
perfumes	p ɝ f j uː m z
perfumes	p ɚ f j uː m z
perfumes	p ə ɹ f j uː m z
persians	p ɝ ʒ ə n z
persians	p ɝ ʒ n̩ z
persians	p ɝ ʃ n̩ z
petrovna	p ɛ t ɹ ə v n ə
petrovna	p ɛ t ɹ oʊ v n ə
petrovna	p ə t ɹ ə v n ə
petted	p ɛ t ɪ d
petted	p ɛ t ə d
petted	p ə ʃ eɪ d
ploughed	p l aʊ d
ploughed	p l ʌ f t
ploughed	p l ɑ f t
poisoning	p ɔɪ z ə n ɪ ŋ
poisoning	p ɔɪ z n ɪ ŋ
poisoning	p ɔɪ z n i ŋ
poring	p ɔː ɹ ɪ ŋ
poring	p ɔ ə ɹ ɪ ŋ
poring	p ə ɹ ɪ ŋ
portico	p ɔ ɹ t ə k oʊ
portico	p ɔː ɹ t ə k oʊ
portico	p ɔː ɹ t i k oʊ
pretends	p ɹ ɪ t ɛ n d z
pretends	p ɹ iː t ɛ n d z
pretends	p ɹ ə t ɛ n d z
prostrated	p ɹ ɑ s t ɹ eɪ t ɪ d
prostrated	p ɹ ə s t ɹ eɪ t ɪ d
prostrated	p ɹ ɑː s t ɹ eɪ t ɪ d
pulses	p ʌ l s ɪ z
pulses	p ʌ l s iː z
pulses	p ʌ l s i z
quickening	k w ɪ k ə n ɪ ŋ
quickening	k w ɪ k ɪ n ɪ ŋ
quickening	k w ɪ k ə n i ŋ
quietness	k w aɪ ə t n ə s
quietness	k w aɪ ɪ t n ə s
quietness	k w aɪ ə t n ɪ s

By generating three candidates, we are able to capture a number of sources of variability including dialect variation ( paupers as /p ɔ p ɚ z/ or /p ɑ p ɚ z/), different stress for different grammatical categories (adjective incarnate as /ɪ n k ɑ ɹ n ɪ t/ and verb incarnate as /ɪ n k ɑ ɹ n eɪ t/), and also just random orthography-phonology nonsense that English has ( ploughed in the list above has the first candidate as /p l aʊ d/, but really /p l ɑ f t/ isn’t a bad guess considering coughed ). Again for languages with more straightforward orthography-phonology mappings, you could probably get away with just one candidate.

Update dictionary with new pronunciations ¶

For selecting candidates, I try to balance speed of adding entries with the following criteria:

including common dialectal variations (like both /ɔ/ and /ɑ/ in caught - cot merger words, though there may be some inconsistency on my part as I don’t have the distinction)
including unstressed vowel variation ( pronounce as /p ɹ ə n aʊ n s/ or /p ɹ oʊ n aʊ n s/)
some less common dialectal variation (like news as /n uː z/ or /n j uː z/, straw as /s t ɹ ɑ/ or /s tʃ ɹ ɑ/)
some phonetic variation, mostly around common deletions ( smouldering as /s m oʊ l d ə ɹ ɪ ŋ/ or /s m oʊ l d ɹ ɪ ŋ/)

But again, speed is the main criteria as there are a lot of missing words. The dictionary is available on GitHub here . Feel free to download, take a look, and if you find any issues, feel free to create a pull request (or let me know).

The dictionary can be downloaded to use with MFA alignment with the command:

    mfa download dictionary english_us_ipa

   

and the G2P model with the command:

    mfa download g2p english_us_ipa_g2p

   

Final thoughts ¶

I hope this is useful as both a tutorial and a starting point for using IPA in English alignment. The eventual goal of my tinkering with this is to include a bunch of corpora together and create an IPA-based acoustic model to use with any language that has a dictionary in IPA. Of course, I think that’s still a bit of a pipe dream given variation in IPA, but the hope is that with the phone clustering under the hood for triphones in Kaldi, along with speaker-adaptation and other feature transformations, we might be able to get relatively good performance.

I have some thoughts and opinions on having an IPA parser inside of MFA that takes out some diacritics so that phonesets are more comparable:

length markers could be internally discarded, and then added back when creating TextGrids
diphthongs and affricates could be split up internally so that you’re not reliant on a single language providing data for something like /tʂ/, and could instead leverage other language data more straightforwardly for /ʂ/
stripping out tone marking since it’s a segmental alignment (particularly cases where it’s a separate symbol in the transcription)

I’m planning on doing more of these blog posts for other languages (and most of them I don’t speak, so I’ll need some feedback on what makes sense and what doesn’t for those). I’m hoping also that the dictionary in this post can be a bit of a living resource as I and hopefully others update it, but at the very least I hope this is useful as a tutorial on using MFA’s G2P functionality to bootstrap and speed up dictionary development.

How to create synthetic speech continua using STRAIGHT

2017-02-23T00:00:00-08:00

[TOC]

Overview ¶

What is STRAIGHT? ¶

STRAIGHT is a program that analyzes speech, decomposes it into source and filter characteristics, and then recombines it after some manipulation or interpolation.

This tutorial is more of a pratical guide for using STRAIGHT to create various synthetic speech. More detailed information and publications about STRAIGHT are available on the official website .

What is STRAIGHT good for? ¶

STRAIGHT is best used for creating continua between two naturally-produced endpoints. This allows for holistic morphing between the two, rather than just mixing one part of the sound file, so aspects of coarticulation can be better preserved. Alignment between sound files is both temporal in the waveform and also in the frequency domain, allowing for peaks in the spectra to be shifted around, rather than just mixed linearly.

What is STRAIGHT not good for? ¶

Parametric synthesis
- Can’t specify formant values
- Can’t specify values for manipulation precisely (click and drag interface)
  - At least in the GUI (Matlab source has greater control, but less user-friendly)
Articulatory synthesis
Transient acoustic event synthesis
- Waveform gets averaged out, stops don’t have as strong of bursts
Non-speech synthesis
- Explicitly assumes source-filter approach

STRAIGHT Basics ¶

Synthesis using STRAIGHT involves analyzing a wav file of naturally produced speech and then resynthsizing a new wav file.

There are three major components involved that are covered in depth:

Wav file analysis
Morphing
Manipulation

The workflow that I’ve found works best to do all the analysis of wav files first, save the output of each file (preferrably in a backed up location, i.e. Dropbox folder) before doing any morphing. You can do the analysis to continuum for each recorded end point, but I find this slower overall.

One important thing to note is how dialogs interact with each other. In general, the changes you do in one dialog (i.e., analysis or anchoring) will not be transferred to the main dialog (Morphing Menu, for instance) unless a specific button is pressed (“Set up anchors” or “Finish/Upload” or “Update frequency anchors”). I tend to recommend that the output of each step be saved to a mat file, and then loaded in the main dialog, just so that intermediate steps get saved and so that you’re sure the main dialog has the correct data.

Analysis ¶

There are three steps to analyze a file, which involve separating a waveform into source and filter characteristics. Source and filter can then be manipulated separately and recombined in the final synthesis.

F0/F0 structure extraction
Aperiodicity extraction
STRAIGHT spectrum

Analysis can be done either from the MorphingMenu program, or from the TandemSTRAIGHTHandler program. I recommend doing the analysis of all sound files in one big batch through the TandemSTRAIGHTHandler program, save the results as .mat files, and then load those .mat files as needed in the MorphingMenu to create continua.

F0/F0 structure extraction ¶

This step extracts the voiced source characteristics of the signal, for all voiced segments in a file. To begin extracting the F0 structure, press the “F0/F0 structure extraction” button to open the F0 extraction dialog.

{: .center-image } The following screen shot is for the asi_initialstress.wav file. You’ll notice in the waveform at the bottom of the dialog that there are two vowel portions, a glottal stop at the beginning, and an sibiliant between the vowels.

{: .center-image }

The first step is press the “Calculate” button to generate an initial pitch track and voiced/unvoiced regions of the waveform.

{: .center-image }

This pitch track is generally pretty good, but clearly has some issues in silent and unvoiced sections. Likewise, there’s a few errors in which frames are voiced versus unvoiced, namely with missing voiced frames in the middle of vowels.

The next step is to apply “Auto tracking” to clean up the pitch track.

{: .center-image }

The pitch contour is now much cleaner, and the missing voiced frames in the middle of vowels are now there. However, extra voiced frames are now present in the middle of the sibilant, which should be corrected manually.

To change voicing regions, you can click the edge of a voiced region and drag it around. To delete these regions, I find it easiest to drag one of the voiced regions from the vowel over the erroneous regions, and then drag it back to where it should be. You can also click the right side of a voiced interval, and drag it all the way to the left side of it, which will remove it as well. Also, refer to the F0 extraction interface page for more information about interacting with the dialog.

{: .center-image }

Once you’re happy with the voiced/unvoiced parts of the file, press the “Finish/upload” button to send what you’ve done back to the main analysis window.

Aperiodicity extraction ¶

Aperiodic portions of the signal are extracted and analyzed. Aperiodic portions are usually obstruents, but it can also (unintentionally) model noise in the recording.

{: .center-image }

This is simple from the user’s point of view, no extra dialog is opened or extra input needed.

STRAIGHT spectrum ¶

This step extracts the filter characteristics of the signal. This is similar to an LPC spectrum or a power spectrum lacking source information like harmonics or noise sources.

{: .center-image }

Like the aperiodicity extraction, no extra dialog is opened when calculating the spectrum.

Once a waveform has been fully analyzed, it is ready to be manipulated and resynthesized or morphed with another analyzed sound file to create a continuum.

The most common use case is to create a continuum via morphing.

Morphing ¶

The most common way to use STRAIGHT in speech perception experiments is to morph between two similar sound files that differ in a key way.

In this page, we’ll morph between asi ( /'ɑsi/ ) and ashi ( /'ɑʃi/ ) with stress on the initial syllable. The wav files are asi_initialstress.wav and ashi_initialstress.wav , and the pre-analyzed mat files are StrObjasi_initialstress.mat and StrObjashi_initialstress.mat .

Load these files into the morphing menu, either as the mat files, or analyze the wav files.

{: .center-image }

There are three steps to creating a continuum:

Alignment of the two files
Setting up morphing rate
Generating the continuum

Alignment of the two files ¶

To begin with, once the sound files are loaded and analyzed, press the “Open anchoring interface” button to specify the alignment between the two. For details on how to perform various actions in the anchoring interface, please see here .

{: .center-image }

This will open a window with the two waveforms and spectrograms (asi on top as “Speaker A” and ashi on bottom as “Speaker B”).

{: .center-image }

Next, click on the “Calculate distance matrix” button, which will compare each frame of audio from one file to each frame in the other.

In the distance matrix on the right side, dark blue represents small distances (very similar frames of audio) and dark red represents large distances (very different frames of audio). In general, you’ll notice stripe or checkboard patterns, and the boundaries of these stripes correspond roughly to segment boundaries.

{: .center-image }

The initial alignment is the straight line across the two files. In general, this should be pretty close to the path we’ll eventually choose, because we want the sound files we’re morphing to be as similar as possible. In this case, I’ve produced the sounds with the same stress pattern and similar timings, though /s/ in general tends to be longer than /ʃ/.

The temporal anchors should be placed such that they line up to points in the sound file that have the same “meaning” in both files. So minimally there should be an anchor for the onset of the /ɑ/, the boundary between the /ɑ/ and the sibilant, the boundary between the sibilant and the /i/, and the offset of the /i/.

{: .center-image }

We also want to make sure that there are frequency anchors for the sibilant and the formant transitions, to make those more natural. Without frequency anchors, any differences will be mixed linearly in the frequency domain, similar to what you would get if you made the stimuli in Praat by extracting the /s/ and /ʃ/ and mixing them at different percentages. Instead we can shift the spectral peak around in STRAIGHT. The differences between mixing and shifting are not super pronounced when synthesizing sibilant continua, but make a huge difference in synthesizing vowel continua (shifting formants around).

{: .center-image }

Once you have an anchor in the middle of the sound to be morphed, we want to anchor the peaks in the frequency domain. To do so, hold alt and click on the point in the center of the sibilants (the cursor will turn to a pointing hand over the anchor). This will bring up a display of the spectra for the two wave files at that point in time.

{: .center-image }

In this case we’ve aligned the peaks of the sibilants, so that they’ll get shifted around when morphing, rather than averaged. See the interface controls page for more details on how to interact with frequency anchors.

Once you’re happy with the frequency anchors, press the “Update frequency anchors” button in the lower right of the window, and that will save what you have anchored and pass it back to the main anchoring dialog.

Once all the anchors are set up to your satisfaction, you can press the “Set up anchors” button to send the work back to the main morphing dialog. I also recommend saving the anchors via the “Save morphing substrate” button, like the asi-ashi_initialstress.mat file.

Setting up morphing rate ¶

Once anchoring is finished, or the morphing substrate mat file from a previous session is loaded, you’ll have to set up the two end points of the continuum you want to generate.

To do so, first press the “Initialize morphing rate” button and then the “Edit morphing rate” button, which will open the morphing rate dialog.

{: .center-image }

In the dialog window, there are a lot of options for what can be done. For the continuum we’re creating, we’re going to be ignoring most everything here, but see the morphing rate interface page for more details on what aspects of it do.

{: .center-image }

What we want to do is create a substrate that is wholly “A” (i.e. asi) and another substrate that is wholly “B” (i.e. ashi). To do so, click on one of the points in the “Spec” figure (the blue boxes and lines should turn green) and drag it all the way up to the “A” side.

{: .center-image }

Once that’s done, press the “Set up morphing rate” button on the right to send it back to the main dialog. Click the “Save morphing substrate” button to save the A subtrate as a mat file, like asi-ashi_initialstress_A.mat .

{: .center-image }

Once the A substrate is saved, we’ll make the B substrate by clicking the “Edit morphing rate” button, and then clicking and dragging a point in the “Spec” figure (along with all the rest of the points) all the way to the B side.

{: .center-image }

Once more, click the “Set up morphing rate” button to send it back to the main morphing window, and then click the “Save morphing substrate” button to save the B subtrate as a mat file, like asi-ashi_initialstress_B.mat .

Generating the continuum ¶

Once the end points of the continuum are all set up, we’re ready to generate the steps. From the main morphing menu, click the “Continuum” button at the bottom right of the menu.

{: .center-image }

This brings up a dialog to generate the continuum. You can edit where the continuum will be save here, but I find it easier just to use the defaults and move the continuum to a backed up location once you’re happy with the results. You might have to go back and edit the anchors a couple of times and STRAIGHT is not good about remembering paths you’ve entered before. You can also change the number of the steps that will be generated if you want more or less than the default 11. I do recommend changing the file name root to something more informative than “stimulus”, in this case I used “asi-ashi_initialstress”.

{: .center-image }

When you click the “Generate continuum” button, it will prompt you to find the A substrate mat file that we saved earlier, followed by the B substrate mat file. Once both have been selected, it will generate the continuum (and play them on top of each other). To actually get a sense of how well they turned out, I would recommend opening them in Praat and playing around with them there. The continuum steps that I’ve generated is below.

Step	File	Step	File	Step	File
1	asi-ashi_initialstress001.wav	2	asi-ashi_initialstress002.wav	3	asi-ashi_initialstress003.wav
4	asi-ashi_initialstress004.wav	5	asi-ashi_initialstress005.wav	6	asi-ashi_initialstress006.wav
7	asi-ashi_initialstress007.wav	8	asi-ashi_initialstress008.wav	9	asi-ashi_initialstress009.wav
10	asi-ashi_initialstress010.wav	11	asi-ashi_initialstress011.wav

I’ve written a batch script in Matlab for generating lots of continua given a folder with *A.mat and corresponding *B.mat substrates. If you’re generating a lot of continua, this might be faster and more flexible than using the dialog for each one. The script is available here

Manipulation ¶

The manipulation dialog is accessed through the TandemSTRAIGHTHandler program, rather than the morphing menu. Near the bottom after analyzing a sound file or loading a pre-analyzed sound file, you can click the “Open manipulation GUI” to manipulate a given sound file.

{: .center-image }

When you load into the dialog, it shows you the spectrogram along with figures for intensity, F0, size ratio, duration ratio, and AP ratio.

{: .center-image }

From here you can use the pen tool to draw contours and patterns for something like F0 or intensity.

{: .center-image }

You can also use a rectangle selector (selected via the radio buttons on the right side of the interface. If you want to make the first word longer by some amount, you can select all the points and then raise their duration ratio.

{: .center-image }

To generate the synthesized wave file, press the “Synthesize” button to hear what the current manipulation sounds like. When you’re happy with the synthesis, press the “Save results” button to save a .mat file and a wav file. The save box makes it seems like only a .mat file will be saved, but a wav file with the same name will be saved in the same place.

Examples ¶

This section contains a number of examples demonstrating the use of STRAIGHT to generate continua between end points, or resynthesize a single file.

Sibilant continua
- Similar to the walkthrough
- More endpoint examples
Stress shift continua
- /’ɑsi/ to /ɑ’si/
- /’ɑʃi/ to /ɑ’ʃi/
Stop voicing continua
- “task” /tæsk/ to “dask” /dæsk/
Vowel continua
- “bit” /bɪt/ to “bat” /bæt/
- Other bVt words to play around with
Intonation continua
- “My dog ate my homework.” (declaritive) to “My dog ate my homework?” (yes-no)
Pitch manipulation
- “My dog ate my homework?” to user-specified declaritive intonation

Synthesizing a sibilant continuum ¶

This kind of continuum is covered in depth in the morphing tutorial .

If you would like to synthesize other similar continua, I have the following wav/mat files available:

The _bad versions are cases where I produced the word with final stress, but with a fairly reduced /ɑ/. The morphing of those would be one instance where it would probably be better to rerecord rather than morph as is, but are provided as examples for teaching.

Synthesizing a stress shift continuum ¶

The continuum for stress shift can be generated using the same files as the sibilant continua, because each token was produced with stress either on the initial syllable or the final syllable.

In this example, we’ll be morphing asi_initialstress.wav to asi_finalstress.wav.

Below you’ll see the temporal anchors for the morphing. You’ll notice that there are relatively large deviations in opposite directions of a straight line across for where the stressed syllable is, due to duration being a large cue to stress in English.

{: .center-image }

There is also some spectral differences in the second syllable, due to the effect of stress. As such, we’d like to make sure that formants are properly aligned during the second syllable as below.

{: .center-image }

Below you’ll find the full morphed continuum from initial to final stress on “asi”.

Step	File	Step	File	Step	File
1	initial-final_asi001.wav	2	initial-final_asi002.wav	3	initial-final_asi003.wav
4	initial-final_asi004.wav	5	initial-final_asi005.wav	6	initial-final_asi006.wav
7	initial-final_asi007.wav	8	initial-final_asi008.wav	9	initial-final_asi009.wav
10	initial-final_asi010.wav	11	initial-final_asi011.wav

Synthesizing a voicing continuum ¶

This page focuses on making a continuum between “task” and “dask”. Below you’ll find the wav files used, as well as a prevoiced version of “dask”, for if you’d like to play around with it.

You’ll see below the temporal anchors for the morphing substrate. The main focus of the anchors is around when the burst and aspiration occur, so anchors have been placed at various points: before the burst, after the burst, when voicing begins, and when formant transitions are complete.

{: .center-image }

It could be argued that rather than placing a boundary where voicing begins, that boundary should be placed relative to the formant transitions. In “dask”, that aligns with voicing, but in “task” formant transitions begin during the aspiration. This would result in a continua centered around the source going from aperiodic to periodic, rather than the duration of the aperiodic aspiration. I’m going to leave this as a task to you if you’re interested in comparing the resulting continua. It should be less natural, because the amplitude of prevoicing would weaken as the VOT gets shorter, which doesn’t make sense from an articulatory standpoint. A potentially better way would be to create two continua next to one another, one from aspirated to unaspirated and then from unaspirated to prevoiced.

Below you’ll find the full morphed continuum from aspirated “task” to unaspirated “dask”.

Step	File	Step	File	Step	File
1	task-dask001.wav	2	task-dask002.wav	3	task-dask003.wav
4	task-dask004.wav	5	task-dask005.wav	6	task-dask006.wav
7	task-dask007.wav	8	task-dask008.wav	9	task-dask009.wav
10	task-dask010.wav	11	task-dask011.wav

One thing to note, is that due to the way STRAIGHT does its synthesis, quick transients like stop bursts can get weakened compared to their original form. If we look at the natural stop burst for “task”, we see that it is higher in amplitude than the following aspiration.

{: .center-image }

However, the same cannot be said for the synthesized version, where the burst amplitude has been reduced and is comparable to the aspiration amplitude.

{: .center-image }

In general, while STRAIGHT excels at modelling continous sources like voicing and turbulence, it’s not as good at modelling transient sources like bursts.

Synthesizing a vowel continuum ¶

This page will focus on making a continuum between “bit” and “bat”, primarily to illustrate the benefits of frequency anchoring. However, I have many /bVt/ words recorded that can be used as end points for many different continua around the English vowel space:

The temporal structure for creating a continuum follows that in the morphing tutorial .

{: .center-image }

You’ll notice that the region corresponding to the vowel in each sound files shows a fair bit of difference, indicative of their different formant structure. Rather than mixing the sound files directly, we’d like to shift the formants around, so we have to align the formant peaks during the vowel as below.

{: .center-image }

The continuum created for the “bit” and “bat” endpoints using morphing substrates of bit-bat_A.mat to bit-bat_B is below.

Step	File	Step	File	Step	File
1	bit-bat001.wav	2	bit-bat002.wav	3	bit-bat003.wav
4	bit-bat004.wav	5	bit-bat005.wav	6	bit-bat006.wav
7	bit-bat007.wav	8	bit-bat008.wav	9	bit-bat009.wav
10	bit-bat010.wav	11	bit-bat011.wav

Listen specifically to steps 6 and 7 to hear something approximately “bet”-like, as the continuum goes through the /ɛ/ formant region.

In contrast, a continuum created without frequency anchoring sounds using morphing substrates of bit-bat_nofreqeuencyalignment_A.mat to bit-bat_B is below.

Step	File	Step	File	Step	File
1	bad_bit-bat001.wav	2	bad_bit-bat002.wav	3	bad_bit-bat003.wav
4	bad_bit-bat004.wav	5	bad_bit-bat005.wav	6	bad_bit-bat006.wav
7	bad_bit-bat007.wav	8	bad_bit-bat008.wav	9	bad_bit-bat009.wav
10	bad_bit-bat010.wav	11	bad_bit-bat011.wav

The boundary between the two is much more abrupt and the synthesis quality is generally not as good.

Synthesizing an intonation continuum ¶

This example uses the following sound files:

homework_s.wav
- StrObjhomework_s.mat
- “My dog ate my homework.” (declarative intonation)
homework_q.wav
- StrObjhomework_q.mat
- “My dog ate my homework?” (yes-no question intonation)

The temporal structure for creating a continuum follows that in the morphing tutorial .

{: .center-image }

There’s a fair bit of spectral difference in the final word (“homework”) which is to be expected given that I tend to use creaky voice more at the end of the sentence, so it might be worthwhile re-recording this pair. However, the continuum generally sounds pretty good to me.

The continuum created for the “homework_s” and “homework_q” endpoints using morphing substrates of s-q_homework_A.mat to s-q_homework_B is below.

Step	File	Step	File	Step	File
1	s-q_homework001.wav	2	s-q_homework002.wav	3	s-q_homework003.wav
4	s-q_homework004.wav	5	s-q_homework005.wav	6	s-q_homework006.wav
7	s-q_homework007.wav	8	s-q_homework008.wav	9	s-q_homework009.wav
10	s-q_homework010.wav	11	s-q_homework011.wav

Pitch manipulation ¶

This example uses the following sound file:

homework_q.wav
- StrObjhomework_q.mat
- “My dog ate my homework?” (yes-no question intonation)
- Resynthesized version

The resynthesized version is the result of drawing the downward trending line from the manipulation page .

The original pitch curve is shown below.

{: .center-image }

By drawing a line going down over the course of the sentence, we can change the question intontation into a (slightly unnatural sounding) declarative intonation.

{: .center-image }

Interface controls ¶

This section contains information on the less obvious ways over interacting with the various dialogs that STRAIGHT uses.

F0 extraction interface controls ¶

{: .center-image }

Click on the time axis to zoom in
- Hold shift and click to zoom out
- Click and drag to move the visible part of the waveform
Click and drag a boundary to extend/reduce a voiced portion
- Moving a boundary over another voiced portion will overwrite it, which can help with deleting lots of small voiced portions
Click and drag a point to mark a frame as either unvoiced or voiced
- Drag to the center to mark as unvoiced
- Drag to top or bottom to mark as voiced

Anchoring interface controls ¶

{: .center-image }

Click on the time axis of either sound file to zoom in
- Hold shift and click on the time axis of either to zoom out
- When zoomed in, you can click and drag the spectrograms, waveforms or distance matrix to move the visible areas
Click on the white alignment line across the distance matrix to add a new anchor
- Mouse cursor will change to have a plus
Click and drag on an anchor to move it around (you can also click and drag the lines in the spectrogram view)
- Hold shift and click to remove an anchor
  - Mouse cursor will change to an eraser
- Hold alt and click to bring up the frequency anchor interface
  - Mouse cursor will change to a pointing hand

Frequency anchor interface controls ¶

{: .center-image }

Click on the frequency axis to zoom in
- Hold shift and click to zoom out
- Click and drag the white background to move the visible part of the spectrum
Click on either frequency spectrum to add an anchor
- Mouse cursor will change to have a plus
Click and drag on the dots to change what parts of the spectrum are aligned
- Hold shift and click to remove an anchor
  - Mouse cursor will change to an eraser
- Graphical elements can interfere with one another, so you may have to move something out of the way first

Troubleshooting STRAIGHT ¶

Recording quality ¶

The source end points should be recorded in as high a quality as possible with a high SNR. STRAIGHT scales the amplitudes, so if your stimuli were recorded with a lower gain, some noise might be introduced from the microphone, leading to artifacts in the eventual synthesis.

Additionally, the recording environment should not have noise in the background. STRAIGHT can remove some low frequency hums, but any noise in the frequency range of speech is going to be analyzed as speech and resynthesized.

Pronunciations ¶

The two end points that you’re generating a continuum for should be maximally similar, except in the one way you want the continuum along. So for a continuum from “bet” to “bat”, the /b/ and /t/ should be pronounced as identically as possible. In English, /b/ can be realized (albeit uncommonly) with prevoicing, and coda /t/ can be realized in a variety of different ways: released, unreleased, glottalized released, or glottal stop. If the two end points have different realizations, generating a continuum is much trickier (you can look at using bet_creaky.wav and bat.wav as the end points). In this case, I would recommend re-recording the end points to be more similar. Side-by-side productions can be used to elicit more similar pronunciations, as well as coaching the speaker about releasing all stops, for instance.

I would also recommend consistent modal voice quality, since pitch tracking is generally easier.

Random error in generating continua ¶

I’ve run into a random error that sometimes pops up on some versions of the Matlab code, where it will give a message about different dimensions of matrices. The fastest solution I’ve found is to redo the anchoring completely, and then run the continuum again and it should work. I think this is only relevant to the batch script that I’ve wrote here , I don’t think I’ve encountered the bug in the standalone versions.