Home › Forums › OpenEars › Hand modifying the .languagemodel and .dic files.

This topic has 2 voices, contains 1 reply, and was last updated by  Halle 301 days ago.

Viewing 2 posts - 1 through 2 (of 2 total)
Author Posts
Author Posts
July 22, 2011 at 3:21 pm #7346

rmangino

Hi,

I have a very long (~5,000) list of words that are not in the cmu word dictionary (these are all domain specific terms).

I used the lmtools to auto-generate .lm (languagemodel) and .dic files. Then I went in and hand tweaked the phones in the .dic file so that they represent the roper pronunciations.

Three questions:

1) Is it ok for me to hand edit the phones in the .dic file? If I change the phones in the .dic file do I also have to make some type of corresponding change in the .languagemodel file?

2) New terms will frequently need to be added to the vocabulary. Does this mean that I have to start all over (new list -> lmtools, fix up the phones for all words, etc.) every single time? It doesn’t seem to be possible to add new words to the .dic file without breaking the values in the .languagemodel file.

3) Is there a better way to do this?

Any pointers would be much appreciated.

Thank you
July 22, 2011 at 4:22 pm #7347

Halle

Hi,

You could probably make this part of a pretty easy workflow by exploiting the fact that LanguageModelGenerator can be run fast on the Simulator and uses a language model generation library (MITLM) that gives equivalently good results to the CMU tool. This would need a bit of hacking, but basically what you would do is create your own master dictionary (to substitute for cmu07a.dic) and remove the ranges that LanguageModelGenerator’s NSScanner uses to constrain searches, which is an optimization that is specific to cmu07a.dic. If you use the same formatting and rules for multiple pronunciations as cmu07a.dic it shouldn’t be too hard to get LanguageModelGenerator finding the appropriate entries in your master dictionary, and then creating new lms and dics is as easy as adding the word to the master dictionary and to your input array and building and running.

Then you just have to open up the simulator app install’s docs folder and get your documents out.

I will take this moment to subtly plug my blog post with my Applescript tool for automatically opening the install folder of the last app you ran in the Simulator here since I use it for tasks like this:

http://www.politepix.com/2011/05/13/open-the-simulator-sandbox-folder-of-the-app-you-just-built-and-ran/

To address your main question, the language model and the phonetic dictionary are only interdependent to the extent that any word which appears in the language model must appear in the phonetic dictionary with the identical spelling and in uppercase (or the identical spelling plus the (2), (3) etc that are appended for multiple pronunciations). There can be more words in the phonetic dictionary than are used by the language model, but not vice versa.

But it’s very important that the phoneme system is unchanged, because it is matched to the acoustic model. If you are creating the dictionary by hand, this might be an area in which bugs sneak in. The .dic file also must be in alphabetical order (which LanguageModelGenerator should do automatically if your master dictionary is in alphabetical order).
Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.