[Resolved] CMUSphinx 0.7

Dear Halle,

Thank you for your consideration. Actually, I need to use OpenEars with a larger vocabulary (almost 5k) and they have claimed that the new version dramatically works faster on large vocabularies.

I have tried WSJ 5K vocabulary using Sphinx Knowledge Base Tool as you have recommended.

However, the program immediately crashes after I speak something, receiving EXC_BAD_ACCESS signal at the IF statement within the following line of ps_start_utt function.

if (ps->search == NULL) {
E_ERROR(“No search module is selected, did you forget to “
“specify a language model or grammar?n”);
return -1;
}

Have you ever experimented with a larger vocabulary? or do you have any idea how can I resolve the problem.

Note: Sorry, if it is inappropriate not opening a separate thread for this question. I also have not tried yet your logging function, I can try to give more details by logging if you can not give an immediate answer.

Note 2: I do not think that I have opened MDEF file but I will try to re-install the project anyway. (Note 3: I have just tried after installing everything again but I am still getting the same error.)

Dear Halle,

I have created a language model with 5,000 words from WSJ dictionary, using CMU Language Tool (http://www.speech.cs.cmu.edu/tools/lmtool.html).

You may download the language model that I have created from this url: http://goo.gl/C6FKY

Actually, there was an already existing 5k language model at .DMP format within VocalKit, which I could not figure out how to use with OpenEars.

I am now trying with logging functionality as well to see if I can catch a proper explanation for both of us.

Thank you very much, you are amazingly helpful.

This seems to be problem but I do not know why the languagemodel is not found:

ERROR: “ngram_model_arpa.c”, line 465: File /var/mobile/Applications/FAF4CCE8-21FF-47EF-800F-8F65D7C467CC/OpenEarsSampleProject.app/7224.languagemodel not found
ERROR: “ngram_model_dmp.c”, line 105: Dump file /var/mobile/Applications/FAF4CCE8-21FF-47EF-800F-8F65D7C467CC/OpenEarsSampleProject.app/7224.languagemodel not found
ERROR: “ngram_search.c”, line 210: Failed to read language model file: /var/mobile/Applications/FAF4CCE8-21FF-47EF-800F-8F65D7C467CC/OpenEarsSampleProject.app/7224.languagemodel

I am just modifying your sample app for now. I have not tried VocalKit model because it was in .DMP format, as I said before. I have put .languagemodel and .dic file generated using CMU Language Tool to the Language Model dictionary under Classes. Then, I have modified kLanguageModelName and kDictionaryName for the new values.

Try the following files from the OpenEars + Pocketsphinx distribution that should be a good match for the default acoustic model:

OpenEars/CMULibraries/pocketsphinx-0.6.1/model/lm/en_US/hub4.5000.DMP
OpenEars/CMULibraries/pocketsphinx-0.6.1/model/lm/en_US/cmu07a.dic

You can always use a .DMP file in place of a .languagemodel file and for large grammars it’s obligatory.

BTW, trying to recognize a complete vocabulary with the iPhone is going to be a memory and CPU time hog, but you probably already know that.

FWIW, I’m also mainly interested in OpenEars for the JSGF support, I expect I’m not the only one – it’s not necessarily a very small number. In fact given the performance available on older iPhones at least, most would probably have to use grammars just because a more open model just means more options, and doesn’t seem feasible the performance is enough of a challenge with JSGFs. (Or am I misjudging and it would actually be faster the other way? Please let me know if not – but as far as restricting syntax of allowable commands, building a grammar works just fine for me.)

I’m interested enough in the supposedly faster 0.7 JSGF performance to have started looking over it and the difficulty of porting it myself for OpenEars (not looking easy, I agree), or trying VocalKit, or just using it directly – except I’m much better with ObjC interfaces than C ones. One thing that concerns me is that something I saw (I forget where) about pocketsphinx that suggested that the compilation of JSGF grammars might be much faster in 0.7 – which implies that could be the speedup, and that recognition might not be much faster.

Any thoughts on an ETA for 0.7 support, or tips on how I might go about patching it in myself if I get to that point? I’m not eager to do that, but I’m sure it’d be faster to try to upgrade OpenEars than to start from pocketsphinx and flite myself, and basically have to rewrite OpenEars.. Thanks in advance for any help you have time to give. I’m very thankful for OpenEars, and especially your excellent documentation – it got me up and running in a matter of hours.

A couple of your comments that I missed:

In fact given the performance available on older iPhones at least, most would probably have to use grammars just because a more open model just means more options, and doesn’t seem feasible the performance is enough of a challenge with JSGFs.

I haven’t tested JSGF versus ARPA for a similarly-sized grammar so I don’t know which is faster on any given iPhone model, but they are both equally open. For one you need to write correct markup and for the other you need to have a fast-yet-memory-frugal word probability calculator. But making either of those processes self-evidently simple, reliable and fast via an API designed to be used by developers with different experience levels is not overtly easy.

One thing that concerns me is that something I saw (I forget where) about pocketsphinx that suggested that the compilation of JSGF grammars might be much faster in 0.7 – which implies that could be the speedup, and that recognition might not be much faster

My reading of the release notes for Pocketsphinx .7 based on having spent a fair amount of time with the Pocketsphinx JSGF code was that recognition for large JSGF grammars that already exist would be faster.

Edgecase, it’s coming. Whether I fork Pocketsphinx 0.6.1 so Halle doesn’t have to try upgrading to 0.7, or whether we can coax him up to 0.7, it’s coming.

Halle, I have tested FSG vs. LM for similar sized applications. FSG, when done right, is much, much better. And FSG grammar runs when the lmtool LMs, with too much emphasis on 1 and 2 grams, fall down flat. I’m not sure if the ngram support for LMs could be improved within the Sphinx framework.

But the Pocketsphinx wasn’t behaving the way I expected. There is no way a low perplexity FSG, run by a user staying within the FSG, should ever be beat by an LM built for a simple command and control grammar. But the FSG was running slower and less accurately than the LM. I got the same sort of weirdness that people were talking about in this thread…

http://www.politepix.com/forums/topic/how-to-use-jsgf/

So, I set out to find out why.

There’s a couple of interesting things I’ve learned along the way. The most important is that Pocketsphinx JSFG support in version 0.6 and 0.7 is broken. It’s apparently been broken a lot longer than three years (0.6).

The broken part is the JSGF to FSG engine that is used both inside pocketsphinx_continuous (the core of OpenEars) and in the standalone Sphinx_JSGF2FSG tool in the Pocketsphinx distribution. Something important is literally missing, so it never could really have worked right. I’ll have a fix soon, I think, but in the meantime, I’ve made a minor change to OpenEars so I can use FSG grammars built on a tool other than CMU’s stuff.

The results are like night and day. The FSG version returns quickly and much more accurately than the LM, now. So, I conclude that any reports of LMs outperforming FSGs in small, containable applications are purely a side-effect of the broken JSGF support.

I’m currently working on an extremely hostile environment iOS recognition application. The task is well suited for good old FSG grammars, it breaks down into habitable grammars and we have a cooperative user pool.

The other thing I learned is that enabling FSG model switching isn’t anywhere near as complicated it sounds. ;)

• This reply was modified 270 days ago by  Joseph S. Wisniewski.

Also I’d be very interested to hear how you managed to achieve FSG switching, that would allow me to reduce the size of my grammar by quite a bit by splitting the user interaction into modes and switching instead of collating and ignoring what isn’t applicable.

I manually update to .7 and I did not find obvious improvement on accuracy. So I rolled back to 0.61.

I agreed with Halle about upgrade policy.

Hi,

OK, I am taking moving to .7 under advisement for a future release if I get enough testing feedback from this post (if I don’t get feedback, or if the feedback is that it doesn’t affect anything, I’m going to assume that it doesn’t make sense to add it). Here are instructions you can follow to use .7 right now.

I am also repeating my offer of letting you test out an early iteration of JSGF-switching code. It’s attached after the .7 install instructions.

These steps are for configuring a brand-new install. You will follow the regular Getting Started install instructions exactly, but you will download pocketsphinx-0.7.tar.gz and sphinxbase-0.7.tar.gz instead of pocketsphinx-0.6.1.tar.gz and sphinxbase-0.6.1.tar.gz, and before you perform any of the installation steps you will make the following changes to the files in the OpenEars.zip download distribution.

1. Open up InstallOpenEars.pl in a text editor and replace every occurence of the text string

-0.6.1

with

-0.7

2. Still in InstallOpenEars.pl, find this code block and remove it entirely:

my $ms_gaudenfilename = 'pocketsphinx-0.6.1/src/libpocketsphinx/ms_gauden.c';
my $ms_gaudenfind1 = 'for \(i = 0\; i \< n_feat\; i\+\+\)';
my $ms_gaudenreplace1 = 'for (i = 0; i < n_feat; i++) {';
my $ms_gaudenfind2 = 'printf\(\" \%dx\%d\", n_density, veclen\[i\]\)\;';
my $ms_gaudenreplace2 = <<"GAUDENTOKEN";
// printf(" %dx%d", n_density, veclen[i]);
}
GAUDENTOKEN
;

local @ARGV = ($ms_gaudenfilename);
local $^I = '.bak';
while( <> ){
	s/$ms_gaudenfind1/$ms_gaudenreplace1/ig;
	s/$ms_gaudenfind2/$ms_gaudenreplace2/ig;
	print;
};

3. Right-click on [OPENEARS]/OpenEarsLibrary/OpenEarsLibrary.xcodeproj and select Show Package Contents. In a text editor, open up the file project.pbxproj within and make the following changes:

Change the two lines which read:

"../CMULibraries/sphinxbase-0.6.1/include",

to instead read:

"../CMULibraries/sphinxbase-0.7/include/**",

and then replace every occurence of the string

-0.6.1

with the string

-0.7

4. Right-click on [OPENEARS]/OpenEarsSampleProject/OpenEarsSampleProject.xcodeproj and select Show Package Contents. In a text editor, open up the file project.pbxproj within and make the following changes:

Change the two lines which read:

"../CMULibraries/sphinxbase-0.6.1/include",

to instead read:

"../CMULibraries/sphinxbase-0.7/include/**",

and then replace every occurence of the string

-0.6.1

with the string

-0.7

This will let you continue with the normal installation steps but end up with a .7 install.

To test out the JSGF-switching code, make the following changes to ContinuousModel.mm:

Replace the entire contents of the method

- (void) changeLanguageModelForDecoder:(ps_decoder_t *)pocketsphinxDecoder languageModelIsJSGF:(BOOL)languageModelIsJSGF

with the following code:

- (void) changeLanguageModelForDecoder:(ps_decoder_t *)pocketsphinxDecoder languageModelIsJSGF:(BOOL)languageModelIsJSGF {

    if(languageModelIsJSGF == TRUE) {

		NSArray *dictionaryArray = [[NSString stringWithContentsOfFile:self.dictionaryFileToChangeTo encoding:NSUTF8StringEncoding error:nil] componentsSeparatedByCharactersInSet:[NSCharacterSet newlineCharacterSet]];

		int updateValue = 0;
		int count = 0;
		int add_word_result = 0;

		for(NSString *string in dictionaryArray) {

			NSArray *lineArray = [string componentsSeparatedByCharactersInSet:[NSCharacterSet whitespaceCharacterSet]];

			NSMutableString *mutablePhonesString = [[NSMutableString alloc] init];
			int i;
			for ( i = 0; i < [lineArray count]; i++ ) {
				if(i > 0) [mutablePhonesString appendString:[NSString stringWithFormat:@"%@ ",[lineArray objectAtIndex:i]]];
			}

			NSRange deletionRange = {[mutablePhonesString length]-1,1};
			[mutablePhonesString deleteCharactersInRange:deletionRange];

			if(count < [dictionaryArray count]) {
				updateValue = 0;
			} else {
				updateValue = 1;
			}

			add_word_result = ps_add_word(pocketsphinxDecoder,(char *)[[lineArray objectAtIndex:0] UTF8String], (char *)[[lineArray objectAtIndex:1] UTF8String],updateValue);
			[mutablePhonesString release];

			if(add_word_result > -1) {
				OpenEarsLog(@"%@ was added to dictionary",[lineArray objectAtIndex:0]);
			} else {
				OpenEarsLog(@"%@ was not added to dictionary, perhaps because it is already in the dictionary",[lineArray objectAtIndex:0]);
			}

			count++;
		}

		OpenEarsLog(@"A request has been made to change a JSGF grammar on the fly.");
		fsg_set_t *fsgs = ps_get_fsgset(pocketsphinxDecoder);

		jsgf_t *jsgf;
		fsg_model_t *fsg;
        jsgf_rule_t *rule;
		char const *path = (char *)[self.languageModelFileToChangeTo UTF8String];

        if ((jsgf = jsgf_parse_file(path, NULL)) == NULL) {
			OpenEarsLog(@"Error: no JSGF file at path.");
		}
        rule = NULL;

		jsgf_rule_iter_t *itor;

		for (itor = jsgf_rule_iter(jsgf); itor;
			 itor = jsgf_rule_iter_next(itor)) {
			rule = jsgf_rule_iter_rule(itor);
			if (jsgf_rule_public(rule))
				break;

            if (rule == NULL) {
                OpenEarsLog(@"Error: No public rules found in %s", path);
            }
        }

		fsg = jsgf_build_fsg(jsgf, rule, pocketsphinxDecoder->lmath, fsgs->lw);
        if (fsg_set_add(fsgs, fsg_model_name(fsg), fsg) != fsg) {
			OpenEarsLog(@"Error: could not add finite state grammar to set.");
        } else {

		}
        if (fsg_set_select(fsgs, fsg_model_name(fsg)) == NULL) {
			OpenEarsLog(@"Error: could not select new grammar.");
		}

		ps_update_fsgset(pocketsphinxDecoder);

	} else {

		OpenEarsLog(@"A request has been made to change an ARPA grammar on the fly. The language model to change to is %@", self.languageModelFileToChangeTo);
		NSNumber *languageModelID = [NSNumber numberWithInt:999];
		NSFileManager *fileManager = [[NSFileManager alloc] init];
		NSError *error = nil;
		NSDictionary *fileAttributes = [fileManager attributesOfItemAtPath:self.languageModelFileToChangeTo error:&error];
		if(error) {
			OpenEarsLog(@"Error: couldn't get attributes of language model file.");
		} else {
			OpenEarsLog(@"In this session, the requested language model will be known to Pocketsphinx as id %@.",[fileAttributes valueForKey:NSFileSystemFileNumber]);
			languageModelID = [fileAttributes valueForKey:NSFileSystemFileNumber];
		}

		[fileManager release];

		ngram_model_t *baseLanguageModel, *newLanguageModelToAdd;

		newLanguageModelToAdd = ngram_model_read(pocketsphinxDecoder->config, (char *)[self.languageModelFileToChangeTo UTF8String], NGRAM_AUTO, pocketsphinxDecoder->lmath);

		baseLanguageModel = ps_get_lmset(pocketsphinxDecoder);

		OpenEarsLog(@"languageModelID is %s",(char *)[[languageModelID stringValue] UTF8String]);
		ngram_model_set_add(baseLanguageModel, newLanguageModelToAdd, (char *)[[languageModelID stringValue] UTF8String], 1.0, TRUE);
		ngram_model_set_select(baseLanguageModel, (char *)[[languageModelID stringValue] UTF8String]);

		ps_update_lmset(pocketsphinxDecoder, baseLanguageModel);

		int loadingDictionaryResult = ps_load_dict(pocketsphinxDecoder, (char *)[self.dictionaryFileToChangeTo UTF8String],NULL, NULL);

		if(loadingDictionaryResult > -1) {
			OpenEarsLog(@"Success loading the dictionary file %@.",self.dictionaryFileToChangeTo);
		} else {
			OpenEarsLog(@"Error: could not load the specified dictionary file.");
		}

		NSDictionary *userInfoDictionary = [NSDictionary dictionaryWithObjects:[NSArray arrayWithObjects:@"PocketsphinxDidChangeLanguageModel",self.languageModelFileToChangeTo, self.dictionaryFileToChangeTo,nil] forKeys:[NSArray arrayWithObjects:@"OpenEarsNotificationType",@"LanguageModelFilePath",@"DictionaryFilePath",nil]];
		NSNotification *notification = [NSNotification notificationWithName:@"OpenEarsNotification" object:nil userInfo:userInfoDictionary];
		[[NSNotificationCenter defaultCenter] performSelectorOnMainThread:@selector(postNotification:) withObject:notification waitUntilDone:NO];

		self.languageModelFileToChangeTo = nil;
		self.thereIsALanguageModelChangeRequest = FALSE;

		OpenEarsLog(@"Changed language model. Project has these words in its dictionary:\n%@", [self compileKnownWordsFromFileAtPath:self.dictionaryFileToChangeTo]);
	}
}

And replace the following lines:

			if(languageModelIsJSGF == FALSE) {
				[self changeLanguageModelForDecoder:pocketSphinxDecoder languageModelIsJSGF:languageModelIsJSGF];
			} else {
				OpenEarsLog(@"Sorry, switching JSGF grammars on the fly isn't supported in this version of OpenEars. You can dynamically generate and switch ARPA grammars.");
			}
	

So that they are as follows:

			//if(languageModelIsJSGF == FALSE) {
            [self changeLanguageModelForDecoder:pocketSphinxDecoder languageModelIsJSGF:languageModelIsJSGF];
			//} else {
			//	OpenEarsLog(@"Sorry, switching JSGF grammars on the fly isn't supported in this version of OpenEars. You can dynamically generate and switch ARPA grammars.");
			//}

I’m still working on my fix for the Sphinx JSGF bug. Because Sphinx generates such bad FSGs from JSGFs (either using the command-line tool on the JSGF then loading the FSG in Sphinx like I do, or telling Sphinx to load a JSGF, like Halle does in OpenEars) the advantages of 0.7 are largely masked.