Tutorials for the OpenEars Platform

To get your own customized tutorial showing you how to use any feature of OpenEars or its plugins in your app, just flip the switches for the features you want. Anywhere that you see the icon you can click it to have the code which follows it copied to your clipboard so you can paste it into your app. Remember, never test recognition on the Simulator since it doesn't use the real OpenEars audio driver.

Show me a 5-minute tutorial for an OpenEars-enabled app with the following features:

Preparing to use OpenEars

1

• Create your own app. Download the OpenEars distribution and unpack it.
• Inside your downloaded OpenEars distribution there is a folder called "Framework". Drag that folder into your app project in Xcode. Make absolutely sure that in the add dialog "Create groups for any added folders" is selected and NOT "Create folder references for any added folders" because the wrong setting here will prevent your app from working. If you are using Xcode 5 with a build number of 5A1413 or later, it has a bug which results in frameworks linked by reference being changed to link at incorrect URL paths, so it is necessary for you to also check the box that says "Copy items into destination group's folder (if needed)", or you may receive errors that header files can't be found. If you receive this kind of error with Xcode 5, it means that your Framework Search Path for the added frameworks has been changed to an invalid URL, so you may have to open that Build Setting for your app target and change it back to a correct path.
• Optional: to save space in your app binary, go to the build settings for your app target and search for the setting "Deployment Postprocessing" and set it to "Yes".
• Add the iOS frameworks AudioToolbox and AVFoundation to your app.

Preparing to use Plugins

1

First download demo versions of your plugin or plugins:
Then open up the Build Settings tab of the target app project and find the entry "Other Linker Flags" and add the linker flag "-ObjC":
And then drag your downloaded demo framework into your app project.

Using LanguageModelGenerator

1

Add the following to your implementation (the .m file): Under the @implementation keyword at the top:
#import <OpenEars/LanguageModelGenerator.h>
Wherever you need to instantiate the language model generator, do it as follows:
LanguageModelGenerator *lmGenerator = [[LanguageModelGenerator alloc] init];

2

In offline speech recognition, you define the vocabulary that you want your app to be able to recognize. A good vocabulary size for an offline speech recognition app on the iPhone, iPod or iPad is between 3 and 300 words.

In the method where you want to create your language model (for instance your viewDidLoad method), add the following method call (replacing the placeholders like "WORD" and "A PHRASE" with actual words and phrases you want to be able to recognize):

NSArray *words = [NSArray arrayWithObjects:@"WORD", @"STATEMENT", @"OTHER WORD", @"A PHRASE", nil];
NSString *name = @"NameIWantForMyLanguageModelFiles";
NSError *err = [lmGenerator generateLanguageModelFromArray:words withFilesNamed:name forAcousticModelAtPath:[AcousticModel pathToModel:@"AcousticModelEnglish"]]; // Change "AcousticModelEnglish" to "AcousticModelSpanish" to create a Spanish language model instead of an English one.


NSDictionary *languageGeneratorResults = nil;

NSString *lmPath = nil;
NSString *dicPath = nil;
	
if([err code] == noErr) {
	
	languageGeneratorResults = [err userInfo];
		
	lmPath = [languageGeneratorResults objectForKey:@"LMPath"];
	dicPath = [languageGeneratorResults objectForKey:@"DictionaryPath"];
		
} else {
	NSLog(@"Error: %@",[err localizedDescription]);
}
If you are using the default English-language or Spanish-language model generation, it is a requirement to enter your words and phrases in all capital letters, since the model is generated against a dictionary in which the entries are capitalized (meaning that if the words in the array aren't capitalized, they will not match the dictionary and you will not have the widest variety of pronunciations understood for the word you are using). If you need to create a fixed language model ahead of time instead of creating it dynamically in your app, just use this method (or generateLanguageModelFromTextFile:withFilesNamed:) to submit your full language model using the Simulator and then use the Simulator documents folder script to get the language model and dictionary file out of the documents folder and add it to your app bundle, referencing it from there.

Using PocketsphinxController

1

To use PocketsphinxController, you need a language model and a phonetic dictionary for it. These files define which words PocketsphinxController is capable of recognizing. They are created above by using LanguageModelGenerator. You also need an acoustic model. OpenEars ships with an English and a Spanish acoustic model.

2

Add the following lines to your header (the .h file). Under the imports at the very top:
#import <OpenEars/PocketsphinxController.h>
#import <OpenEars/AcousticModel.h>
In the middle part where instance variables go:
PocketsphinxController *pocketsphinxController;
In the bottom part where class properties go:
@property (strong, nonatomic) PocketsphinxController *pocketsphinxController;

3

Add the following to your implementation (the .m file): Under the @implementation keyword at the top:
@synthesize pocketsphinxController;
Among the other methods of the class, add this lazy accessor method for confident memory management of the object:
- (PocketsphinxController *)pocketsphinxController {
	if (pocketsphinxController == nil) {
		pocketsphinxController = [[PocketsphinxController alloc] init];
	}
	return pocketsphinxController;
}

4

In the method where you want to recognize speech (to test this out, add it to your viewDidLoad method), add the following method call:
[self.pocketsphinxController startListeningWithLanguageModelAtPath:lmPath dictionaryAtPath:dicPath acousticModelAtPath:[AcousticModel pathToModel:@"AcousticModelEnglish"] languageModelIsJSGF:NO]; // Change "AcousticModelEnglish" to "AcousticModelSpanish" to perform Spanish recognition instead of English.

Using FliteController

1

To use FliteController, you need to have at least one Flite voice added to your project. When you added the "framework" folder of OpenEars to your app, you already imported a voice called Slt, so these instructions will use the Slt voice. You can get eight more free voices in OpenEarsExtras, available at https://bitbucket.org/Politepix/openearsextras

2

Add the following lines to your header (the .h file). Under the imports at the very top:
#import <Slt/Slt.h>
#import <OpenEars/FliteController.h>
In the middle part where instance variables go:
FliteController *fliteController;
Slt *slt;
In the bottom part where class properties go:
@property (strong, nonatomic) FliteController *fliteController;
@property (strong, nonatomic) Slt *slt;

3

Add the following to your implementation (the .m file): Under the @implementation keyword at the top:
@synthesize fliteController;
@synthesize slt;
Among the other methods of the class, add these lazy accessor methods for confident memory management of the object:
- (FliteController *)fliteController {
	if (fliteController == nil) {
		fliteController = [[FliteController alloc] init];
	}
	return fliteController;
}

- (Slt *)slt {
	if (slt == nil) {
		slt = [[Slt alloc] init];
	}
	return slt;
}

4

In the method where you want to call speech (to test this out, add it to your viewDidLoad method), add the following method call:
[self.fliteController say:@"A short statement" withVoice:self.slt];

Using OpenEarsEventsObserver

1

Add the following lines to your header (the .h file). Under the imports at the very top:
#import <OpenEars/OpenEarsEventsObserver.h>
at the @interface declaration, add the OpenEarsEventsObserverDelegate inheritance. An example of this for a view controller called ViewController would look like this:
@interface ViewController : UIViewController <OpenEarsEventsObserverDelegate> {
In the middle part where instance variables go:
OpenEarsEventsObserver *openEarsEventsObserver;
In the bottom part where class properties go:
@property (strong, nonatomic) OpenEarsEventsObserver *openEarsEventsObserver;

2

Add the following to your implementation (the .m file): Under the @implementation keyword at the top:
@synthesize openEarsEventsObserver;
Among the other methods of the class, add this lazy accessor method for confident memory management of the object:
- (OpenEarsEventsObserver *)openEarsEventsObserver {
	if (openEarsEventsObserver == nil) {
		openEarsEventsObserver = [[OpenEarsEventsObserver alloc] init];
	}
	return openEarsEventsObserver;
}
and then right before you start your first OpenEars functionality (for instance, right before your first self.fliteController say:withVoice: message or right before your first self.pocketsphinxController startListeningWithLanguageModelAtPath:dictionaryAtPath:languageModelIsJSGF: message) send this message:
[self.openEarsEventsObserver setDelegate:self];

3

Add these delegate methods of OpenEarsEventsObserver to your class:
- (void) pocketsphinxDidReceiveHypothesis:(NSString *)hypothesis recognitionScore:(NSString *)recognitionScore utteranceID:(NSString *)utteranceID {
	NSLog(@"The received hypothesis is %@ with a score of %@ and an ID of %@", hypothesis, recognitionScore, utteranceID);
}

- (void) pocketsphinxDidStartCalibration {
	NSLog(@"Pocketsphinx calibration has started.");
}

- (void) pocketsphinxDidCompleteCalibration {
	NSLog(@"Pocketsphinx calibration is complete.");
}

- (void) pocketsphinxDidStartListening {
	NSLog(@"Pocketsphinx is now listening.");
}

- (void) pocketsphinxDidDetectSpeech {
	NSLog(@"Pocketsphinx has detected speech.");
}

- (void) pocketsphinxDidDetectFinishedSpeech {
	NSLog(@"Pocketsphinx has detected a period of silence, concluding an utterance.");
}

- (void) pocketsphinxDidStopListening {
	NSLog(@"Pocketsphinx has stopped listening.");
}

- (void) pocketsphinxDidSuspendRecognition {
	NSLog(@"Pocketsphinx has suspended recognition.");
}

- (void) pocketsphinxDidResumeRecognition {
	NSLog(@"Pocketsphinx has resumed recognition."); 
}

- (void) pocketsphinxDidChangeLanguageModelToFile:(NSString *)newLanguageModelPathAsString andDictionary:(NSString *)newDictionaryPathAsString {
	NSLog(@"Pocketsphinx is now using the following language model: \n%@ and the following dictionary: %@",newLanguageModelPathAsString,newDictionaryPathAsString);
}

- (void) pocketSphinxContinuousSetupDidFail { // This can let you know that something went wrong with the recognition loop startup. Turn on OPENEARSLOGGING to learn why.
	NSLog(@"Setting up the continuous recognition loop has failed for some reason, please turn on OpenEarsLogging to learn more.");
}
- (void) testRecognitionCompleted {
	NSLog(@"A test file that was submitted for recognition is now complete.");
}

Using LanguageModelGenerator+RuleORama

1

First, find the line
#import <OpenEars/LanguageModelGenerator.h>
in your app and add the following line right underneath it:
#import <RuleORamaDemo/LanguageModelGenerator+RuleORama.h>
Next, change this line where you create a language model:
NSError *err = [lmGenerator generateLanguageModelFromArray:words withFilesNamed:name forAcousticModelAtPath:[AcousticModel pathToModel:@"AcousticModelEnglish"]];
to use this grammar and method instead:

NSDictionary *grammar = @{
     ThisWillBeSaidOnce : @[
         @{ OneOfTheseCanBeSaidOnce : @[@"HELLO COMPUTER", @"GREETINGS ROBOT"]},
         @{ OneOfTheseWillBeSaidOnce : @[@"DO THE FOLLOWING", @"INSTRUCTION"]},
         @{ OneOfTheseWillBeSaidOnce : @[@"GO", @"MOVE"]},
         @{ThisWillBeSaidOnce : @[
             @{ OneOfTheseWillBeSaidOnce : @[@"10", @"20",@"30"]}, 
             @{ OneOfTheseWillBeSaidOnce : @[@"LEFT", @"RIGHT", @"FORWARD"]}
         ]},
         @{ ThisCanBeSaidOnce : @[@"THANK YOU"]}
     ]
 };
 
    NSError *err = [lmGenerator generateFastGrammarFromDictionary:grammar withFilesNamed:name forAcousticModelAtPath:[AcousticModel pathToModel:@"AcousticModelEnglish"]];


This will allow you to recognize statements in accordance with this grammar, such as: HELLO COMPUTER DO THE FOLLOWING MOVE 10 LEFT THANK YOU or GREETINGS ROBOT INSTRUCTION MOVE 20 RIGHT but it will not recognize individual words or words in orders outside of the grammar. Please note that unlike the JSGF output type in stock OpenEars, RuleORama doesn't support the rule types with optional repetitions. Rules defined with repetitions will be compressed into a rule with a single repetition.

Using LanguageModelGenerator+Rejecto

1

First, find the line
#import <OpenEars/LanguageModelGenerator.h>
in your app and add the following line right underneath it:
#import <RejectoDemo/LanguageModelGenerator+Rejecto.h>
Next, change this line where you create a language model:
NSError *err = [lmGenerator generateLanguageModelFromArray:words withFilesNamed:name forAcousticModelAtPath:[AcousticModel pathToModel:@"AcousticModelEnglish"]];
to use this method instead:

NSError *err = [lmGenerator generateRejectingLanguageModelFromArray:words
 withFilesNamed:name 
 withOptionalExclusions:nil
 usingVowelsOnly:FALSE 
 withWeight:nil 
 forAcousticModelAtPath:[AcousticModel pathToModel:@"AcousticModelEnglish"]]; // Change "AcousticModelEnglish" to "AcousticModelSpanish" to create a Spanish Rejecto model.

You will use the same array for languageModelArray (again, the words and phrases in your array must be in all-capital letters) and the same files name for fileName as you did with the old generateLanguageModelFromArray method, and to get started you can use the value "nil" for optionalExclusions, vowelsOnly, and weight, since they are there to help you refine your results and might not be needed. You can learn more about fine-tuning your results with those optional parameters in the Rejecto documentation.

Using PocketsphinxController+RapidEars

1

Like PocketsphinxController which it extends, we need a language model created with LanguageModelGenerator before using PocketsphinxController+RapidEars. We have already completed that step above.

2

Add the following to your implementation (the .m file): Under the @implementation keyword at the top, after the line #import <OpenEars/PocketsphinxController.h>:
#import <RapidEarsDemo/PocketsphinxController+RapidEars.h>
Next, comment out all calls in your app to the method
startListeningWithLanguageModelAtPath:dictionaryAtPath:languageModelIsJSGF:
and in the same part of your app where you were formerly using this method, place the following:
[self.pocketsphinxController setRapidEarsToVerbose:FALSE]; // This defaults to FALSE but will give a lot of debug readout if set TRUE
[self.pocketsphinxController setRapidEarsAccuracy:10]; // This defaults to 20, maximum accuracy, but can be set as low as 1 to save CPU
[self.pocketsphinxController setFinalizeHypothesis:TRUE]; // This defaults to TRUE and will return a final hypothesis, but can be turned off to save a little CPU and will then return no final hypothesis; only partial "live" hypotheses.
[self.pocketsphinxController setFasterPartials:TRUE]; // This will give faster rapid recognition with less accuracy. This is what you want in most cases since more accuracy for partial hypotheses will have a delay.
[self.pocketsphinxController setFasterFinals:FALSE]; // This will give an accurate final recognition. You can have earlier final recognitions with less accuracy as well by setting this to TRUE.
[self.pocketsphinxController startRealtimeListeningWithLanguageModelAtPath:lmPath dictionaryAtPath:dicPath acousticModelAtPath:[AcousticModel pathToModel:@"AcousticModelEnglish"]]; // Starts the rapid recognition loop. Change "AcousticModelEnglish" to "AcousticModelSpanish" in order to perform Spanish language recognition.

If you find that sometimes you are getting live recognition and other times not, make sure that you have definitely replaced all instances of startListeningWithLanguageModelAtPath: with startRealtimeListeningWithLanguageModelAtPath:.

Using OpenEarsEventsObserver+RapidEars

1

At the top of your implementation after the line
#import <OpenEars/OpenEarsEventsObserver.h>
Add the line
#import <RapidEarsDemo/OpenEarsEventsObserver+RapidEars.h>
And after this OpenEarsEventsObserver delegate method you added when setting up your OpenEars app:
- (void) pocketSphinxContinuousSetupDidFail { 

}
Just add the following extended delegate methods:
- (void) rapidEarsDidReceiveLiveSpeechHypothesis:(NSString *)hypothesis recognitionScore:(NSString *)recognitionScore {
    NSLog(@"rapidEarsDidReceiveLiveSpeechHypothesis: %@",hypothesis);
}

- (void) rapidEarsDidReceiveFinishedSpeechHypothesis:(NSString *)hypothesis recognitionScore:(NSString *)recognitionScore {
    NSLog(@"rapidEarsDidReceiveFinishedSpeechHypothesis: %@",hypothesis);
}

- (void) rapidEarsDidDetectBeginningOfSpeech {
    NSLog(@"rapidEarsDidDetectBeginningOfSpeech"); 
}

- (void) rapidEarsDidDetectEndOfSpeech {
    NSLog(@"rapidEarsDidDetectEndOfSpeech");
}

Using FliteController+NeatSpeech

1

FliteController+NeatSpeech preconditions

In order to use NeatSpeech, as well as importing the framework into your OpenEars-enabled project, it is also necessary to import the voices and voice data files by dragging the "Voice" folder in the disk image into your app project (once your app is working you can read more here about how to remove the elements you don't need in order to have a small app binary size). Very important: when you drag in the voices and framework folders, make sure that in Xcode's "Add" dialog, "Create groups for any added folders" is selected. Make sure that "Create folder references for any added folders" is not selected or your app will not work. For the last step, change the name of the implementation source file in which you are going to call NeatSpeech methods from .m to .mm (for instance, if the implementation is named ViewController.m, change its name to ViewController.mm and verify in the Finder that the name of the file has changed) and then make sure that in your target Build Settings, under the section "C++ Standard Library", the setting "libstdc++ (Gnu C++ standard library)" is selected. If you receive errors like "Undefined symbols for architecture i386: std::basic_ios >::widen(char) const", that means that this step needs special attention.

2

FliteController+NeatSpeech implementation

FliteController+Neatspeech just replaces FliteController's voice type with the advanced NeatSpeech voice type and it replaces FliteController's say:withVoice: method with NeatSpeech's sayWithNeatSpeech:withVoice: method. Before adding NeatSpeech, make sure that you either change the name of the View Controller that is hosting NeatSpeech from .m to .mm (so if it is called ViewController.m you should change it to ViewController.mm) or as an alternative to that you can add the linker flag -lstdc++ to "Other Linker Flags" under "Build Settings". Then, in order to use NeatSpeech in a project that has FliteController enabled, you will only have to do the following: In your header replace this:
#import <Slt/Slt.h>
#import <OpenEars/FliteController.h>
with this:
#import <Emma/Emma.h>
#import <OpenEars/FliteController.h>
#import <NeatSpeechDemo/FliteController+NeatSpeech.h>
and replace this:
Slt *slt;
with this:
Emma *emma;
and replace this:
@property (strong, nonatomic) Slt *slt;
with this:
@property (strong, nonatomic) Emma *emma;
in your implementation, replace this:
@synthesize slt;
with this:
@synthesize emma;
and replace this:
- (Slt *)slt {
	if (slt == nil) {
		slt = [[Slt alloc] init];
	}
	return slt;
}
with this:
- (Emma *)emma {
	if (emma == nil) {
		emma = [[Emma alloc]initWithPitch:0.0 speed:0.0 transform:0.0];
	}
	return emma;
}
and replace this:
[self.fliteController say:@"A short statement" withVoice:self.slt];
with this:
[self.fliteController sayWithNeatSpeech:@"Alice was getting very tired of sitting beside her sister on the bank, and having nothing to do: once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, and what is the use of a book, thought Alice, without pictures or conversations?" withVoice:self.emma];
And replace any other calls to say:withVoice with sayWithNeatSpeech:withVoice: Once this is definitely working you can remove the Slt or other Flite voice frameworks from your app to reduce app size. You can replace references to the Emma framework and object with any of the other voices to try them out. The available voice frameworks you'll find in the Voices folder in the distribution are as follows: Emma (US English, female) EmmaAlternate (US English, female) William (US English, male) WilliamAlternate (US English, male) Beatrice (UK English, female) Elliott (UK English, make) Daniel (Castilian Spanish, male) Martina (Castilian Spanish, female) Mateo (Latin American Spanish, male) Valeria (Latin American Spanish, female) You can also change the speaking speed, pitch of the voice, and inflection of each voice using the voice's initializer arguments speed, pitch and transform respectively. As an example, to initialize the Emma voice with a higher pitch you could use the following initialization: Emma *emma = [[Emma alloc]initWithPitch:0.2 speed:0.0 transform:0.0]; Once you know how your project is to be configured you can remove the unused voices following these instructions in order to make your app binary size as small as possible. You can pass the sayWithNeatSpeech:withVoice: method as much data as you want at a time. It will process the speech in phases in the background and return it for playback once it is ready. This means that you should rarely experience long pauses while waiting for synthesis, even for very long paragraphs. Very long statements need to include pause indicators such as periods, exclamation points, question marks, commas, colons, semicolons, etc. To interrupt ongoing speech while it is in progress, send the message [self.fliteController stopSpeaking];. This will not interrupt speech instantaneously but halt it at the next available opportunity.

Using OpenEarsEventsObserver+SaveThatWave

1

At the top of your implementation after the line
#import <OpenEars/OpenEarsEventsObserver.h>
Add the line
#import <SaveThatWaveDemo/OpenEarsEventsObserver+SaveThatWave.h>
And after this OpenEarsEventsObserver delegate method you added when setting up your OpenEars app:
- (void) pocketSphinxContinuousSetupDidFail { 

}
Just add the following extended delegate method:
- (void) wavWasSavedAtLocation:(NSString *)location {
    NSLog(@"WAV was saved at the path %@", location);
    
}

Using SaveThatWaveController

1

Add the following lines to your header (the .h file). Under the imports at the very top:
#import <SaveThatWaveDemo/SaveThatWaveController.h>
In the middle part where instance variables go:
SaveThatWaveController *saveThatWaveController;
In the bottom part where class properties go:
@property (strong, nonatomic) SaveThatWaveController *saveThatWaveController;

2

Add the following to your implementation (the .m file): Under the @implementation keyword at the top, add this line:
@synthesize saveThatWaveController;
Among the other methods of the class, add this lazy accessor method for confident memory management of the object:
- (SaveThatWaveController *)saveThatWaveController {
	if (saveThatWaveController == nil) {
		saveThatWaveController = [[SaveThatWaveController alloc] init];
	}
	return saveThatWaveController;
}
and change this block of code:
- (PocketsphinxController *)pocketsphinxController {
	if (pocketsphinxController == nil) {
		pocketsphinxController = [[PocketsphinxController alloc] init];
	}
	return pocketsphinxController;
}
to this:

- (PocketsphinxController *)pocketsphinxController {
	if (pocketsphinxController == nil) {
		pocketsphinxController = [[PocketsphinxController alloc] init];
		pocketsphinxController.outputAudio = TRUE; // This setting is not necessary if you are only using SaveThatWave with RapidEars
	}
	return pocketsphinxController;
}
Alternately, if you only want to record speech and you don't want to perform local speech recognition on it, use the following settings instead:

- (PocketsphinxController *)pocketsphinxController {
	if (pocketsphinxController == nil) {
		pocketsphinxController = [[PocketsphinxController alloc] init];
		pocketsphinxController.outputAudio = TRUE; // This setting is not necessary if you are only using SaveThatWave with RapidEars
		pocketsphinxController.processSpeechLocally = FALSE; // This shuts off offline speech recognition.
	}
	return pocketsphinxController;
}
Then, after this line for OpenEars:
[self.pocketsphinxController startListeningWithLanguageModelAtPath:lmPath dictionaryAtPath:dicPath acousticModelAtPath:[AcousticModel pathToModel:@"AcousticModelEnglish"] languageModelIsJSGF:NO];
or this line for RapidEars:
[self.pocketsphinxController startRealtimeListeningWithLanguageModelAtPath:lmPath dictionaryAtPath:dicPath acousticModelAtPath:[AcousticModel pathToModel:@"AcousticModelEnglish"];
You can add the line:
[self.saveThatWaveController start]; // For saving WAVs from OpenEars
or
[self.saveThatWaveController startForRapidEars]; // For saving WAVs from RapidEars