Using OpenEars in unit tests

[fushunpoon]

PocketSphinxController provides a useful function called runRecognitionOnWavFileAtPath:. I would like to execute this as a part of my
Kiwi test suite.

I have a unit test target set up called “MyProjectTests” but it cannot seem to find any headers associated with the framework, nor can the OpenEars.framework be added to the list of frameworks and libraries to link with in Build Phrases for that target.

How may I accomplish this?

Thanks,

Hok

[Halle Winkler]

Hi Hok,

OpenEars.framework is a totally normal compiled framework with public headers (but it is not a static library project), so I would just continue troubleshooting your issue of not being able to find the headers. Maybe you need to add the headers to Header Search Paths, or maybe you are trying to integrate the framework project as if it were a static library project instead of integrating the compiled framework as if it were something like AVFoundation.framework.

[fushunpoon]

Thanks for your feedback; I managed to re-organise my test target completely and added the OpenEars and Slt frameworks to the project separately. Now it’s possible to reference OpenEars in test targets.

Cheers!

Hok

[Halle Winkler]

That is great news, I’m glad it’s working for you.

[fushunpoon]

Another question for you, Halle.

It would be very useful to be able to write automatic tests that verify any one particular language model recognises pre-recorded segments of speech. However, the only function that is able to perform this task, runRecognitionOnWavFileAtPath:, DOES NOT RETURN A HYPOTHESIS!

In order for a unit test to verify that a wav file is correctly interpreted as a the expected sentence, the only way I see it being possible is to capture NSLog into a file first (see http://parmanoir.com/Redirecting_NSLog_to_a_file) and then opening that file to make assertions afterwards.

The problem with that approach is that while I can stomach the horribleness, XCode relies on NSLog to be written to the console to provide accurate test reporting so even redirecting NSLog is not an attractive option.

Please return a hypothesis or something so that automated testing can be done. Thanks!

[Halle Winkler]

Why can’t you use the delegate method where the hypothesis is returned? How do you test other results that are returned in delegate methods?

[fushunpoon]

Okay, so it occurred to me that the OpenEarsEventsObserver class registers itself for NSNotifications on the main thread when it’s initialised, which is how it hears about events. This is asynchronous whereas unit test execution is just synchronous.

The solution was to use so that we assert that the value _eventually_ becomes the one we are looking for. The timeout needs to be set to at least 1s, so that the delegate has a chance to receive the message: in the delegate method I assign onto a property that I’m asserting over.

You can still use runRecognitionOnWavFileAtPath:: the documentation claims that it’s ‘synchronous’, which may have been the thing that misled me.
So good stuff that that’s working.

Halle, I noticed that OpenEars can’t be tested in a Logic Test with this method (see the definitions of Application and Logic tests here.

Is there a way to make OpenEars recognition work without having the application host our tests (i.e. setting TEST_HOST to the application binary)? In principle I think checking whether a WAV test resource is recognised correctly is pure enough to be a ‘Logic’ test rather than an ‘Application’ test.

I’ll be happy to provide some example code if it helps you debug this.

Cheers.

[Halle Winkler]

Nope, not currently possible. It isn’t a bug but I’ll enter it as a feature request.

[Halle Winkler]

BTW, please be so kind as to read these posts on the pocketsphinx board about what you can expect when enabling an automated WAV test with pocketsphinx and what you need to take into account in order to get deterministic results, so we can avoid the non-constructive tone that this request for assistance took on:

http://sourceforge.net/p/cmusphinx/discussion/search/?q=different+results&project=0

[Author]

Posts