ARG – Multiple conflicting APIs

[geareddev]

Hello,

I am working on an augmented reality game which uses a binaural audio engine called Papa Engine and a low level audio processing api called Novocaine. Novocaine reroutes the microphone data through the users headphones. I mix binaural sound effects into that to create the illusion that these sounds are real (and in 3D space).

Novocaine is open source, but Papa Engine is not. It took me a while, but I managed to get the two of those to play nicely together. Enter OpenEars. I also want my ARG to do voice recognition. During the experience, there is a narrator which tells you what to do. When I disable Novocaine and Papa Engine, OpenEars works great. When I enable either of those first, the application crashes when I call “pocketsphinxController startListeningWithLanguageModelAtPath.” If I enable OpenEars first, the other two work, and pocketsphinxController says that it is listening but voice recognition stops functioning.

Is there any way to setup the audio sessions so that I can get all three of these to play nice?

[Halle Winkler]

Welcome,

Sorry, this is unlikely to work although I wish I could give you more encouraging news. Here is a bit more about it from the FAQ: https://www.politepix.com/openears/support/#Q_I8217m_trying_to_use_a_sound_framework_like_Finch_or_another_OpenAL_wrapper_and_things_aren8217t_working_as_expected

Basically, all of these frameworks are making a bid for the session and they all probably require it to a similar extent because low-latency audio i/o is resource-demanding. But, you might be able to find out a bit more about the precise point at which it is breaking if you enable logging: https://www.politepix.com/forums/topic/install-issues-and-their-solutions/ and you can also try searching these forums for some info about PocketsphinxController’s audioSessionMixing property, which is designed to allow session mixing (although this scenario with two other frameworks may still be a bit too much).

[Halle Winkler]

Wanted to add: if you search these forums for ideas about audioSessionMixing and session mixing in general, it’s probably best to stick with posts from the last year or so, since I migrated those methods into PocketsphinxController and there will probably be some out-of-date suggestions recommending accessing the session via AudioSessionManager properties directly (not advisable now).

[geareddev]

At this point, I don’t even need PapaEngine and OpenEars to work at the same time. I just need my app to not crash when I activate OpenEars after PapaEngine. Unfortunately, PapaEngine is a singleton class, and there’s no real way to turn it off. I’m not sure how it holds onto the audio session, or how to temporarily disable it from the outside.

Everything appears to work up until this point:

[11748:60b] PreferredBufferSize is now on the correct setting of 0.128000.
:: Buffer::copyHeaderFrom(ch 2 2, fr 256 4096)
:: Error: An error occured in the CSL nextBuffer method: Can’t reallocate buffers at run-time

[11748:60b] preferredSampleRateCheck is incorrect, we will change it.

It is at this point that the app crashes. The crash occurs inside:
AudioToolbox`AURemoteIO::PerformIO

Is there anyway to prevent this crash? I am okay with the two APIs not working simultaneously, but I need to be able to alternate between the two.

[Halle Winkler]

Sorry, that looks like it is due to PapaEngine not being able to share the session or (as far as I can tell) release the remoteIO input while it isn’t using it, so I’m not really able to offer help from this side of the picture if it runs as a singleton that can’t be stopped.

It’s unlikely to work to run a generalized engine that requires complete control over the session and i/o at the same time as performing low-latency speech recognition from a mic stream. OpenEars is designed to recover gracefully from other audio session usages when it starts listening, but it can’t avoid taking over the audio stream while it is being asked to perform recognition.

Did you try my suggestion of researching session mixing results in these forums?

[geareddev]

When I add this code:

[[AudioSessionManager sharedAudioSessionManager] setSoundMixing:true];
self.pocketsphinxController.audioSessionMixing = TRUE;

It only crashes about 50% of the time. When it doesn’t crash, I get a constant stream of this in the console:

:: Buffer::copyHeaderFrom(ch 2 2, fr 256 2048)
:: Error: An error occured in the CSL nextBuffer method: Can’t reallocate buffers at run-time

But otherwise both APIs seem to function. I get binaural sounds, and speech detection works. If I can manage to hack this together so that it doesn’t crash at all, that would be great. I have tried modifying constants such as those in AudioConstants.h but the OpenEars logging seems to suggest that my values are being ignored. I have a gut feeling that if I could match up the buffer sizes that I could make these two play nice.

[geareddev]

I went into OpenEars and modified kBufferLength to match the buffer used by PapaEngine.

//changed from 0.12800
#define kBufferLength 0.005805

Both APIs play nicely now, but the speech recognition isn’t functioning all that well. It only detects a word about 20% of the time, and is often wrong. Will playing with any of the other constants improve the speech recognition and return it to its functional state?

[Halle Winkler]

Sorry, those settings are not optional. They are required for recognition. I’m afraid troubleshooting this is a bit beyond the support I can give you since it is an issue originating from another framework. I would focus the research on how to shut PapaEngine off or suspend it while another framework is making required changes to the session.

[Author]

Posts