Can't recognize speech after playing video with AVPlayer

Hi Aleksey,

I sent you a note — I’ve heard about this issue before but I’ve never seen any code or logs that would help me replicate, so maybe you can show me some of your AVPlayer code via email or here if you don’t mind. Aside from that, what jumps out at me looking at the logs are the five instances of LanguageModelGenerator. Is that intentional, and is the statement “Ready to play video!” the indicator of where in the process the AVPlayer is being used?

It’s extremely likely that it has something to do with the audio session if it works on the simulator (no audio sessions there, totally different audio driver for Pocketsphinx) and doesn’t work on the device (audio driver for Pocketsphinx that is totally dependent on the audio session). But that is weird if running AudioSessionManager a second time doesn’t help. Is the logging you showed above the logging that you get when you try to run AudioSessionManager startAudioSession a second time after the AVPlayer is finished?

Hiya,

Thanks for sending me the example and for the helpful logging output. Having looked at the project, I misunderstood your question a bit from the title, since the issue is with running AVPlayer and recognition simultaneously, not consecutively.

There are a couple of AV objects that override the audio session no matter what, like a UIWebView with a video in it, AVPlayer, and I believe MPMediaPlayer as well. Presumably they need to plug in to a system resource in a particular way, and my understanding is that they change the audio session type from one which can record to one which can only play back, which always hoses subsequent uses of PocketsphinxController until this is fixed because you can’t open a mic input unless you are using a recording audio session type. You unfortunately can’t run these few kinds of objects simultaneously with speech recognition, because they do not release their audio session settings while they are instantiated. But, it shouldn’t be an issue to fix the audio session settings afterwards by running AudioSessionManager startAudioSession a second time after you have completely put them away and then starting recognition after that.

Yup, nothing is different when you change the audio session category to play and record because that is the category already for OpenEars.

The issue is that it is being changed to something else while your video is being played and it is the video player’s settings which are winning out. It is probably working with the other ASR because it isn’t using an audio driver that makes use of the audio session settings, and it probably doesn’t need very low-level audio access. Is using that ASR instead of OpenEars not an option?

I think the hardest thing to change in OpenEars would be its audio driver, or the relationship between its audio driver and the continuous loop, and it is the kind of thing I’d be unable to help you out with. But I also don’t want to be discouraging. Something entirely different you could attempt would be to see if you can just use the AudioQueue fallback driver that is used by the simulator. It isn’t as good as the audio unit driver in a couple of different ways, and it isn’t very tested, but if it works on the simulator despite the AVPlayer, it will probably work on the device with the same driver. You would basically need to look for all of the preprocessor conditionals that check for #if defined TARGET_IPHONE_SIMULATOR && TARGET_IPHONE_SIMULATOR and change it to some other constant that you define, so that it is always trying to use the Simulator driver.

Nope, there isn’t anything in the SpeakHere code that is applicable to .911 other than the fact that it uses AudioQueue Services.

If it would be worth the work for you in trying to adapt sample code, I would definitely recommend instead that you stick with trying to get the fallback driver working since it’s the cleanest possible approach for getting a drop-in driver that uses AudioQueue Services (like SpeakHere) but is designed for use with the OpenEars continuous listening loop.

Another option is to use the previous OpenEars version 0.902 (it can be downloaded from the top of the Getting Started page) which uses a simpler AudioQueue driver. That doesn’t necessarily mean it will work simultaneously with an AVPlayer (there’s a difference between what the SpeakHere audio queue has to be able to do, which is basically start an audio queue and stop it on button press, and what the OpenEars driver is responsible for), but it should be a bit less sensitive to the AVPlayer overriding the audio session.

But, I would also just say to use the other ASR you found that is compatible with your requirement since it’s already working. Those are the only three suggestions I have — hope one of them is an option for you.

Hi Aleksey,

Just wanted to draw your attention to the fact that in the post above this one, there is a question from a developer who seems to be using AVPlayer simultaneously with PocketsphinxController, albeit with iPod song playback instead of silent video playback:

http://www.politepix.com/forums/topic/speech-recognition-during-play-ipod-music

Maybe this is a good opportunity to ask him any questions, since both threads are active.

Hi Aleksey,

There is a new OpenEars version .912 out now which will reset the audio session successfully when [audioSessionManager startAudioSession] is run a second time after an AVPlayer or media player has completed playback. In some cases (specifically, when the audio session changes caused an interruption message to be sent to PocketsphinxController so that it exited its loop) it is then necessary to restart the listening loop — experiment to find out whether it is needed.

I have tried this with the sample app you sent me and I saw it resume listening on the device, with the audio unit driver, without needing a restart of the loop. I will email my modified version to you so you can check it out (but make sure to install a new .912 version in the library folder so there’s no altered .911 code in there). Inline resumption of listening as the video ends isn’t intended to be a feature of the library so there won’t be a lot of support for it (the intended and supported feature is to be able to restart the loop after stopping a playback object that has overridden the audio session), but it seems to be working in this case.

Hi Aleksey,

Yup, you may need to restart the speech recognition loop after the AVPlayer instead of just resetting the audio session inline. Having them both function simultaneously despite the audio session mismatch would be nice if it worked but it isn’t a feature of the library.

Thank you very much, that’s exactly what I needed. I noticed that the fallback approach was working in the simulator, so I wanted to try that too, just didn’t know a quick way to do it. Its working on my ipod 3rd gen now, so thank you.

~Santiago