- This topic has 28 replies, 3 voices, and was last updated 9 years, 3 months ago by .
-
Posts
-
Hi Halle,
Overall this update is a nice big step forward for voice recognition. I thought the 1.x series was good, this is just that much better. Great work.
Something I’m now running into is the .vadThreshold setting. In my testing if this is left as the default or even 2.0, my 6 Plus is constantly entering the speech detect loop with the slightest noise. That’s not really a problem as I’m seemly getting MUCH better recognition across the room in a quiet environment which is going to be excellent for some of my customers.
If I use the low end of 1.5-2.0 for vadThreshold then where there is noise in the room the recognition engine seems to get pretty unreliable and flooded with the noise and has a really hard time detecting speech. If I bump this up to the max of 3.9 (looks like the framework has an upper limit of 4.0) then I can have lots of noise in the room without OE entering into it’s speech detecting loop until I say something directly into the mic. Again, this is pretty good behavior consider all the noise/music I’ve got playing and seeing it actually work.
The problem comes in when the environment has variable noise levels. Quiet at some parts of the day and noisy during other parts. (My customers use my app in a wall mounted scenario). Maybe I’m wrong on this, but I thought that the OE 1.x series did an auto-tune (calibration) for background noise and would make continual internal adjustments based on noise/recognition. Do I have that right? If so, is that something that can be turned on in 2.0 or is this behavior now up to us to implement by fine tuning the .vadThreshold in real time?
Thanks Halle!
WesHi Wes,
Thank you for your kind words. It should still be reacting to the environment and updating itself – the vadThreshold isn’t an absolute volume level, but a s/n ratio setting, so it should continue to make sense as those values change. But you might not be able to use a value which is as aggressive as 3.9, since it will tend to reject real speech under normal circumstances.
What is your experience under changing circumstances using a value like 3.0? Your feedback is appreciated on this, since even though I have a lot of different audio in my tests, there’s no substitute for real-world feedback and this is part of the new Pocketsphinx code.
I checked in with the CMU project and verified that this is correct (I’ll probably post the response later if I get permission to quote) – recalibration is definitely happening as you’ve seen, and 3.0 is probably the highest value that is correct to use. It is designed to be adaptive to changing environments but expects stationary noise, i.e. no dramatic oscillations that need to be reacted to in very short timeframes (this was also the case that would get the old VAD stuck, so we have an improvement if there’s no stuckness but recognition is sub-optimal).
It might be possible to change the VAD timeslice although it’s probably dangerous or possibly pointless to optimize in that area at the same time it continues to be developed by the Sphinx project.
If you feel like recompiling the framework, there are some config settings you can look at in OEPocketsphinxRunConfig.h related to VAD activity:
// #define kVAD_PRESPEECH //”-vad_prespeech”, int, default ARG_STRINGIFY(DEFAULT_PRESPCH_STATE_LEN), Num of speech frames to trigger vad from silence to speech.
// #define kVAD_POSTSPEECH //”-vad_postspeech”, int, default ARG_STRINGIFY(DEFAULT_POSTSPCH_STATE_LEN), Num of speech frames to trigger vad from speech to silence.Or if the issue is that recognition is getting stuck, you can also reduce this check for a stuck utterance in OEContinuousModel.m to something lower than 25:
if(([NSDate timeIntervalSinceReferenceDate] – self.stuckUtterance) > 25.0)
Remember that the framework project has to be archived rather than just built or it won’t build a universal framework and you’ll get object errors with either a device or a simulator, depending.
Question, does your app play back audio or does it just take in mic audio?
Great, I will be happy to hear about what you discover. This is a .0 version so as more info comes in from real-world usage there can be adjustments where it makes sense.
To add to this thread: in my (somewhat limited) testing with both my app and the sample app, the threshold between 2.5 and 3 works well. Default value of 1.5 seems to be too low.
(my testing was with Apple’s headset; with default levels ‘speech’ gets detected even with minor noise that’s far from the mic)
The main issue with the lower values is the end-pointing, which can affect the flow of the application. In other words, even if there is a false speech detection trigger (e.g. noise, bgnd speech, etc.), decoder will typically deal without any problems with that. But, those noise levels that triggered vad and started recognition will also prevent it to end, after user said what they were supposed to say.
This may not be an issue with RapidEars where decoding is done in the real time (so decoder is effectively doing VAD).
Yup, I’ve raised vadThreshold to 2.0 for the current version and we’ll see what the feedback on that is. Here is the CMU commentary on the VAD:
Our VAD does track the noise level continuously, it updates noise estimation every frame with sliding average of about 5 seconds.[….] it tracks the noise level and raises speech signal when the signal in some frequency band is higher than threshold * noise.
On the other hand, the VAD is designed to work with slowly changing colored (different levels in different bands) noise. It is not supposed to deal with non-stationary noise. The recommended threshold is about current value (2.0) or it could be 3.0 if you expect slightly more noise variation. Values over 3 are not very reasonable. The value of threshold describes how the noise changes (in what boundaries you consider the change as noise), not how the speech change so it should not be tuned.
I’d give it a try since it will change how transient the speech/silent values need to be to trigger. It might not be useful, but it’s the next thing I would personally investigate under the same circumstances.
The reason I asked about audio is because I would also be curious to know whether switching to a voice processing io audio unit would help you (this is something I’m going to investigate for 2.1 once I know everything has settled down from this big release). However, it will reduce the output levels of outgoing audio.
Oh, I think you might also be able to get the VPIO noise-suppression effect with using audioMode = @”VoiceChat” without having to do a recompile, not 100% sure. I’m curious about whether combining hardware noise suppression and software noise suppression gives you the results you’re looking for, and I think now the VAD probably won’t object to receiving pre-noise-suppressed audio from the mic unlike the old VAD.
The audioMode settings are still essentially experimental from my perspective since they don’t really seem to offer what I’d call an API contract, but it would be very interesting to know your experiences trying @”VoiceChat” in your challenging environment.
I’m actually using the VoiceChat mode almost exclusively since the app can also stream IP Camera video/audio. That seems to work as it did before.
Good to know, please mention things like this in troubleshooting – interpreting reports of results with processed audio from an experimental setting is going to be different from interpreting reports of unprocessed audio and a default setting and I’m in the process of evaluating the defaults, so I want to make sure I’m correlating experiences with those settings to common/uncommon use cases as the case may be. Thanks!
Hi Wes,
Thanks for hanging in there with the new setup – that sounds to me like a very good solution. I appreciate the update. I will definitely be considering your experiences and other developers’ feedback about vadThreshold, so there may be further developments on that front as I start to get some ideas about it. For now I’m still just taking in the feedback until there’s enough broad-based info to synthesize into a sense of what the developer experience and user experience with it is.
Hi Wes,
I was giving this a bit of thought:
The reason I need to use the audioMode value of “VoiceChat” is so I can issue system sounds while the mic is on. With using the “default” value for the audioMode, I have to use an AVAudioPlayer object which causes a lag or delay in the issuance of the sound which won’t work in my application.
Out of curiosity, have you checked out whether this is still necessary for system sounds with iOS8 + OpenEars 2.0? I don’t know if iOS8 has any fixes for that Core Audio PlayAndRecord bug/mal-feature with system sounds, but OpenEars 2.0 sets a somewhat different and more-minimal audio session now in order to improve coexistence, so it’s probably worth a quick check to see if the default mode behavior there has improved (maybe you’ve already done so, in which case no worries).
OK, thanks very much for confirming for me that it is the same and also that the VoiceChat mode fixes this symptom in the current version – it’s good info to have even if it isn’t the ideal situation.
Hi Wes,
Just wanted to let you know that the new version 2.01 out today has a fix for a VAD behavior bug in OpenEars that could affect accuracy and detection under some circumstances, so although your question wasn’t specifically about accuracy, you might want to check it out and see if it gives you improved results with any of the things you’ve mentioned in this thread.
That’s great! Yes, the fixes up to and including 2.03 would fix exactly the issue you described above, along with some subtler ones. In my own testing, it also did better with noisy environments with lower vadThreshold settings as a result of the bugs causing stuck VAD behavior (which sometimes looked like oversensitivity) being fixed. Thank you for the feedback.
Most of the audio session settings are the same in both versions, so if it isn’t just a new device difference or new iOS SDK version difference masquerading as an OpenEars difference (volume levels vary a lot from SDK version to SDK version and from device to device in my experience), I think it will tend to come down to the new version of OpenEars defaulting to audio session mixing now since that’s the only dramatic change. If you wanted to verify that, you could remove the option ‘AVAudioSessionCategoryOptionMixWithOthers’ from the audio session initialization in OEContinuousAudioUnit.m and archive, and see if it returns to the behavior you saw in 1.7. At the moment there is a bug in archiving which means you need to keep your original acoustic models and then drag them back over the archiving-created models after you’ve made your archive; otherwise you can get an error when uploading your app.
A near-future OpenEars version will have the ability to opt out of mixing since I’ve had a case reported where it was desirable to suppress, but I wanted to square away the accuracy/VAD issues first before making any API additions.
Sorry, no other ideas – in my experience, when it comes down to these SDK-related volume fluctuations it’s good to go with defaults. It’s an area where there is basically no API contract from Apple (like, if you read the audio mode and audio unit type descriptions they usually describe categories of apps and use the word “may” and refer to “signal processing”, rather than describing concrete behaviors, DSPs, or expectations). I had an unexpected development with really low output volume and VPIO in a very early version of OpenEars which led me to the general conclusion that default audio gets the most QA coverage and has the most “contract” to it, and even when it isn’t ideal for the application, it’s often the prudent call for that reason.
Since you are going to be adding in new API’s here, can I request an API to disable Bluetooth?
Sure, nothing speaks against it now since it should be semantically and behaviorally very similar to the mixing opt-out. I’ll add a ticket and barring any unexpected discoveries related to the feature that would cause me to revise my idea that it is straightforward, I will add it at the same time.
- You must be logged in to reply to this topic.