The symptoms goes away if I DON’T make the stopListening call when going into the background. Now I’m only calling stopListening during an audio interruption or route change. I believe that my AVAudioPlayer instances were interfering with the stopListening in some way. I had been stopping the instances right before making the stopListeningCall. The trick turned out be to stop them in the didStopListening delegate call back. This way I only try to stop them after I know OEPocketsphinxController has itself stopped avoiding the conflict. This seems to work fine for the audio interruption related situations but for not going to the background.
I think this solves this issue, thanks for your help!