Ok, will do this evening or tomorrow morning. I am using the built-in mic at the top of the iPad. Also one thing I found by going through the official demo code was that after making the call below it worked perfectly. So I plugged that into my demo code just to see and it also worked perfectly every time.
[self.pocketSphinxController changeLanguageModelToFile:self.pathToDynamicallyGeneratedGrammar withDictionary:self.pathToDynamicallyGeneratedDictionary];