OK, I think this is probably going to be the root of the issue – RapidEars is only designed for use with the shipped acoustic models and with the normal OpenEars-required language model maximum size of around 200-500 words depending. I can imagine your specs working some of the time on a 5S, but it’s too much overhead for an iPhone 4 to support. I think you’re just seeing a delayed search time on a slow device for a 16khz model doing large vocabulary recognition – speed considerations are the reason that even stock OpenEars doesn’t use 16khz models and big vocabularies, and RapidEars is more sensitive to performance considerations since it has to do much more.
If you’re finding the performance acceptable on devices later than the iPhone 4, I think what I would do is to make a downsampled version of your acoustic model, ship it alongside your 16khz model, and have the iPhone 4 and earlier fall back to using the 8khz model. This might help your issue sufficiently, but it also might be necessary for you to have the iPhone 4 fall back to a smaller vocabulary as well in order to achieve normal performance. Those would be my starting suggestions so that you can preserve your ideal performance for your users whose devices can support it without giving the older phones a job to do that they aren’t up for.