Yep, it would create search spaces of truly remarkable size :) .
However, if your RuleORama grammar has a bit of complexity it ought to already be ignoring intermittent noises and utterances.
I.e., if the grammar has very short possible hypotheses it may sometimes respond to random stuff, but if the minimum allowable utterance has at least a few syllables it ought to be ignoring shorter intermittent sounds and utterances pretty decently. If you have a very over-reactive grammar you might also want to investigate whether you have the right vadThreshold value – it could be too low for your use case.