Is the bigger issue the memory overheard or the time to return a hypothesis? A language model switch has to occur after some kind of final hypothesis return since it’s like a micro-restart, so a pause has to have occurred. Just to verify, have you tried this using a statistical model rather than a grammar?