Yes, set vadThreshold higher than the default, not lower, so it will perform recognition on fewer non-speech noises. It’s a good idea to test this at different settings to find the right level. If it was set to .5 instead of the default of 2.0 that will be the primary cause of CPU overuse since that will cause it to perform recognition on everything.
Latency tuning is even more important since it is expressly designed for reducing RapidEars’ CPU usage across the board. Take a look at RapidEars’ documentation for information about its properties – it’s in the header and PDF and on politepix.com/rapidears.