The solution for the link you gave me actually helped with this issue. I terminate the voice recognition after 2 sec.
I am still using the background thread that checks the inputLevel and restarts if it doesn’t change. But the number of times that happens now has decreased very much (specially after a running a while when it seemed to happen all the time).
Maybe the terminating the voice recognition lets the buffer “breathe” a little.