I’m glad to hear it. The only reason it isn’t designed for that and is designed for the use of suspend/resume during internal speaker output is just because it doesn’t work well to do speech recognition using a mic that is right next to a speaker from which continuous sound is emitting. But your case with an audio out is a different situation that can work better, if the audio out is leading to speakers that are not next to the mic and not closer to the mic than the user is, and isn’t operating with similar volume at distance to user speech. But it does have to be notably quieter at the mic input or it will cause a lot of recognition confusion.
So in that case the answer to your question above would be that the calibration will adapt to the decrease in background noise. The case you were seeing in which there was no adaptation was because of a stuck value in the VAD (the only circumstance I’m aware of this happening currently is when there is a fast, persistent and very large increase in background noise levels, but in the cases in which this happens there will now be a quick recalibration as soon as the “stuckness” gets on the radar after a few seconds). Otherwise the expected behavior during increases or decreases of background noise levels is adaptation, although the adaptation will not be instantaneous.