I can only speculate at the moment and I’m not in front of the code, but IIRC the audio unit has both an input and an output and the audio session category is also input and output, meaning that the audio unit callback buffer is doing double duty. In Core Audio the callback buffer size is just a request, not a contract, so my suspicion is about whether a backlog of unsent output buffers or a changing buffer size is having an effect on the input buffer size or ability to call back. There is headroom in the ringbuffer for changing input buffer sizes but ultimately it’s a fixed amount.
In this case I would expect more reported errors in the callback, but if I could check this out right now, this is where I would be looking. You can experiment with turning off the output in the audio unit to see if it helps (you basically already know that disabling output helps because using a different output from AirPlay helps, so that is where I’d look).