Music playing on other devices

So I have a bunch of Google devices in my house and if you want to cast to one of them I always just selected it, otherwise music will play through the speakers.of the phone. With the S25 it's not doing that. I play YouTube music on the phone and it picks a random Google speaker instead. How can I stop this?