There really isn’t a more granular way to ask for permission in this case. Giving access to scanning for signals around you enable the app’s developers to triangulate your position. There isn’t really any room for granularity in this system as you can’t separate one purpose from the other.
You could conceivably let the OS handle scanning in a dedicated UI and let users choose the device they want to connect to from a list. However, hardly anyone would want to use a solution as complicated as that. Modern devices keep changing their MAC every couple of hours too so you can’t really grant access to a specific device based on their MAC address.
A better design would probably involve connecting the sound system to your local network and let devices discover it using mDNS and DNS-SD instead. Apps can currently see all services on the network, and the OS could handle more granular app permissions by proxying service discovery queries and resolution. Apps could request permission (through their manifest and probably not by prompting users to specific service instance names such as _googlecast._tcp or _printer.tcp depending on their actual needs.