I haven’t tried removing the band from my watch yet, partly because I don’t have a secondary band to swap out and partly because the watch and the band feel like a singular unit. Even though they are designed for customisation, they feel very integrated. Apple has trained me for the last five years (maybe more) to not fiddle with the hardware they produce. I don’t want to detach my Sport band because it is uncomfortable, it feels like the wrong thing to do.
In fact, when I showed the watch to my dad, it didn’t cross his mind that the watchband is independent from the body. His preconception, validated by every other Apple product in recent history, is that Apple products are not to be tampered with. Although I haven’t popped the band out, I do know that it is possible. I bet a good portion of Apple Watch owners never realise that they can change the strap.