On/Off switch doesn't seem to do anything

Hello, I just got around to trying out the newer version of the power hat with the switch. It seemed to work as expected, however after I separated the battery holder from the power hat itself, the On/Off switch no longer functions. If there is a battery in the holder, it will send power to the Pi regardless of the state the switch is in.

Looking for a possible explanation or solution. Is this its intended design?

Thanks!