You only lose protection if you upgrade your town hall to level 5.