the answer is very clear, the game allows no1 to control a barb town therefore it is anybodies..according to the game itself (its like hitting a no allaince member when u know they merging from one freindly to another freindly and they are in that transition period....the game lets u do it so its fair game..lol)

however since barb towns are a source of food for a certain player/players, expect there to be contentions over who farms them first, and who is getting the food, so there may be consequences to attacking an npc that some1 has enjoyed farming for free up until that point (just like there is probally going to be consequences for hitting that merging friendly player)

my best answer and fairest answer, if the game lets u do it, u can do it...but there may be consequences!