ya its a fair game when you can handle it , if you can beat him , and farm his city , i say take his lakes and valleys too and farm him till he drop dead ( if he is an enemy)
if he is a friendly and the NPC is closer to him than you , he's right , its his territory
there comes the factor : who was there first ?
if he was there first , yes he can claim it cause he built it. this in the case he's a friendly and only that. like i said , if he's red or grey , do whatever you like
cheers