You get less prestige as you rank up and add more cities.