It has to do with how many cities you have, your rank, and your title. The higher/more of each, the less prestige you get for each action.