Your military should be limited by your overall population. It makes no sense to have 1 million troops in a city that only has 12k pop for a player that only has 80k pop total.

Want a larger army? Build more cottages, want to build a smaller army faster? Build as you do now.

If population became the regulatory method for army max, I'd tear down 24 of my lvl 9 barracks and put cottages in their place.