If I were you, I will try to conquer or farm my victim instead of spending my time, "colonizing" them. "Colony" is a very useless gaming feature.