How many colonists until you feel overwhelmed?

I wanted to create a city, a large settlement ruled by a few but manned by many, but I often leave my colonies without finishing their story, I realized I enjoy the early game much much more. Does anyone have had this experience of trying to make large settlements and then backing away from it once it gets too clustered and mind fucking to manage. Is there any mods that makes this easier, should I just be content with a few pawns per colony.