On a finite planet, nothing physical can grow indefinitely. The more of us there are, the fewer resources there are for each of us and for members of other species with which we share the planet.
The question of human population size is fundamentally one of sustainability. Only resources that can be sustained indefinitely can support humanity in the long term. Given this limitation, we must consider carefully our consumption, what living standards are acceptable, what technologies make best use of the resources available and how to maintain the ecosystems on which we depend.