Hi there! I have made infinite villager breeder [one designed by ImpluseSV] and enhanced it a bit to prevent villagers stuck in the farming area (by watching SerpentDagger's video on the topic). I made the breeder, made the village underneath by placing six doors and leaving a villager, giving him access to direct sunlight, all and everything and it produced the first two villagers very quickly but then it suddenly stopped. It's about 4 hours and I'm not able to get anything done.
I have done some testing which indicates that it is a combination.
Villagers on the same level as the village center seem to be counted if they are within the village border even if they aren't near doors.
So, yes, you definitely should move your collection center.
Additionally, villagers more than 6 blocks above or below the center seem to be counted if they are close to a door, and it doesn't even have to be a valid door.
(My results could possibly be caused by unvalid doors within 5 blocks of a villager being able to affect the position of the village center but unless I see that suggested somewhere I'm ignoring that possibility.)