5 blocks in which direction?
What do you mean by village? The trading hall?
Are there any doors in your trading hall?
If the villagers in the trading hall aren't at least 5 blocks above or below the doors of the breeder then they will be counted in the population cap and probably stop the breeding.
If you add doors closer than 66 blocks at any height other than the original doors then you are moving volume in which villagers are counted and may well stop the breeding.