I have had trouble with mine where a water elevator was allowing baby villagers to glitch their heads into blocks and die. I didn't notice for a long time because I'd load my chunk by working around my base and checking every few minutes. It wasn't until I was lucky enough to come by at exactly the right time that I realized what was happening. If this does happen, it can slow the breeder way down and then when they so finally breed the baby dies quickly resting the whole issue.
Also how sure are you that there are no other villages near the doors within five blocks in the Y direction?