Siege zombies are bastards. They ignore light levels, can spawn in bottom slabs (normally those prevent spawns), and essentially spawn inside the village, which might be inside the safe area. I know you said you used an iron door, but you also said you're working on an iron farm nearby, which would require a bunch of wooden doors. (Plus any other random wooden door nearby could create a village, too.)
Another possibility is social spawning. If you're within 40 blocks of their bunker and smack a zombie, his buddies might well decide to spawn inside the bunker. Social zombies can spawn at light level 9, rather than the usual 7.
To solve both of these issues, build their entire bunker floor out of transparent blocks such as glass with no solid blocks immediately below it. Put some solid blocks and torches down a few blocks lower if there's any chance of a creeper blowing up nearby.
Another likely possibility is lightning strikes nearby killing your villagers outright or turning them into witches (which won't attack villagers but will eventually despawn). Put a 2nd roof 4+ blocks above the existing roof to combat this.
And, since the recent addition of entity cramming, make sure you don't try to shove more than (by default) 25 of them into a single space, or they will take damage and eventually start to die.