This may be related to a similar issue with peaceful mobs despawning.
If you have a fenced in area that contains peaceful mobs, for example cows. If the pen is larger than 20 blocks in any direction, it is possible for the cows to despawn. If it's 19 blocks in either direction, the cows will not despawn.
If the villagers are using the same code mechanics as other peaceful mobs then they probably can despawn if they are in an area that is larger than 19 blocks in either direction. Since the villagers did not despawn while you had them locked in their houses it makes me think this may be the case as well.