There is a breeding cap and a hard cap. They will stop making new babies when the population reaches 34. The hard cap of 50 is in any given loaded area and if you exceed that number villagers MAY despawn. The despawning behavior seems to change from update to update. If you need more villagers you need to send the ones you want to keep into unloaded chunks.