How far from other villagers is the farm? If the farm is within 42 blocks, the zombie my be locked onto a villager outside the farm. From the wiki. https://minecraft.gamepedia.com/Zombie#Attacking_villagers
Zombies target villagers within 42 blocks, and they can always see villagers through walls. Once a zombie has focused on a villager, the zombie ignores any other villagers and the player, until its target is dead or the zombie is attacked
More likely, it is probably this. The villages can't stay constantly scared anymore from a 1.14.3 pre 2 update. https://minecraft.gamepedia.com/Villager#History
Panicked villagers now have to work and sleep, so they cannot be in a state of panic all the time.