The villagers that generate with villages do not despawn. They either die from taking too much damage, get turned into zombie villagers and then despawn per normal hostile rules, or the escape their enclosures to wander freely around the world (usually dying quickly because of zombies or falling off ledges repeatedly/jumping into lava lakes/getting stuck underwater in pools we've torn away sand and clay and other resources from without filling those holes back in). In some cases, they didn't despawn so much as get overwritten due to world-saving bugs that have corrupted a particular region. This will revert the area back to the last saved state or even all the way back to the original worldgen state before you did ANYTHING there.
Wandering Traders, a new villager mob introduced in 1.14 that comes with trader llamas, DO despawn, however, along with the llamas. These don't count towards village populations, however, so if you didn't want them around feel free to murder them. You might be able to nametag them if you DID want to keep them around.