Nope. I was highly annoyed with the one who spawned by our main base. I killed him and walled in the llamas. They despawned and later others showed up at our village roughly 800 blocks away. The 4th one finally had something I wanted, coral, so I traded with him. Sometimes the Wandering Traders spawn in the village palisade and sometimes outside the palisade.