Make sure you sprint at the target or get a critical hit to guarantee that a sweep attack doesn't occur. I use a toggle-sprint mod that I turn on if I only want to hit 1 target, and often stand still or walk backwards to sweep attack.
Alternatively you could try using an axe as your weapon, although you won't have access to half the enchantments a sword has.
Ax here, too. I mainly use axes against single opponents or when culling the herds or flocks on my farms. It's precise and does a lot of damage, especially on a crit. I only use swords vs. groups of opponents (zombies,) or if I have to attack quickly multiple times. I use bows vs. creepers, cave spiders, and ghasts.
Side effect is that my sword and bow tend to last a long time. I go through axes much faster.