I also agree with the idea that they should be scarce, and much more powerful. If they were as rare as the skeleton jockeys and incredibly strong, they could as well methodically attack player structures, tearing them apart rapidly unless, say, they're made out of obsidian.
Though I don't have a problem with them as they are. I built my skull fort middle of ocean, lava flowing out of it's eyes and I plan to make whirlpools around it. If I were to make, say, a giant statue, I imagine I would do the same. And it was hilarious to walk in a lava hole made by an Enderman.