I'm surprised that nobody has mentioned that you can use Customized to disable their generation since the OP just seems to not like how they appear in the world. If you do still want to be able to find them you could set their max height to a low value, such as 8 (count should also be reduced to 2-3, making them more concentrated but not so much that they mostly overwrite themselves with the variant that generates last), then they will only generate below y=8, below the level of most caves (mineshafts can go deeper but they are rare), and even without any naturally generating you can still craft them.
Also, you can remove them from an existing world by using MCEdit to replace all stone blocks (make sure all subtypes is checked) with stone:0; to prevent them from generating in new chunks you can edit level.dat to change the world type to Customized and add the appropriate preset to generatorOptions (it is worth noting that even if you downgrade (not recommended!) they would not disappear and simply turn into stone since the actual block ID still exists; they even reappear again if you upgrade as metadata is not updated unless the block is changed).
I voted for no. I wouldn't mind if the textures were tweaked and I would like it if we got matching stairs/slabs, but I don't hate the -ite blocks as they are.
I use the polished version of diorite as an accent in dark oak structures/interiors and personally like the contrast between them. Also, early game, if one doesn't have quartz, diorite is fairly acceptable for Greco-Roman temple style building, especially as something atop a hill to be viewed from a distance below. The texture is a bit busy up close.
But...when was dark oak added...was it after the -ite blocks?
I also like the raw and polished versions of andesite as a block to mix with other grey/gray masonry/stone.
Granite is a challenging one for me, but I know rose/pink granite is a real thing and one would be missing it if they tried to build Egyptian monument style struuctures in a desert biome, because it was used inside pyramids or in some temples. I think it's more challenging to place in other styles of building because it's a very salmon pink and those warm/orange tones are tricky. I would guess it looks best with oak (of the woods.) Jungle? (I don't think I tried this.)
I'm using BDCraft and the Torchwall Castle from Marketplace.
The "sewer" levels and much of the castle makes use of undyed terracotta and polished granite. The effect works, and it is far better than "yet more stone brick" in the build. They mix between the terracotta and polished granite, sometimes more of one, sometimes the other at random. It really works as an alternative to "just more drab grey" or tons of expensive quartz. No, I don't want the blocks removed, and I don't see Mojang ever doing such a thing as it would be a disaster for any pre-existing build.
I'd just like to have the things tweaked on appearance and made generation "biome specific".