If I remember correctly, years ago goblins were going to be in the game but were replaced by the Squidward villagers we know and love today. Personally I would love to find some goblins in a cave. This is a pretty good idea!
I'm sure it would be easily mitigated with cave villages, which would fit the same theme and perform the same thing, robes and all.
Again though, I don't think cults quite evoke *trader*. A cave village with a nether portal would be much more fitting than a troglodytic cult.