The best idea to make this work would be make it so the player is the one summoning the raid. If they manage to beat it, they should get some sort of combat-oriented loot, but its also not mandatory to do it.
That way, builders dont have to deal with it at all, and the PvE'ers that want to try out their combat/defense skills can get some bonus for it.
That actually sounds great! That way, it can technically happen in all modes. That way, no griefing for people who don't want this, and fishg and I get our epic fight!