Use a flipflop to make a door spammer (noise maker) activate every time someone enters the house through a pressure plate or lever (or button). Then have it flip back to off when they leave :3
Leave a hidden entrance so you can get in and turn it off when the prank gets old.
Grab yourself a toggle sneak mod and go camp inside of their house. Dig to their chests and wait for them to start moving stuff and then take their most valuable things out and move them while they are watching. So long as you are crouched (with the toggle sneak) they won't know what it messing with their chests.
How about a large labyrinth at the edge of map with dozens of traps? Your friends should think you've stopped pranking and working on a project. And they will be surprised when you teleport them to the labirynth. That could be a nice event, through. And make reward chest filled with diamonds for winners, who reached the exit.
Tell your friends to start mining downward in a staircase formation. Tell them you'll stay up to stop monsters from attacking you. Then when they're very deep in the earth, pour lava down the hole. They'll be so mad!
Another thing to do would be to walk far away from everyone else, and start saying that your minecraft is glitching up in weird ways. After some buildup, say "OMG IT'S H" and disconnect very quickly after that. Then change your skin to herobrine very quickly, and come back on the server. When people ask questions, just don't answer. Run up to or teleport to all the people on the server and start attacking them. This might not work if your friends aren't that gullible, but it's worth a shot!
Try out the mod called Trap craft their are some great things to prank people with such a the one way glass which is basically like a trick mirror theirs one side that has glass on it and you can see through to the other side and the other side has a cobblestone layer that you can't see through. Theirs sticky sand does just as it says. Then the other one that is quite harmless are trap chests basically the chest is like a lever when its off its like a normal chest but when someone tries to use it it turns on great prank item. Oh I had not noticed it was not for SMP yet but I'm sure it will be soon. I have also seen a simple mod called hidden levers it basically disguises levers as a torch when their off and a burnt out torch when their on. Their is also an invisible pressure plates mod.
Their is also this mod where you can take structures granted their not to big and make them basically float and you are able to control them its called Zeppelin don't get it confused with the other Zeppelin mod.
Their is a Combustible Lemon mod inspired by portal 2 not exactly sure what happens when its used as theirs no video but it looks to me like the mod fills trees with lemons your friends will be like what the hell and you shear them off the trees.
Their is a laser mod now that you could use to surround your friends houses in lasers and make it like mission impossible although I'm sure the lasers can kill you but come on who does not want to see someone try to dodge lasers getting to their house.
Their is a mod called Wolfgun it basically looks like a regular bow an arrow but it shoots out eggs and then those eggs turn into wolves.
Their is a mod the bookshelf activator it really helps to make hidden passage ways if any of your friends has a bookshelf wall or something you could create a secret passage disguise as a creeper somehow hide your name and when he/she comes in their house scare the crap out of them by coming out of the secret passage. Their is also the secret bookshelf.
You could play a prank with the anti- gravity mod which allows you to lift entire areas of blocks
their is also this other mod I cannot for the life of me remember the name of but it basically allows you to make doors out of any material.
Then theirs another mod that allows you to make half doors that would be quite funny your friends go to open their door or doors and they only open half of it I think this one was called multiple use door mod or something.
Then I know theirs this mod that has a bunch of trap blocks and theirs this one block that you lay down and when you step on it you become surrounded by leaves its spose to be like a net really easy to break out of but at first its like whats going on here.
Then I hear the portal gun mod is coming to SMP sometime so what you could do when it comes is put a portal in your friends house at the block right next to the door so when they walk in they fall into the portal. Then what you do is build a hole far down underground and make sure its enclosed and if you don't want them to possibly die light it up then you put another portal in that room on the ceiling don't make it to high up then when they come on they fall into the portal you then quickly get rid of the portals and he/she is then trapped in an enclosed room somewhere underground. Or since you can shoot arrows through the portals now you can mess with your friend when their backs are turned and shoot random arrows at them then make the portals disappear and make them think a skeleton is stalking them.
or theirs this other cool door mod I saw called secret door basically what it does is you make the door and where ever you place it it blends in with those blocks so say you cut down part of a tree place the door their it should look like the wooden blocks you chopped down so you can essentially make fake trees it does however need to be updated last I saw.
Place all sorts of noteblocks on circuits under their house and cover it all up with dirt/stone so they have trouble finding and silencing it. Or, build an exact copy of their house next to their house and trap that with TNT.