Quote from DiligentDiamonds
If I'm not mistaken, It's impossible for people with mods to not download worlds. If I'm wrong please correct me with proof.
Diligent Diamonds Team
I do not believe it is possible due to the client rendering the server as well and therefore the file is also on their computer. However, I may be wrong.