Whenever you look at something, Minecraft performs ray traces to get the target. These are just Vectors pointing along the player's view vector with a predefined length. Look at the RayTraceResult class for more info
Doing this may violate the Mojang EULA, but you are safe as long as it's not meant to be played on a public server that doesn't have this mod installed. You have been warned.
Your code checks if the player is pointing AT A PLAYER and then attack the entity... You should check if the entity is NOT the player (no instanceof checks here, because you want to be able to attack other players!)
I'm a bit confused on how you want this to work. You can certainly make it so a player's reach is extended, in fact I have a tutorial on how to do that, however, you can only do it if you can mod the server. The client side really only processes the player input (i.e. that the player has clicked on something to attack) and then send a packet to the server to say "I am interacting with the target" but the server has code that will double-check that the reach is valid and won't do anything. In other words, like any client-server MMO game the server doesn't just let the client tell it when things happen.