Well, you can use different hacking detectors like NoCheat. Otherwise, simply ghost mode + no pickup and follow people mining around. If they hit a LOT of valuable ores in a short time, then be suspicious.
But how do i use ghost mode, i been having this server for about 5 months but, i don't use plugins. mods ect. i just play normal, but with friends, but i think one of them is using hacks and if i confront them they are just going to deny everything so i want to catch them. If you dont mind telling me which one should i use.