not at all. just has to be below layer 40, and a light level of 7 and up. You can check your debug screen by clicking "f3" on windows, or "fn+f3" on mac. Hope i helped!
also, make sure to put golems on slabs so that they don't get killed by cactus! xD