Yea just move them away and make a small house or something to do while your there. I recently made a iron golem farm for the first time and to curb the immense time of waiting. I made a 20 by 20 cow farm and planted wheat by the river and sugar cane. Basically I kill time making cows breed and waiting for the wheat to breed them. Also I have a sugar cane farm for the books/leather. All this while I am waiting for my iron golems to spawn to their respective deaths.