I harvested a small magical forest and got three saplings from five silver wood trees. Three trees gave me four saplings. Those four gave two saplings. Those two gave me zero. I have a hard time believing a farm of silver wood is feasible. It wasn't in TC4 either. Which sucks because I've not been able to find more.
For some reason, an axe of the stream with fortune 3 seems to give me a net income of saplings, along with making leaf clean-up easy. Maybe you could try that and see if you have better luck with it. And just use the axe on the wood, don't bother breaking the leaf blocks with it.