Hello, I have built a villager farm, i've inserted some screenshots. The farmer has got one wheat seed in their inventory as far as i know and has carrots planted all around them but they aren't harvesting the carrots at all. I've thrown a few carrots on the ground and they pick them up but don't throw them to the other villager until a few minutes have passed. I've also put a composter down but the villager farmer just likes to jump on it.
I've also attached another image to show how far away the other librarian villagers are. I'm on java edition 1.15.2 with optifine.
It's hard to tell from a screenshot but I'd say the farm is probably close enough to the bed by the librarians that he'd want to get to the "village", I'd try adding a bed or two to the farm to keep him happy where he is..
If you trade with him he ought to restock his trades at the composter but I'm pretty sure they don't compost seeds until 1.16