What is your render distance (or server view distance)? If it is less than 10 mobs won't despawn before they reach "lazy chunks" (entities are only ticked if a 5x5 area around them is loaded; 128 blocks = 8 chunks so you need another 2 chunks loaded out. This has been a thing for a long time but only became an issue in singleplayer after 1.7.4, which changed server-side chunk loading to match the client-side render distance in singleplayer (from 1.3 to 1.6.4 the server-side view distance was always 10 so this never happened even on Tiny render distance, though Far was limited to 10 chunks because of this).
MC-2536 View distance affects mob spawning
MC-42053 Low mob spawn rates on low render distances
Unless there is a mod that changes the entity (de)spawn radius (both will need to be changed so they can't spawn in lazy chunks) you'll have to increase the render distance. I don't know of any such mods though (other than my own, non-Forge mod for 1.6.4, which reduces it to 96 blocks/6 chunks to increase mob density without increasing the cap), only that Spigot servers allow you to change this.
That said, I do have a non-Forge mod which increases the minimum server-side view distance to 10, allowing you to use a lower render distance to improve client-side performance (however, server-side performance will always be at least that of 10, thus it is more ideal to reduce the (de)spawn radius); I don't know if it is compatible with Forge though (at the least, you'll need to add "-Dfml.ignorePatchDiscrepencies=true -Dfml.ignoreInvalidMinecraftCertificates=true" to the JVM arguments):
https://www.minecraftforum.net/forums/mapping-and-modding-java-edition/minecraft-mods/1292668-1-7-1-13-themastercavers-mods-and-tweaks ("Render Distance-Mob Spawning Fix mod" near the bottom)
Thanks! I had render distance at 8 and I didn't know it's a vanilla server problem, not only modded. I changed my server's view distance and my client's render distance to 10 and my mob farm is even faster than the old design in single player. There was so much XP that I couldn't pick it all up and my screen was only XP orbs... but it's an expensive farm and it'll take long time to build it, especially on survival. Now everything works better than I expected and the server's CPU was almost all the time at 100% so I hope it won't catch fire. Also I found one small flaw... yes, it can be even faster. You know what's also fast? Your reply, I thought this place is kinda dead, but it turns out my earlier problems (all about my modpack) were too hard to answer or I didn't have the luck and it went unnoticed. This was probably the last problem with my modpack so I can do the final test after I modify my mob farm. Thanks again.