I am new to modding Minecraft servers and am a bit confused.
I had been playing on a server while using mods through my client and they worked fine. I later started a modded server and found out that all of the players have to have forge and the mods installed on their computers. So whats the point of modding the server? Am I missing something?
Well, it depends on the server and the mods. Forge is indeed known to use mods which mostly work on the client, so if you want to play together with those mods (think about tech mods and such) then everyone, including the server, needs to run those same mods.
But there are also mods which are used on the server itself, so that even vanilla clients can connect and use those. This is usually the case for the Spigot project. Mods like WorldGuard or GriefPrevention for example; these allow players to claim and/or protect land. mypet, a plugin which gives players the opportunity to keep a pet (basically a mob which will keep following them around).
So it heavily depends on the server and the mods you're using.
Modding your server can make your game worth it. I see surprisingly more players playing on modded servers that I run across. Yes there is a deterral, as players will have to first download all the mods required to play on your server, then try them out and the server, and if they dislike it, they wasted their time and computer space (although this should take less than 20 minutes). But at the same time, the people who do join will most likely be players happy to give the mods a try, players who enjoy playing modded, and those players may enjoy the game more and stay for longer than if you did it regular (you still honestly still need modifications with regular to make it good, such as some form of anticheat and grief prevention). What you should do to make a good modded server is first thoroughly consider all of the mods you're adding. Do they add cool features that add to the game (blow-darts, dragons, custom structures, well-designed mobs), or superfluous boring convenience features (mega backpacks with all these add-ons, factory generators, etc)? Every mod you add should be worth downloading and should add mostly good features and few pointless or grinding ones. You also don't want to make the game suddenly too complicated with a ton of mods that don't fit together. If you want an interesting and nice modded theme, try out themed dimensional mods such as Twilight Forest, The Aether II, or (AMAZING DivineRPG and Advent of Ascension)!
The goal of modded servers is to bring people who like playing those certain mods together to play, just because your server supports the mods, doesn't mean that everyone can connect without the mods. Mods are mostly client-sided, as everyones game needs the proper textures etc.