I run a small server for < 10 people with about 4G ram allocated. The server ran fine for weeks. When my roommates joined, the server lagged for them and all other players. The same happened when I used the IP to join but not on localhost. (the server showed 1G free memory and tps was fine at all times).
The problem was that they and I connected from the same network that the server was hosted from. And when they turned on a vpn, everything was fine. Same when I used localhost instead of the IP.
So if possible, get your friends and family on the same network as you to use a vpn or a different internet connection (like a hotspot from their phone*). Better yet, if you have multiple people on the same network and you have an alternative connection thats good enough, host the server on that. And always use localhost to connect.
*Some carriers will charge you for using a hotspot, especially US ones
Anyway thanks for reading I hope this helps somebody. Took us way too long to figure out.