The server memory usage looks very different than the prior version. Is there some memory management issue in this version, or was memory management updated in this version? Attached is a snapshot of typical memory usage history.
It's likely the 1.13 code is far from optimized, and things will probably improve as it gets updates (server and client).
Are you running the Mojang server?
SpigotMC has 1.13 builds available now, with frequent updates. They are working hard on fixes and improvements. I built it 3 days ago, and last night it said I was 28 versions behind. I'm running a SpigotMC 1.13 server on an old broken Sony laptop, and it's been pretty stable for me. It's just not ready to up-convert older worlds yet.