I've tried every suggestion, tweaking every launch parameter, nothing makes any difference. Running java server on linux box, I have all sorts of monitoring logs and graphs, the box runs nothing else, it's multi-core with lots of RAM. Running minecraft causes no spike in usage of any CPU, RAM, RAM bandwidth or IO. Minecraft runs fine - until someone joins. Then within seconds, I start getting "server overloaded" messages in the log, with varying amounts of "ticks behind" which generally climb until it's at 60 sec and then boom, crash. At NO TIME does CPU usage spike, it never gets over 5-7 percent. At NO TIME does RAM usage or RAM I/O spike, it is never off the floor. At NO TIME does disk I/O spike. It's as if the machine is doing nothing, yet the software whines and complains about "server overload" and the one user on it experiences increasing lag until it finally crashes. I've been running Minecraft on Linux for years and this problem has been constant through many upgrades of hardware, underlying O/S and Minecraft version without relief. What gives?
You'll have to tell me if there are any particular logs in which you're interested. Mostly the logs look like this:
[00:51:23] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 2053ms or 41 ticks behind
[00:51:52] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14317ms or 286 ticks behind
[00:52:20] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12652ms or 253 ticks behind
[00:52:58] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 23187ms or 463 ticks behind
[00:53:31] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 17464ms or 349 ticks behind
[00:54:04] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 18350ms or 367 ticks behind
[00:54:36] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 17260ms or 345 ticks behind
[00:55:11] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 19508ms or 390 ticks behind
[00:55:41] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 15816ms or 316 ticks behind
[00:56:11] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14076ms or 281 ticks behind
[00:56:34] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 8595ms or 171 ticks behind
[00:57:04] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 15188ms or 303 ticks behind
[00:57:28] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 9218ms or 184 ticks behind
[00:57:54] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 10955ms or 219 ticks behind
[00:58:24] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14275ms or 285 ticks behind
[00:58:52] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13324ms or 266 ticks behind
[00:59:21] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14178ms or 283 ticks behind
[00:59:49] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13354ms or 267 ticks behind
[01:00:19] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14572ms or 291 ticks behind
[01:00:47] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13059ms or 261 ticks behind
[01:01:16] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14408ms or 288 ticks behind
[01:01:44] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12587ms or 251 ticks behind
[01:02:13] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14042ms or 280 ticks behind
[01:02:42] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14448ms or 288 ticks behind
[01:03:08] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 11087ms or 221 ticks behind
[01:03:36] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12228ms or 244 ticks behind
[01:04:06] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 15716ms or 314 ticks behind
[01:04:36] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14379ms or 287 ticks behind
[01:05:06] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 15464ms or 309 ticks behind
[01:05:34] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12692ms or 253 ticks behind
[01:05:59] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 10565ms or 211 ticks behind
[01:06:25] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 10578ms or 211 ticks behind
[01:06:49] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 9241ms or 184 ticks behind
[01:07:15] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 10653ms or 213 ticks behind
[01:07:36] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 6062ms or 121 ticks behind
[01:07:56] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5423ms or 108 ticks behind
[01:08:19] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7861ms or 157 ticks behind
[01:08:41] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7370ms or 147 ticks behind
[01:09:10] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13428ms or 268 ticks behind
[01:09:39] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13733ms or 274 ticks behind
[01:10:05] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 11467ms or 229 ticks behind
[01:10:27] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7537ms or 150 ticks behind
[01:10:50] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7959ms or 159 ticks behind
[01:11:19] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13309ms or 266 ticks behind
[01:11:46] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12737ms or 254 ticks behind
[01:12:15] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13233ms or 264 ticks behind
[01:12:42] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12446ms or 248 ticks behind
[01:13:11] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13863ms or 277 ticks behind
[01:13:39] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13379ms or 267 ticks behind
[01:14:07] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12620ms or 252 ticks behind
[01:14:36] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14553ms or 291 ticks behind
[01:15:04] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12705ms or 254 ticks behind
[01:15:29] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 10440ms or 208 ticks behind
[01:15:58] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13647ms or 272 ticks behind
[01:16:26] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13380ms or 267 ticks behind
[01:16:55] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13788ms or 275 ticks behind
[01:17:26] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 15487ms or 309 ticks behind
[01:17:57] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 16544ms or 330 ticks behind
[01:18:28] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 16110ms or 322 ticks behind
[01:18:57] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14234ms or 284 ticks behind
[01:19:25] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12634ms or 252 ticks behind
[01:19:55] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 15264ms or 305 ticks behind
[01:20:28] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 18051ms or 361 ticks behind
[01:20:55] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12097ms or 241 ticks behind
[01:21:21] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 10340ms or 206 ticks behind
[01:21:46] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 9918ms or 198 ticks behind
[01:22:11] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 9994ms or 199 ticks behind
[01:22:35] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 9076ms or 181 ticks behind
[01:22:56] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 6939ms or 138 ticks behind
[01:23:19] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7383ms or 147 ticks behind
[01:23:41] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7401ms or 148 ticks behind
[01:24:06] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 9444ms or 188 ticks behind
[01:24:28] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7291ms or 145 ticks behind
[01:24:48] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5151ms or 103 ticks behind
which goes on and on until it spikes and crashes.
I just logged in and walked around for a minute and logged out. At this time I'm the only person on the server. Here is the latest.log for that time period:
Here is a CPU snapshot at that time:
CPU not busy. Here is disk I/O:
Disk not busy. Here is network I/O:
This machine is on the same VLAN as the client in this case, with 2Gbit/sec of bidirectional bandwidth and nothing else happening. So, network not busy either.
Nothing is happening, as far as the computer hardware is concerned, but as far as Minecraft Java Server is concerned, the sky is falling.
I can't even begin to list all the FAQs I've read and parameters I've adjusted over the years. I've also upgraded the hardware of the host more than once, and I migrated it to a 64-bit Linux OS in the interest of expanding the RAM I can make available to the Java runtime process, which had zero effect.
Here is the server log from the last crash (last night, when I had two players in). I'm only including the tail of this log, which is thousands of lines of "server overloaded" messages:
And here is the crash log file referenced by the server log. Some names have been obfuscated to protect the innocent:
cat crash-2019-07-27_23.59.40-server.txt
---- Minecraft Crash Report ----
// Oh - I know what I did wrong!
Time: 7/27/19 11:59 PM
Description: Watching Server
java.lang.Error
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at ql.b(SourceFile:100)
at ql.a(SourceFile:117)
at aiu.dw(SourceFile:2783)
at aiu.dy(SourceFile:2795)
at aiu.h(SourceFile:2187)
at aiv.h(SourceFile:294)
at vi.a(SourceFile:580)
at vi$$Lambda$2359/1187365954.accept(Unknown Source)
at bhm.a(SourceFile:676)
at vi.a(SourceFile:383)
at net.minecraft.server.MinecraftServer.b(SourceFile:824)
at ui.b(SourceFile:343)
at net.minecraft.server.MinecraftServer.a(SourceFile:763)
at net.minecraft.server.MinecraftServer.run(SourceFile:631)
at java.lang.Thread.run(Thread.java:748)
A detailed walkthrough of the error, its code path and all known details is as follows:
---------------------------------------------------------------------------------------
-- Head --
Thread: Server Watchdog
Stacktrace:
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at ql.b(SourceFile:100)
at ql.a(SourceFile:117)
at aiu.dw(SourceFile:2783)
at aiu.dy(SourceFile:2795)
at aiu.h(SourceFile:2187)
at aiv.h(SourceFile:294)
at vi.a(SourceFile:580)
at vi$$Lambda$2359/1187365954.accept(Unknown Source)
at bhm.a(SourceFile:676)
at vi.a(SourceFile:383)
at net.minecraft.server.MinecraftServer.b(SourceFile:824)
at ui.b(SourceFile:343)
at net.minecraft.server.MinecraftServer.a(SourceFile:763)
-- Thread Dump --
Details:
Threads: "Server-Worker-1748" Id=1785 WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Server-Worker-1747" Id=1784 TIMED_WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Server-Worker-1746" Id=1783 WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Server-Worker-1745" Id=1782 WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Server-Worker-1743" Id=1780 WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Server-Worker-1741" Id=1778 WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Server-Worker-1739" Id=1776 WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Netty Epoll Server IO #15" Id=796 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #14" Id=291 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #13" Id=138 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #12" Id=137 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #11" Id=136 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #10" Id=120 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #9" Id=119 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #8" Id=111 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #7" Id=84 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #6" Id=49 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #5" Id=48 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #4" Id=47 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #3" Id=46 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #2" Id=31 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #1" Id=30 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Server Watchdog" Id=29 RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
at ul.run(SourceFile:45)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #0" Id=28 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"ObjectCleanerThread" Id=27 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@5cb35343
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@5cb35343
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
at io.netty.util.internal.ObjectCleaner$1.run(ObjectCleaner.java:54)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
"DestroyJavaVM" Id=26 RUNNABLE
"Server console handler" Id=25 RUNNABLE (in native)
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked java.io.BufferedInputStream@588cdc0d
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked java.io.InputStreamReader@1d8e5978
at java.io.InputStreamReader.read(InputStreamReader.java:184)
...
"Server thread" Id=22 RUNNABLE
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at ql.b(SourceFile:100)
at ql.a(SourceFile:117)
at aiu.dw(SourceFile:2783)
at aiu.dy(SourceFile:2795)
at aiu.h(SourceFile:2187)
at aiv.h(SourceFile:294)
...
"Server Infinisleeper" Id=23 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at ui$1.run(SourceFile:93)
"Snooper Timer" Id=21 WAITING on java.util.TaskQueue@6a36caeb
at java.lang.Object.wait(Native Method)
- waiting on java.util.TaskQueue@6a36caeb
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
at java.util.TimerThread.run(Timer.java:505)
"Signal Dispatcher" Id=5 RUNNABLE
"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@2cd9a5a5
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@2cd9a5a5
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@128b6887
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@128b6887
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Stacktrace:
at ul.run(SourceFile:61)
at java.lang.Thread.run(Thread.java:748)
-- System Details --
Details:
Minecraft Version: 1.14.3
Operating System: Linux (amd64) version 2.6.32-754.14.2.el6.x86_64
Java Version: 1.8.0_212, Oracle Corporation
Java VM Version: OpenJDK 64-Bit Server VM (mixed mode), Oracle Corporation
Memory: 5422678576 bytes (5171 MB) / 8367636480 bytes (7980 MB) up to 8589934592 bytes (8192 MB)
JVM Flags: 6 total; -Xms1024M -Xmx8192M -XX:+UseG1GC -XX:+CMSIncrementalPacing -XX:ParallelGCThreads=8 -XX:+AggressiveOpts
Player Count: 2 / 20; [vj['*******'/813922, l='*****', x=211.57, y=64.00, z=996.17], vj['**********'/851787, l='*****', x=-134.30, y=49.00, z=-146.69]]
Data Packs: vanilla
Is Modded: Unknown (can't tell)
Type: Dedicated Server (map_server.txt)
****** END INSERTED LOG FILE
At the time of this log, the CPU, disk and network IO were all just as quiet as the ones posted earlier in this thread.
It's backed up nightly, but this problem has been going on for years.
What really makes no sense is, corrupted or not, how can the software complain about server overload on a server that's not overloaded? In fact, you'd have a hard time seeing there was anything running at all, from the server hardware point of view.
I've tried every suggestion, tweaking every launch parameter, nothing makes any difference. Running java server on linux box, I have all sorts of monitoring logs and graphs, the box runs nothing else, it's multi-core with lots of RAM. Running minecraft causes no spike in usage of any CPU, RAM, RAM bandwidth or IO. Minecraft runs fine - until someone joins. Then within seconds, I start getting "server overloaded" messages in the log, with varying amounts of "ticks behind" which generally climb until it's at 60 sec and then boom, crash. At NO TIME does CPU usage spike, it never gets over 5-7 percent. At NO TIME does RAM usage or RAM I/O spike, it is never off the floor. At NO TIME does disk I/O spike. It's as if the machine is doing nothing, yet the software whines and complains about "server overload" and the one user on it experiences increasing lag until it finally crashes. I've been running Minecraft on Linux for years and this problem has been constant through many upgrades of hardware, underlying O/S and Minecraft version without relief. What gives?
G.
[00:51:52] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14317ms or 286 ticks behind
[00:52:20] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12652ms or 253 ticks behind
[00:52:58] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 23187ms or 463 ticks behind
[00:53:31] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 17464ms or 349 ticks behind
[00:54:04] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 18350ms or 367 ticks behind
[00:54:36] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 17260ms or 345 ticks behind
[00:55:11] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 19508ms or 390 ticks behind
[00:55:41] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 15816ms or 316 ticks behind
[00:56:11] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14076ms or 281 ticks behind
[00:56:34] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 8595ms or 171 ticks behind
[00:57:04] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 15188ms or 303 ticks behind
[00:57:28] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 9218ms or 184 ticks behind
[00:57:54] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 10955ms or 219 ticks behind
[00:58:24] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14275ms or 285 ticks behind
[00:58:52] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13324ms or 266 ticks behind
[00:59:21] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14178ms or 283 ticks behind
[00:59:49] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13354ms or 267 ticks behind
[01:00:19] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14572ms or 291 ticks behind
[01:00:47] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13059ms or 261 ticks behind
[01:01:16] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14408ms or 288 ticks behind
[01:01:44] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12587ms or 251 ticks behind
[01:02:13] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14042ms or 280 ticks behind
[01:02:42] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14448ms or 288 ticks behind
[01:03:08] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 11087ms or 221 ticks behind
[01:03:36] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12228ms or 244 ticks behind
[01:04:06] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 15716ms or 314 ticks behind
[01:04:36] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14379ms or 287 ticks behind
[01:05:06] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 15464ms or 309 ticks behind
[01:05:34] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12692ms or 253 ticks behind
[01:05:59] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 10565ms or 211 ticks behind
[01:06:25] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 10578ms or 211 ticks behind
[01:06:49] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 9241ms or 184 ticks behind
[01:07:15] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 10653ms or 213 ticks behind
[01:07:36] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 6062ms or 121 ticks behind
[01:07:56] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5423ms or 108 ticks behind
[01:08:19] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7861ms or 157 ticks behind
[01:08:41] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7370ms or 147 ticks behind
[01:09:10] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13428ms or 268 ticks behind
[01:09:39] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13733ms or 274 ticks behind
[01:10:05] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 11467ms or 229 ticks behind
[01:10:27] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7537ms or 150 ticks behind
[01:10:50] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7959ms or 159 ticks behind
[01:11:19] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13309ms or 266 ticks behind
[01:11:46] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12737ms or 254 ticks behind
[01:12:15] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13233ms or 264 ticks behind
[01:12:42] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12446ms or 248 ticks behind
[01:13:11] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13863ms or 277 ticks behind
[01:13:39] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13379ms or 267 ticks behind
[01:14:07] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12620ms or 252 ticks behind
[01:14:36] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14553ms or 291 ticks behind
[01:15:04] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12705ms or 254 ticks behind
[01:15:29] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 10440ms or 208 ticks behind
[01:15:58] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13647ms or 272 ticks behind
[01:16:26] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13380ms or 267 ticks behind
[01:16:55] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 13788ms or 275 ticks behind
[01:17:26] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 15487ms or 309 ticks behind
[01:17:57] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 16544ms or 330 ticks behind
[01:18:28] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 16110ms or 322 ticks behind
[01:18:57] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 14234ms or 284 ticks behind
[01:19:25] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12634ms or 252 ticks behind
[01:19:55] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 15264ms or 305 ticks behind
[01:20:28] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 18051ms or 361 ticks behind
[01:20:55] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 12097ms or 241 ticks behind
[01:21:21] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 10340ms or 206 ticks behind
[01:21:46] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 9918ms or 198 ticks behind
[01:22:11] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 9994ms or 199 ticks behind
[01:22:35] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 9076ms or 181 ticks behind
[01:22:56] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 6939ms or 138 ticks behind
[01:23:19] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7383ms or 147 ticks behind
[01:23:41] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7401ms or 148 ticks behind
[01:24:06] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 9444ms or 188 ticks behind
[01:24:28] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7291ms or 145 ticks behind
[01:24:48] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5151ms or 103 ticks behind
which goes on and on until it spikes and crashes.
I just logged in and walked around for a minute and logged out. At this time I'm the only person on the server. Here is the latest.log for that time period:
Here is a CPU snapshot at that time:
CPU not busy. Here is disk I/O:
Disk not busy. Here is network I/O:
This machine is on the same VLAN as the client in this case, with 2Gbit/sec of bidirectional bandwidth and nothing else happening. So, network not busy either.
Nothing is happening, as far as the computer hardware is concerned, but as far as Minecraft Java Server is concerned, the sky is falling.
I can't even begin to list all the FAQs I've read and parameters I've adjusted over the years. I've also upgraded the hardware of the host more than once, and I migrated it to a 64-bit Linux OS in the interest of expanding the RAM I can make available to the Java runtime process, which had zero effect.
Any productive suggestions would be appreciated.
Sincerely,
G.
And here is the crash log file referenced by the server log. Some names have been obfuscated to protect the innocent:
cat crash-2019-07-27_23.59.40-server.txt
// Oh - I know what I did wrong!
Time: 7/27/19 11:59 PM
Description: Watching Server
java.lang.Error
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at ql.b(SourceFile:100)
at ql.a(SourceFile:117)
at aiu.dw(SourceFile:2783)
at aiu.dy(SourceFile:2795)
at aiu.h(SourceFile:2187)
at aiv.h(SourceFile:294)
at vi.a(SourceFile:580)
at vi$$Lambda$2359/1187365954.accept(Unknown Source)
at bhm.a(SourceFile:676)
at vi.a(SourceFile:383)
at net.minecraft.server.MinecraftServer.b(SourceFile:824)
at ui.b(SourceFile:343)
at net.minecraft.server.MinecraftServer.a(SourceFile:763)
at net.minecraft.server.MinecraftServer.run(SourceFile:631)
at java.lang.Thread.run(Thread.java:748)
A detailed walkthrough of the error, its code path and all known details is as follows:
---------------------------------------------------------------------------------------
-- Head --
Thread: Server Watchdog
Stacktrace:
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at ql.b(SourceFile:100)
at ql.a(SourceFile:117)
at aiu.dw(SourceFile:2783)
at aiu.dy(SourceFile:2795)
at aiu.h(SourceFile:2187)
at aiv.h(SourceFile:294)
at vi.a(SourceFile:580)
at vi$$Lambda$2359/1187365954.accept(Unknown Source)
at bhm.a(SourceFile:676)
at vi.a(SourceFile:383)
at net.minecraft.server.MinecraftServer.b(SourceFile:824)
at ui.b(SourceFile:343)
at net.minecraft.server.MinecraftServer.a(SourceFile:763)
-- Thread Dump --
Details:
Threads: "Server-Worker-1748" Id=1785 WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Server-Worker-1747" Id=1784 TIMED_WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Server-Worker-1746" Id=1783 WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Server-Worker-1745" Id=1782 WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Server-Worker-1743" Id=1780 WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Server-Worker-1741" Id=1778 WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Server-Worker-1739" Id=1776 WAITING on java.util.concurrent.ForkJoinPool@5274b77d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.ForkJoinPool@5274b77d
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"Netty Epoll Server IO #15" Id=796 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #14" Id=291 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #13" Id=138 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #12" Id=137 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #11" Id=136 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #10" Id=120 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #9" Id=119 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #8" Id=111 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #7" Id=84 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #6" Id=49 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #5" Id=48 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #4" Id=47 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #3" Id=46 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #2" Id=31 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #1" Id=30 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"Server Watchdog" Id=29 RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
at ul.run(SourceFile:45)
at java.lang.Thread.run(Thread.java:748)
"Netty Epoll Server IO #0" Id=28 RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
"ObjectCleanerThread" Id=27 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@5cb35343
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@5cb35343
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
at io.netty.util.internal.ObjectCleaner$1.run(ObjectCleaner.java:54)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
"DestroyJavaVM" Id=26 RUNNABLE
"Server console handler" Id=25 RUNNABLE (in native)
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked java.io.BufferedInputStream@588cdc0d
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked java.io.InputStreamReader@1d8e5978
at java.io.InputStreamReader.read(InputStreamReader.java:184)
...
"Server thread" Id=22 RUNNABLE
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at ql.b(SourceFile:100)
at ql.a(SourceFile:117)
at aiu.dw(SourceFile:2783)
at aiu.dy(SourceFile:2795)
at aiu.h(SourceFile:2187)
at aiv.h(SourceFile:294)
...
"Server Infinisleeper" Id=23 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at ui$1.run(SourceFile:93)
"Snooper Timer" Id=21 WAITING on java.util.TaskQueue@6a36caeb
at java.lang.Object.wait(Native Method)
- waiting on java.util.TaskQueue@6a36caeb
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
at java.util.TimerThread.run(Timer.java:505)
"Signal Dispatcher" Id=5 RUNNABLE
"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@2cd9a5a5
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@2cd9a5a5
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@128b6887
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@128b6887
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Stacktrace:
at ul.run(SourceFile:61)
at java.lang.Thread.run(Thread.java:748)
-- System Details --
Details:
Minecraft Version: 1.14.3
Operating System: Linux (amd64) version 2.6.32-754.14.2.el6.x86_64
Java Version: 1.8.0_212, Oracle Corporation
Java VM Version: OpenJDK 64-Bit Server VM (mixed mode), Oracle Corporation
Memory: 5422678576 bytes (5171 MB) / 8367636480 bytes (7980 MB) up to 8589934592 bytes (8192 MB)
JVM Flags: 6 total; -Xms1024M -Xmx8192M -XX:+UseG1GC -XX:+CMSIncrementalPacing -XX:ParallelGCThreads=8 -XX:+AggressiveOpts
Player Count: 2 / 20; [vj['*******'/813922, l='*****', x=211.57, y=64.00, z=996.17], vj['**********'/851787, l='*****', x=-134.30, y=49.00, z=-146.69]]
Data Packs: vanilla
Is Modded: Unknown (can't tell)
Type: Dedicated Server (map_server.txt)
****** END INSERTED LOG FILE
Does it happen on one world or are others affected? And use paste.ubuntu.com to prevent huge text walls.
There is only one world.
Then try a new world and see if it happens there.
Pretend it doesn't. What next?
Then that one world might be corrupted.
How can it be repaired? It's a world upon which the users have worked for at least 8 years. Last thing I want to do is trash it.
Restore from backup. Make regular backups.
It's backed up nightly, but this problem has been going on for years.
What really makes no sense is, corrupted or not, how can the software complain about server overload on a server that's not overloaded? In fact, you'd have a hard time seeing there was anything running at all, from the server hardware point of view.