"IPC Server handler 9 on 8021" daemon prio=10 tid=0x00007f2f4863e000 nid=0x4245 in Object.wait() [0x00007f2f2fcfa000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.hadoop.hdfs.DFSOutputStream.waitAndQueueCurrentPacket(DFSOutputStream.java:1384)
        - locked <0x000000073be74098> (a java.util.LinkedList)
        at org.apache.hadoop.hdfs.DFSOutputStream.writeChunk(DFSOutputStream.java:1452)
        - locked <0x000000073bec2280> (a org.apache.hadoop.hdfs.DFSOutputStream)
        at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:161)
        at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:136)
        - locked <0x000000073bec2280> (a org.apache.hadoop.hdfs.DFSOutputStream)
        at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:125)
        - locked <0x000000073bec2280> (a org.apache.hadoop.hdfs.DFSOutputStream)
        at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:116)
        at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:90)
        - locked <0x000000073bec2280> (a org.apache.hadoop.hdfs.DFSOutputStream)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:54)
        at java.io.DataOutputStream.write(DataOutputStream.java:107)
        - locked <0x000000073bee40f0> (a org.apache.hadoop.hdfs.client.HdfsDataOutputStream)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:220)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:290)
        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:294)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:140)
        - locked <0x000000073bee40d0> (a java.io.OutputStreamWriter)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
        at java.io.BufferedWriter.flush(BufferedWriter.java:253)
        - locked <0x000000073bee40d0> (a java.io.OutputStreamWriter)
        at java.io.PrintWriter.flush(PrintWriter.java:293)
        - locked <0x000000073bee40a0> (a java.io.BufferedWriter)
        at java.io.PrintWriter.checkError(PrintWriter.java:330)
        at org.apache.hadoop.mapred.JobHistory.log(JobHistory.java:765)
        at org.apache.hadoop.mapred.JobHistory$Task.logFinished(JobHistory.java:1905)
        at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2558)
        - locked <0x000000073bda1148> (a org.apache.hadoop.mapred.JobInProgress)
        at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1202)
        - locked <0x000000073bda1148> (a org.apache.hadoop.mapred.JobInProgress)
        at org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:4280)
        at org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3216)
        - locked <0x00000006000173b8> (a org.apache.hadoop.mapred.JobTracker)
        at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2915)
        - locked <0x00000006000173b8> (a org.apache.hadoop.mapred.JobTracker)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)

   Locked ownable synchronizers:
        - None


ALL other RPC threads blocked like this for extended periods of time:


"IPC Server handler <N> on 8021" daemon prio=10 tid=0x00007f2f4863c800 nid=0x4244 waiting for monitor entry [0x00007f2f2fdfc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2847)
        - waiting to lock <0x00000006000173b8> (a org.apache.hadoop.mapred.JobTracker)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)

   Locked ownable synchronizers:
        - None


WHO WRITES THIS SHIT?!