Been constantly receiving email from Linode recently about the high CPU usage of my server. Kudos to Linode, even with 100% CPU usage and >20 load average the server was still pretty solid. Though the service response time was definitely slow.
The high CPU usage was from Java so a thread dump was in order, and identifying the busy (maybe too busy) thread followed suit.
I think every Java Dev can do or find out how to do a thread dump. I was surprised that I actually needed more than a few minutes to accomplish that. I want to share a few tips here:
- “Kill -3” works (Google for the command), but if you run Java VM with a designated user, like “tomcat”, make sure run “sudo user …”. Otherwise some error like this will prevent generating the thread dump: “Unable to open socket file: target process not responding or HotSpot VM not loaded”.
- To use jstack, you’ll have to install openjdk-devel package. openjdk only gives you a JRE installation.
- “top -H” give you the actual thread in each process. Use the thread Id to match the thread in the thread dump result. Make sure convert the decimal value to hexadecimal.
In my case, the culprit was this:
“Thread-12552” #18161 daemon prio=5 os_prio=0 tid=0x00007fdf68117000 nid=0x3c31 runnable [0x00007fdf39d18000]
When dealing with large String my process seems to choke on the search replace operation.
I did come up a quick fix but it’s interesting and a little surprising to me where the problem is. I probably need to come up with a better algorithm instead of relying on “StringUtils.replaceEachRepeatedly”. This is definitely one of the area that Java, by design, hard to over perform C, or PHP.