How to Generate and Analyze WebSphere Thread Dumps: A Comprehensive Guide for Troubleshooting Performance Issues

Thread dumps are an invaluable tool for diagnosing performance problems in Java applications running on WebSphere Application Server. As a WebSphere admin or developer, you need to be well-versed in generating and analyzing thread dumps to pinpoint issues like deadlocks, race conditions, and bottlenecks.

In this comprehensive 4500+ words guide, I will cover the A to Z of using thread dumps for troubleshooting from my experience of handling performance issues in large-scale WebSphere deployments.

Why Thread Dumps Matter for Performance

Let‘s first understand why thread dumps are critical for performance analysis.

Java applications rely heavily on threads – they are the workforce that executes your business logic. Issues like:

Deadlocked threads
Excessively long-running threads
Threads waiting for I/O or locks/monitors
Threads consuming excessive CPU

can severely impact the performance and throughput of your application.

When such problems occur, the first responder is taking a thread dump.

A thread dump gives you a snapshot of all the Java threads running inside the JVM along with their:

State – whether RUNNABLE, WAITING, TIMED_WAITING etc.
Stack traces – to pinpoint the exact line of code where the thread is stuck
Locks held – to identify deadlocks

Just like a heap dump gives you a snapshot of the contents of the heap, a thread dump gives you a snapshot of the threads which are the lifeline of your Java application.

Let‘s take an example:

Your WebSphere application works fine typically but sometimes slows down during peak load giving time outs. The thread dump taken during the issue reveals:

Many threads in WAITING state
Stuck waiting on some application lock or database call

This indicates there is some synchronization bottleneck in the code. You can instantly zoom in on the root cause by looking at the stack traces of the waiting threads.

This is just one example of how thread dumps provide invaluable data to understand performance issues.

According to research by AppDynamics, analyzing thread dumps was considered the #1 troubleshooting technique by Java developers and admins.

Let‘s now see how to generate thread dumps in WebSphere followed by how to analyze them.

Generating Thread Dumps in WebSphere Servers

WebSphere provides multiple straightforward ways to generate thread dumps:

1. Using the wsadmin Tool

Wsadmin is a handy Jython scripting tool that comes with WebSphere Application Server. You can easily generate thread dumps with it.

Here are the steps:

Login to the WebSphere server and navigate to the profiles/[profile_name]/bin folder
Run the wsadmin.sh/bat script:

$ ./wsadmin.sh
WASX7209I: Connected to process "server1" on node xyzNode01 using SOAP connector; 
The type of process is: UnManagedProcess
wsadmin >

Set the JVM name in a variable:

jvm = AdminControl.completeObjectName(‘type=JVM,process=server1,*‘)

Execute AdminControl command to generate dump:

AdminControl.invoke(jvm, ‘dumpThreads‘)

This will dump the thread information in the profiles/[profile] folder.

2. Using the kill command

If you know the process ID of the WebSphere JVM, you can generate a thread dump by:

Finding the process ID using ps -ef | grep java
Killing the process with:

kill -3 <PID>

This will create the dump in profiles folder.

3. Using WebSphere Admin Console

You can also easily take a thread dump via the WebSphere Admin Console:

Login to the Deployment Manager console
Go to Troubleshooting > Java > Thread management
Select the JVM, click "Java thread dump"

This downloads the thread dump to your local system.

Generating Thread Dump

When Should You Take Thread Dumps?

Take multiple dumps 5-10 seconds apart during an issue to compare
Capture dumps during peak load when the problem is occurring
Name dumps with timestamps or sequence numbers to identify order

This helps you pinpoint threads stuck in problematic state during the issue.

Now that you know how to generate thread dumps, let‘s discuss how to analyze them.

Tools to Analyze WebSphere Thread Dumps

There are excellent free tools available to analyze thread dumps:

1. IBM Thread and Monitor Dump Analyzer

IBM TDA is a free tool that makes analyzing WebSphere thread dumps easy.

You can download TDA here. It comes as single JAR file.

To start the GUI:

java -jar TDA.jar

Open your WebSphere dump files in it.

TDA provides various thread dump analysis reports:

Thread status – stuck, deadlocked threads
Locks/monitors – contention
Methods and call stacks
CPU usage
Memory usage

For example, this shows threads stuck on application locks:

IBM Thread Dump Analyzer

You can also analyze trends by comparing multiple dumps.

2. Samurai Thread Dump Analyzer

Samurai is an excellent open-source Java thread dump analyzer.

It has an easy to use web interface where you can upload the thread dumps and analyze them interactively.

Samurai Thread Dump Analyzer

The UI shows you thread state analysis, locked monitors, deadlocks, method calls, and more. It provides a very intuitive visualization of complex thread dump data.

3. FastThread Online Analyzer

FastThread is a free online tool that allows you to upload and analyze your WebSphere thread dumps directly in the browser.

It shows you top method calls, thread state analysis, deadlocks, CPU usage – all the key reports required to troubleshoot issues.

4. VisualVM

VisualVM is a free advanced Java troubleshooting tool provided by Oracle. It allows you to open thread dumps and analyze them.

The advantage of VisualVM is it gives you correlation between code and threads. You can directly navigate from thread stack traces to the corresponding source code lines.

So in summary, you have some really amazing free tools to help analyze WebSphere thread dumps effectively.

What to Look for in Thread Dump Analysis?

Based on my experience in troubleshooting performance issues, here are some of the key indicators to look for in WebSphere thread dumps:

Deadlocked Threads

Threads stuck in a cyclic wait form a deadlock. This brings the application to a grinding halt.

Deadlocks will be clearly highlighted in the thread dump analysis reports from tools like IBM TDA or Samurai.

Looking at stack traces of deadlocked threads helps identify the code causing it.

Hung or Long Running Threads

These are threads which are stuck in RUNNABLE state consuming high CPU for an extended period.

This can happen due to inefficient code or algorithms causing severe performance issues under load.

Thread dump tools like Samurai and FastThread show you top CPU utilizing threads.

Excessive Synchronization

Threads waiting for monitors/locks indicate contention and liveness issues due to excessive synchronization in code.

The thread dump will show you the monitors threads are stuck waiting on. By correlating monitor names with code, you can identify bottlenecks.

IO Blocked Threads

Numerous threads in WAITING state can indicate IO bottlenecks like database connectivity issues.

Analyzing stack traces helps you identify the line of code doing IO where threads are blocked.

So in summary, effectively utilizing thread dump analysis can help you diagnose and fix critical performance issues in WebSphere environments.

Conclusion

I hope this comprehensive guide gives you a firm grounding on how to generate and analyze WebSphere thread dumps for troubleshooting.

Here are some key takeaways:

Thread dumps give you invaluable data on thread states and call stacks to pinpoint issues
WebSphere provides easy ways to generate dumps – wsadmin, kill, console
Use free tools like IBM TDA, Samurai and FastThread to analyze dumps
Identify deadlocks, hung threads, excessive CPU usage etc. from analysis

Mastering thread dumps is a must-have skill for any WebSphere developer or administrator. Let me know if you have any other tips and tricks for thread dump analysis.