1. First, get the PID of your Mule runtime

The easiest method to obtain the process ID in the operating system is to use the jps tool provided in the JDK used to run Mule. The line that contains the MuleContainerBootstrap is the one that indicates the PID. Will refer to this value as MULE_PID later.

Example (MULE_PID in bold):

$ jps
31077 Jps
30636 MuleContainerBootstrap

Alternatives ways to find the PID are using the operating system tools. For example the ps command in Linux/Mac/Unix and in Windows the Task Manager's Processes tab.

2. Obtaining a thread dump

A thread dump is a text snapshot of the stack for every thread present in the JVM. It includes information on the exact execution point of each thread and the locks (if any) on which these threads are waiting. There are several methods to capture a Java thread dump. We'll mention the more commonly used.

2.1 Using the JDK jstack utility

Any modern version of Oracle JDK or OpenJDK provides a utility called jstack which can take a thread dump of a running Java application as the Mule Runtime/API Gateway.

usage:

jstack -l MULE_PID > filename.tdump

Note: older Java versions might not provide all the necessary information with their jstack utilities. In such a case it is recommended to upgrade the Java version. If not possible please see one of the other methods mentioned below.

The jstack utility is only provided with Java JDK. A Java JRE doesn't contain this or other tools needed for troubleshooting.

2.2 On Unix-like operating system (including Linux and Mac OS X)

JVMs provide a thread dump on the standard output when they receive the QUIT signal. This will not cause the Mule runtime to stop. Use the following command to send the signal: kill -quit MULE_PID. This thread dump will be sent to standard output if the JVM is running in an interactive console, or to Mule's log (mule_ee.log) file if it is running as a background process (using mule start to start the server)

To create a thread dump, press CTRL-\ when running Mule in a console, or use mule dump when running Mule as a service.

When Mule runs as a service, the Java Service Wrapper logs standard output from the JVM through its own Log4J service, with the INFO logging level. If you have disabled INFO logging in your wrapper's logger configuration, you will not see the output in your log files.

To solve this problem make sure that your wrapper.conf's logging section has the following values set:

wrapper.console.loglevel=INFO
wrapper.logfile=%MULE_EXE%/../logs/mule.log
wrapper.logfile.format=M
wrapper.logfile.loglevel=INFO

2.3 On Windows operating systems

When running Mule in a console press CTRL-Break to create a thread dump.

When running as a Windows service then some previous configuration is required. Follow this procedure to enable thread dumps:

1. stop the Mule Runtime service (from Windows Services window or using command line mule stop)
2. uninstall the Mule Runtime service (command: mule remove)
3. add the following line to the configuration in wrapper.conf: wrapper.thread_dump_control_code=255
4. install the Mule Runtime service (command: mule install)
5. start the Mule Runtime service (from Windows Services window or using command line mule start)

After this procedure is completed you can create a thread dump sending the control code 255 to the service with the following Windows command:

c:\> sc control mule_ee 255

The thread dump will be sent to the mule_ee.log.

2.4 Obtaining a thread dump when executing Mule Runtime as a service

Usually, services execute under a special user account on the operating system for security reasons. This requires a different method to execute the commands for each operating system.

2.4.1 Windows:

Use Microsoft psexec utility (https://technet.microsoft.com/en-us/sysinternals/bb897553.aspx) and execute the Java SDK jstack utility. Specify the full path to jstack. Define an appropriate output filename to write the heap dump.

psexec -s %JAVA_HOME%\bin\jstack.exe -l MULE_PID > filename.tdump

Example:

psexec -s %JAVA_HOME%\bin\jstack.exe -l 12345 > \mydir\myapp-20160509-235959.tdump

2.4.2 Linux/Mac/Unix:

sudo -u USERNAME $JAVA_HOME/bin/jstack.exe -l MULE_PID > filename.tdump

where USERNAME is the operating system user that is used to execute the service. This can be determined with the ps command.

Example:

sudo -u johndoe $JAVA_HOME/bin/jstack -l 12345 > /home/john/myapp-20160509-235959.tdump

2.5 Using kill -3

In some scenarios you are not able to use any of the previous methods. In this case, you can use kill -3 like this:
 

kill -3 MULE_PID

Beware that the output of the thread dump will be sent to mule_ee.log

3. Obtaining a heap dump

A heap dump is a snapshot of a JVM's heap space, complete with all object references. This is a binary file (often very large) and needs special tools to be analyzed.

Note: Generating a heap dump may pause and stop the processing of the JVM while it is being generated which could take some minutes for large heap sizes. The JVM also might get restarted if the JVM pause is more than "wrapper.ping.timeout" property from MULE_HOME/conf/wrapper.conf file.

To obtain a heap dump of an existing JVM process on Sun / Oracle JVMs use the jmap tool bundled with all VMs. The syntax is simple:

$ jmap -dump:format=b,file=snapshot.hprof MULE_PID

If you require to take a heapdump on an out-of-memory error (OOM), please refer to this article.

Some versions of the JDK don't bring the jmap command anymore, but they contain the jcmd which is equivalent:

$  jcmd MULE_PID GC.heap_dump filename=heapdump.hprof

3.1 IBM JVM

To obtain a heap dump on IBM's Java implementations, you will need to add some Java arguments first before starting the Mule Runtime.

Example:

# disable default IBM configuration for dumps
wrapper.java.additional.15=-Xdump:none
# configure dumps
wrapper.java.additional.16=-Xdump:java+heap+system:events=user,request=exclusive+prepwalk+compact
# log current options at startup to verify it is correctly configured
wrapper.java.additional.17=-Xdump:what

Note that the numbers used in the example might need to be modified to non-used indices in your wrapper.conf file.

and then, to get the dump by sending a QUIT signal to the JVM:

$ kill -QUIT MULE_PID

The JVM will generate several file named "core.xxxxxxxxx.dmp", "heapdump.xxxxxxxxx.phd" and "javacore.xxxxxxxxx.txt", where "xxxxxxxxx" is a set of digits specifying a timestamp, and the process number of the JVM from which the snapshot was taken.

If you contact MuleSoft Support to ask for assistance, please mention that you took these snapshots. They will be very valuable in finding the cause of your problem.

3.2 Obtaining a heap dump when executing Mule Runtime as a service

Usually, services execute under a special user account on the operating system for security reasons. This requires a different method to execute the commands for each operating system.

3.2.1 Windows:

Use Microsoft psexec utility (https://technet.microsoft.com/en-us/sysinternals/bb897553.aspx) and execute the Java SDK jmap utility. Specify the full path to jmap. Define an appropriate output filename to write the heap dump.

psexec -s %JAVA_HOME%\bin\jmap.exe -dump:format=b,file=filename MULE_PID

Example:

psexec -s c:\Program Files\Java\jdk1.8.0_91\bin\jmap.exe -dump:format=b,file=myapp.hprof 12345

3.2.2 Linux:

sudo -u USERNAME $JAVA_HOME/bin/jmap -dump:format=b,file=filename MULE_PID

where USERNAME is the operating system user that is used to execute the service. This can be determined with the ps command.

Example

sudo -u johndoe $JAVA_HOME/bin/jmap -dump:format=b,file=myapp.hprof 12345

Note: if the environment variable JAVA_HOME is not defined you'll need to replace it with the absolute path to the correct version of the Java SDK installation directory.

3.3 Obtaining a heap dump via JMX and JConsole

If the above options are not possible or present issues, you can alternatively try to take a heap dump via JMX. JMX allows for monitoring and managing and applications from afar. Either via the local machine or accessing remotely, you can access the Java runtime via JMX and control it with JConsole to take the heap dump. If JMX is currently disabled, one drawback is that the Mule Runtime must be restarted before JMX can be enabled. Also, for remote connection, the remote machine must have network connectivity with the Mule Runtime and the port specified for JMX.

Here are instructions on how to enable JMX:
1) Via Mule runtime's wrapper.conf: How to enable JMX using wrapper parameters
2) Official documentation of enabling JMX from Oracle: https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html

Then, you can manage the Java runtime from JConsole and take a heap dump. Here are instructions under the header HotSpot Diagnostic MXBean:
https://docs.oracle.com/javase/8/docs/technotes/guides/management/jconsole.html

3.4 Obtaining a heap if "Premature EOF" exception is thrown

It was observed, sometimes, when heap size allocated is too large, e.g. 10GiB, the heap dump generation may throw the following exception: 

-bash-4.1$ jmap -dump:format=b,file=/tmp/runtime_heap.hprof 17790
Dumping heap to /tmp/runtime_heap.hprof ...
Exception in thread "main" java.io.IOException: Premature EOF
at sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:248)
at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:199)
at sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:217)
at sun.tools.attach.HotSpotVirtualMachine.dumpHeap(HotSpotVirtualMachine.java:180)
at sun.tools.jmap.JMap.dump(JMap.java:242)
at sun.tools.jmap.JMap.main(JMap.java:140)

In this case, a forced garbage collection before heap dump generation may help.

4. Preserve logs

Mule logs all exceptions to the various global and per-application log files, so if anything goes wrong, it will be there together with a timestamp and information on what Mule was doing at the time of the event.
Logs could rotate and be overwritten. It is best to preserve the whole logs files at the time of the incident. To reduce space usage logs can be compressed with zip. Usually, this results in a good compression ratio.

5. Obtaining extra info from operation system

6. Run a fine-grain JVM-level performance profiling

This is to detect performance hotspots by using Java Flight Recorder on Hotspot JVM either on Windows or Linux operating systems.

When using HotSpot JVM (from Oracle Vendor),  taking a flight recording is extremely helpful when trying to identify the root cause for a performance issue.  The recording itself will record a lot of vital run-time information from the JVM by instrumenting it. 

The steps required to take a flight recording are different depending on whether you are running JVM version 7 or 8.

JDK7

Prior to taking the flight recording, you must ensure the following flags are present on the JVM command line:

-XX:+UnlockCommercialFeatures
-XX:+FlightRecorder

If those options are not set, you will not be able to start a flight recording on the target JVM.  In this case, we recommend following the method given in step 7 which does not require any JVM utility.

JDK8

Issue the following command on Linux:

$JAVA_HOME/bin/jcmd <pid of mule JVM> VM.unlock_commercial_features

Or its counterpart on Windows:

%JAVA_HOME%\bin\jcmd <pid of mule JVM> VM.unlock_commercial_features

This command will enable flight recording on the target JVM.

To start the flight recording, on either JDK7 or JDK8, run the following command on Linux:

$JAVA_HOME/bin/jcmd <pid of mule JVM> JFR.start duration=<time in seconds>s settings=profile filename=<target directory>/recording.jfr

Or its counterpart on Windows:

%JAVA_HOME%\bin\jcmd <pid of mule JVM> JFR.start duration=<time in seconds>s settings=profile filename=<target directory>\recording.jfr

Set <target directory> to a folder belonging to a filesystem with at least 10Gb of free space;  duration specifies the duration of the recording in seconds.  It's recommended to record at least 30 minutes of activity (1800s) as a default value to capture a considerable amount of JVM events.  The size of the recording file being generated depends on both the current JVM load and the profile being used.   The CPU overhead when running the flight recording is usually less than 2%, so it is not harmful in terms of performance detriment.
 
It's recommended to have the recording file to be compressed and attached to the case for further analysis.

A JFR recording has a lot of useful information which will allow our engineers to easily identify any CPU / memory / I-O  which could be causing the performance hit.  It gives a deeper view of JVM internals that usually cannot be seen when using any other toolset.  So we largely recommend to take a flight recording when under a performance degradation scenario.

You can complement this information with the given on step 7.

7. Getting the most CPU consuming Java methods by using the TTOP utility

As an alternative to method 6, you can get the hottest CPU methods (those which are spending the most CPU cycles ), by using the TTOP utility.  This utility will allow you to sample the hottest threads and record them on a file, by using a sampling algorithm largely known as the "poor man's" algorithm. The information given by ttop is very limited in comparison with a flight recording because a flight recording comprehends different JVM aspects, not only CPU usage, but when identifying CPU hotspots, it's efficient and detailed.

You can find instructions on how to run this tool and how to download the script on the following KB:
How to identify top CPU consumer threads and hotspots effectively using the TTOP utility on Linux

8. Glossary

• PID: Process ID. It is the identifier of an operating system process. In Unix-like operating systems it can be found with the ps command. In Windows it can be find through the Task Manager.

• MULE_PID: Process ID of the Java process used to execute the Mule runtime process.

• JFR:  Acronym for Java Flight Recorder