JVM received a signal SIGKILL (9)

SYMPTOM

Your Mule core runtime got restarted and you read the following lines in your mule_ee.log file:

JVM appears hung: Timed out waiting for signal from JVM. Requesting thread dump.
Dumping JVM state.
JVM appears hung: Timed out waiting for signal from JVM. Restarting JVM.
JVM received a signal SIGKILL (9).

CAUSE

The "sigkill (9)" signal, a.k.a "kill -9" is an exit signal sent by some other layer in your environment.
There are several known scenarios that could be triggering it:

• The operating system (OS) could kill the runtime in case it's consuming too many resource
• The Tanuki Wrapper is in charge of monitoring the runtime, and could kill it if the runtime is not responding
• Some third-party tools or external monitoring agents could kill the process on demand
• A user with the necessary security rights could trigger the kill command from within the server 
• The operating system (OS) services manager killing the Mule service due to timeouts

SOLUTION

• For the Operating System, please read the following article
  • How to Detect When Linux Kills Mule Runtime Process Due to Native Memory OOMKiller Error
• For the Tanuki Wrapper, please read the following articles:
  • Adjusting the timeout behaviour of the Mule Service wrapper to prevent unexpected restarts
  • How to prevent a Mule Runtime restart when the JVM becomes unresponsive
  • Mule Runtime's JVM Restarts Every 10 Minutes
• For any other third-party tool or monitoring agent, please contact the third party tool/agent Support team
  • If you don't know if you have some, usually those tools/agents are configured in the $MULE_HOME/conf/wrapper.conf file, or they are readable from the wrapper-additional.conf file at the same folder (that last file is read only and created upon each execution of the runtime)
• To check if any user triggered the kill command, please engage your SysAdmin team

• To confirm if the issue is the service or not please run the mule runtime as "$MULE_HOME/bin/mule" without the "start" to run it in standalone mode. If it works, then the issue is running it as a service, you should ping your SysAdmin to verify your service configuration in case it has a default timeout. If running the mule in standalone mode also fails, then is not a service issue.
  • For Red Hat, please check: Mule Service Fails to Start with systemd: Failed to start MuleSoft Mule ESB Application Container

ADDITIONAL INFORMATION

• Most programs will try to exit the runtime first, and will send the sigkill command after the first one not responding. So instead of "JVM received a signal SIGKILL (9)", you may get the following one:

  JVM exited after being requested to terminate.

• It's normal to read that a thread dump was requested, but taking the dump will depend on your JVM health, so if your runtime was unresponsive, it might not be possible to get a thread dump at all.

  JVM appears hung: Timed out waiting for signal from JVM. Requesting thread dump. 

  If the Java JDK is able to take a thread dump from the unresponsive Mule runtime, it will be appended to the mule_ee.log file.