Take Heap Dump of an Application in RTF

QUESTION

How to take heap dump from a running application in RTF?

ANSWER

1. Use rtfctl

• Install rtfctl or upgrade rtfctl to the latest version

$ sudo ./rtfctl update
Current:   v1.0.113
Latest:    v1.0.118
Updating rtfctl for darwin
Downloading v1.0.118...
Still downloading https://anypoint.mulesoft.com/runtimefabric/api/download/rtfctl-darwin/latest (5 seconds elapsed)
Replaced /usr/local/bin/rtfctl.

• Run rtfctl to generate a heap dump

# Check the help 
$ ./rtfctl heapdump -h
Trigger a JVM heap dump

Usage:
  rtfctl heapdump APP_NAME OUTPUT [flags]

Examples:
  # Take a JVM heap dump for application my-app, saving it as /tmp/dump.hprof, using the first replica by default
  rtfctl heapdump <app_name> <destination_path> --app-namespace <app_namespace> --namespace <rtf_namespace>

  # Take a JVM heap dump for application my-app, saving it as /tmp/dump.hprof, using replica abc in environment env
  rtfctl heapdump <app_name> <destination_path> --pod <replica_name> --app-namespace <app_namespace> --namespace <rtf_namespace>


Flags:
      --app-namespace string   Environment to fetch heapdump for applications in
  -h, --help                   help for heapdump

Global Flags:
      --as string              Username to impersonate for the operation. User could be a regular user or a service account in a namespace.
      --config-file string     Configuration file
  -e, --control-plane string   Set the anypoint control plane environment (default "auto")
      --debug                  Increase the logging verbosity
      --host string            Set anypoint control plane (default "anypoint.mulesoft.com")
  -n, --namespace string       System namespace scope (default "rtf")
      --pod string             If present, the name of a pod

# generate a heap dump 
$ sudo ./rtfctl heapdump hellopsa-new /tmp/dump.hprof --app-namespace xxxxxx-xxxx-xxxxx-xxxx
Dumping heap for hellopsa-new-5fb6f475d5-xgh4r...
Heap dump written to /tmp/dump.hprof

$ ls /tmp/
dump.hprof

2. Take an OOH (Out of Heap) event generated heap dump

If an app container is crashed due to OOH, a heap dump will be generated in the /tmp inside the pods, referring to How to check if an application is restarted due to memory issues in RTF (Runtime Fabric)

• Use kubectl command to copy the dump

$ kubectl exec -it -n 2abc2ce6-cce6-463b-afaa-01234567890abcd demoapp1-6c949cdcf7-cbvm2 -c app -- /bin/bash
app@demoapp1-6c949cdcf7-cbvm2:/tmp$ ls -al
total 148
drwxrwxrwx. 6 root root    253 Feb  8 03:12 .
drwxr-xr-x. 1 root root     43 Feb  8 03:12 ..
-rw-r--r--. 1 app  app       0 Feb  7 19:32 MIME5463303917572216231.tmp
-rw-r--r--. 1 app  app       0 Feb  8 03:12 MIME7136087841279184546.tmp
drwxr-xr-x. 2 app  app      45 Feb  8 03:12 heap_dumps
-rw-r--r--. 1 app  app  147593 Feb  8 03:12 hs_err_pid_50.log
drwxr-xr-x. 2 app  app      16 Feb  8 03:12 hsperfdata_app
drwxr-xr-x. 2 app  app       6 Feb  7 19:32 jetty-exec-_-any-4780254465244538951.dir
drwxr-xr-x. 2 app  app       6 Feb  8 03:12 jetty-exec-_-any-860175366609163137.dir
-rw-r--r--. 1 app  app       0 Feb  8 03:20 readiness

# copy the heap out 
$ kubectl cp  -n 2abc2ce6-cce6-463b-afaa-a1979e3ffd29 -c app demoapp1-6c949cdcf7-cbvm2:/tmp/heap_dumps /tmp/heap_dumps
$ ls /tmp/heap_dumps/
java_dump.hprof.20200208-031218

• Use rtfctl to generate an app package

$ sudo ./rtfctl package -h

Create a debugging information package for an application.

The package includes: thread dump, open file handles, open network connections.
Optional: heap dump, apps, policies, .mule directories

Usage:
  rtfctl package APP_NAME PACKAGE_DIRECTORY [flags]

Examples:
  # Create a package for application my-app, using the first pod by default
  rtfctl package <app_name> <application_package_filename> --app-namespace <app_namespace> --namespace <rtf_namespace>

  # Create a package for application my-app, containing the apps, policies, and .mule directories
  rtfctl package <app_name> <application_package_filename> --apps --policies --dotmule --app-namespace <app_namespace> --namespace <rtf_namespace>

  # Create a package for application my-app in environment env, containing a heap dump
  rtfctl package <app_name> <application_package_filename> --heap-dump --app-namespace <app_namespace> --namespace <rtf_namespace>


Flags:
      --app-namespace string   Environment to query packages for applications in
      --apps                   Include apps directory (default true)
      --conf                   Include conf directory (default true)
      --dotmule                Include .mule directory (default true)
      --heap-dump              Include heap dump
  -h, --help                   help for package
      --policies               Include policies directory (default true)
      --system-properties      Include JVM system properties

Global Flags:
      --as string              Username to impersonate for the operation. User could be a regular user or a service account in a namespace.
      --config-file string     Configuration file
  -e, --control-plane string   Set the anypoint control plane environment (default "auto")
      --debug                  Increase the logging verbosity
      --host string            Set anypoint control plane (default "anypoint.mulesoft.com")
  -n, --namespace string       System namespace scope (default "rtf")
      --pod string             If present, the name of a pod

$ sudo ./rtfctl package hellopsa-new --heap-dump --app-namespace xxxxxx-xxxx-xxxxx-xxxx
Creating package for replica hellopsa-new-5fb6f475d5-xgh4r...
 Fetching files...
 Dumping threads...
 Inspecting open file handles...
 Listing thread metrics...
 Reporting network connections...
 Dumping heap...
 Package written to /tmp/hellopsa-new-5fb6f475d5-xgh4r-package.tar
Done

$ tar -xf /tmp/hellopsa-new-5fb6f475d5-xgh4r-package.tar
$ ls
heap_dump  heap_dump.hprof  hellopsa-new-5fb6f475d5-xgh4r-package.tar  lsof  netstat  opt  ps-elo  thread_dump  tmp
$ ls tmp/heap_dumps/
java_manual.hprof

ADDITIONAL INFORMATION

Please refer to How to check if an application is restarted due to memory issues in RTF (Runtime Fabric), OOH and OOM are two different events. Only OOH generates a heap dump when crashing. For OOM, you may also need to check the native memory How to Check Application Native Memory Usage in RTF