Loading

How to Correlate Thread Dumps and Heap Dumps to a Specific Kubernetes Pod Replica in Runtime Fabric (RTF)

Veröffentlichungsdatum: Aug 27, 2026
Voraussetzung(en)
* Eclipse Memory Analyzer Tool (MAT) installed * Target Heap Dump (.hprof or binary format) and Thread Dump (.txt) files * Access to the Kubernetes cluster and the rtfctl utility (for RTF report collection)
Aufgabe

Because Kubernetes replicas run identical workloads, verify which dump belongs to which Pod instance for accurate root cause analysis.

 

Step 1: Capture Thread Dumps & Heap Dumps using RTFCTL 

 

Thread Dump Script:

cat get-thread-dumps.sh
NAMESPACE="<app-name-space>"
APP_NAME="<RTF-app-name>"

# Fetch all pod names for this app in the namespace dynamically
PODS=$(kubectl get pods -n "$NAMESPACE" | grep -i "$APP_NAME" | awk '{print $1}')

for pod in $PODS; do
  echo "Capturing dump for $pod..."
  rtfctl threaddump "$APP_NAME" --pod "$pod" --app-namespace "$NAMESPACE" > "threaddump_${pod}.txt"
done

 

Script Output as follows:

Capturing dump for simple-app-6bfb55c884-nd7qj...
Capturing dump for simple-app-6bfb55c884-nlw4b...
Capturing dump for simple-app-6bfb55c884-sjj7f...

 

Heap Dump Script:

% cat get-heapdumps.sh
NAMESPACE="<app-name-space>"
APP_NAME="<RTF-Mule-app>"

# 1. Fetch only the pods that match the specific ReplicaSet hash
PODS=$(kubectl get pods -n "$NAMESPACE" | grep 'simple-app-6bfb55c884' | awk '{print $1}')

# 2. Loop through and capture the heap dump for each pod
for pod in $PODS; do
  echo "Capturing heap dump for $pod (this may take a minute)..."

  # Define a unique output file name for this specific pod
  HEAP_OUTPUT_FILE="heapdump_${pod}.hprof"

  # Execute the rtfctl command
  rtfctl heapdump "$APP_NAME" "$HEAP_OUTPUT_FILE" --pod "$pod" --app-namespace "$NAMESPACE"

  echo "Saved to $HEAP_OUTPUT_FILE"
done

 

Script Output as follows:

Capturing heap dump for simple-app-6bfb55c884-nd7qj (this may take a minute)...
Dumping heap for simple-app-6bfb55c884-nd7qj...
Heap dump written to heapdump_simple-app-6bfb55c884-nd7qj.hprof
Saved to heapdump_simple-app-6bfb55c884-nd7qj.hprof

Capturing heap dump for simple-app-6bfb55c884-nlw4b (this may take a minute)...
Dumping heap for simple-app-6bfb55c884-nlw4b...
Heap dump written to heapdump_simple-app-6bfb55c884-nlw4b.hprof
Saved to heapdump_simple-app-6bfb55c884-nlw4b.hprof

Capturing heap dump for simple-app-6bfb55c884-sjj7f (this may take a minute)...
Dumping heap for simple-app-6bfb55c884-sjj7f...
Heap dump written to heapdump_simple-app-6bfb55c884-sjj7f.hprof
Saved to heapdump_simple-app-6bfb55c884-sjj7f.hprof

 

Step 2: Map the Heap Dump to the Replica ID

  • Open Eclipse MAT.
  • Go to File -> Open Heap Dump... and select your .hprof or binary file.
  • Wait for MAT to parse the file. If a Getting Started wizard appears, click Cancel to go to the Overview tab.
  • On the Overview tab, click the OQL (Object Query Language) icon in the toolbar.
  • In the OQL editor, run this query to search memory strings for the Kubernetes Pod name pattern.
SELECT DISTINCT toString(s) FROM java.lang.String s WHERE (toString(s) LIKE ".*simple-app-[a-z0-9]{10}-[a-z0-9]{5}.*")

Note: Replace simple-app in the regex with the actual RTF application name.

  • Check the query results. The output shows the exact Replica ID string, for example simple-app-6bfb55c884-sjj7f. This identifies the Pod that produced the heap dump.

Step 3: Map the Thread Dump (.txt) to the Heap Dump

Standard text thread dumps do not include the Pod name. Use Hexadecimal Object Matching to confirm a .txt thread dump matches a specific heap dump.

  1. Open the .txt thread dump and find an active application thread, for example java.lang.Thread.State: WAITING.
    "GeoDb-Scheduled-1" #37 prio=5 os_prio=0 cpu=0.14ms elapsed=2122.17s tid=0x00007f2e76f8f560 nid=0x7e waiting on condition [0x00007f2e5d1fc000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@17.0.15/Native Method) - parking to wait for <0x00000000f7ac7e18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@17.0.15/LockSupport.java:341)
  2. Find a line where the thread is locking or waiting on an object and copy the hexadecimal address. Example: <0x00000000f7ac7e18>.
  3. In Eclipse MAT, open the OQL tool for the verified heap dump and query that exact memory address.
select * from objects 0x00000000f7ac7e18
  1. Validation. If MAT returns the matching object, for example java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject @ 0xf7ac7e18 , the thread dump and heap dump are a confirmed match for the same JVM instance.

Why Capture Both Thread Dumps and Heap Dumps

Thread Overview. A heap dump contains java.lang.Thread objects and their stack traces. MAT can show a thread overview from the heap dump.

 

  1. Locking and deadlock visibility. The .txt thread dump prints memory addresses of locks (for example - locked <0x...>) and shows which threads wait for them. The plain text format makes deadlocks and blocked threads faster to identify.
  2. Native OS context. The .txt file is generated by native JVM tools and includes low-level OS details that a heap dump may not show, such as the Native Thread ID (nid) and OS-level thread priorities.
  3. Performance impact. Taking a .txt thread dump is lightweight and takes milliseconds. Taking a heap dump requires a Stop-The-World pause and may freeze the application for several seconds or longer while memory is written to disk. Capture thread dumps first to check system health without impacting traffic.

Thank you for your attention.

Schritte

1. Collect the thread dump and heap dump for a specific replica.

  • Run rtfctl commands with the --pod flag and the application namespace.
  • Include the target pod replica ID so the dump is tied to that replica.

2. Collect dumps for every replica when required.

  • Iterate through all pods in the application namespace with kubectl or oc.
  • Run the heap dump command for each pod individually.

3. Validate correlation between thread dumps and heap dumps.

  • Open heap dumps in MAT and use OQL to locate replica-specific environment variables.
  • Match each thread dump to its heap dump by comparing hexadecimal memory addresses of locked objects.
Nummer des Knowledge-Artikels

005389195

 
Laden
Salesforce Help | Article