Because Kubernetes replicas run identical workloads, verify which dump belongs to which Pod instance for accurate root cause analysis.
cat get-thread-dumps.sh
NAMESPACE="<app-name-space>"
APP_NAME="<RTF-app-name>"
# Fetch all pod names for this app in the namespace dynamically
PODS=$(kubectl get pods -n "$NAMESPACE" | grep -i "$APP_NAME" | awk '{print $1}')
for pod in $PODS; do
echo "Capturing dump for $pod..."
rtfctl threaddump "$APP_NAME" --pod "$pod" --app-namespace "$NAMESPACE" > "threaddump_${pod}.txt"
done
Script Output as follows:
Capturing dump for simple-app-6bfb55c884-nd7qj...
Capturing dump for simple-app-6bfb55c884-nlw4b...
Capturing dump for simple-app-6bfb55c884-sjj7f...
% cat get-heapdumps.sh
NAMESPACE="<app-name-space>"
APP_NAME="<RTF-Mule-app>"
# 1. Fetch only the pods that match the specific ReplicaSet hash
PODS=$(kubectl get pods -n "$NAMESPACE" | grep 'simple-app-6bfb55c884' | awk '{print $1}')
# 2. Loop through and capture the heap dump for each pod
for pod in $PODS; do
echo "Capturing heap dump for $pod (this may take a minute)..."
# Define a unique output file name for this specific pod
HEAP_OUTPUT_FILE="heapdump_${pod}.hprof"
# Execute the rtfctl command
rtfctl heapdump "$APP_NAME" "$HEAP_OUTPUT_FILE" --pod "$pod" --app-namespace "$NAMESPACE"
echo "Saved to $HEAP_OUTPUT_FILE"
done
Script Output as follows:
Capturing heap dump for simple-app-6bfb55c884-nd7qj (this may take a minute)...
Dumping heap for simple-app-6bfb55c884-nd7qj...
Heap dump written to heapdump_simple-app-6bfb55c884-nd7qj.hprof
Saved to heapdump_simple-app-6bfb55c884-nd7qj.hprof
Capturing heap dump for simple-app-6bfb55c884-nlw4b (this may take a minute)...
Dumping heap for simple-app-6bfb55c884-nlw4b...
Heap dump written to heapdump_simple-app-6bfb55c884-nlw4b.hprof
Saved to heapdump_simple-app-6bfb55c884-nlw4b.hprof
Capturing heap dump for simple-app-6bfb55c884-sjj7f (this may take a minute)...
Dumping heap for simple-app-6bfb55c884-sjj7f...
Heap dump written to heapdump_simple-app-6bfb55c884-sjj7f.hprof
Saved to heapdump_simple-app-6bfb55c884-sjj7f.hprof
.hprof or binary file.SELECT DISTINCT toString(s) FROM java.lang.String s WHERE (toString(s) LIKE ".*simple-app-[a-z0-9]{10}-[a-z0-9]{5}.*")Note: Replace simple-app in the regex with the actual RTF application name.
simple-app-6bfb55c884-sjj7f. This identifies the Pod that produced the heap dump.Standard text thread dumps do not include the Pod name. Use Hexadecimal Object Matching to confirm a .txt thread dump matches a specific heap dump.
.txt
thread dump and find an active application thread, for example java.lang.Thread.State: WAITING.
"GeoDb-Scheduled-1" #37 prio=5 os_prio=0 cpu=0.14ms elapsed=2122.17s tid=0x00007f2e76f8f560 nid=0x7e waiting on condition [0x00007f2e5d1fc000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.15/Native Method)
- parking to wait for <0x00000000f7ac7e18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@17.0.15/LockSupport.java:341)
select * from objects 0x00000000f7ac7e18
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject @ 0xf7ac7e18
, the thread dump and heap dump are a confirmed match for the same JVM instance.Thread Overview. A heap dump contains java.lang.Thread objects and their stack traces. MAT can show a thread overview from the heap dump.
- locked <0x...>) and shows which threads wait for them. The plain text format makes deadlocks and blocked threads faster to identify.Thank you for your attention.
1. Collect the thread dump and heap dump for a specific replica.
--pod flag and the application namespace.2. Collect dumps for every replica when required.
kubectl or oc.3. Validate correlation between thread dumps and heap dumps.
005389195

We use three kinds of cookies on our websites: required, functional, and advertising. You can choose whether functional and advertising cookies apply. Click on the different cookie categories to find out more about each category and to change the default settings.
Privacy Statement
Required cookies are necessary for basic website functionality. Some examples include: session cookies needed to transmit the website, authentication cookies, and security cookies.
Functional cookies enhance functions, performance, and services on the website. Some examples include: cookies used to analyze site traffic, cookies used for market research, and cookies used to display advertising that is not directed to a particular individual.
Advertising cookies track activity across websites in order to understand a viewer’s interests, and direct them specific marketing. Some examples include: cookies used for remarketing, or interest-based advertising.