This procedure documents the diagnosis and remediation steps for CrashLoopBackOff incidents on DIAS ingestor pods (dias-prov-k8s-am-ingestor-comp-0/1/2 in namespace dias) caused by disk exhaustion on the /var/lib/data persistent volume. The root cause is typically an unrotated logstash/logs/fallback directory accumulating stale metric-buffer files over months, which fills the PVC and prevents Logstash from opening its persistent queue on startup, triggering an indefinite restart loop. This runbook provides step-by-step commands to confirm the failure signature, identify the disk consumer, safely purge stale files, restart the affected pod(s), and check sibling replicas for the same condition — along with escalation criteria and a noted action item to automate log rotation going forward.
Step 1 — Confirm the pod is crash-looping
kubectl get pod -n dias | grep ingestor-comp
kubectl describe pod dias-prov-k8s-am-ingestor-comp-<N> -n dias
Look for Warning BackOff events and a high restart count.
Step 2 — Pull last crash logs
kubectl logs dias-prov-k8s-am-ingestor-comp-<N> -n dias -c application --previous
Confirm the signature error:
ERROR org.logstash.execution.AbstractPipelineExt - Logstash failed to create queue.
java.io.IOException: Unable to allocate ... more bytes for persisted queue ...
If this exact error is not present, STOP — this procedure applies only to this disk-exhaustion signature. Escalate separately.
Step 3 — Check disk usage via the sidecar container
The application container is down, so use telegraf-ingestor (shares the same /var/lib/data volume):
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- df -h /var/lib/data
Usage above ~85–90% confirms the disk-exhaustion cause.
Step 4 — Identify the largest space consumer
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- sh -c "du -sh /var/lib/data/logstash/logs/* | sort -rh | head -20"
Expected finding: /var/lib/data/logstash/logs/fallback is disproportionately large (tens of GB).
Step 5 — Confirm the fallback files are stale
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- sh -c "ls /var/lib/data/logstash/logs/fallback | wc -l"
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- sh -c "ls /var/lib/data/logstash/logs/fallback | tail -5"
Confirm the most recent file is old relative to today's date — this indicates the directory is not actively consumed and is safe to prune.
Step 6 — Purge stale fallback files (30-day retention)
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- sh -c "find /var/lib/data/logstash/logs/fallback -name '.metric' -mtime +30 -delete"
Do NOT apply this same command to logstash-json-metrics-buffer-*.metric files without first checking timestamps (Step 4 output) — these may be actively written/consumed. If cleanup is needed there, use a longer window (60–90 days):
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- sh -c "find /var/lib/data/logstash/logs -name 'logstash-json-metrics-buffer-.metric' -mtime +60 -delete"
Step 7 — Verify space freed
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- df -h /var/lib/data
Target: usage back below ~80%.
Step 8 — Restart the pod
kubectl delete pod dias-prov-k8s-am-ingestor-comp-<N> -n dias
kubectl get pod dias-prov-k8s-am-ingestor-comp-<N> -n dias -w
Wait for READY to reach full count (e.g. 3/3) and STATUS = Running, with restart count not climbing.
Step 9 — Confirm clean startup
kubectl logs dias-prov-k8s-am-ingestor-comp-<N> -n dias -c application --tail=50
Confirm no Unable to allocate ... bytes for persisted queue error on this run.
Step 10 — Check sibling pods
This issue tends to affect multiple StatefulSet replicas on the same schedule, even if only one has crashed. Repeat Steps 3–7 (diagnosis at minimum) against all other replicas:
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-0 -c telegraf-ingestor -- df -h /var/lib/data
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-1 -c telegraf-ingestor -- df -h /var/lib/data
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-2 -c telegraf-ingestor -- df -h /var/lib/data
Apply Steps 6–8 to any pod at or above ~85%.
005460618

We use three kinds of cookies on our websites: required, functional, and advertising. You can choose whether functional and advertising cookies apply. Click on the different cookie categories to find out more about each category and to change the default settings.
Privacy Statement
Required cookies are necessary for basic website functionality. Some examples include: session cookies needed to transmit the website, authentication cookies, and security cookies.
Functional cookies enhance functions, performance, and services on the website. Some examples include: cookies used to analyze site traffic, cookies used for market research, and cookies used to display advertising that is not directed to a particular individual.
Advertising cookies track activity across websites in order to understand a viewer’s interests, and direct them specific marketing. Some examples include: cookies used for remarketing, or interest-based advertising.