Loading

DIAS Ingestor Disk-Exhaustion CrashLoopBackOff — Remediation - PCE

Data pubblicazione: Sep 25, 2026
Descrizione

 This procedure documents the diagnosis and remediation steps for CrashLoopBackOff incidents on DIAS ingestor pods (dias-prov-k8s-am-ingestor-comp-0/1/2 in namespace dias) caused by disk exhaustion on the /var/lib/data persistent volume. The root cause is typically an unrotated logstash/logs/fallback directory accumulating stale metric-buffer files over months, which fills the PVC and prevents Logstash from opening its persistent queue on startup, triggering an indefinite restart loop. This runbook provides step-by-step commands to confirm the failure signature, identify the disk consumer, safely purge stale files, restart the affected pod(s), and check sibling replicas for the same condition — along with escalation criteria and a noted action item to automate log rotation going forward.

Risoluzione

Step 1 — Confirm the pod is crash-looping
kubectl get pod -n dias | grep ingestor-comp
kubectl describe pod dias-prov-k8s-am-ingestor-comp-<N> -n dias
Look for Warning BackOff events and a high restart count.

Step 2 — Pull last crash logs
kubectl logs dias-prov-k8s-am-ingestor-comp-<N> -n dias -c application --previous
Confirm the signature error:
ERROR org.logstash.execution.AbstractPipelineExt - Logstash failed to create queue.
java.io.IOException: Unable to allocate ... more bytes for persisted queue ...
If this exact error is not present, STOP — this procedure applies only to this disk-exhaustion signature. Escalate separately.

Step 3 — Check disk usage via the sidecar container
The application container is down, so use telegraf-ingestor (shares the same /var/lib/data volume):
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- df -h /var/lib/data
Usage above ~85–90% confirms the disk-exhaustion cause.

Step 4 — Identify the largest space consumer
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- sh -c "du -sh /var/lib/data/logstash/logs/* | sort -rh | head -20"
Expected finding: /var/lib/data/logstash/logs/fallback is disproportionately large (tens of GB).

Step 5 — Confirm the fallback files are stale
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- sh -c "ls /var/lib/data/logstash/logs/fallback | wc -l"
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- sh -c "ls /var/lib/data/logstash/logs/fallback | tail -5"
Confirm the most recent file is old relative to today's date — this indicates the directory is not actively consumed and is safe to prune.

Step 6 — Purge stale fallback files (30-day retention)
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- sh -c "find /var/lib/data/logstash/logs/fallback -name '.metric' -mtime +30 -delete"
Do NOT apply this same command to logstash-json-metrics-buffer-*.metric files without first checking timestamps (Step 4 output) — these may be actively written/consumed. If cleanup is needed there, use a longer window (60–90 days):
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- sh -c "find /var/lib/data/logstash/logs -name 'logstash-json-metrics-buffer-.metric' -mtime +60 -delete"

Step 7 — Verify space freed
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-<N> -c telegraf-ingestor -- df -h /var/lib/data
Target: usage back below ~80%.

Step 8 — Restart the pod
kubectl delete pod dias-prov-k8s-am-ingestor-comp-<N> -n dias
kubectl get pod dias-prov-k8s-am-ingestor-comp-<N> -n dias -w
Wait for READY to reach full count (e.g. 3/3) and STATUS = Running, with restart count not climbing.

Step 9 — Confirm clean startup
kubectl logs dias-prov-k8s-am-ingestor-comp-<N> -n dias -c application --tail=50
Confirm no Unable to allocate ... bytes for persisted queue error on this run.

Step 10 — Check sibling pods
This issue tends to affect multiple StatefulSet replicas on the same schedule, even if only one has crashed. Repeat Steps 3–7 (diagnosis at minimum) against all other replicas:
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-0 -c telegraf-ingestor -- df -h /var/lib/data
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-1 -c telegraf-ingestor -- df -h /var/lib/data
kubectl exec -n dias dias-prov-k8s-am-ingestor-comp-2 -c telegraf-ingestor -- df -h /var/lib/data
Apply Steps 6–8 to any pod at or above ~85%.

Numero articolo Knowledge

005460618

 
Caricamento
Salesforce Help | Article