DAT Solutions Internal

Back
Nov 3, 2025 6:27 AM PST

OTEL Collector deployment in CrashLoopBackOff

Updates

  • Nov 3, 2025 11:09 AM PST

    Freight posting service and Freight truck posting service triggered alarms for Golden Signals Throughput being low Customer impact: No external customer were impacted
    At around 7:12 AM CDT, several teams reported that metrics were not appearing in Datadog.
    The MOB team created an incident and invited SRE to investigate.
    Troubleshooting summary:
    Cole and I began reviewing the issue.
    We observed that all OTEL Collector pods were in CrashLoopBackOff, initially due to CPU pressure.
    After restarting the pods, they immediately began failing again — this time due to OOM (Out of Memory) errors.
    We increased vertical resources (instead of horizontal scaling, since the load wasn’t distributed evenly across pods):
    Memory: 4Gi → 8Gi
    CPU: 3 cores → 4 cores
    After the resource adjustment, the pods stabilized and metrics began flowing normally again.
    Resolution: Issue has been fully resolved.
    PR applied: https://github.com/dat-freight-and-analytics/gitops-platform/pull/514
    Root cause (still under investigation):
    Autoscaling for the OTEL Collector did not trigger as expected, The workload was not distributed across pods. We’re still investigating why the autoscaler did not initiate.
    We will continue to dig into the autoscaling behavior and provide updates once we identify the exact cause

    Resolved
  • Nov 3, 2025 7:16 AM PST (3 hours and 52 minutes earlier)

    Freight posting service and Freight truck posting service triggered alarms for Golden Signals Throughput being low The incident was mitigated by increasing the resources allocated to the OTEL Collector, which resolved the CPU overload and eliminated the OOMKilled status, allowing the deployment to run as expected. Monitoring of the deployment was initiated to ensure stability before officially updating the incident.

    Monitoring
  • Nov 3, 2025 6:27 AM PST (49 minutes earlier)

    Freight posting service and Freight truck posting service triggered alarms for Golden Signals Throughput being low

    Investigating