DAT Solutions Internal

Back
Sep 30, 2026 11:47 AM PDT

Network Node capacity loss causing cascading pod scheduling failures and elevated 5xx errors

Updates

  • Sep 30, 2026 11:50 AM PDT

    Elevated 5xx errors on network-tagging-system via emissary (monitor network-tagging-system:emissary:500Errors flapped 7x through the day, peaking at 2,801 matched error events at 11:10 EDT), plus degraded scheduling/availability for office-service and company-service in network-prod. Mitigation achieved at 10:22 MDT on 2026-09-30, with no customer impact reported. The report does not specify the exact remediation steps or final root cause.

    Resolved
  • Sep 30, 2026 11:47 AM PDT (3 minutes earlier)

    Elevated 5xx errors on network-tagging-system via emissary (monitor network-tagging-system:emissary:500Errors flapped 7x through the day, peaking at 2,801 matched error events at 11:10 EDT), plus degraded scheduling/availability for office-service and company-service in network-prod. Mitigation isn’t apparent in the report; the listed actions are to capture current customer impact, schedule a Post-Incident Review, and ensure the PIR is complete. There is no indicated step or signal showing that the elevated 5xx errors or scheduling failures were stopped.

    Monitoring