DAT Solutions Internal

Back
Mar 7, 2022 7:00 AM PST

INC-230 Visible slowness in Prod Freight cluster

Updates

  • Mar 8, 2022 9:51 AM PST

    This incident has been resolved.

    Resolved
  • Mar 7, 2022 12:25 PM PST (21 hours and 25 minutes earlier)

    After new instance of Ambassador were deployed onto new nodes, thing have improve. Ambassador CPU usage, still showing some fluctuations, but overall we are not spiking up to the max we allocate for the nodes.

    Monitoring
  • Mar 7, 2022 12:09 PM PST (15 minutes earlier)

    CPE team is looking into increasing replica count in the freight cluster

    Identified
  • Mar 7, 2022 8:48 AM PST (3 hours and 21 minutes earlier)

    Looks like we back to working state again. The cycle of the nodes got us back into stable state with ambassador PODS.
    This was affecting multiple products - TAPI (LB4T, TE), Mobile BFF (DAT One Mobile)
    We are currently monitoring them.

    Monitoring
  • Mar 7, 2022 8:34 AM PST (13 minutes earlier)

    The issue has been identified and a fix is being implemented.

    Identified
  • Mar 7, 2022 7:00 AM PST (1 hour and 34 minutes earlier)

    INC-230

    Investigating