DAT Solutions Internal

Back
Mar 9, 2026 12:34 PM PDT

High latency/timeouts for crt-insurance-api

Updates

  • Mar 10, 2026 6:53 PM PDT

    Seeing a high number of 504s with the crt-insurance-api. At this point, not hearing of any customer concerns or issues reported in any of the customer facing channels (customer-impact, cms_support..) Script completed execution and a rolling restart was performed as well. Database connections and cluster cpu usage returned to normal levels.

    Resolved
  • Mar 9, 2026 2:19 PM PDT (1 day, 4 hours, and 34 minutes earlier)

    Seeing a high number of 504s with the crt-insurance-api. At this point, not hearing of any customer concerns or issues reported in any of the customer facing channels (customer-impact, cms_support..) Rolling restart initiated after traffic returned to normal levels from earlier. Database connections returned to normal levels as did CPU. Metrics around requests have returned to normal.

    Will watch to see if this occurs tomorrow, the spike in traffic we saw was unexpected.

    Monitoring
  • Mar 9, 2026 12:34 PM PDT (1 hour and 44 minutes earlier)

    Seeing a high number of 504s with the crt-insurance-api. At this point, not hearing of any customer concerns or issues reported in any of the customer facing channels (customer-impact, cms_support..)

    Investigating