DAT Solutions Internal
High latency/timeouts for crt-insurance-api
Updates
-
Mar 10, 2026 6:53 PM PDT
Seeing a high number of 504s with the crt-insurance-api. At this point, not hearing of any customer concerns or issues reported in any of the customer facing channels (customer-impact, cms_support..) Script completed execution and a rolling restart was performed as well. Database connections and cluster cpu usage returned to normal levels.
Resolved -
Mar 9, 2026 2:19 PM PDT (1 day, 4 hours, and 34 minutes earlier)
Seeing a high number of 504s with the crt-insurance-api. At this point, not hearing of any customer concerns or issues reported in any of the customer facing channels (customer-impact, cms_support..) Rolling restart initiated after traffic returned to normal levels from earlier. Database connections returned to normal levels as did CPU. Metrics around requests have returned to normal.
Will watch to see if this occurs tomorrow, the spike in traffic we saw was unexpected.
Monitoring -
Mar 9, 2026 12:34 PM PDT (1 hour and 44 minutes earlier)
Seeing a high number of 504s with the crt-insurance-api. At this point, not hearing of any customer concerns or issues reported in any of the customer facing channels (customer-impact, cms_support..)
Investigating