DAT Solutions Internal

Back
Jun 11, 2026 9:21 AM PDT

Location updates not flowing from TT iPhone app to Backend API

Updates

  • Jun 15, 2026 9:15 PM PDT

    Location updates not flowing from TT iPhone app to Backend API. Error says - "The network connection was lost."
    There are other multiple different error msgs as well shown in datadog Resolution summary:

    • What happened: A 3-hour backlog occurred because auto-scaling failed to trigger, leaving ~3.2 million messages unprocessed in TT backend services/HTProg/Monorail queue. Clients were still sending location updates, but processing was halted.
    • How it was resolved: The queue was manually scaled up to 40+ instances, after which message processing resumed and the backlog began to decline.
    • Outcome: All data from the outage window is recoverable as the backlog is processed.
    • Time: Mitigated at 2026-06-11T09:05:00-06:00 MDT.
    • Remaining items: Root cause of the auto-scaling trigger failure to be determined; follow-ups include PIR planning and documenting the incident RCA, plus ensuring recovery steps and safeguards are in place.
    Resolved
  • Jun 11, 2026 9:21 AM PDT (4 days, 11 hours, and 53 minutes earlier)

    Location updates not flowing from TT iPhone app to Backend API. Error says - "The network connection was lost."
    There are other multiple different error msgs as well shown in datadog Mitigation summary:

    • Action taken: Manually scaled the TT backend queue processing to 40+ instances.
    • Effect: Backlog began processing again; the queue message count declined as messages were consumed, confirming backlog clearance.
    • Outcome: All data from the ~3-hour outage window will be recovered as the backlog is processed.

    Root cause and follow-ups:

    • Root cause identified in the timeline: auto-scaling did not trigger. The exact reason was not determined during the meeting and is a follow-up item.

    Recommended next steps:

    • Schedule and run a Post-Incident Review (PIR).
    • Update the PIR document and capture the customer impact.
    • Investigate and implement a fix for the auto-scaling trigger, plus safeguards (alerts, retries) to prevent recurrence.
    • Verify end-to-end data recovery once backlog processing completes.
    Monitoring