DAT Solutions Internal

Updates

• Jul 1, 2026 9:40 AM PDT

  Reports from TQL that users can't use multi-shipments, able to replicate as well. Summary of resolution

  • What happened: A defect in the cargo feature flag service caused upstream connection errors and OOM issues, leading to failures for multi-shipments via D1W and internal DAT users. 429s and SDK queue growth contributed to degraded web/app experience.

  • How it was resolved:

    • Implemented a Breaking Glass workaround to reset feature flag timing and restart the feature flag service.
    • Merged and deployed fixes:
    • FES-1227 (feature flag timing reset)
    • Cargo PR 975 and gitops Freight PR 7504 to apply the fix in production
    • Deploy workflow:
    • Deploy to test, promote to staging, then promote to prod
    • Configuration fix:
    • Corrected flag configurations for bulk upload, addressing misconfig detected during rollout
    • Additional stability: PR 7518 to reduce update time for Google Sheets-related updates to improve responsiveness

  • Result: Incident mitigated at 2026-07-01 09:05 MDT. The changes stabilized the service, and the incident status was set to mitigated. Follow-ups were planned (PIR scheduling, PIR doc updates, and customer impact capture).

  • Next steps (post-incident):

    • Schedule and run the Post-Incident Review
    • Complete PIR documentation
    • Capture current customer impact and distribute findings
    • Monitor for any residual issues and confirm stability in prod

  Resolved
• Jul 1, 2026 9:14 AM PDT (25 minutes earlier)

  Reports from TQL that users can't use multi-shipments, able to replicate as well.

  Monitoring
• Jul 1, 2026 6:28 AM PDT (2 hours and 46 minutes earlier)

  Reports from TQL that users can't use multi-shipments, able to replicate as well. The feature flag timing was reset, which restarted the affected production service and restored the Multi-Shipments functionality, including the ability for brokers to utilize Workgroups.

  Monitoring