DAT Solutions Internal

Updates

• Jun 26, 2026 12:27 PM PDT

  Customers are reporting that CarrierWatch / Keypoint is incorrectly flagging some carriers as having inactive authority and sending a large volume of automated notifications.

  Impact includes customer confusion, excessive CarrierWatch email traffic, and some dispatch-blocking reports.

  Current working theory from engineering is that a recent Motus-related change caused an updates-only file to be processed like a full dataset, which incorrectly triggered downstream logic and led to false status evaluations / notifications.

  This appears to be related to the broader Motus transition work, but it is not the same underlying issue as INC-1296. Recommendation is to track this as a separate incident and link INC-1296 as related context.

  Motus file handling issue causing false CarrierWatch inactive alerts and email flood Short answer: The incident was mitigated, but not fully resolved.

  What happened:
  - Status: mitigated (as of 2026-06-23 11:55 MDT)
  - Root cause (working theory): Motus-related changes caused an updates-only file to be processed like a full dataset, triggering false inactive authority assessments and mass emails.
  - Mitigation and fix steps:
  - Implemented a job-side mitigation to stop mass emails by running the job without the send-emails flag.
  - Deployed patches to address data corrections; Round 1 reduced unauthorized carriers from ~1,615,952 to ~1,156,191; Round 2 patches started 2026-06-25 to handle multiple authority types.
  - Customer communications remained active; status page updated.
  - Current state (as of the latest updates in the report): mass email flood stopped; partial data corrections underway; underlying data fix still incomplete (DOT 1223051 issue persists in prod; ~2,344 carriers with multiple authorities still affected). A broader fix to handle any number of authority types was in progress.
  - Related follow-up: INC-1309 for a Keypoint SOAP import issue spun off.

  In brief: Incident stopped the email flood and began data corrections via patches, but a complete resolution (full data correctness and handling of all authority-type scenarios) was ongoing and not yet finished. A PIR is slated to ensure full closure.

  Resolved
• Jun 23, 2026 11:11 AM PDT (3 days, 1 hour, and 16 minutes earlier)

  Customers are reporting that CarrierWatch / Keypoint is incorrectly flagging some carriers as having inactive authority and sending a large volume of automated notifications.

  Impact includes customer confusion, excessive CarrierWatch email traffic, and some dispatch-blocking reports.

  Current working theory from engineering is that a recent Motus-related change caused an updates-only file to be processed like a full dataset, which incorrectly triggered downstream logic and led to false status evaluations / notifications.

  This appears to be related to the broader Motus transition work, but it is not the same underlying issue as INC-1296. Recommendation is to track this as a separate incident and link INC-1296 as related context.

  Motus file handling issue causing false CarrierWatch inactive alerts and email flood Mitigation summary for this incident:

  • Job-side mitigation implemented: The affected job can run without the send-emails flag, which should prevent another wave of CarrierWatch emails once the authority-side fix is applied.
  • Patch timeline: A mitigation patch is being pushed this afternoon; the patch is not expected to take effect until the next scheduled job run tomorrow morning.
  • Email-flood control: CSB-2894 has been opened to put registry event emails behind a flag, enabling pause/controls on email floods going forward.
  • Observability/triage: Plan to enable Datadog Sendgrid integration to pull logs/metrics for better visibility during and after the patch.
  • Root cause direction: Working theory identifies Motus file handling as the cause (updates-only files being processed like full datasets, triggering downstream logic). Fix path targeted to prevent this mis-triggering.
  • Customer communications: Statuspage updates are active; continued customer/status communications underway.

  Next steps:
  - Deploy the mitigation patch this afternoon; verify its effect on the next job run tomorrow morning.
  - Confirm registry flag behavior and ensure no additional email floods occur.
  - Complete PIR-related tasks: schedule Post-Incident Review, capture current customer impact, update PIR document.
  - Validate logs/metrics after patch rollout to verify the fix and monitor for any reoccurrence.

  Monitoring