DAT Solutions Internal

Updates

• Aug 3, 2026 5:41 AM PDT

  TIM became unusable for all reported agents. Users cannot save notes or other changes, including work on OCR exceptions. Insurance certificates processed successfully by OCR are not appearing in TIM and are consequently not being published to CarrierWatch.

  Customer-impacting examples include:

  Dayas Freight Carrier LLC — DOT 2436684

  Rodolfo Jasso Jr. — DOT 2794852

  OCR returned success for both certificates, but neither appeared in TIM or CarrierWatch. This is blocking insurance-certificate processing and publication for Legacy CarrierWatch customers on month-end. Resolution summary:

  • The TIM/SQL failure was mitigated and end-to-end certificate processing was restored for OCR → TIM → CarrierWatch.
  • A backlog of approximately 64 affected OCR batches was identified. The plan is to re-submit these via the DICE hot-folder replay process and reprocess in a controlled manner.
  • Manual replay tests through the DICE hot folder succeeded, confirming the recovery path.

  What happened (highlights)-

  • Root causes identified:
    • OCR processing server (bvtocr03.tcore.com) disk full (149 GB C: drive).
    • SQL log drive (E:) nearly exhausted; backups hadn’t run since 06/18, contributing to disk pressure.
  • Immediate actions:
    • Freeed disk space and restarted the OCR server (07:09 PT).
    • Restarted the SQL Server Agent on philwcrbsql04 and started backups; expanded E: drive by 50 GB.
  • Recovery steps:
    • New certificate processing validated end-to-end (test certificate published).
    • Previously processed certificates that failed did not automatically replay; initiated manual replay via DICE hot-folder.
    • A manual replay test succeeded through OCR, TIM, and CarrierWatch.
  • services have been restored and stable.

  Resolved
• Jul 31, 2026 9:00 AM PDT (2 days, 20 hours, and 40 minutes earlier)

  TIM became unusable for all reported agents. Users cannot save notes or other changes, including work on OCR exceptions. Insurance certificates processed successfully by OCR are not appearing in TIM and are consequently not being published to CarrierWatch.

  Customer-impacting examples include:

  Dayas Freight Carrier LLC — DOT 2436684

  Rodolfo Jasso Jr. — DOT 2794852

  OCR returned success for both certificates, but neither appeared in TIM or CarrierWatch. This is blocking insurance-certificate processing and publication for Legacy CarrierWatch customers on month-end. mitigation summary

  • Immediate resource remediation:

    • OCR processing server (bvtocr03.tcore.com): freed disk space by adding capacity and restarted around 07:09 PT.
    • SQL server (philwcrbsql04): restarted SQL Agent; backups resumed; expanded E: drive by 50 GB to prevent log-fill.

  • Issue status during mitigation:

    • TIM save operations and OCR-to-TIM publication to CarrierWatch were still failing after the restart; HTTP 500s observed on /Carrier/NewNote. Root cause under investigation (TIM application vs SQL/log capacity).

  • Validation and backlogged work:

    • End-to-end certificate processing validated for new certificates: a test certificate published to TIM and CarrierWatch successfully.
    • Backlog handling: began manual reprocessing via the hot folder; first test certificate picked up by OCR and monitored downstream.

  • Current recovery stance:

    • New certificate processing works end-to-end for new items.
    • Previously processed certificates that failed are not automatically replayed; backlog reprocessing is ongoing and full recovery requires reprocessing and verification of the backlog.

  • Coordination and next steps:

    • Continue coordination across TIM/BIS, OCR, Windows/infrastructure, and database teams.
    • Complete backlog reprocessing and confirm full stability.
    • Schedule Post-Incident Review and confirm customer impact reconciliation.

  Monitoring