DAT Solutions Internal

Back
May 11, 2026 7:54 AM PDT

TIM and DICE outage preventing new request processing and uploads

Updates

  • May 11, 2026 3:00 PM PDT

    TIM and DICE appear to be down at the moment, preventing users from accessing DICE and blocking the processing or upload of any new requests. Legacy CarrierWatch is still working, but new requests cannot move forward until TIM and DICE are restored. The issue is actively impacting operations and needs investigation from the owning team. Resolution: The outage was caused by the TIM_PROD transaction log filling the E: drive on philwcrbsql04, preventing TIM UI access and new requests. Short-term mitigation added 50 GB of space to the E: drive enabling TIM/DICE to restore (~118 minutes MTTR), and a long-term plan to shrink the SQL logs and implement regular backups was put in place to prevent recurrence.

    Resolved
  • May 11, 2026 7:54 AM PDT (7 hours and 5 minutes earlier)

    TIM and DICE appear to be down at the moment, preventing users from accessing DICE and blocking the processing or upload of any new requests. Legacy CarrierWatch is still working, but new requests cannot move forward until TIM and DICE are restored. The issue is actively impacting operations and needs investigation from the owning team. Short-term mitigation: 50 GB of space was added to the E: logs drive to address the TIM_PROD transaction log full condition, after which TIM UI access and TIM/DICE functionality began restoring. Long-term mitigation: Sam will work with Henry to shrink the SQL log files and clean up space to prevent recurrence.

    Monitoring