DAT Solutions Internal

Back
May 4, 2026 10:09 AM PDT

FMIC AWS FSx File Share Outage – \pawswfmcalx01\share

Updates

  • May 4, 2026 1:55 PM PDT

    FMIC’s AWS FSx file share \pawswfmcalx01\share is down due to a failed FSx–Active Directory connection, leaving the share inaccessible since roughly 12 a.m. PT on 5/2 and impacting IQ dashboard processing for 80+ customers. On-call has verified DC/DNS reachability; multiple FSx recovery attempts have failed, and further recovery vs. restore actions are pending with the Platform team. Resolution: The FSx security group sg-0cff90e1c8704fb64 was updated to allow TCP/UDP 464 to the Domain Controllers, enabling the misconfigured state recovery process. After connectivity was verified by Eddie Boss, the incident was mitigated and marked resolved at 4:05 PM UTC.

    Resolved
  • May 4, 2026 10:09 AM PDT (3 hours and 46 minutes earlier)

    FMIC’s AWS FSx file share \pawswfmcalx01\share is down due to a failed FSx–Active Directory connection, leaving the share inaccessible since roughly 12 a.m. PT on 5/2 and impacting IQ dashboard processing for 80+ customers. On-call has verified DC/DNS reachability; multiple FSx recovery attempts have failed, and further recovery vs. restore actions are pending with the Platform team. - The mitigation was performed by updating the FSx security group (sg-0cff90e1c8704fb64) to allow TCP/UDP 464 to the Domain Controllers, enabling the misconfigured state recovery process.
    - After verification that connectivity was restored (Eddie’s confirmation), the incident was marked mitigated.

    Monitoring