DAT Solutions Internal

Back
Oct 19, 2026 2:42 AM PDT

FMIC AWS FSx File Share Outage – \pawswfmcalx01\share

Updates

  • Oct 19, 2026 7:16 AM PDT

    Summary: The FMIC AWS FSx file share \pawswfmcalx01\share (file system fs-050a21643fdfc8a3b, prod-analytics account) became inaccessible after the weekly FSx maintenance window on Sat 17 Oct 2026. The L: and N: drives were unavailable, the scheduled FMIC List runners could not run, and the weekly Legacy BA Portal and iFMI API releases were delayed. Platform on-call retried FSx Misconfigured State Recovery, and the share returned to Available at 09:37 UTC on 19 Oct (15:07 IST / 02:37 PDT). Total outage: roughly 50 hours.

    Customer impact: Per Ashwini M V (Analytics), iQ dashboard processing was blocked for customers who submit data on Sundays (her estimate: about 75-80% of those customers). Workflows and List runners are running normally again. Customers who submitted on Sunday will see their iQ dashboards updated on Tuesday instead of Monday, a one-day delay. Weekly releases were delayed by the N: drive outage.

    Detection: First reported in the platform Slack channel at 10:42 IST on 19 Oct (about 22:12 PDT on 18 Oct) after the FMIC/Analytics team noticed the share was down. FSx's last contact with the domain controllers was 17 Oct 07:39 UTC, so the problem went undetected for about two days. No alert fired.

    Timeline (UTC):
    - 17 Oct 07:30 - weekly FSx maintenance window starts
    - 17 Oct 07:39 - FSx's last contact with the domain controllers
    - 19 Oct 05:12 - outage reported (10:42 IST)
    - 19 Oct 08:14 - Platform on-call (Erick Stubbs) retries Misconfigured State Recovery
    - 19 Oct 08:26 - FSx resumes contacting the domain controllers
    - 19 Oct 09:37 - FSx Available; N: and L: confirmed working
    - 19 Oct 09:42 - incident marked mitigated
    - [time] - incident resolved after monitoring

    What was ruled out: Network path to the DCs (Reachability Analyzer and VPC flow logs; no network changes since 14 Oct). The AWSSync service account (works, and the Finance FSx uses it without issues). The FSx computer object (present). DC03, DC04, and root DC security logs (no failures). An early suspicion about a missing route was not supported.

    Root cause: Not confirmed. Working theory: the file server did not rejoin Active Directory after the 17 Oct maintenance. AWS Support case [number] is open for confirmation. INC-1273 (May 2026) may have started at the same Saturday maintenance window, but our records for that incident conflict.

    Resolution: Platform retried FSx Misconfigured State Recovery. FSx resumed contacting the domain controllers and returned to Available. Analytics confirmed L: and N: work and the workflows and List runners run as expected. Stable since 02:37 PDT, monitored until [time]. Resolution: Platform retried FSx Misconfigured State Recovery. FSx resumed contacting the domain controllers and returned to Available. Analytics confirmed L: and N: work and the workflows and List runners run as expected. Stable since 02:37 PDT

    Resolved
  • Oct 19, 2026 2:42 AM PDT (4 hours and 33 minutes earlier)

    Summary: The FMIC AWS FSx file share \pawswfmcalx01\share (prod-analytics account) is inaccessible. The L: drive used by FMIC/Analytics workflows is unavailable, so the scheduled List runners can’t run.

    Customer impact: Per Ashwini M V (Analytics), work is blocked now. Sun–Tue is the peak window when customers submit data, and about 75–80% of the customers who submit data on Sundays expect it in their iQ dashboards on Monday. Processing can’t resume until the share is back. Company holidays tomorrow and the day after mean pending work should be completed today.

    Detection: DAT employee (reported in the platform Slack channel at 10:42 IST, 19 Oct 2026 / about 22:12 PDT, 18 Oct). The actual start time is not yet known.

    Findings so far (Windows team, Darin McIlroy): The FSx subnet’s route table (rtb-07a10ccd3e3a8cd22, VPC vpc-0465e2b963d715aed, subnet subnet-012cdd6edf50fa24e) has no specific route to the domain controller range 10.36.236.0/22 in Corp prod sysops (VPC vpc-0ee1d7890d690d244). It does have 0.0.0.0/0 via tgw-051aeaa520bacd32c, so the transit gateway attachments and routes still need to be checked. A recent networking change is suspected but not confirmed.

    Working hypothesis (unconfirmed): FSx cannot reach the AD domain controllers, so it has dropped out of the domain and is in a misconfigured state.

    Related incidents: INC-1273 (May 2026, same share, FSx–AD failure) and INC-981 (Mar 2025, AD service account disabled). Previous fix, per Slack: opening TCP/UDP 464 to security group sg-0cff90e1c8704fb64 and starting “Misconfigured state recovery” on the file system.

    Current actions: Platform on-call (Erick Stubbs) is investigating the network path (TGW, route tables, security groups, CloudTrail). Mitigation Update:
    The FMIC FSx file share \pawswfmcalx01\share is accessible again. FSx came back Available at about 15:07 IST (02:37 PDT), and BB & analytics team confirms the N: drive works.
    Impact: The share was down from about 17 Oct 07:39 UTC (around the Saturday maintenance window), roughly 50 hours. FMIC/iQ data processing was blocked, and the weekly Legacy BA Portal and iFMI API releases were delayed.
    What fixed it: Platform (Erick) retried FSx Misconfigured State Recovery. FSx resumed contacting the domain controllers at 08:26 UTC and then returned to Available.
    Cause: Not confirmed. Working theory is that the file server didn’t rejoin AD after the 17 Oct maintenance. The network, the AWSSync account, and the DC logs all checked out. The AWS Support case stays open for root cause.
    Still to confirm today: @ashwini.m v, please verify L:, run a List runner, and tell us the backlog time.

    Monitoring