DAT Solutions Internal

Back
Aug 7, 2025 7:07 AM PDT

Elevated Login Block Activity Affecting iOS/macOS Users

Updates

  • Aug 8, 2025 3:29 PM PDT

    Issue is now resolved

    Resolved
  • Aug 8, 2025 3:26 PM PDT (2 minutes earlier)

    We are still seeing spikes, which means this incident is not fully resolved. Teams are currently investigating. A patch was deployed on the client code and teams no longer see \"user is blocked\" messages over the past hour

    Monitoring
  • Aug 8, 2025 11:48 AM PDT (3 hours and 38 minutes earlier)

    We've rolled back the offending code on the client side. Users that previously downloaded that version of the code will need to refresh the browser to be delivered the fixed version of the code. We're also investigating any other avenues to speed up client uptake of the fixed code. We're also investigating ways to mitigate this kind of long tail in the future.

    Monitoring
  • Aug 7, 2025 1:19 PM PDT (22 hours and 28 minutes earlier)

    Over the past week, we’ve observed elevated login block activity, though not the highest within the last month. Customer Support has reported an unusual increase in user calls - over 6 in the last hour - primarily from iOS/macOS users. There was also a known fraud spike in July, which may or may not be related to the current issue. Investigation is ongoing. The incident was mitigated by rolling back the problematic deployment that contributed to the elevated login block activity and implementing a script to unblock previously affected users. Additionally, monitoring was established to track call volumes, which began to decline, indicating successful mitigation efforts. We will, however, continue to monitor for the remaining of the day.

    Monitoring
  • Aug 7, 2025 12:05 PM PDT (1 hour and 14 minutes earlier)

    We waited an hour to monitor the situation, but the spike in calls has not decreased as expected. We will continue monitoring for another hour then proceed with mitigation. Team members in India have been informed to keep an eye on the graphs and report any significant changes.

    Monitoring
  • Aug 7, 2025 10:51 AM PDT (1 hour and 14 minutes earlier)

    We will reconvene in an hour to analyze call volume, which has already started to decline. In the meantime, another team is working on a script to unblock the previously affected users. Customer Support will also follow up with some users directly to manually restore access, which should further reduce call volume.

    Identified
  • Aug 7, 2025 10:03 AM PDT (47 minutes earlier)

    We are currently investigating elevated login block activity affecting users on iOS and macOS devices. We've identified that a recent deployment by our SWAT team involving some non-production code may have contributed to this issue. This code has since been rolled back, and we are monitoring the situation closely. Our team is working diligently to resolve the ongoing latency issues in our identity services, which appear to have exacerbated the login challenges.

    Identified
  • Aug 7, 2025 8:08 AM PDT (1 hour and 54 minutes earlier)

    We are currently investigating this issue.

    Investigating
  • Aug 7, 2025 7:07 AM PDT (1 hour and 1 minute earlier)

    Over the past week, we’ve observed elevated login block activity, though not the highest within the last month. Customer Support has reported an unusual increase in user calls - over 6 in the last hour - primarily from iOS/macOS users. There was also a known fraud spike in July, which may or may not be related to the current issue. Investigation is ongoing.

    Investigating