DAT Solutions Internal

Updates

• Jun 3, 2026 6:21 PM PDT

  The Posting System has been experiencing intermittent dips in availability / spikes in 5xx errors for the past 30-45 minutes. The root cause of these spikes appear to be failed calls to the account-service, which has also been experiencing spikes in OOM errors during the same timeframe. Resolution was achieved by stopping memory-intensive processing (scaling the account-consumer to zero replicas), upgrading the Aurora MySQL DB with a blue-green upgrade, and gradually bringing services back online (excluding the batch job) to reduce risk. The posting system stabilized with account view access restored and healthy pods, marking inc-1288 as resolved, with a PIR to be documented and reviewed.

  Resolved
• Jun 3, 2026 2:15 PM PDT (4 hours and 5 minutes earlier)

  The Posting System has been experiencing intermittent dips in availability / spikes in 5xx errors for the past 30-45 minutes. The root cause of these spikes appear to be failed calls to the account-service, which has also been experiencing spikes in OOM errors during the same timeframe. Mitigation steps included scaling down the Salesforce account-consumer service to zero replicas to stop memory-intensive processing, upgrading the database instance type to provide additional RAM via an infrastructure PR, and performing a blue-green database upgrade (writer instance modified, readers reconfigured) with in-place Terraform changes. A recovery plan was then executed to bring up all services except the batch job service initially to reduce risk, followed by restoring account view access after the BFF restart.

  Monitoring