DAT Solutions Internal

Updates

• Apr 29, 2026 12:19 PM PDT

  Since ~17:20 PT on 2026-04-28, Salesforce/DAT Rep → CSR synchronization has been intermittently failing, leaving some accounts active in Salesforce/DAT Rep but inactive or missing seats in CSR, which prevents users from logging in or seeing postings.
  Salesforce secureproxy is returning ERR_CONNECT_FAIL when connecting to connector.dat.com from aws-prod21-useast2/core1, resulting in a growing backlog of unsynced transactions and expected customer-impacting contacts as volumes increase toward 8am EST; NOC and the owning team are being paged to investigate network/ACL issues on connector.dat.com and restore sync. Resolution: The incident was mitigated by reverting the Salesforce custom label to the correct connector.dat.com endpoint and applying the endpoint update via a release, allowing CSR synchronization to resume and reducing the backlog from 643 to about 544 records. Root cause: a stale label value in the repository was overwritten by a recent label change, with a follow-up PR (BusApps 4132) planned to align production labels and prevent recurrence.

  Resolved
• Apr 28, 2026 6:30 PM PDT (17 hours and 49 minutes earlier)

  Since ~17:20 PT on 2026-04-28, Salesforce/DAT Rep → CSR synchronization has been intermittently failing, leaving some accounts active in Salesforce/DAT Rep but inactive or missing seats in CSR, which prevents users from logging in or seeing postings.
  Salesforce secureproxy is returning ERR_CONNECT_FAIL when connecting to connector.dat.com from aws-prod21-useast2/core1, resulting in a growing backlog of unsynced transactions and expected customer-impacting contacts as volumes increase toward 8am EST; NOC and the owning team are being paged to investigate network/ACL issues on connector.dat.com and restore sync. - The mitigation fixed the issue by reverting the Salesforce custom label to the correct connector.dat.com endpoint and applying the endpoint update via a release ticket to restore CSR synchronization.
  - Validation was planned (ETR ~15 minutes); partial recovery was observed with some syncs resuming and the unsynced/backlog decreasing from 643 to about 544 records.

  Monitoring