DAT Solutions Internal

Updates

• Aug 5, 2026 10:25 AM PDT

  CSR support staff (Miriel Robles) reported errors when trying to search for users on CSR (initially thought to also affect DAT Rep, but DAT Rep was confirmed working by Mohamed Suhail Z and Akash Roshan). Miriel confirmed CSR itself was down for the team ("Its just csr that's down for now"), impacting the ability to search users, add/remove seats, and manage NA account services. Support noted downstream effects: users stuck in free demo status may not be creatable, and seat discrepancies between DATrep/CSR can't be troubleshot.

  Datadog logs confirm active 500 errors on rest.dat.com:8443/TcoreRequest/rest/auth/externalIdentity/csbId/{id}?token=csrpages, consistent with the Auth0Api layer failing, likely due to a null client_id (java.lang.IllegalArgumentException: 'client id' cannot be null at Auth0Api init).

  Timing coincides with the CSB release deployed the evening of Aug 4 (build 654 / CAB-11966), which included CSR Pages code changes and logged auth-service restarts during rollout. Troy Folger noted the release was tested and working immediately after deploy last night, so the correlation is not yet confirmed — root cause is still under investigation with CSB on-call Incident mitigated at 08:04 MDT. Root cause was a missing auth0.mgmt.clientId value in the production G-stack AWS Secrets Manager compound secret, inadvertently omitted during the 8/4 CSB Cascade-git release (Build 654). This caused the Auth0Api layer to fail initialization, resulting in 500 errors on the CSR Pages REST path and blocking user search/creation/deletion.

  Mitigation steps:

  Identified the missing secret key via Datadog log analysis (null client_id pattern on rest.dat.com externalIdentity endpoint)

  Govinda (SRE) and Troy (CSB) restored the missing value in AWS Secrets Manager

  CSB directory and CSR Pages restarted to pick up the updated secret

  Functionality verified by CS reps post-restart

  Follow-up: PIR document in progress. Post-release smoke test automation for CSB-dependent endpoints to be tracked as an action item.

  Resolved
• Aug 5, 2026 8:04 AM PDT (2 hours and 20 minutes earlier)

  CSR support staff (Miriel Robles) reported errors when trying to search for users on CSR (initially thought to also affect DAT Rep, but DAT Rep was confirmed working by Mohamed Suhail Z and Akash Roshan). Miriel confirmed CSR itself was down for the team ("Its just csr that's down for now"), impacting the ability to search users, add/remove seats, and manage NA account services. Support noted downstream effects: users stuck in free demo status may not be creatable, and seat discrepancies between DATrep/CSR can't be troubleshot.

  Datadog logs confirm active 500 errors on rest.dat.com:8443/TcoreRequest/rest/auth/externalIdentity/csbId/{id}?token=csrpages, consistent with the Auth0Api layer failing, likely due to a null client_id (java.lang.IllegalArgumentException: 'client id' cannot be null at Auth0Api init).

  Timing coincides with the CSB release deployed the evening of Aug 4 (build 654 / CAB-11966), which included CSR Pages code changes and logged auth-service restarts during rollout. Troy Folger noted the release was tested and working immediately after deploy last night, so the correlation is not yet confirmed — root cause is still under investigation with CSB on-call Mitigation summary:

  • Likely root cause: missing CSR Pages/Auth0 client_id in the production G-stack AWS Secrets Manager entry, causing 500 errors in the Auth0/Auth integration layer.

  • Mitigation actions (in progress as of the latest update):

  • 1) CSB team identified the missing client_id and prepared a fix.

  • 2) Updated the compound secret in AWS Secrets Manager for the production G-stack.

  • 3) Applied the CSB patch/update to restore proper CSR Pages authentication flow.

  • 4) SRE involvement (Govinda) to assist with the Secrets Manager update.

  verified and the CSB user service has no errors and CSR pages are working as expected

  Monitoring