DAT Solutions Internal

From this morning, shipment service keep getting 429s from calling negotiation service and all request seems to be from one user

Since ~17:20 PT on 2026-04-28, Salesforce/DAT Rep → CSR synchronization has been intermittently failing, leaving some accounts active in Salesforce/DAT Rep but inactive or missing seats in CSR, which prevents users from logging in or seeing postings.
Salesforce secureproxy is returning ERR_CONNECT_FAIL when connecting to connector.dat.com from aws-prod21-useast2/core1, resulting in a growing backlog of unsynced transactions and expected customer-impacting contacts as volumes increase toward 8am EST; NOC and the owning team are being paged to investigate network/ACL issues on connector.dat.com and restore sync. Resolution: The incident was mitigated by reverting the Salesforce custom label to the correct connector.dat.com endpoint and applying the endpoint update via a release, allowing CSR synchronization to resume and reducing the backlog from 643 to about 544 records. Root cause: a stale label value in the repository was overwritten by a recent label change, with a follow-up PR (BusApps 4132) planned to align production labels and prevent recurrence.

Okta regularly checks with UKG for changes and it appears to be suspending all employees in UKG. - Luke disabled the UKG→Okta updating and all suspended accounts were unsuspended.
- Root cause identified: a UKG report filter change by Roper caused deactivations; the fix is rebuilding the UKG report to filter by company codes so the integration can be reinstated.

For example: https://scotlynn.truckertools.com is responding with a 500 error. This appears to be something related to third party host Netlify. Most of these sites haven't been deployed in 3-6+ months and now randomly all requests giving 500 errors. Affecting 11 customer sites. All sites are running normally again. Currently still have a support ticket open with Netlify to understand root cause. It appears Netlify functions/workers may have been updated/changed by Netlify behind the scenes causing errors. Redeploy of the same project and version stopped all errors.

We are getting calls because customers can't log in. Their contract renewed yesterday and everything look good in SF but when we look in CSR pages, there isn't anything there. I've listed the accounts we have fixed and a couple of the ones that I've just found looking at contract renewals.

Fixed accounts:
0001372109 - SF Data Migration Renewal Opportunity - Closed Date 4/18/2026
0001597906 - Rebuild Contract - Add 8th User Renewal Opportunity - Closed date 4/18/2026
0002023232 - Self reactivation - SSU - Power Select Carrier - 0015G00001etvQKQAY Renewal Opportunity_Reactivated - Closed Date 4/18/2026
0001847860 - SF Data Migration Renewal Opportunity - Closed Date 4/18/2026
0002179003 -Self reactivation - SSU - DAT Load Board - TE Enhanced - 0015G000023Ei3QQAS Renewal Opportunity, Close Date 4/18/2026
0001783586 - SF Data Migration Renewal Opportunity - Closed date 4/18/2026
0002167871 - SSU - DAT Load Board - TE Standard - 0015G000020vrinQAA Renewal Opportunity- Closed 4/18/2026

And these are 2 that we found that haven't been fixed yet:
0002444042 - SSU - DAT Load Board - TE Standard - 001Vo00001NUhTCIA1_Reactivated - Closed date 4/18/2026 (not fixed yet)
0001389385 - SF Data Migration Renewal Opportunity - Closed date 4/18/2026 (not fixed yet) Mitigation occurred by 4/20/2026 after TRX pod resets and cascading fixes to CSR/DAT Rep synchronization, with the root cause traced to Kafka IAM/policy settings. The CSB release completed and the incident was officially marked resolved on 4/29/2026.

Several customers have reported a number of different bugs and issues. Impact for any one of those issues is relatively limited but as a whole we need to track this set of issues with a detailed coordination plan to make sure nothing is dropped.

Unable to connect to CarrierWatch on 4/14
Similar to INC-1256

Weekly(current) model is delayed in L-BA by a day since the verification file did not match

High Auth issues / errors

Users reporting being kicked out of D1W when editing postings Mitigation: Increased memory for the association service pods from 256MB to 1GB, after which the service started responding normally. The incident was closed as resolved with the Search team re-enabling the experiment and Network Nomads taking ownership of monitoring.

DBT cloud is facing some issue. We are getting 502-BadGateWay error
N the airflow jobs are also returning same error. DBT Updates: Deployed a fix for the outage in US Multi-tenant impacting the availability . The functionality has returned to its normal state.

Latency at the Emissary level that is not present in the application. First noticed within Sv4 when publishing to Notification System. During our traffic peak for thursday morning, latency stayed consistent for affected applications.

FMIC (Legacy benchmark) with 60 customers are affected.

DAT Artifactory shows Compromised axios 1.14.1 package was downloaded
https://www.stepsecurity.io/blog/axios-compromised-on-npm-malicious-versions-drop-remote-access-trojan Resolution: The investigation concluded there was no evidence of payload execution or C2 across environments, despite observed downloads. Containment actions were completed and a Post-Incident Review is planned

Unable to connect to CarrierWatch

Users are getting disconnected to Keypoint and are getting an error when connecting to "contact the administrator." We are unable to connect to our gw01 server. Resolved: The gw01 gateway was rebooted and connectivity validated, with affected users able to reconnect by about 07:28 PDT and the incident mitigated by 07:34 PDT. The root cause wasn’t definitively identified, but evidence pointed to a transient network/AWS blip rather than a gateway-domain controller failure, with monitoring to continue.

This incident has been resolved

tri rates table is not updating with customer data within the SLA

Auth0 issues with the create user process that started around 3:30pm pacific time.

There are no rates available for any lane in Rateview. RateView was resolved by correcting the readiness probes to target the proper internal CSB endpoints (WSDL/CSB hosts), updating CSR/WSA configurations, and deploying these changes via GitOps (Argo CD), while throttling the rate-lookup-service and restarting CSR/CarrierWatch/WSA to stabilize. The services returned to normal operation with no timeouts and CSR/CarrierWatch were restored, after which the incident was closed as resolved.

missing data from DS Tri Rates

[TEST] sev2, team = Security, types = Security

turned on page, create confluence doc, escalation path

[TEST] sev3, team = Security, types = Security we restarted...

Seeing a high number of 504s with the crt-insurance-api. At this point, not hearing of any customer concerns or issues reported in any of the customer facing channels (customer-impact, cms_support..) Script completed execution and a rolling restart was performed as well. Database connections and cluster cpu usage returned to normal levels.

Some of the DAGs in airflow did not run since the time change skipped an hour. Critical data has already been addressed. This is to address lower tier data. missed jobs have now all been rerun

Noticed some customers are experiencing issues with posting loads. we assume it's related to DTS for now.

Some customers are unable to use front end bulk upload when they are picking a particular date

We turned on a new pipeline that feeds MCI data based on the new fact and the results look inaccurate

We are seeing a recurring pattern of "phantom" credit usage across multiple accounts. Customers are being debited for RateView queries that do not appear in their transaction logs or manual activity history.

We have receive a report from both customer and internal, that the option for "Extend Posting Date" is not working correctly. Doesn't matter the date or time of the posting we are getting the message unable to extend pickup date. We can replicate the issue.

Customers uploading CarrierWatch files since 2/13 have not had changes take effect due to problems uploading the file from the AWS Transfer Family to CSB

This affects business reporting for objects relying on the fact_transactions table The issue was resolved by backfilling the missing data: reprocessing the missed days in fact_transactions and re-running downstream pipelines. Snowpipe and the Lambda function were verified as working, and the downstream data loads completed successfully (through 2026-03-03 and today’s data) with final sanity checks confirming resolution.

This incident has been resolved.

I need to create an incident for Keypoint Anywhere. DB pawsukpaora01 is not responding. I have tried restarting it twice now. I cannot connect to it. We need someone from the DB and Linux team for assistance. The outage was resolved by replacing the affected server with a new instance, updating the DNS to the new IP, and bringing the Oracle DB and backend services back online, after which reports and user access were restored. Additionally, Jasper server issues were addressed with startup scripts (no reboot needed), and the team deemed the incident resolved.

This incident has been resolved.

The Events section and load details on the new Beta page are not loading properly, and the page is experiencing noticeable latency issues.

Observed Behavior:
• Beta page takes an extended time to load.
• When the page does load, the Events section is missing.

Comparison:
• ✅ Legacy page: functioning normally with no issues.
• ❌ Beta page: latency present and Events not displaying.

Example Impacted Load:
• Customer: LTID13982
• Load #: 127929170

Testing Performed:
Issue reproduced during internal testing with the same results.
The incident was resolved by rolling back the PR deployed to the API servers, which removed the dynamic-text issue causing latency and blank Events on the Beta page. A more durable fix (request-level cache optimization for dynamic text) was developed and merged, and the incident was marked mitigated once the rollback completed.

CW shows no insurance on file 1st screen shot however it is in fact on file in TIM which feeds CW so it should be showing. The team forced the insurance import job to run and monitored it until completion (81560 of 81561 certificates). The root cause was the TIM database move to AWS causing slowness, and once the import finished CW showed the insurance on file, resolving the incident.

RV transaction has stopped reporting for customers from both sales and support Resolution: The issue was fixed by correcting Stitch data access, creating the necessary backfill views, and performing a large Stitch backfill of Rate Lookup Transactions (Jan 9 onward) followed by the merge to restore reporting. After the backfill completed and RV billing data was restored (28 billion billed rows since 2/16), the incident was officially marked resolved.

philudbapql01 csbprod postgres database filesystem filling and need more space added. Resolution: The incident was mitigated by pausing the new rate_lookup_transactions_with_latest_updates extraction, killing the runaway PostgreSQL queries, and expanding disk headroom; disk usage stabilized and the csbprod DB remained online. INC-1216 was moved to Resolved/Closed, with long‑term fixes tracked separately.

Testing the rootly retrospective steps

After deploying a new index we lost connection to DB Resolution was achieved by rolling back CAB-11033, which restored the database connection; a test load post-rollback succeeded and systems returned to normal. The team then rolled out index changes in stages and, as of 2/27/2026, confirmed the incident resolved.

notice a spike of 403 errors in one web bff service https://app.datadoghq.com/logs?query=service%3Aone-web-bff%20env%3Aprod%20status%3Aerror%20%22failed%20to%20get%20metrics%20for%20postingids%22&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&messageDisplay=inline&refresh_mode=sliding&storage=hot&stream_sort=time%2Casc&viz=stream&from_ts=1771521938567&to_ts=1771522838567&live=true Root cause: an internal frontend change caused one-web-bff to call the negotiation metrics endpoint without the required permissions, triggering 403s. Resolution: a frontend fix was deployed and validated via Datadog traces/logs and Slack confirmations; error volume returned to baseline with no confirmed customer impact, and the incident was closed as resolved.

We are receiving an error when working Dice. Dice has an impact on CarrierWatch. If Dice is not working we cannot get insurance certificates to go through to OCR to be able to then publish into CarrierWatch. Resolution was achieved by temporarily rolling OCR back to the Hillsboro server to re-establish ABBYY licensing and clear the backlog, then restarting DICE to process certificates. After the licensing and connectivity issues were resolved, OCR was migrated back to AWS and DICE was brought online with end-to-end validation showing the DICE → OCR → CarrierWatch pipeline functioning again.

We have not received calc_market_rate data yet for 2026-02-12. The job dwh1000_2000_daily_rate_forecasts has failed. Resolved by manually triggering the Stitch data load, which began delivering the 12th data and allowed downstream jobs to be retried; data science jobs completed and data was confirmed. A root-cause follow-up with CSB is pending, but the incident is mitigated and can be closed.

Customers are encountering an error when attempting to enter their insurance agent's email address as part of the P2H flow on mobile. Updated version of crt-insurance-api has been deployed that no longer makes real time calls to Certificial to perform enrollments, these instead will be handled asynchronously.

Multiple enterprise RateView customers report lane/credit counts not showing correctly; reproducible on internal account.
Issues: credits not showing or lower than contracted, potential lockout from multilane or premature overage charges. The incident was resolved by restoring and correcting the lane/credit counts for several affected enterprise accounts, including Amerilux Logistics, Strategic Logistics Company, SELECT TRANSPORT, and Ascent Global Logistics LLC. Confirmation from users indicated that the issues were mitigated for most accounts, allowing the incident to be marked as resolved.

50% of Trucker Tool users unable to login The incident was resolved by deploying a fix that addressed the login issues experienced by 50% of Trucker Tool users. This resolution was confirmed by Deborah Haynes shortly after the fix was implemented.

Enrichment Service is taking too long to process messages and this impacts posting creation to available in search time The issue self-resolved; the first spike was caused by a Geo v3 pod returning 504s and cycling, and the second intermittent latency did not persist, with no lasting root cause found. Mitigations included relaxing enrichment duration thresholds and continuing follow-ups in the respective backlogs, and INC-1205 was closed.

There have been reports of slowness in DAT One web. The incident was resolved by deploying a hotfix to stabilize the UI performance for users, which was confirmed to have improved the situation by Landstar agents. The final status was marked as resolved after confirming that the memory/performance issues were no longer present.

rate-engine-history is sending massive amount of logs to Datadog The incident was resolved by updating the logging level of the rate-engine-historic service to WARN, thereby significantly reducing the volume of logs sent to Datadog. Additionally, there were recommendations made for future logging policies and monitoring to prevent similar incidents.

Users are unable to access Keypoint. The incident was resolved when it was determined that users could access Keypoint again after approximately 10 minutes due to the resetting of the live database following interference from a test restore. Further investigation into the logs was planned to prevent future occurrences.