DAT Solutions Internal

Customer Support has reported that some load postings remain visible to carriers and are still generating calls after the customer deleted/cancelled them. Initial examples include:

ZS19C1Wf

ZS19B1ub

ZS19BJPW

These postings appear to be removed from the customer’s view (e.g. via CSR/D1 Web/TMS), but are still exposed in search results to carriers, consistent with prior “stuck load” incidents where deleted postings continued to appear on the load board and in SV4 search.

At this time, impact appears limited to a handful of brokers, but there is potential for broader customer impact if this is another systemic issue with posting delete/cancel propagation. The incident was resolved by performing a rollback of the changes that led to the issue and manually cancelling the affected postings, as confirmed by Keegan Brown. The status of the incident is marked as resolved.

In production, user is getting data that does not belong to them.

Beginning ~1/21 our truck postings are down from Load Link

Spinning up INC to bring together various channels, conversations, research and to drive resolution LoadLink deployed a fix and DAT has verified counts are back to a normal level

dwhprod_db.dw_sf_admin.ds_ps_rv_tdz_best_match is missing data for the last 2 days, we need it to be resolved today so it doesn't interrupt Monday morning deliverables. The incident was resolved by backfilling the missing data and adjusting the job schedule to successfully run at an earlier time of 3:15 AM PT. Confirmation of successful runs and availability of data led to the closure of the incident.

Drivers are not able to start tracking on the app. It gets stuck loading.
Also the Load Tracking admin site is down. The incident was resolved by killing a MySQL job that was causing a table lock, which restored services on the mobile application for Trucker Tools Load Tracking. There are no indications of any unresolved issues following the incident.

There is an issue with the data needed for ratecast The incident was resolved by deploying a production fix that updated the date criteria for the Rateview and CMR jobs from data_interval_start to data_interval_end. After deployment and verification, both jobs (ratecast-reverse-etl and dwh1000_2000_daily_rate_forecasts) are functioning as expected.

The datalake connector for Search V4 data has become disconnected and kafka lag for that consumer group is rising.

Snowflake provider is down The incident was resolved by isolating the issue to the Snowsight application, restarting the related systems, and reverting recent application releases, resulting in restored access. By the end of the timeline, it was confirmed that access was looking good again.

Multiple reports of enterprise accounts with incorrect (much lower) lane credits for multilane. Marking INC‑1193 as resolved: incorrect Multilane lane credits introduced by the TRX migration (CAB‑10719) have been repaired via targeted and deep-cleaning scripts, no new bad data is being generated, and remaining work is limited to billing remediation tracked outside this incident.

Another test just a test

Key Accounts are unable to find recent matches, only receives matches that are 10+ hours old.

I was not able to replicate, but some of our agents are seeing the same issue

Getting reports of account getting stuck during reactivation. Also amending the contract. The incident was resolved by addressing the Heroku outage that caused accounts to get stuck during reactivation, and manual cleanup of the affected accounts was completed successfully. The status is noted as resolved, indicating that the issues were fully addressed.

We have not received the latest calc market rate data needed for Ratecast.

csbprod.CALCULATED_MARKET_RATE is only populated up to 2026‑01‑14 17:56:19 The incident was resolved by initiating the \"persistRates\" job and ensuring that data loads were successfully completed from Stitch to Snowflake, restoring the flow of market rate data. Communication throughout the process helped manage customer expectations regarding the delay, and the status was later confirmed as resolved.

Multiple brokers’ credit scores and Days to Pay history were not appearing in the Directory, though they were visible on the Load Board. Examples included several DOT/CID accounts, and reports came from multiple DAT employees. Root cause: A SQL bug in the Ansonia credit-load job cleared the credit profile table but failed before reloading it, leaving Directory/My Trucks/Search Loads without Credit Score and Days to Pay data for all brokers.

Fix: Engineering corrected the job and reran it after receiving the latest Ansonia file. Credit profile tables are now repopulated and credit data is again visible in Directory and related views, confirmed by product support and users. No further issues observed; incident resolved.

Trendlines Regional Rates Data files are not available in S3 bucket after 09 Jan The job, which was on hold, was run again, and the data is now in place.

Some are customers using locally (on premise) and some are Keypoint Anywhere. When they go to import a carrier with insurance it should take around 1.5 to 2 seconds, but they are telling us it is sometimes 10 seconds. The incident was resolved by adding two indexes, which improved system performance and significantly reduced the lag reported by customers. Follow-up testing confirmed stabilization, leading to the closure of the incident.

missing data 1/10 and duplicates duplicates were removed and the script has been updated so that this will not occurr again. 1-10 best match has been updated and pushed downstream

Team can't access PUD within SF. this is affecting internal team members. We can't access the product usage dashboard for any account. When we click we are getting the error: "An error occurred. An email has been sent to the Dashboard Administrators. You may try refreshing the page once to see if this was a temporary issue." The incident was resolved by removing the old Product Usage Dashboard (PUD) link from Salesforce, which was causing the access issues, and updating the page layout to redirect users to the new PUD in the Rep application. Users were informed to refresh their pages to clear any cached links, and subsequent checks confirmed that the old link had disappeared, allowing access to the new PUD as intended.

data warehouse jobs unsuccessful
Impact:
On Ratecast, the customer impact is minimal; they'll see one fewer prediction at the end of a series of them. For MCI, there are currently no forecasts (and that includes today's value, which is a forecast). ds_tri_rates is deduped and the new script has been confirmed to work

we observed that REST→CSB fallback calls in Prod are failing with InvalidAuthentication / “Invalid Session Token” responses from CSB for all JWT‑based requests, causing affected customer requests to fail. This behavior matches previously closed defect TMS‑8281, where newly added customers received InvalidAuthentication errors from CSB after a fallback from the Assets Mapper and their requests were not processed successfully. The issue is customer‑impacting (dashboard currently marked yellow) and the application team has engaged CSB to investigate; as a contingency, traffic can be rolled off the adapter to avoid further impact if a fix is not in place before monday morning The incident was resolved when a property change, which had not been committed to the repository, was manually applied and production adapters were restarted, confirming that the issue was resolved. Success was verified by team members, and the incident status was marked as resolved.

It was identified that the run book was not fully followed during execution. Remediation is in progress, starting with adding foreign keys to Postgres, followed by a review and completion of the remaining run book steps to ensure full compliance. At this time, customer impact is difficult to determine. There is no indication of direct availability impact; however, there is a potential risk of data consistency issues. The incident was resolved by adding the necessary foreign keys to Postgres and validating that the affected systems were functioning correctly after remediation. Additionally, a follow-up ticket was created for further monitoring, and it was confirmed that the FTP/SFTP services were fully recovered.

Direct access to legacy signup.dat.com URLs is intermittently failing with blank pages / net::ERR_CONNECTION_RESET instead of redirecting users into ssu.dat.com as expected.
The incident is not related to the CSB DB cutover. login.dat.com now redirects to ssu.dat.com via a rewrite rule in Cloudflare. SRE has documented the steps for this fix and we will eventually remove it in the future after links on dat.com sites are updated.

After database migration, we are seeing intermittent issues with the negotiation service / bookable loads.(Trailblazers) The incident was resolved by determining that the majority of transactions succeeded without issue and that the problems were outliers, with a follow-up ticket created for future investigation. The incident status is marked as resolved.

After database migration, no successful webhook requests from the private network system to search v4 Association service was modified to properly set headers for all endpoint processors

When masquerading as a Carrier in OnBoard and I click on Contracts I get this error the screen for Contracts will not load up The incident was resolved by deploying a fix that allowed the brokers tab to load successfully, while the contracts page remains slow but functional and will be tracked as a separate performance issue. Therefore, since there is no longer a broken customer flow, the incident marked as resolved.

New directory not loading after database migration was done

CSB and Aurora Borealis Team to work on this tomorrow morning The incident regarding the New Directory not loading was resolved upon updating the search_path, as indicated by John Freeman's verification that the issue appeared to be fixed, followed by a confirmation from Alex indicating no issues were observed after testing. The status of the incident is marked as resolved.

Registration / Onboarding bug.

Onboarding registration will not work post cutover (csb call to register). Reviewing fix and mitigation for related issues found with tables/sequences migrated as views.

CSB team will work with DBAs on Saturday. Waiting on customer impact to evaluate SEV 2 or 3 status The incident was resolved by performing a schema consolidation and testing the end-to-end flow, which fixed the onboarding registration issue. Julie Mather confirmed the issue was fixed, and the incident was subsequently closed.

Unable to establish connections to oracle and it's impacting hexarocket, which will impact cutover tonight. The incident was resolved by restarting the CSB Oracle database, which involved shutting down services and causing a planned 30-minute outage. After the restart and validation of services, operations were resumed, and the incident was marked as resolved.

Users are seeing error message The incident was resolved by correcting the access-service configuration to remove references to the removed migration service secrets, followed by redeploying the access-service, which restored user login functionality to D1 Web and Mobile. The incident was officially marked as Resolved in the change ticket CAB-10711.

Customers that were migrated yesterday to the REST adapter are experiencing forbidden issues All customers were rolled back. Confirmed with support prior issues are resolved.

Major security vulnerability found regarding Mongodb servers.

This is a test for the incident workflow

Customers are reporting loads are showing no pings since 12/31/25. Most all loads are showing tracking interrupted. The incident was resolved by adding a partition to the database, which allowed data to flow again. However, there was data loss during the downtime that could not be restored.

Kafka lag for the fraud detection consumer has been spiking since ~8:52am PT and is not self-resolving.

https://app.datadoghq.com/monitors/244711610?from_ts=1766076931626&to_ts=1766078731626&live=true The incident was resolved by updating the node affinity to pin the fraud detection consumer to 15 pods, which mitigated the Kafka lag issue. This change was successfully merged, allowing the consumer to properly sync and manage load in the future.

Starting Dec 16, 2025 02:55 UTC, Snowflake reported an active service incident (INC0148543) impacting query execution in some regions. We’re seeing SQL execution internal error: Processing aborted due to error 300002:xxxxxxxxxx with multiple Snowflake incident IDs. As a result, our dbt/Airflow jobs and Stitch loads are failing (intermittently/consistently) due to the provider outage. Snowflake has stated there is no workaround at this time and a fix is in progress. We’ve paused the failed Airflow schedules to prevent repeated retries and are monitoring Snowflake status The incident was resolved by resuming paused Airflow and dbt jobs after Snowflake confirmed the incident's mitigation, with no new failures reported. Additionally, the MCI, Ratecast, and DAS/Snowflake views began populating correctly for the affected dates, confirming the restoration of service.

DAT One Mobile on Android is currently having an outage The incident was resolved through a rollback of the BFF code, which restored normal functionality, and the deployment of a code fix (CAB-10655) to prevent future issues.

Upstream tasks are not finished when data merge is executed so the pipeline has been loading blank tables since 12/1 Data added to upstream table

user account appears to be compromised. Compliance has removed the fraudulent postings, but some loads remain visible in Search/Load Board (stuck exposures), causing ongoing risk. requesting clear any exposed postings from Search. The incident was resolved by manually deleting the affected postings from the database after confirming their removal. Although the incident status is marked as resolved, there is a note indicating that the root cause remains unclear, and additional follow-up may be needed for complete resolution.

Customers using TMS are reporting issues connecting to FTP for posting loads. The incident was resolved by the Auth team preparing a pull request to remove SFTP client from MFA, which addressed the connection issues affecting multiple users. It was confirmed that the problem was resolved for customers, and follow-up communication was initiated to verify this.

Customer's have a significant lag on logging in, searching, and posting. Customer's initially not getting search results, have to search several times back to back to receive results. The incident was resolved by identifying that the significant lag experienced by customers was likely due to the use of Surf Shark VPN, which was affecting their ability to log in and search effectively. The stakeholders agreed to close the incident after determining it wasn't a systemic service failure, but rather a compliance-related issue.

Testing out workflow for metrics

ATS (customer) is unable to repost some loads via REST because they are trying to reuse reference ids from previously FTP'd loads. The incident was resolved after confirming that the customer, ATS, was able to successfully repost loads by deleting their outdated postings, either on their own or with assistance from Customer Support. There have been no further errors or reports from the customer since the mitigation steps were implemented.

Resolved – customer impact mitigated; systemic behavior tracked in HELP‑73576 / VA follow‑ups; guidance documented for future FTP→REST migrations.

After rolling back CAB-10575, there are some files that are stuck in processing. Need production write access to invoke the lambda and reprocess the files that are stuck in the s3 bucket The incident was resolved by successfully redeploying the bulk-ratecast-lambda and reprocessing the 107 user files, with all customer issues being fixed and no further problems reported. The team remains on alert for any related customer issues but considers the incident resolved.

Posting service experiencing a OwnershipParsing error.

Ownership only has UserID The incident was resolved through a code change that addressed the underlying issue causing the OwnershipParsing error, which was successfully released on January 15, 2026. Additionally, the immediate user impact was mitigated as the issue seemed to self-resolve over time.

Loadlink has reported issues sending files to DAT via FTP The incident was resolved by resetting the primary firewall, which restored connectivity issues with the SFTP endpoint, allowing Loadlink to successfully send files to DAT. Loadlink confirmed everything was functioning well on their side following the resolution.

Incident Summary – Tracking Data Not Reaching TMS

Impacted Customers:
Traffix (LTID 2810)
Example affected loads: CCA1002AM, CCA1003AM

JEAR (LTID 13060)
Example affected load: 1063940

Issue Description:
Both Traffix and JEAR are reporting that no tracking data is being received in their TMS, despite the loads showing as tracking normally within Trucker Tools.

At this time, these are the only two customers reporting the issue, but both are high-volume accounts.

Initial Assessment:

Load tracking appears healthy within Trucker Tools.

The issue is isolated to tracking updates not being delivered or ingested by the customers’ respective TMS systems.

Further investigation is required to determine whether this is:
A Trucker Tools outbound webhook/payload failure,
A delivery issue, or
A TMS-side ingestion/parsing issue. The incident was resolved by catching up on the backlog of tracking data processing, with the system fully operational again after approximately five hours. The resolution included ensuring minimal missing geolocation data and completing the backfill process successfully.

The PATCH load posting >=3s alert has been triggered off-and-on since yesterday (11/23/25) evening. Initial signs point to the posting-change-stream-consumers as having issues. The availability of the Freight Posting Service is still looking okay, but I am opening this incident as I continue to investigate the root cause.

RDP to pawswkpagw01 is not happening The incident was resolved by rebuilding the gateway server (GW01) and limiting inbound traffic, which enabled successful logins to the keypoint application for users who were previously having issues. The status was officially marked as resolved on 11/22/2025 at 11:32 PM UTC.

Rate limiting rule set in Cloudflare

Crowdstrike detected potential malicious code being run on user's machine. Determined that malicious parts of the code were not executed and no data was found to be exfiltrated and no malicious payloads were delivered.

MCI Job got failed in production. The incident was resolved by re-running the failed jobs, which successfully updated the necessary data. It was confirmed that the data reflected correctly for the reports and customer access points following the resolution.

Cloudflare outage causing Load Tracking issues. Creating incident as will need to work with Marketing on Comms

The json stream read was corrected. The data has been backfilled and current data is flowing as expected.

Both APIs causing duplicate stops during load creation.

The incident was resolved by disabling live matching for 30 to 60 seconds, which eliminated lag, and subsequent processing of the backlog restored system functionality. The status indicates that the issue is resolved.

The incident has been resolved.

Freight posting service and Freight truck posting service triggered alarms for Golden Signals Throughput being low Customer impact: No external customer were impacted
At around 7:12 AM CDT, several teams reported that metrics were not appearing in Datadog.
The MOB team created an incident and invited SRE to investigate.
Troubleshooting summary:
Cole and I began reviewing the issue.
We observed that all OTEL Collector pods were in CrashLoopBackOff, initially due to CPU pressure.
After restarting the pods, they immediately began failing again — this time due to OOM (Out of Memory) errors.
We increased vertical resources (instead of horizontal scaling, since the load wasn’t distributed evenly across pods):
Memory: 4Gi → 8Gi
CPU: 3 cores → 4 cores
After the resource adjustment, the pods stabilized and metrics began flowing normally again.
Resolution: Issue has been fully resolved.
PR applied: https://github.com/dat-freight-and-analytics/gitops-platform/pull/514
Root cause (still under investigation):
Autoscaling for the OTEL Collector did not trigger as expected, The workload was not distributed across pods. We’re still investigating why the autoscaler did not initiate.
We will continue to dig into the autoscaling behavior and provide updates once we identify the exact cause