DAT Solutions Internal

There are users in search V4 that can’t be mapped in Snowflake, causing us to begin losing insight to Search behavior in the new search system. The incident was resolved by creating a new dimension (DIM) table with modified logic to better map searches to users, which enhanced data integrity and allowed for effective mapping of Search V4 users in Snowflake. The changes implemented a temporary dataset and established a path for a long-term solution, effectively addressing the issue.

The incident was resolved when the office service was deployed to production on 9/24/2025, allowing all 48 affected accounts to properly utilize the Prefer/Block feature. However, it is noted that further work is necessary for Loadlink offices, which will be addressed separately.

This incident has been officially closed with the team's agreement that a post-incident review was not necessary. Please refer to Security Team for additional details.

An issue has arisen wherein deleted loads are still visible on the Load Board and search page, raising concerns about compliance and operational visibility, particularly with fraudulent loads appearing. This problem impacts users who have reported that the deletion process, especially for loads deleted via bulk actions, is not functioning correctly. A temporary workaround has been implemented where the Discovery team manually removes postings from the production database. The incident was reported by Chetan Narasimhamurthy and involves discussions among several team members including Vince Molieri and Keegan Brown, highlighting potential root causes related to system communications and an AWS outage that has led to elevated error rates. Further analysis and permanent solutions are yet to be determined.

The incident report indicates that the high latency issue was resolved by provisioning a new reader instance to manage increased load, stabilizing the service. Ongoing monitoring and enhancements have also been implemented to ensure the solution's effectiveness.

Due to AWS ongoing outage, instances aren't able to auto-scale, causing a delay in message processing. DB is having intermittent network connection issues prevent data from being saved. Monitoring has shown stable, AWS has resolved the services outage in us-east-1 along with the ec2 instance issues. We've also processed all queued jobs that backed up during this event

Seems like aws IAM does not work correctly. Consumers could not connect to Kafka brokers The incident was resolved by restarting multiple Kafka consumer pods, which restored AWS IAM authentication functionality and resolved connectivity issues to Kafka brokers. The incident status was marked as \"resolved\" following successful mitigation and monitoring showed no further issues.

404 errors regarding amplify and cloudflare changes We've monitored for a few hours and the issue is resolved. The Amplify project was reconfigured again on a call with AWS and the issue resolved itself. There appears to be some internal issue with Amplify as the configuration itself didn't change. Will continue to follow up with an in-depth investigation and postmortem.

If the search trucks page is refreshed the search results disappear and the search results count changes to 0. Navigating away from search trucks and back restores the search results. The incident was resolved by implementing a fix that addressed the issue of disappearing search results upon page refresh, which was released to production. Additionally, a separate ticket was created to address the root cause, allowing the incident to be marked as resolved.

DAT ASSURANCE link included with a load post is broken somehow and generating an error - team is able to easily reproduce. The incident was resolved by identifying and fixing the issue with the DAT ASSURANCE link, specifically by updating how client-support replaced spaces in URLs, which restored link functionality. A JIRA ticket has been created to monitor the issue for future occurrences.

Multiple customer reports of loads seeming to hang on creation.

Some postings are showing as cancelled in CSR pages, but showing as exposed in SV4. This is likely related to a database change deployed yesterday. The incident was resolved by developing and deploying a script that successfully addressed the exposure issue with the affected postings and manually handling individual cases as needed. The resolution was completed at 12:05:54 AM UTC on 10/11/2025.

Appear to be having an issue with Company Search. It is not displaying results Company search is back in working state after the rollback completed in production.

Issue has cleared up after migrating to Let's Encrypt SSL cert instead of Google Trust Services or SSL.com

unknown yet how widespread the issue is, appears to be limited to D1Mobile and not D1Web. Inability to view lane rates in D1Mobile for some lanes Summary of this incident (let me know @matt.lang and/or @doug.burke and/or @jon.rich if you have anything else to add):
it can be closed. Mitigated by setting the feature-flag for using the new Rate Lookup API V1 temporarily to 0%
the net effect for our users was worse for the d1mobile app, less bad for one-web:
for lanes < 250 miles in length, users of d1mobile did not see any lane rates
for lanes < 250 miles in length, users of one-web see rates, and are only missing a fuelSurcharge in a float-over div
MOBS and ATM teams are deciding how to best solve the 'problem' whereby the new API omits fuelSurcharge 'per mile' for lanes <250 miles (it does still include a per-trip fuelSurcharge, so the BFFs could calculate the per-mile)

The BA_ANALYTICS freshness check failed — no new records were found from ba_files give a header to this The incident was resolved by confirming that all records that were previously not processed were now visible to customers, as they were passed through to the user interface. Additionally, the delay in data arrival was identified and logged for further investigation to prevent future occurrences.

This is another rootly training The incident was resolved by rolling back the change that caused the issue, as confirmed by Fab Herrera after monitoring the situation post-rollback. Fab Herrera also indicated that the incident was deemed resolved after the rollback was successfully completed and monitored.

The harvester job that collects billing data from the asset_post_txns table of the origami schema of the bapprod database for Extended Network Load postings has not run successfully since October 31, 204. The estimated revenue impact is approximately $10k. The current working theory is that the table associated with this data has grown so large that the harvester job is timing out and thus data is not being sent to Salesforce for billing.

This incident has been resolved.

The incident was resolved after confirmation that agents were no longer experiencing issues with making or answering calls, which Fab Herrera noted before mitigating the incident. While the situation will continue to be monitored for any updates from NICE, the immediate problem was addressed and declared resolved. NICE also confirmed Service Restoration and After 72 hours of monitoring and additional troubleshooting NICE believe the impact has been resolved.

The Truck Posting DocumentDB to kafka lambda has not run since we rotated the secret earlier this evening The incident was resolved by updating the password for the Lambda function, redeploying it, and processing all previously unprocessed records. The team confirmed a stable flow of data afterward, leading to the decision to mark the incident as resolved.

Currently facing an issue with the production ETL jobs due to a failure in the Wherescape scheduler. The incident was resolved by the Wherescape support team clearing the log files that contributed to the scheduler failures, which enabled the team to sequentially trigger the ETL jobs and successfully populate data to the UI. All jobs were reported to have run and completed, leading to the incident being marked as resolved.

one customer experiencing this issue The incident was resolved by rolling back the problematic change (CAB-009867), which was confirmed as effective by the customer. The status was updated to \"Resolved\" after this confirmation.

Postings on from the posting system are not meeting their api spec and are therefore being dropped from SV4. The incident was resolved by implementing a hotfix that adjusted the validation process, which eliminated the dropped postings and ensured compliance with the API specifications. Monitoring indicated stability post-fix, allowing the team to confirm the resolution.

So far 2 - 5 users are reporting the spinning wheel when trying to access DAT One Web The incident was resolved by turning off the feature flag for the User API, which allowed all user requests to be processed by the CSB instead, leading to some users successfully accessing DAT One Web. The status is marked as resolved, indicating that the issue has been addressed. Teams will create action items to follow up with a long-term solution.

Getting paged for low availability in load and truck posting The incident was resolved when both Load and Truck posting services returned to normal availability, as indicated by the alarm status changing to green. Incident was tracked for the following 2 hours with no issues.

When posting to the Private Network and DAT Loadboard, the post is going to the Private network only and has to be edited/updated to post the the public DAT loadboard/Extended network https://transcore.jira.com/jira/servicedesk/projects/CAB/queues/issue/CAB-10056

We are seeing a big drop in posting starting 9/12 in load_postings_stream_parse DWH table

The incident was resolved after the upstream DNS issues impacting Heroku were confirmed to be fully addressed, leading to normal operations for processing amendments and activations. Additionally, the team completed the cleanup of any affected quotes, indicating that all tasks related to the incident were finalized.

The daily census file that is consumed by the company system to insure the latest data is in place for consumers of the service is not available and therefore was not processed early this morning The incident was resolved by confirming that the missing census file from FMCSA was made available again and the data was successfully republished in the company's system. The issue was marked as mitigated while monitoring continued, and it was concluded that the incident could be closed when the data was confirmed to be updated.

Testing has shown that the CSB config changes have had the desired effect, functionality is restored

An incident occurred in SV4 that significantly delayed the posting updates ingestion, causing users to report that loads were not refreshing and displaying outdated information. The issue was attributed to a Kafka heartbeat timeout set too low, which led to repeated batch processing and system lag. Actions were taken to mitigate the situation, including shutting down the live match until recovery was seen, tuning Kafka configurations, and tuning notification service to improve performance. The incident was reported by Keegan Brown, and despite not having formally assigned roles for incident management, several team members collaborated to address the issue, with an anticipatory plan for a post-incident review. The incident was resolved by tuning Kafka configurations, adjusting timeout settings, and implementing circuit-breaking measures to manage incoming loads and performance. The team reported that the system performed well after these changes, indicating a successful resolution.

NPM packages in an upstream provider have been compromised. We need to validate if they are used in our environment and mitigate. After extensive searching and waiting to see if vulnerable packages were found on users’s machines, it was determined that none of these vulnerable packages was found in our environment.

Issue is occurring when trying to update a contact from SF to DAT Rep

The incident involving Trucker Tools relates to the system returning a series of 400 error responses to a subset of customers, primarily due to issues with requests routed through Cloudflare, which showed intermittent problems in certain regions while direct requests to CloudFront functioned properly. The mitigation involved reverting users to the previous stack while investigating the root cause, which included examining server headers and traffic logs that indicated errors when passing through Cloudflare. Steps taken included toggling Cloudflare’s proxy feature to validate its effect on error occurrence and engaging multiple team members, including Luis Diaz Ramirez, who reported the issue, and several responders who contributed to the analysis and resolution efforts. The incident status was marked as mitigated after confirming improvements in system behavior when Cloudflare was disabled, but further examination into addressing security postures, including the Web Application Firewall, continues.

• Meeting
• Severity: SEV3
• Incident Types: Default AWS Amplify applied to a fix to how they read ASCII characters in request headers.

Customers can update credit card details, but payments cannot be processed, receiving the message "payment could not be processed The incident was resolved by increasing the event delivery limits with Salesforce, which allowed synchronization to resume and previously held events to be replayed. This adjustment led to successful account reactivations and confirmed accurate account statuses during spot checks, marking the issue as resolved.

Due to a recent batch job that was run to update a large number of Account records in Salesforce, we have exceeded our daily limit of delivered platform events and change data capture events. This means that external subscribers to those events will no longer receive them until the rolling 24-hour reset occurs. This limit will recover gradually over the next 24 hours, and we expect full recovery by this time tomorrow.

Note: This does not impact internal Salesforce processing of those events, and does not affect inbound API calls or inbound delivery of those events to Salesforce. The incident was resolved by increasing the event delivery limits within Salesforce, which allowed for the synchronization to resume and previously held events to be replayed. Following this action, events began flowing through, confirming that account statuses were accurate.

Test

Issue identified and resolved

The incident involves the Multilane system on IQ and Rateview not providing results, affecting only one customer so far. The incident was resolved by rolling back recent updates that were causing issues with the Multilane system on IQ and Rateview, which restored functionality for pulling up reports. Communication continued to address residual issues, particularly with the delete function, but the primary incident was marked as resolved.

Keypoint customers have been unable to consistently connect to remote desktop services due to ongoing licensing and remote resource errors, which started on September 5, 2025, and have led to at least 40 active users being unable to log in following a reboot intended to resolve the issue. The original troubleshooting identified configuration issues within CrowdStrike security policies that were blocking RDP access for service accounts, prompting adjustments and a full reboot of the Remote Desktop Services environment. Despite initial improvements, the underlying problems persisted, leading to the reopening of the incident on September 6, and the team is continuing to investigate and monitor the situation while implementing further corrective actions.

• Meeting
• Severity: SEV1 (Major)
• Incident Types: Default The incident was resolved by implementing maintenance fixes across all RDP servers, which resulted in 526 users being successfully logged in, and no new issues were reported. The incident was ultimately classified as mitigated on September 22, 2025, after confirming stable operations with no further customer complaints.

customers reporting that their removed loads are still receiving calls, and load posting system cannot find any logs for deleting those postings. The incident was resolved by manually removing the affected postings from the SV4 database, ensuring they were no longer visible to customers, and confirmation was received that the calls to those postings had stopped. Follow-up actions for a permanent fix were planned, but the immediate customer impact was addressed.

A job responsible for extracting and processing BA_files—customer-supplied shipment and cost data—has failed. As a direct result, the BA Shipper production database has not been updated since August 27. This failure disrupts the daily data pipeline that translates raw customer files into actionable analytics within our shipper platform. there will be 100 shipper notice this issue tomorrow as today is holiday. The incident was resolved by deploying changes to fix the infrastructure issues that disrupted the data pipeline and successfully reprocessing the missed 6,134 files. Validation of the data confirmed that everything was functioning correctly after the reprocessing, leading to the incident being officially closed.

An incident has arisen due to the deletion of the user account TQLBHughes during a user cleanup process, resulting in complications in recreating the account. This issue is affecting operations for the stakeholder involved. The incident was resolved by successfully restoring the TQLBHughes user account, which became active in the system following modifications made to the user status. The customer confirmed that they were no longer experiencing issues, and the NOC will continue to monitor the situation.

Koola Logistics is not seeing the "days to pay" and "credit score" attribute showing in Search V4. This appears to be due to an account that was deleted and recreated, causing a data integrity issue. 1 account and apx. 65 users are impacted with this feature not being available.

Customers are currently unable to receive multi-factor authentication (MFA) codes via text message due to issues with the SMS provider, Twilio, which is causing delays and failures in code delivery. The incident was reported by Miriel Robles and is classified as Severity 2. The team, including responders such as Evie May, Miles Johnson, and others, is working with Twilio to resolve the issue, while customers are advised to use alternative authentication methods. 3rd party provider resolved the incident

We are experiencing network connectivity issues in Hillsboro. One of the leaf switches has gone down, impacting host hilcsbcore02-a. While other servers remain online due to redundant connections, the affected server is unreachable (no SSH access). The issue appears isolated but is affecting production connectivity. The incident was resolved by physically swapping network cables on Core02 and replacing a failed fiber cable on Core03, restoring functionality to both servers. Additionally, the leaf switch was replaced, ensuring the network was fully operational again.

3 customers are impacted since yesterday for data being now shows for last two days. company id value in SF is not present , a hotfix was rolled out but we are getting null in the company id for those records. documenting with incident to investigate further.

The incident was resolved when the issue was corrected, as indicated by the message stating \"issue solved, test is done.\" Additionally, a Post-Incident Review meeting has been scheduled for further evaluation.

The incident was resolved by identifying connection pool errors as the likely cause of the latency, and the Salesforce team provided documentation on avoiding such issues in the future.

The TriHaul service is currently down, affecting users with subscriptions by preventing access to additional features, although they can still search for regular loads. This incident was triggered by alerts indicating that no data is being returned from the CSB when utilizing the Trucker API for TriHaul data. As of now, the team is investigating the issue, and a potential cause linked to a change (CAB-9828) has been reverted. Deborah Haynes is leading the incident management efforts while Dugan Kniesteadt is actively responding to the situation. The incident was resolved by reverting a previous change and restarting the necessary services, which ultimately restored access to the TriHaul features for users with subscriptions. The team's ongoing collaboration and troubleshooting efforts led to a successful fix, as confirmed by user reports of restored functionality.

Can some declare an incident for us, postings ingestion is delayed in Search V4

Commercial Partners: MercuryGate
In-House Partner: Keypoint

Retrieval of CarrierWatch data is failing for all SOAP integrated customers using Keypoint or MercuryGate The incident report indicates that the issue was resolved after reverting a previous change (CAB-9820) that had pointed the service to a reverse proxy; confirmation was sought from affected customers to ensure the problem had stopped. Additionally, the incident status is marked as resolved, confirming the situation was addressed.

After a EKS update from 1.32 to 1.33, we have found the datadog pods crashing with the error:
2025-08-16 02:15:40 UTC | CORE | ERROR | (comp/aggregator/demultiplexer/demultiplexerimpl/demultiplexer.go:81 in newDemultiplexer) | Error while getting hostname, exiting: unable to reliably determine the host name. You can define one in the agent config file or in your hosts file
Error: Error while getting hostname, exiting: unable to reliably determine the host name. You can define one in the agent config file or in your hosts file

Some users are experiencing intermittent issues with searching for loads, categorized as a SEV2 incident that began on August 11, 2025, at 5:07 PM UTC, triggered by an alert from the workflow. The lack of an assigned incident commander or responders has impeded immediate resolution efforts, and no tasks have been assigned for managing the incident or conducting a post-incident review. Deborah Haynes reported the incident and noted that the functionality was restored briefly but then broke again shortly after. This is now resolved

We had a data issue on 7/18 where not all data was loaded into the Data warehouse table. It also occurred on 8/1. It was recently identified and fixed. The incident was resolved as noted in the report, indicating that the data loading issue was identified and fixed. There are no specific action items assigned to individuals.

Issue is now resolved