DAT Solutions Internal

Summary: The FMIC AWS FSx file share \pawswfmcalx01\share (file system fs-050a21643fdfc8a3b, prod-analytics account) became inaccessible after the weekly FSx maintenance window on Sat 17 Oct 2026. The L: and N: drives were unavailable, the scheduled FMIC List runners could not run, and the weekly Legacy BA Portal and iFMI API releases were delayed. Platform on-call retried FSx Misconfigured State Recovery, and the share returned to Available at 09:37 UTC on 19 Oct (15:07 IST / 02:37 PDT). Total outage: roughly 50 hours.

Customer impact: Per Ashwini M V (Analytics), iQ dashboard processing was blocked for customers who submit data on Sundays (her estimate: about 75-80% of those customers). Workflows and List runners are running normally again. Customers who submitted on Sunday will see their iQ dashboards updated on Tuesday instead of Monday, a one-day delay. Weekly releases were delayed by the N: drive outage.

Detection: First reported in the platform Slack channel at 10:42 IST on 19 Oct (about 22:12 PDT on 18 Oct) after the FMIC/Analytics team noticed the share was down. FSx's last contact with the domain controllers was 17 Oct 07:39 UTC, so the problem went undetected for about two days. No alert fired.

Timeline (UTC):
- 17 Oct 07:30 - weekly FSx maintenance window starts
- 17 Oct 07:39 - FSx's last contact with the domain controllers
- 19 Oct 05:12 - outage reported (10:42 IST)
- 19 Oct 08:14 - Platform on-call (Erick Stubbs) retries Misconfigured State Recovery
- 19 Oct 08:26 - FSx resumes contacting the domain controllers
- 19 Oct 09:37 - FSx Available; N: and L: confirmed working
- 19 Oct 09:42 - incident marked mitigated
- [time] - incident resolved after monitoring

What was ruled out: Network path to the DCs (Reachability Analyzer and VPC flow logs; no network changes since 14 Oct). The AWSSync service account (works, and the Finance FSx uses it without issues). The FSx computer object (present). DC03, DC04, and root DC security logs (no failures). An early suspicion about a missing route was not supported.

Root cause: Not confirmed. Working theory: the file server did not rejoin Active Directory after the 17 Oct maintenance. AWS Support case [number] is open for confirmation. INC-1273 (May 2026) may have started at the same Saturday maintenance window, but our records for that incident conflict.

Resolution: Platform retried FSx Misconfigured State Recovery. FSx resumed contacting the domain controllers and returned to Available. Analytics confirmed L: and N: work and the workflows and List runners run as expected. Stable since 02:37 PDT, monitored until [time]. Resolution: Platform retried FSx Misconfigured State Recovery. FSx resumed contacting the domain controllers and returned to Available. Analytics confirmed L: and N: work and the workflows and List runners run as expected. Stable since 02:37 PDT

This incident has been resolved.

Customers report search table glitching while scrolling The incident was mitigated at 11:35 MDT on 2026-10-16. The report does not specify the exact remediation actions or the root cause; it only notes mitigation and that customer impact was present.

This incident has been resolved.

This incident has been resolved.

DAT One Mobile users are not seeing their saved loads in My Loads nor are they able to save loads to the worklist. Mitigation completed on 2026-10-05 at 11:28 MDT; the incident status is mitigated. The record does not specify the remediation steps or confirm full resolution, and no follow-ups are listed—please review the PIR to verify closure and customer impact.

Root cause: ingress, not company-service. Six emissary pods (replicaset emissary-ingress-67c85f7574: j24gx, 47pq2, fwncr, 4dd4h, 778qg, ncmn4) all run on one node, ip-10-248-140-130 (c5.4xlarge). That node became CPU-bound (system.cpu.user ~45% → ~76%, emissary CPU ~6.5 → ~12.5 cores, no restarts), and those pods got slow (p90 80–290ms).
- Evidence it's not the service:
- Latency rose evenly across all ~40 company-service pods.
- Most of the time is spent before the request reaches the service: p90 duration ~110ms vs upstream_service_time ~35ms.
- The same 15 pods of that replicaset on other nodes stayed normal (17–27ms).
- Blast radius: every service routed through those pods got slower: company-service (18 → 82ms p90), office-service (21 → 115ms), association-service (23 → 165ms), network-tagging-system (15 → 95ms), client-support. Services routed through other emissary replicasets were not affected.
- Likely trigger: traffic shifted onto the pods on that node (~177k vs ~95k req/min), and the node ran out of CPU.
- Action (platform / cloud_engineering): cordon/drain ip-10-248-140-130 or move the emissary pods off it. Add pod anti-affinity / topologySpreadConstraints so emissary replicas spread across nodes.
- company-service: no action needed.

SSU IS IMAPCTED FOR CUSTOMERS

Elevated 5xx errors on network-tagging-system via emissary (monitor network-tagging-system:emissary:500Errors flapped 7x through the day, peaking at 2,801 matched error events at 11:10 EDT), plus degraded scheduling/availability for office-service and company-service in network-prod. Mitigation achieved at 10:22 MDT on 2026-09-30, with no customer impact reported. The report does not specify the exact remediation steps or final root cause.

Falcon Overwatch has identified malicious activity Medium detection on PAWSWKPAGW01 will cause interruptions. Account disabled. Resetting customer password.

Kafka lag for Load and Truck Posting enrichment has been growing since ~5:30am PT, causing elevated posting enrichment duration. I am also seeing elevated latency from downstream services in the network domain (Company and Office service)

Custmers are reporting dat is unable to load

Customers on the legacy version of Trendlines are seeing Reefer rate data regardless of the equipment type (Van, Flatbed, etc.) selected from the dropdown. Investigation traced this to requests hitting the new /trendlines/{equipment}/... URLs, which return a 301 redirect (X-Redirect-By: WordPress) to the /trendlines-legacy/... structure — the new path lacks the matching WordPress parent/child page hierarchy, triggering a canonical redirect. Responses are cached by Pantheon/Varnish, so the cache will need purging after the fix

Insurance certificates are not publishing to legacy CarrierWatch, impacting all brokers subscribed to CarrierWatch. Certificates worked via both OCR and Manual DE are getting stuck — new InsuranceCertificate records are being created, but the underlying Document data isn't saving. Issue appears tied to a recent server reboot: two services are showing red in the management console post-reboot, and users are intermittently seeing "unable to save insurance update" errors when connecting to TIM The incident is marked as mitigated as of 2026-09-23 15:56 MDT. It appears linked to a recent server reboot that left two services red and caused “unable to save insurance update” errors; the report does not specify the exact remediation steps taken to fully resolve the issue.

ISSUE:
Carrier authority updates made in Motus (source of truth) are not syncing to
CarrierWatch. Carriers marked as "active" in Motus show as "inactive" in CarrierWatch, creating a data integrity issue for all affected carriers.

CUSTOMER IMPACT:
- Affected: All carrier customers who have updated their authority status in Motus
- What they see: Incorrect authority status (active/inactive mismatch) in CarrierWatch
- Business impact: Data integrity issue affecting carrier searchability and compliance visibility
- Workaround: NONE — Support team cannot manually correct via CSR/CarrierWatch UI because
CarrierWatch pulls 100% of its data from Safer, not directly from Motus.

ROOT CAUSE:
Unknown. suspected Motus to Safer sync pipeline failure or missing sync logic

Three customers have reported failing to create tracks from their MercuryGate TMS. Initial findings appear to be related to infrastructure changes that are causing a legacy MercuryGate API endpoint https://loadtracking.truckertools.com/loadTrack/loadTrackCreateJSONService.php to 404 Fix was implemented and all API traffic to the legacy MG endpoint was showing a HTTP 200 OK after prod release for the nginx rewrite rules

SOAP API users are unable to set a password — temp password fails validation. Reported by Dave McClure, who also encountered this at the McLeod user conference. 2 accounts affected so far

Unable to login to key point

Customer subscription renewal automation is completely non-functional. Renewals are not being triggered for affected accounts, causing customers to lose active service access. All affected accounts are on credit card billing (no recent changes). Support team cannot manually repair contracts or add services. Issue first reported 17:33 IST, scope expanding (now 4+ accounts: 797083, 797106, 755868, 763659). Revenue at immediate risk.

recent change deactivated automation impacting signup. and sales workflows.case management Resolution summary

• The incident was resolved by re-enabling Salesforce integrations. During a review of OwnBackup backups and tests, an option to disable Salesforce integrations was inadvertently checked. Re-enabling the integrations fixed downstream errors, and the incident was mitigated at 16:11 MDT on 2026-09-14.

Root cause

• A configuration change during backup/testing caused Salesforce integrations to be disabled, which deactivated automation affecting signup and sales workflows.

Timeline highlights

• 2026-09-14 15:55 MDT: Incident created (alert from workflow).
• 2026-09-14 16:07 MDT: Salesforce team reviewed backups and ran tests; an option to disable integrations was enabled.
• 2026-09-14 16:11 MDT: Integrations re-enabled; downstream errors fixed. Incident mitigated.

Impact

• Automation impacted (signup and sales workflows). Customer impact noted as Yes in incident fields.

Action Items

If you’d like, I can draft the PIR (Post-Incident Report) outline or help schedule the PIR meeting.

Box has stopped sending us webhook events since Sep 1

During TT Salesforce migration (Sept 11-12), 62 active customer broker accounts
lost MC/DOT data. These fields are required for load postings to appear to carriers.

• Spike: 68 affected accounts (23x normal; 62 active, 6 test)
• Root Cause: TT data migration — MC/DOT field mapping failed
• Customer Impact: Brokers cannot post loads; carriers won't see postings
• Complaints: 1 confirmed; more expected
• Workaround: None (MC is required)

Last modified by: LeanData Integration (09/11), TT Data Migration (09/11),
Integration Admin (09/12)

The authentication service is failing to generate valid tokens, preventing
API requests from the new Trendlines page from completing successfully.
Data fails to load on the page UI. This is blocking today's production
deployment of the Trendlines feature.

Initial Investigation: Token generation issue identified on auth side.
Impact: Trendlines page returns no data; feature cannot ship to production today.

AWS is reporting an issue with STS in the us-west-2 region. As many services rely on this endpoint to access other AWS resources, we should expect customer impact. Some services are already showing increased latency/timeouts After monitoring traffic during our typical morning ramp up we saw no noticeable impact, did find some remaining services that needed a restart. Resuming normal monitoring stance.

During a routine deployment, the secret "/freight-negotiation/prod/service/client-api-keys/" was accidentally deleted/recreated by terraform.

The scheduled maintenance has been completed.

At 3PM MT, we detected that Box.com began returning APIErrors for upload requests. The last Api Error we received was at 3:21PM MT. This is confirmed by the status page for box.com (https://status.box.com). Will continue to monitor for service health. 3rd party site resolved their incident. Box.com and are continuing to monitor https://status.box.com/

Users are getting logged out of rate view

Johns Manville (AN#0001670298) reports that their entire team is receiving a redirection notification when accessing DAT iQ, landing on a Dashboard view overlaid with an "Upgrade to IQ" prompt instead of their subscribed product content. This pattern matches prior cases where DAT iQ's capability/entitlement-based routing misfired due to a feature flag or entitlement-sync issue rather than a service outage The dashboard_ssu_enabled feature flag, shipped in CAB-11986 to gate a DAT Bid SSU upsell modal to internal users only, was misconfigured and enabled for all users, blocking real customers (Johns Manville, Smithfield Foods, and USA Truck users, among others) from their DAT iQ Dashboard behind an upgrade modal. Fix: Flag disabled by Orr Epstein — no code rollback needed and the incident marked as Mitigated.
customer confirmed that their services are restored and working as expected. PIR will be scheduled upon full PIR documentation to address staging test-coverage gaps and modal-targeting logic.

Multiple customers are getting 503 Server unavailable errors when trying to sync or create load tracks.

The Preferred & Blocked Service has degraded performance after upgrading the Preferred & Blocked DocumentDB from 3.x to 5.0 The Preferred & Blocked Service experienced elevated latency after the production DocumentDB 3.x to 5.0 upgrade. Investigation identified an issue in the count wrapper invocation: the query filter and index hint were passed in the wrong argument positions, preventing the count query from using the intended index efficiently.

The wrapper was corrected so the filter is passed as the query and the index hint is passed through the options argument. The fix was deployed to production. After deployment, preferred and blocked list requests returned normally and the service response time improved.

Datadog validation for the reviewed production window showed 0 error logs and 0 warning logs, with normal info-level activity. CloudWatch and AWS-side metrics could not be independently validated due to access was unavailable; this does not block resolution but remains a follow-up validation item.

Follow-up items:
Confirm CloudWatch and DocumentDB Performance Insights metrics after the observation window.
Run and retain a production explain for the office-ID query to confirm the intended index is being used.
Remove the temporary Performance Insights access/configuration after the agreed observation window.
Add regression coverage for the count-wrapper query and hint argument shape, including the $in path.
Complete the PIR.

All BIPD-related admin reports in CarrierWatch (Cancelled BIPD, No BIPD, Bad BIPD Policy) are returning HTTP 500 errors due to a JSP compilation failure on the CarrierWatch Tomcat application. The Reporter class cannot be resolved at runtime, causing the /reports/cancelledBipd.jsp (and likely related BIPD report pages) to fail to render. This prevents admin and operations staff from running insurance compliance reports used to identify and remove carriers with inactive or cancelled BIPD insurance policiess Commit 9d8ce6b6 (CSB-2863 treg logging, PR #1254) deleted treg/src/java/com/tcore/treg/Reporter.java from the source tree. Nine BIPD report JSPs continued to reference new Reporter() via import com.tcore.treg.*. Because JSP scriptlets compile lazily at first HTTP request rather than at build time, the Java-only CI compile remained green and the missing class was not detected until the report pages were accessed in production, resulting in HTTP 500 errors across all affected BIPD report endpoints.

Resolution:
Reporter.java was restored from the pre-deletion commit (9d8ce6b6^) and its internal logging dependency was migrated from the deleted com.tcore.log.Logger to slf4j/lombok. The restored file was reintroduced to the source tree, a hotfix jar was produced, and the patch was hot-deployed to the CarrierWatch Tomcat (treg) host without a full stack redeployment. All affected BIPD report pages were verified as operational following the patch.

What Changed:

Restored Reporter.java to the treg/src/java/com/tcore/treg/ source tree
Migrated logging from com.tcore.log.Logger → slf4j/lombok

A PIR will be documented and action items will be assigned.

CSR support staff (Miriel Robles) reported errors when trying to search for users on CSR (initially thought to also affect DAT Rep, but DAT Rep was confirmed working by Mohamed Suhail Z and Akash Roshan). Miriel confirmed CSR itself was down for the team ("Its just csr that's down for now"), impacting the ability to search users, add/remove seats, and manage NA account services. Support noted downstream effects: users stuck in free demo status may not be creatable, and seat discrepancies between DATrep/CSR can't be troubleshot.

Datadog logs confirm active 500 errors on rest.dat.com:8443/TcoreRequest/rest/auth/externalIdentity/csbId/{id}?token=csrpages, consistent with the Auth0Api layer failing, likely due to a null client_id (java.lang.IllegalArgumentException: 'client id' cannot be null at Auth0Api init).

Timing coincides with the CSB release deployed the evening of Aug 4 (build 654 / CAB-11966), which included CSR Pages code changes and logged auth-service restarts during rollout. Troy Folger noted the release was tested and working immediately after deploy last night, so the correlation is not yet confirmed — root cause is still under investigation with CSB on-call Incident mitigated at 08:04 MDT. Root cause was a missing auth0.mgmt.clientId value in the production G-stack AWS Secrets Manager compound secret, inadvertently omitted during the 8/4 CSB Cascade-git release (Build 654). This caused the Auth0Api layer to fail initialization, resulting in 500 errors on the CSR Pages REST path and blocking user search/creation/deletion.

Mitigation steps:

Identified the missing secret key via Datadog log analysis (null client_id pattern on rest.dat.com externalIdentity endpoint)

Govinda (SRE) and Troy (CSB) restored the missing value in AWS Secrets Manager

CSB directory and CSR Pages restarted to pick up the updated secret

Functionality verified by CS reps post-restart

Follow-up: PIR document in progress. Post-release smoke test automation for CSB-dependent endpoints to be tracked as an action item.

classification tagging effecting shares Root cause: Snowflake masking/tagging policy changes introduced dangling tag references on shared DB objects, breaking the morning replication/refresh to customer shares and emptying the Broker Benchmark table in iQ.

Resolution: Tags were rolled back; the database refresh was manually rerun across all affected regions (AWS us-east-2, Azure East 2). Broker Benchmark recovered (~46M rows, iQ access restored). Brindha ran the Shipper refresh manually; completed and UI confirmed updated with new records. Raven did not run during the affected window and was unaffected.

All customer-facing data confirmed restored. Monitoring follow-ups (zero-row ETL alerts, tagging guardrails, PIR) tracked as separate action items.

Trendlines is displaying incorrect month-over-month and year-over-year percentage changes for the spot-rate metrics. The calculation currently uses only the base USD rate and excludes the fuel surcharge, although the spot rate should include both components. This produces inaccurate trend percentages, including the July 2026 comparisons against June 2026 and July 2025. The issue is longstanding in the Trendlines calculation logic and is unrelated to the load-to-truck ratio issue. Resolution: Trendlines calculation errors (inc-1336)

• Status: Mitigated at 2026-08-03 08:42 MDT
• Root cause: Trendlines was calculating MoM and YoY changes using only the base rate, excluding the fuel surcharge. The complete spot rate (base + surcharge) should be used.
• What was done (mitigation):
  • Manually retrieved corrected values from the Linehaul API and updated DynamoDB.
  • Focused backfill on key comparison months: June 2026, July 2026, and July 2025.
  • Estimated initial correction completion: 2–3 hours.
  • Backfilled remaining historical data the following day.
• Permanent fix plan:
  • Implement, test, and document a backend change to use the complete spot rate (base + surcharge) for MoM and YoY calculations.
  • Add regression validation for the next monthly data load (van, flatbed, reefer).
  • Targeted completion: backend logic corrected and deployed within ~1 week; next monthly calculation on Sept 1 using the corrected logic.
• Impact to customers: No customer impact reported. Action Items (as listed in the incident)

Negotiation Consumer Service started to error out with MSK connection errors. Summary: negotiation-service-consumer lost connectivity to MSK broker 2 (negotiation-prod-kafka) starting ~8:34 PM PDT Aug 2, correlating with a CPU spike to 92.8% on broker 2 at ~9:11 PM (brokers 1 and 3 stayed normal the whole time). No customer-facing impact was confirmed and no other services on the cluster showed alerts.

Mitigation: Broker 2 was manually rebooted at 1:52 AM PDT Aug 3. Connection errors (ETIMEDOUT, BrokerPool timeouts) stopped immediately after and have not recurred since, over 24 hours of monitoring.

Root cause: Not fully confirmed, but the evidence points strongly to a JVM/GC-driven acceptor thread hang on broker 2. Eli's Datadog analysis found a ~30 min metric reporting gap (8:32-9:06 PM) consistent with a severe JVM stall, followed by broker 2 staying alive for replication but unable to accept new client connections until the reboot cleared it. Traffic was normal/low at the time, ruling out a load-driven spike. Log compaction, ISR catch-up, and leader rebalance remain unruled-out alternatives. Follow-up questions are still open with Platform (broker Kafka version, GC logs, aws.kafka.request_handler_avg_idle_percent for that window) to pin this down, but that investigation will continue as a follow-up rather than blocking resolution given the sustained recovery.

Tracked follow-ups already filed:

OS-3276: Discovery adding MSK broker-health alerting for negotiation-prod-kafka (this incident only had symptom-level alerting, not broker health).

Orient Express alert-routing cleanup: several negotiation-service alert routes were still pointing at the deactivated OE schedule instead of VA/Discovery. Vince and Alastair are reassigning these.

Thanks to Eli Hurst and Chathura Bamunusinghe for the quick work overnight and the follow-through this morning