DAT Solutions Internal

TIM became unusable for all reported agents. Users cannot save notes or other changes, including work on OCR exceptions. Insurance certificates processed successfully by OCR are not appearing in TIM and are consequently not being published to CarrierWatch.

Customer-impacting examples include:

Dayas Freight Carrier LLC — DOT 2436684

Rodolfo Jasso Jr. — DOT 2794852

OCR returned success for both certificates, but neither appeared in TIM or CarrierWatch. This is blocking insurance-certificate processing and publication for Legacy CarrierWatch customers on month-end. Resolution summary:

• The TIM/SQL failure was mitigated and end-to-end certificate processing was restored for OCR → TIM → CarrierWatch.
• A backlog of approximately 64 affected OCR batches was identified. The plan is to re-submit these via the DICE hot-folder replay process and reprocess in a controlled manner.
• Manual replay tests through the DICE hot folder succeeded, confirming the recovery path.

What happened (highlights)-

• Root causes identified:
  • OCR processing server (bvtocr03.tcore.com) disk full (149 GB C: drive).
  • SQL log drive (E:) nearly exhausted; backups hadn’t run since 06/18, contributing to disk pressure.
• Immediate actions:
  • Freeed disk space and restarted the OCR server (07:09 PT).
  • Restarted the SQL Server Agent on philwcrbsql04 and started backups; expanded E: drive by 50 GB.
• Recovery steps:
  • New certificate processing validated end-to-end (test certificate published).
  • Previously processed certificates that failed did not automatically replay; initiated manual replay via DICE hot-folder.
  • A manual replay test succeeded through OCR, TIM, and CarrierWatch.
• services have been restored and stable.

The scheduled maintenance has been completed.

some searches are failing with high latency to downstream services Root Cause

ScaleOps rollout in freight-prod (2026-07-24) reduced the cluster's node count over the weekend, bin-packing search-service pods more tightly. When Monday's traffic spike hit, search-service was heavily CPU throttled and couldn't scale horizontally in time, causing health-check failures and cascading pod restarts. This was compounded by Emissary circuit breakers and upstream connection pressure from the network-tagging call rollout (ramped to 100% on 7/14).

Key Mitigations

Set rollout percentages for private-network, network-membership, network-ownership, and book-now to 0% to cut traffic to the affected upstream path.

Disabled qualification claims and account-company-mapping calls in prod search-service to reduce hanging external calls and CPU/heap pressure.

Raised search-service autoscaling ceiling (75 → 120) and CPU request/limit (3100m → 4000m → 4500m).

Temporarily disabled readiness/liveness probes to stop Kubernetes from killing pods during elevated latency.

Increased Emissary capacity in freight-prod and re-enabled network-domain calls once traffic stabilized.

Rolled back the Datadog Cluster Agent upgrade to reduce circuit-breaker events in Emissary/Freight paths.

Follow-Up / Hardening (tracked post-incident)

Harden search-service to scale horizontally more effectively and avoid limit-based throttling.

Add PriorityClass support (helm-node chart PR open) to protect workloads from eviction, replacing the prior Guaranteed-QoS approach.

services have been restored and stable.

BB team reported that iQ BA for Broker is experiencing random database connection failures in production.
This appears to be an intermittent availability/degradation issue affecting broker workflows, but exact customer impact, affected services, and trigger conditions are not yet confirmed

The marketing site for Outgo.com was compromised, possibly through a developer/contractor credential phishing campaign. The compromised site is redirecting users to a phishing domain that hosts a malicious Android APK.

freight cluster cpu usage is high

There was a major interruption in network services that affected multiple 3rd Party services including AWS and Cloudflare.

This incident has been resolved.

This incident has been resolved.

Posting enrichment service lag is increasing sharply due to company and office system latency.

An external client in AWS east is making >10M requests/month to one-web-bff. Fraud activity that is known.

checking Gmeet compatibility with rootly agent

test incident

For some number of hours google maps was returning a 403 while broker-service was trying to calculate pricing for carrier matches. This led to the convoy fee not being deducted properly and frieght matching over max pay for some brokers.

Internal and external users are reporting. D1W isn't allowing anyone to post loads, also getting 503 errors being reported via TMS postings.

Multiple standard customer accounts are reporting inability to pay bills or add a card in the self-service billing flow. Customer-facing reports in #customer-impact indicate repeated failures with the message Card could not be added, including multiple calls and at least five affected accounts identified so far: 04095728, 04095754, 04095753, 04095759, and 0002420578.

Datadog evidence shows a shared backend failure in payment-service during Salesforce payment-method creation:

Error upserting payment method: Failed to create payment method in Salesforce for account 001f400001M2VpgAAF: [{"statusCode":"INVALID_FIELD","message":"No such column 'blng_BinLookupStatusc' on sobject of type blngPaymentMethod_c","fields":[]}]

This suggests the issue is not limited to front-end user entry alone and may be caused by a Salesforce schema / field availability / deployment mismatch in the billing-payment integration path. The customer-facing Credit card could not be added message is known to be generic and can mask multiple backend failure reasons, so the Datadog error is currently the strongest technical signal. The incident was caused by an incorrect Salesforce field API name used in payment service during payment method creation, which resulted in backend failures and surfaced to customers as inability to add a card or pay bills. The issue correlated with the CAB change. Engineering corrected the field reference, pushed the fix through CI/CD, and deployed the update to production. After deployment, the Datadog error stopped reproducing, and no new customer reports were observed, so the incident has been resolved.

Several large accounts reporting they are unable to post, repost or copy loads via the UI.

Convoy Kingpin is out

Several of your pods are in crash loop backoff and lookup requests are timing out.

Multiple reports indicate that customers successfully completed self-service reactivation payments, but the account reactivation did not complete as expected. Not resolved yet.

What the provided report shows
- Status: started
- No resolution or root-cause identified
- Action items exist, all unassigned with medium priority
- Schedule Post-Incident Review (PIR) meeting
- Ensure PIR document is complete and up to date
- Capture current impact to customers
- No follow-ups recorded

Recommended next steps
- Assign owners to the three action items and set due dates
- Schedule a PIR meeting (e.g., within the next 1–2 days)
- Complete the PIR document with incident timeline, impact, contributing factors, and proposed preventive actions
- Collect and document current customer impact (number of affected accounts, revenue impact if available, service-level effects)
- Begin a root-cause investigation (start with possible payment self-service flow issues, backend reconciliation, and any async processing)
- Prepare a communication plan for customers and internal stakeholders

Would you like me to create these tasks in Rootly, assign owners, and schedule the PIR meeting? If yes, please provide:
- Who should own each item (or should I assign to on-call / incident lead)
- Desired PIR date/time
- Any initial root-cause hypotheses you want documented

Current context note: inc-1316 (Delays in reactivations after successful self-service payments) remains open until these follow-ups are completed.

DAT is experiencing much lower clicks on their loadboard and Convoy is experiencing much lower views. This could be a technical issue, or a broader industry issue. Lets find out! Short answer: there is no formal resolution documented yet in this incident. The report shows a downgrade from Sev-2 to Sev-3 and ongoing investigation, with specific notes thatDAT-related data issues are being handled separately and a potential link to a June 16th event flagged for further investigation. Action items exist (PIR document, PIR meeting, and capturing current customer impact) but have not been completed.

Key points:
- Status downgraded from Sev-2 to Sev-3 due to Datadog data showing no significant active carrier drop; DAT data pipeline issue to be addressed separately.
- Convoy-related investigation still needed; possible connection to prior June 16 event noted for investigation.
- No final root cause or resolution documented.
- Open actions: complete PIR document, schedule Post-Incident Review meeting, capture current impact.

If you want, I can help draft a PIR agenda, create follow-up tasks, or schedule the Post-Incident Review.

Reports from TQL that users can't use multi-shipments, able to replicate as well. Summary of resolution

• What happened: A defect in the cargo feature flag service caused upstream connection errors and OOM issues, leading to failures for multi-shipments via D1W and internal DAT users. 429s and SDK queue growth contributed to degraded web/app experience.

• How it was resolved:

  • Implemented a Breaking Glass workaround to reset feature flag timing and restart the feature flag service.
  • Merged and deployed fixes:
  • FES-1227 (feature flag timing reset)
  • Cargo PR 975 and gitops Freight PR 7504 to apply the fix in production
  • Deploy workflow:
  • Deploy to test, promote to staging, then promote to prod
  • Configuration fix:
  • Corrected flag configurations for bulk upload, addressing misconfig detected during rollout
  • Additional stability: PR 7518 to reduce update time for Google Sheets-related updates to improve responsiveness

• Result: Incident mitigated at 2026-07-01 09:05 MDT. The changes stabilized the service, and the incident status was set to mitigated. Follow-ups were planned (PIR scheduling, PIR doc updates, and customer impact capture).

• Next steps (post-incident):

  • Schedule and run the Post-Incident Review
  • Complete PIR documentation
  • Capture current customer impact and distribute findings
  • Monitor for any residual issues and confirm stability in prod

We continue to see data inconsistencies with FMSA daily census files after the rollout of Motus. The main issue is carriers that appear to be reactivated are only showing up in Motus as such and not in the Safer daily census file. This incident is mainly for tracking the issue is the FMCSA continues to work on it.

The reactivation team is not seeing non-pay / credit-card-decline cancels on the daily list. Initial review indicates the records may be stuck in progress, and the attached Salesforce report filtered to Credit Card Decline for this week shows no new opportunities for the last two days, which is unusual.
This is likely customer impacting for customers attempting self-reactivation after a non-pay cancel; if the underlying Salesforce/reactivation flow is failing, customers may be unable to complete self-reactivation and may instead be directed to contact support. Resolution summary

• Root cause: A Salesforce email template update on 6/24 caused failed payment transactions to become stuck in an In Progress state, preventing notification emails and disrupting the reactivation flow.

• Mitigation and resolution: Reverted the Salesforce email template back to the original version, enabling failed-payment emails to resume and allowing transactions to move out of In Progress. This mitigated the issue and halted the customer impact caused by the stuck records.

• Scope and impact: Approximately 669 accounts were impacted (accounts receiving free services) due to the stalled transactions. The incident status is MITIGATED as of 2026-06-26 14:43 MDT.

• Next steps / remediation plan:

  • Review the 669 impacted accounts to identify which have already paid and exclude them from further action.
  • For remaining accounts, choose a remediation path:
  • Option A: Retry payment and, if it fails, cancel the service.
  • Option B: Immediate cancellation for all remaining impacted accounts.

• Follow-ups and actions created:

  • Schedule Post-Incident Review meeting
  • Ensure PIR (Post-Incident Report) document is complete and up to date
  • Capture current impact to customers

Current status: mitigated (mitigated_at 2026-06-26 14:43 MDT).

Unable to get this certificates in to Dice. Both certs@dat.com and using our Hot Folder are working to get this certificate through to Dice. Additionally this time of month we normally have a lot of certs in Dice to process and we are at 0. From the provided incident report, there is no resolution documented. The incident status is started, with no root cause, corrective actions, or closure notes, and follow-ups are empty. The only documented action items are post-incident review tasks.

What’s needed to identify the resolution:
- Obtain updates from engineering on whether the Legacy Carrierwatch certs are now being processed by DICE (and if not, what workaround or fix was implemented).
- Confirm any root cause and corrective actions taken (e.g., changes to filters, hot folder processing, or ingestion pipeline).
- Update the incident with the final resolution, root cause, containment, and verification steps.

Suggested next steps (I can help execute these if you’d like):
- Schedule a Post-Incident Review and assign owners for the PIR.
- Ensure the PIR document is completed and up to date.
- Capture current customer impact and confirm if it’s fully mitigated or still partial.
- When resolution is known, close the incident and publish the PIR findings.

Would you like me to create/schedule the PIR, assign the tasks, and ping the on-call/eng team for an update? If you have any recent notes or updates from engineering, share them and I can draft an incident resolution summary.

Certain components on the iQ dashboard are not displaying.

Keypoint can no longer import carriers via CarrierWatch (SOAP) using an MC number.  Keypoint can import some information using a DOT number, but not insurance information.  At this point Keypoint cannot get any CarrierWatch information into the system via the SOAP interface.

Users are unable to sign in. They're getting a "Your user account is not associated with any carriers." message.

A couple examples:
Carrier MC 158069 sapatel@duprelogistics.com (Broker / Carrier on Safer and active)
Carrier MC  494689 naaf@nalogistics.com (CARRIER on safer and active) Here’s how the incident was resolved according to the report:

• Resolution status: Mitigated
• Mitigation timestamp: 2026-06-24 08:51:38 MDT
• Scope: The impact appeared limited to Ryder Private Load Board (PLB) users; no broader TT platform outage was observed.
• What happened (summary): Users were unable to sign in with the error “Your user account is not associated with any carriers.” The leading theory tracked to post-authentication: users obtain an auth/Auth0 token, but the subsequent hydration of user data fails when calling back into Smart Capacity, causing the system to fail to associate the user with their carrier.
• What was done to mitigate:
  • Logging for the post-authentication workflow was being enhanced to gain visibility into the root cause.
  • The team updated customers via the Statuspage.
  • Considered bypass strategies (e.g., bypassing the API path and validating against scapi directly) and examined gateway/auth behavior and the post-auth user hydration path as part of the investigation.
• Current status and next steps:
  • The incident was marked as mitigated, but a definitive root cause fix had not been documented in the report.
  • Investigation continues with next steps including confirming the failing component, scope, any recent changes, and likely fix paths.
  • Actions items were created (PIR document completion, capture current impact, schedule Post-Incident Review) to formalize follow-up.

If you want, I can pull out a concise RCA-ready summary or help schedule a Post-Incident Review and capture the detailed timeline and root-cause hypotheses.

Customers are reporting that CarrierWatch / Keypoint is incorrectly flagging some carriers as having inactive authority and sending a large volume of automated notifications.

Impact includes customer confusion, excessive CarrierWatch email traffic, and some dispatch-blocking reports.

Current working theory from engineering is that a recent Motus-related change caused an updates-only file to be processed like a full dataset, which incorrectly triggered downstream logic and led to false status evaluations / notifications.

This appears to be related to the broader Motus transition work, but it is not the same underlying issue as INC-1296. Recommendation is to track this as a separate incident and link INC-1296 as related context.

Motus file handling issue causing false CarrierWatch inactive alerts and email flood Short answer: The incident was mitigated, but not fully resolved.

What happened:
- Status: mitigated (as of 2026-06-23 11:55 MDT)
- Root cause (working theory): Motus-related changes caused an updates-only file to be processed like a full dataset, triggering false inactive authority assessments and mass emails.
- Mitigation and fix steps:
- Implemented a job-side mitigation to stop mass emails by running the job without the send-emails flag.
- Deployed patches to address data corrections; Round 1 reduced unauthorized carriers from ~1,615,952 to ~1,156,191; Round 2 patches started 2026-06-25 to handle multiple authority types.
- Customer communications remained active; status page updated.
- Current state (as of the latest updates in the report): mass email flood stopped; partial data corrections underway; underlying data fix still incomplete (DOT 1223051 issue persists in prod; ~2,344 carriers with multiple authorities still affected). A broader fix to handle any number of authority types was in progress.
- Related follow-up: INC-1309 for a Keypoint SOAP import issue spun off.

In brief: Incident stopped the email flood and began data corrections via patches, but a complete resolution (full data correctness and handling of all authority-type scenarios) was ongoing and not yet finished. A PIR is slated to ensure full closure.

This incident has been resolved.

For 10m~ (Jun 22, 3:54 pm – Jun 22, 4:04 pm) the one-web-bff was unavailable due to a duplication of the /one-web-bff-service route, eliminating the /one-web-bff route. The issue was noticed at 4pm once the datadog processing delay window had passed, the fix was identified, applied, and working 4m after detection. The missing route was restored.

This incident has been resolved.

Yesterday production release introduce a new schema change. and this affected some customers who were not using the API in the correct way. we are rolling back. Resolved by rolling back the production release’s new restricted schema, restoring the previous schema. This rollback mitigated the issue, and the incident was marked as mitigated at 2026-06-19 11:18:40 MDT. The incident affected customers not using the API correctly, and is now resolved. Follow-ups are scheduled (PIR meeting, PIR doc, and capturing current customer impact).

Approximately 200 Shipper customers using iQ Benchmark are experiencing inaccurate data dating back to April 1, 2026. Root cause is identified as a bug in the Raven model (Data Platform). This is not an outage — the product is accessible, but data quality is degraded. A remediation plan has been defined and requires sequential execution across test and production environments before impacted customers are made whole.

Test Incident - Drill drill

Several carriers have reported being unable to sign up for the for the platform.

Rolling database upgrades are impacting our services All databases have been upgraded.

Downstream issue from #1288 impacted Ratecast, Ratecast forecasts were impacted. Ratecast granularity feature was turned on as a break fix and as a result, forecasts were incorrect. Here’s how the Ratecast Issue was resolved:

• Initial mitigation: A hotfix was deployed to cap/outlier forecasts and prevent runaway projections, with a later rollback to remove that cap as a temporary measure. This mitigated the immediate customer impact by stopping the flatlining and extreme forecasts.
• Service restoration: The Auth0 token issue that had disabled the ratecast granularity service was addressed, and the granularity service was re-enabled. This helped restore normal forecast behavior.
• Validation: The team validated that the clamp/flatline was removed and that several lanes no longer exhibited the problematic behavior.
• Long-term fix: A new API flag, divergentForecast, was introduced to surface forecast divergence without masking it. The API now includes this flag to help downstream systems monitor divergence.
• Resolution declaration: On 2026-06-16, with the divergentForecast flag in production and the underlying issues addressed, the incident was marked as resolved.

Key timestamps:
- Mitigated: 2026-06-11 10:31 MDT
- DivergentForecast flag deployed: 2026-06-15/16
- Incident marked resolved: 2026-06-16

Location updates not flowing from TT iPhone app to Backend API. Error says - "The network connection was lost."
There are other multiple different error msgs as well shown in datadog Resolution summary:

• What happened: A 3-hour backlog occurred because auto-scaling failed to trigger, leaving ~3.2 million messages unprocessed in TT backend services/HTProg/Monorail queue. Clients were still sending location updates, but processing was halted.
• How it was resolved: The queue was manually scaled up to 40+ instances, after which message processing resumed and the backlog began to decline.
• Outcome: All data from the outage window is recoverable as the backlog is processed.
• Time: Mitigated at 2026-06-11T09:05:00-06:00 MDT.
• Remaining items: Root cause of the auto-scaling trigger failure to be determined; follow-ups include PIR planning and documenting the incident RCA, plus ensuring recovery steps and safeguards are in place.

Production mutation requests in the Preference Settings System are intermittently returning 504 errors. Initial investigation shows pfs-settings-service pods are healthy enough to continue serving some read traffic, but the Kafka producer in prod is repeatedly failing to resolve the configured MSK broker hostnames and reports host resolution failures and all broker connections down. Based on current evidence, this is impacting write paths that depend on publishing settings change events to Kafka, which is why mutation requests are timing out while some GET requests continue to succeed. The issue is under active investigation, with current focus on validating prod Kafka broker configuration, DNS resolution, and cluster connectivity, while mitigation and rollback options are being evaluated. Resolution summary:

• Root cause: Kafka broker DNS/identity change from prod identity cluster replacement caused pfs-settings-service to point to the old cluster; write path (settings changes) failed while reads remained.

• Mitigation actions completed:

  • Updated pfs-settings-service to use the new Kafka cluster identity (identity-kafka-prod) and added the necessary producer access policy.
  • Created the missing prod topic stream.identity.settings.change.v1.
  • Merged and applied infra fixes to correct broker endpoints and remote-state/configuration.
  • Validated the write path (save/update) works in production and that service health is stable.

• Validation results:

  • Prod write path recovered; reads remained healthy.
  • Bridge validation confirms the save/update path functioning in prod.

Once we reach the max limit, Platform/CDC Events will no longer delivered. Resolution summary:

• The incident was mitigated by temporarily increasing the Salesforce platform events delivery limit, allowing downstream event flow to resume. A fix was put in place to make sure a secret was in place to prevent pod restarts followed by a large amount of Salesforce API calls.
• Salesforce was contacted to raise the limit (a Docusign quote for the limit increase was circulated; awaiting approval).
• The team planned to wait for the limit increase and then verify that events are flowing from the replay number.
• Once older order events are consumed, a follow-up PR would change the replay ID to -1 to ensure only live events are processed going forward.
• The incident status was set to mitigated at 16:30 MDT on 2026-06-09.

load.booking.canceled events aren't being emitted for Pepsi/Rectangle The socket hangups customers/clients were seeing were coming from the ingress layer: we had ingress nodes that got terminated during a node/AMI roll but didn't cleanly deregister from the NLB target groups. These needed to be manually deleted from the NLB.

We are noticing an unusual volume of 500 responses from the Load Posting API's PATCH endpoint which is driving down the service's availability. The impacted postings are missing their ownershipV2, resulting in a TypeError. The impacted postings appear to have originated in the legacy systems during the time window that the Account API was down in INC-1288. Due to the outage, the freight-posting-service was unable to backfill critical information from the Account API onto the postings. Resolved by identifying the 201 impacted postings missing ownershipV2 and manually deleting them to stop the 500 errors on the PATCH API. The team also planned cluster upgrades and a post-incident review to address the Account API outage root cause and prevent recurrence.

The Posting System has been experiencing intermittent dips in availability / spikes in 5xx errors for the past 30-45 minutes. The root cause of these spikes appear to be failed calls to the account-service, which has also been experiencing spikes in OOM errors during the same timeframe. Resolution was achieved by stopping memory-intensive processing (scaling the account-consumer to zero replicas), upgrading the Aurora MySQL DB with a blue-green upgrade, and gradually bringing services back online (excluding the batch job) to reduce risk. The posting system stabilized with account view access restored and healthy pods, marking inc-1288 as resolved, with a PIR to be documented and reviewed.

The issue affecting DAT One has been resolved and services are operating normally. We apologize for any inconvenience this may have caused and thank you for your patience while our team worked to restore full service.

This incident has been resolved.

a sharp drop in paid-user SSU completions. Current signals show SSU conversion fell to about 3% yesterday, versus a typical 30–40%, and the number of completed submissions dropped materially compared with prior Mondays. Early review suggests the break may be occurring around or after the plan picker step, but that has not been confirmed yet and the team is still validating where the funnel is failing. The incident was resolved by rolling back the last night’s deployment; post-rollback Datadog showed no 404s and the system stabilized. Next steps are for Product/CS to validate that paid SSU completions are back to normal and to coordinate outreach to affected customers.

We are getting reports that CSB and Salesforce are not communicating with DAT Rep. Resolved by deploying the PR that enables replay of Salesforce CDC events (gitops-identity PR #2691) and replaying events from the disruption point, which restored communication between CSB/Salesforce and DAT Rep. A root-cause investigation is still needed to determine why the salesforce-account-adapter stopped consuming events, as there may be an underlying issue.

The mci-batch job failed in production today, resulting in incomplete file delivery with one file currently missing. The incident was resolved by rerunning the child job with an override on the threshold checks to bypass the MCI total anomaly and by refiring the 3DZ prev-day 30 job, both of which completed successfully. The UI now shows 19 files for today and the -30D/3DZ data, and the incident was marked as resolved.

Customer Support is seeing Company Search fail for multiple active carrier DOTs, and the issue is reproducible across additional spot checks. This appears systemic rather than account-specific and is impacting carrier validation workflow. Resolved by applying a safe temporary mitigation: pausing the network tagging feature flag/audience so the JWT path requiring company UUID and account UUID is not invoked, which stopped the failing DOT lookups in Company Search. The team is validating the outcome and planning a longer-term fix (updating release/PR and clarifying the audience) while keeping the feature disabled for affected users. More details will be added in PIR document.

The nightly job to refresh the data cleared out the data. Resolution was achieved by an immediate workaround: pointing the 30-day index alias to the 60-day index to restore lane search results with negligible user impact. Data was restored and backfilled, and by 5/14 all lane search indexes were serving complete data, with a post-incident review and a JIRA ticket opened for a longer-term fix.

Sales Tech / Salesforce issue: leads are not being ingested/processed in Salesforce, so new customer inquiries are not reaching the normal routing flow. This is a high-priority blocker impacting same-day SLAs and stalling operations. The incident was resolved by identifying the root cause as the Okta–Salesforce integration/configuration; Salesforce lead processing works when Okta is bypassed, confirming the issue is Okta-related. An Okta support ticket was opened and remediation efforts with Okta are ongoing while authentication tests and environment checks continue.

TIM and DICE appear to be down at the moment, preventing users from accessing DICE and blocking the processing or upload of any new requests. Legacy CarrierWatch is still working, but new requests cannot move forward until TIM and DICE are restored. The issue is actively impacting operations and needs investigation from the owning team. Resolution: The outage was caused by the TIM_PROD transaction log filling the E: drive on philwcrbsql04, preventing TIM UI access and new requests. Short-term mitigation added 50 GB of space to the E: drive enabling TIM/DICE to restore (~118 minutes MTTR), and a long-term plan to shrink the SQL logs and implement regular backups was put in place to prevent recurrence.

The FTP production host is currently down and will be unavailable. This incident is mainly to track customer communications.

A potential compromise of an CSB ftp EC2 instances was detected The incident was contained and resolved through containment of the affected FTP/EC2 instances, removal of anonymous FTP access, deletion of malicious files and test users, restart of the FTP service, and forensic data collection, followed by security hardening (revoking root access, rotating passwords, disabling unused accounts, centralizing evidence). The FTP server was brought back online with the AWS Global Accelerator reenabled and customer communications prepared

FMIC’s AWS FSx file share \pawswfmcalx01\share is down due to a failed FSx–Active Directory connection, leaving the share inaccessible since roughly 12 a.m. PT on 5/2 and impacting IQ dashboard processing for 80+ customers. On-call has verified DC/DNS reachability; multiple FSx recovery attempts have failed, and further recovery vs. restore actions are pending with the Platform team. Resolution: The FSx security group sg-0cff90e1c8704fb64 was updated to allow TCP/UDP 464 to the Domain Controllers, enabling the misconfigured state recovery process. After connectivity was verified by Eddie Boss, the incident was mitigated and marked resolved at 4:05 PM UTC.