DAT Solutions Internal

Back
Sep 10, 2025 8:15 AM PDT

Trucker Tools randomly returning 400s

Updates

  • Sep 29, 2025 9:47 AM PDT

    The incident involving Trucker Tools relates to the system returning a series of 400 error responses to a subset of customers, primarily due to issues with requests routed through Cloudflare, which showed intermittent problems in certain regions while direct requests to CloudFront functioned properly. The mitigation involved reverting users to the previous stack while investigating the root cause, which included examining server headers and traffic logs that indicated errors when passing through Cloudflare. Steps taken included toggling Cloudflare’s proxy feature to validate its effect on error occurrence and engaging multiple team members, including Luis Diaz Ramirez, who reported the issue, and several responders who contributed to the analysis and resolution efforts. The incident status was marked as mitigated after confirming improvements in system behavior when Cloudflare was disabled, but further examination into addressing security postures, including the Web Application Firewall, continues.

    • Meeting
    • Severity: SEV3
    • Incident Types: Default AWS Amplify applied to a fix to how they read ASCII characters in request headers.
    Resolved
  • Sep 25, 2025 7:52 AM PDT (4 days, 1 hour, and 54 minutes earlier)

    We've identified the root cause and will be deployed a fix Monday. The Incident is currently mitigated.

    Identified
  • Sep 11, 2025 3:54 PM PDT (1 week, 6 days, 15 hours, and 58 minutes earlier)

    There are a series of 400 responses being returned to a different set of customers, the mitigation has been to return users to the previous stack. After re-examining Cloudflare and reviewing the headers more carefully, we confirmed that requests routed through Cloudflare show intermittent issues in certain regions (e.g., Turkey, Colombia, El Salvador), while requests going directly to CloudFront work consistently.
    We validated this by toggling Cloudflare’s proxy on and off. With Cloudflare off, the site has run without errors for over an hour.
    Our initial check two weeks ago wasn’t thorough enough, as we hadn’t fully reviewed server headers and cache behavior. With Darin’s help, we confirmed that disabling Cloudflare’s WAF resolves the issue consistently.
    While this appears to mitigate the issue in the short term, we recognize DAT’s security posture depends on Cloudflare and remain committed to identifying the root cause. AWS has logs showing errors on traffic flowing through Cloudflare to Amplify, which may help narrow this down. As an alternative, CloudFront’s WAF could also be considered.

    Monitoring
  • Sep 10, 2025 8:15 AM PDT (1 day, 7 hours, and 38 minutes earlier)

    There are a series of 400 responses being returned to a different set of customers, the mitigation has been to return users to the previous stack.

    Investigating