DAT Solutions Internal

Back
Jun 11, 2026 9:31 AM PDT

Ratecast Issue

Updates

  • Jun 16, 2026 11:06 AM PDT

    Downstream issue from #1288 impacted Ratecast, Ratecast forecasts were impacted. Ratecast granularity feature was turned on as a break fix and as a result, forecasts were incorrect. Here’s how the Ratecast Issue was resolved:

    • Initial mitigation: A hotfix was deployed to cap/outlier forecasts and prevent runaway projections, with a later rollback to remove that cap as a temporary measure. This mitigated the immediate customer impact by stopping the flatlining and extreme forecasts.
    • Service restoration: The Auth0 token issue that had disabled the ratecast granularity service was addressed, and the granularity service was re-enabled. This helped restore normal forecast behavior.
    • Validation: The team validated that the clamp/flatline was removed and that several lanes no longer exhibited the problematic behavior.
    • Long-term fix: A new API flag, divergentForecast, was introduced to surface forecast divergence without masking it. The API now includes this flag to help downstream systems monitor divergence.
    • Resolution declaration: On 2026-06-16, with the divergentForecast flag in production and the underlying issues addressed, the incident was marked as resolved.

    Key timestamps:
    - Mitigated: 2026-06-11 10:31 MDT
    - DivergentForecast flag deployed: 2026-06-15/16
    - Incident marked resolved: 2026-06-16

    Resolved
  • Jun 11, 2026 9:31 AM PDT (5 days, 1 hour, and 34 minutes earlier)

    Downstream issue from #1288 impacted Ratecast, Ratecast forecasts were impacted. Ratecast granularity feature was turned on as a break fix and as a result, forecasts were incorrect. Mitigation summary

    • Immediate containment

      • Overnight hotfix deployed (June 4–5) to cap outlier Ratecast projections, addressing runaway forecasts temporarily.
      • Observed that a hard clamp suppressed analytical trust; decision made to revert once the service was healthy.

    • Restore service and fix root causes

      • Root cause identified: Auth0 token rate-limiting blocked the ratecast granularity service, causing aberrant forecasts.
      • Paul Ganser deployed a fix to handle Auth0 rate limiting properly and re-enabled the ratecast granularity feature flag. Validation showed the feature functioning again via Argo logs across multiple lanes (e.g., Dallas→Albuquerque, Atlanta→LAX).

    • Remove the temporary workaround and validate rollback

      • Triggered batch jobs to remove the clamp/flatline fix and revert to the previous forecast outputs.
      • Focused validation on rollback success and ensuring no remaining broad flatlines; rollback processing initiated (expected ~1.5 hours).

    • Communications and governance

      • Downgraded incident severity to Sev2.
      • Status updates and coordination ongoing; customer-facing messages and banners planned/noted to address the issue with RateView users and Snowflake data consumers.

    • Ongoing and long-term actions

      • Monitoring of problem lanes after cap removal; targeted data analyses to understand model behavior (Dallas vs. Fort Worth lanes).
      • Planning for additional guardrails and model stability improvements; RCA/PIR scheduling and broader post-incident communications.

    Current status
    - The mitigation efforts have focused on restoring accurate forecasting by re-enabling the granularity service with a proper rate-limiting fix and removing the temporary clamp, with validation in progress. Further steps include ensuring rollback effectiveness, continued monitoring, and communicating findings and fixes to customers.

    Monitoring