DAT Solutions Internal

Back
Jun 22, 2026 8:22 AM PDT

Ontime Evaluation Service DB running out of space

Updates

  • Jun 23, 2026 7:19 PM PDT

    This incident has been resolved.

    Resolved
  • Jun 22, 2026 8:22 AM PDT (1 day, 10 hours, and 56 minutes earlier)

    https://dat-solutions.slack.com/archives/C098U9P8RSA/p1781474190068219 Mitigation identified:

    • Immediate action taken: Jack manually killed the active delete process at 10:15am PST. This was intended to stop the long-running delete and relieve backpressure on the vacuum process amid rising storage usage.

    Context:

    • Root cause hypothesis in discussion: storage usage rose from ~270GB to ~290GB since May due to a surge in 3P shipment volume, with auto vacuum lagging behind deletes.
    • Customer impact: No customer impact reported.

    Next steps / follow-up (from the incident):
    - Schedule Post-Incident Review meeting
    - Ensure PIR Document is complete and up to date
    - Capture current impact to customers

    If you’d like, I can add a mitigation note to the incident timeline or help schedule the PIR.

    Monitoring