DAT Solutions Internal
Prod mci-batch job failure causing missing file delivery
Updates
-
May 27, 2026 6:48 AM PDT
The mci-batch job failed in production today, resulting in incomplete file delivery with one file currently missing. The incident was resolved by rerunning the child job with an override on the threshold checks to bypass the MCI total anomaly and by refiring the 3DZ prev-day 30 job, both of which completed successfully. The UI now shows 19 files for today and the -30D/3DZ data, and the incident was marked as resolved.
Resolved -
May 27, 2026 6:46 AM PDT (1 minute earlier)
The mci-batch job failed in production today, resulting in incomplete file delivery with one file currently missing. Mitigation steps: reran the child job with an override on the threshold checks to bypass the MCI total anomaly, and refired the 3DZ prev-day 30 job ignoring threshold checks. The reruns succeeded (19 files today; data appeared in the UI), and the incident was marked as resolved.
Monitoring