DAT Solutions Internal

Updates

• Aug 12, 2026 6:15 AM PDT

  The Preferred & Blocked Service has degraded performance after upgrading the Preferred & Blocked DocumentDB from 3.x to 5.0 The Preferred & Blocked Service experienced elevated latency after the production DocumentDB 3.x to 5.0 upgrade. Investigation identified an issue in the count wrapper invocation: the query filter and index hint were passed in the wrong argument positions, preventing the count query from using the intended index efficiently.

  The wrapper was corrected so the filter is passed as the query and the index hint is passed through the options argument. The fix was deployed to production. After deployment, preferred and blocked list requests returned normally and the service response time improved.

  Datadog validation for the reviewed production window showed 0 error logs and 0 warning logs, with normal info-level activity. CloudWatch and AWS-side metrics could not be independently validated due to access was unavailable; this does not block resolution but remains a follow-up validation item.

  Follow-up items:
  Confirm CloudWatch and DocumentDB Performance Insights metrics after the observation window.
  Run and retain a production explain for the office-ID query to confirm the intended index is being used.
  Remove the temporary Performance Insights access/configuration after the agreed observation window.
  Add regression coverage for the count-wrapper query and hint argument shape, including the $in path.
  Complete the PIR.

  Resolved
• Aug 11, 2026 11:47 PM PDT (6 hours and 27 minutes earlier)

  The Preferred & Blocked Service has degraded performance after upgrading the Preferred & Blocked DocumentDB from 3.x to 5.0 Following the DocumentDB 3.6-to-5.0 upgrade, Preferred and Blocked office lookups experienced elevated latency because the count path was not applying the query filter and index hint correctly. No API or contract changes were required.

  The team corrected the shared DocumentDB count-wrapper argument handling, updated hint typing/key-pattern usage, and deployed the hotfix to production. Post-deployment validation confirmed that Preferred and Blocked lists were loading normally and that collection scans were not increasing. A final production explain check and PIR remain as follow-up items.

  Mitigation status: Customer impact is mitigated; continue normal monitoring pending final query-plan confirmation and PIR follow-up.

  Monitoring