DAT Solutions Internal

Back
Aug 4, 2026 10:39 AM PDT

Data classifications - causing share issue

Updates

  • Aug 5, 2026 10:03 AM PDT

    classification tagging effecting shares Root cause: Snowflake masking/tagging policy changes introduced dangling tag references on shared DB objects, breaking the morning replication/refresh to customer shares and emptying the Broker Benchmark table in iQ.

    Resolution: Tags were rolled back; the database refresh was manually rerun across all affected regions (AWS us-east-2, Azure East 2). Broker Benchmark recovered (~46M rows, iQ access restored). Brindha ran the Shipper refresh manually; completed and UI confirmed updated with new records. Raven did not run during the affected window and was unaffected.

    All customer-facing data confirmed restored. Monitoring follow-ups (zero-row ETL alerts, tagging guardrails, PIR) tracked as separate action items.

    Resolved
  • Aug 4, 2026 10:39 AM PDT (23 hours and 23 minutes earlier)

    classification tagging effecting shares Marking this as mitigated.

    Root cause: a Snowflake masking/tagging policy change left dangling tag references on objects in the shared database, which broke replication/refresh to customer shares and emptied Broker Benchmark in iQ.

    What we did:

    Rolled back the masking/tagging changes causing the dangling references

    Manually reran the refresh across affected regions, AWS us-east-2 and Azure East 2 both confirmed populating with today's data

    Broker Benchmark recovered (~46M rows), iQ access restored

    Shipper was stale (last pickup Aug 2), Brindha ran the Shipper refresh and it's completed, UI now showing new records

    Raven didn't run during the affected window so it wasn't impacted, no rerun needed there

    Customer-facing data is confirmed back (enterprise support thread on this closed out too). Follow ups tracked separately: zero-row ETL monitoring, PIR doc, and capturing customer impact

    Monitoring