DAT Solutions Internal

Updates

• Jun 12, 2026 12:06 PM PDT

  Production mutation requests in the Preference Settings System are intermittently returning 504 errors. Initial investigation shows pfs-settings-service pods are healthy enough to continue serving some read traffic, but the Kafka producer in prod is repeatedly failing to resolve the configured MSK broker hostnames and reports host resolution failures and all broker connections down. Based on current evidence, this is impacting write paths that depend on publishing settings change events to Kafka, which is why mutation requests are timing out while some GET requests continue to succeed. The issue is under active investigation, with current focus on validating prod Kafka broker configuration, DNS resolution, and cluster connectivity, while mitigation and rollback options are being evaluated. Resolution summary:

  • Root cause: Kafka broker DNS/identity change from prod identity cluster replacement caused pfs-settings-service to point to the old cluster; write path (settings changes) failed while reads remained.

  • Mitigation actions completed:

    • Updated pfs-settings-service to use the new Kafka cluster identity (identity-kafka-prod) and added the necessary producer access policy.
    • Created the missing prod topic stream.identity.settings.change.v1.
    • Merged and applied infra fixes to correct broker endpoints and remote-state/configuration.
    • Validated the write path (save/update) works in production and that service health is stable.

  • Validation results:

    • Prod write path recovered; reads remained healthy.
    • Bridge validation confirms the save/update path functioning in prod.

  Resolved
• Jun 11, 2026 6:14 AM PDT (1 day, 5 hours, and 52 minutes earlier)

  Production mutation requests in the Preference Settings System are intermittently returning 504 errors. Initial investigation shows pfs-settings-service pods are healthy enough to continue serving some read traffic, but the Kafka producer in prod is repeatedly failing to resolve the configured MSK broker hostnames and reports host resolution failures and all broker connections down. Based on current evidence, this is impacting write paths that depend on publishing settings change events to Kafka, which is why mutation requests are timing out while some GET requests continue to succeed. The issue is under active investigation, with current focus on validating prod Kafka broker configuration, DNS resolution, and cluster connectivity, while mitigation and rollback options are being evaluated.

  Monitoring