DAT Solutions Internal

Updates

• Aug 4, 2026 4:50 AM PDT

  Negotiation Consumer Service started to error out with MSK connection errors. Summary: negotiation-service-consumer lost connectivity to MSK broker 2 (negotiation-prod-kafka) starting ~8:34 PM PDT Aug 2, correlating with a CPU spike to 92.8% on broker 2 at ~9:11 PM (brokers 1 and 3 stayed normal the whole time). No customer-facing impact was confirmed and no other services on the cluster showed alerts.

  Mitigation: Broker 2 was manually rebooted at 1:52 AM PDT Aug 3. Connection errors (ETIMEDOUT, BrokerPool timeouts) stopped immediately after and have not recurred since, over 24 hours of monitoring.

  Root cause: Not fully confirmed, but the evidence points strongly to a JVM/GC-driven acceptor thread hang on broker 2. Eli's Datadog analysis found a ~30 min metric reporting gap (8:32-9:06 PM) consistent with a severe JVM stall, followed by broker 2 staying alive for replication but unable to accept new client connections until the reboot cleared it. Traffic was normal/low at the time, ruling out a load-driven spike. Log compaction, ISR catch-up, and leader rebalance remain unruled-out alternatives. Follow-up questions are still open with Platform (broker Kafka version, GC logs, aws.kafka.request_handler_avg_idle_percent for that window) to pin this down, but that investigation will continue as a follow-up rather than blocking resolution given the sustained recovery.

  Tracked follow-ups already filed:

  OS-3276: Discovery adding MSK broker-health alerting for negotiation-prod-kafka (this incident only had symptom-level alerting, not broker health).

  Orient Express alert-routing cleanup: several negotiation-service alert routes were still pointing at the deactivated OE schedule instead of VA/Discovery. Vince and Alastair are reassigning these.

  Thanks to Eli Hurst and Chathura Bamunusinghe for the quick work overnight and the follow-through this morning

  Resolved
• Aug 3, 2026 2:09 AM PDT (1 day, 2 hours, and 40 minutes earlier)

  Negotiation Consumer Service started to error out with MSK connection errors. Connection errors stopped after the reboot. No further errors were observed from the monitored services through the end of the call.
  Current status: Mitigated, but root cause remains undetermined. Continued monitoring is required.
  Findings from logs and metrics
  A significant CPU spike correlated with the incident window.
  Broker 2 showed no client connections, while brokers 1 and 3 continued receiving connectivity.
  The overall MSK cluster appeared healthy despite broker 2 symptoms.
  No relevant AWS scheduled event or maintenance activity was identified.
  The earlier assumption that cluster patching caused the issue was corrected; patching was not scheduled until the following week.
  The affected services stopped reporting connection timeouts after the broker reboot.

  Monitoring