DAT Solutions Internal
Fraud consumer Kakfa lag spiking
Updates
-
Dec 18, 2025 10:34 AM PST
Kafka lag for the fraud detection consumer has been spiking since ~8:52am PT and is not self-resolving.
https://app.datadoghq.com/monitors/244711610?from_ts=1766076931626&to_ts=1766078731626&live=true The incident was resolved by updating the node affinity to pin the fraud detection consumer to 15 pods, which mitigated the Kafka lag issue. This change was successfully merged, allowing the consumer to properly sync and manage load in the future.
Resolved -
Dec 18, 2025 10:31 AM PST (3 minutes earlier)
Kafka lag for the fraud detection consumer has been spiking since ~8:52am PT and is not self-resolving.
https://app.datadoghq.com/monitors/244711610?from_ts=1766076931626&to_ts=1766078731626&live=true The incident was mitigated by updating the node affinity to pin the fraud detection consumer to 15 pods, which resolved the Kafka lag issue and was approved despite being within a change freeze. Additionally, the change was successfully merged to ensure the consumer could properly sync and manage load in the future.
Monitoring -
Dec 18, 2025 9:28 AM PST (1 hour and 2 minutes earlier)
Kafka lag for the fraud detection consumer has been spiking since ~8:52am PT and is not self-resolving.
https://app.datadoghq.com/monitors/244711610?from_ts=1766076931626&to_ts=1766078731626&live=true
Investigating