DAT Solutions Internal
High latency in search requests; failed downstream requests to network domain services
Updates
-
Aug 4, 2026 11:55 AM PDT
some searches are failing with high latency to downstream services Root Cause
ScaleOps rollout in freight-prod (2026-07-24) reduced the cluster's node count over the weekend, bin-packing search-service pods more tightly. When Monday's traffic spike hit, search-service was heavily CPU throttled and couldn't scale horizontally in time, causing health-check failures and cascading pod restarts. This was compounded by Emissary circuit breakers and upstream connection pressure from the network-tagging call rollout (ramped to 100% on 7/14).
Key Mitigations
Set rollout percentages for private-network, network-membership, network-ownership, and book-now to 0% to cut traffic to the affected upstream path.
Disabled qualification claims and account-company-mapping calls in prod search-service to reduce hanging external calls and CPU/heap pressure.
Raised search-service autoscaling ceiling (75 → 120) and CPU request/limit (3100m → 4000m → 4500m).
Temporarily disabled readiness/liveness probes to stop Kubernetes from killing pods during elevated latency.
Increased Emissary capacity in freight-prod and re-enabled network-domain calls once traffic stabilized.
Rolled back the Datadog Cluster Agent upgrade to reduce circuit-breaker events in Emissary/Freight paths.
Follow-Up / Hardening (tracked post-incident)
Harden search-service to scale horizontally more effectively and avoid limit-based throttling.
Add PriorityClass support (helm-node chart PR open) to protect workloads from eviction, replacing the prior Guaranteed-QoS approach.
services have been restored and stable.
Resolved -
Jul 27, 2026 9:37 AM PDT (1 week, 1 day, 2 hours, and 17 minutes earlier)
some searches are failing with high latency to downstream services
Monitoring