DAT Solutions Internal

Updates

• Oct 1, 2026 12:02 PM PDT

  Root cause: ingress, not company-service. Six emissary pods (replicaset emissary-ingress-67c85f7574: j24gx, 47pq2, fwncr, 4dd4h, 778qg, ncmn4) all run on one node, ip-10-248-140-130 (c5.4xlarge). That node became CPU-bound (system.cpu.user ~45% → ~76%, emissary CPU ~6.5 → ~12.5 cores, no restarts), and those pods got slow (p90 80–290ms).
  - Evidence it's not the service:
  - Latency rose evenly across all ~40 company-service pods.
  - Most of the time is spent before the request reaches the service: p90 duration ~110ms vs upstream_service_time ~35ms.
  - The same 15 pods of that replicaset on other nodes stayed normal (17–27ms).
  - Blast radius: every service routed through those pods got slower: company-service (18 → 82ms p90), office-service (21 → 115ms), association-service (23 → 165ms), network-tagging-system (15 → 95ms), client-support. Services routed through other emissary replicasets were not affected.
  - Likely trigger: traffic shifted onto the pods on that node (~177k vs ~95k req/min), and the node ran out of CPU.
  - Action (platform / cloud_engineering): cordon/drain ip-10-248-140-130 or move the emissary pods off it. Add pod anti-affinity / topologySpreadConstraints so emissary replicas spread across nodes.
  - company-service: no action needed.

  Resolved
• Oct 1, 2026 9:50 AM PDT (2 hours and 12 minutes earlier)

  Root cause: ingress, not company-service. Six emissary pods (replicaset emissary-ingress-67c85f7574: j24gx, 47pq2, fwncr, 4dd4h, 778qg, ncmn4) all run on one node, ip-10-248-140-130 (c5.4xlarge). That node became CPU-bound (system.cpu.user ~45% → ~76%, emissary CPU ~6.5 → ~12.5 cores, no restarts), and those pods got slow (p90 80–290ms).
  - Evidence it's not the service:
  - Latency rose evenly across all ~40 company-service pods.
  - Most of the time is spent before the request reaches the service: p90 duration ~110ms vs upstream_service_time ~35ms.
  - The same 15 pods of that replicaset on other nodes stayed normal (17–27ms).
  - Blast radius: every service routed through those pods got slower: company-service (18 → 82ms p90), office-service (21 → 115ms), association-service (23 → 165ms), network-tagging-system (15 → 95ms), client-support. Services routed through other emissary replicasets were not affected.
  - Likely trigger: traffic shifted onto the pods on that node (~177k vs ~95k req/min), and the node ran out of CPU.
  - Action (platform / cloud_engineering): cordon/drain ip-10-248-140-130 or move the emissary pods off it. Add pod anti-affinity / topologySpreadConstraints so emissary replicas spread across nodes.
  - company-service: no action needed.

  Monitoring