DAT Solutions Internal

Updates

• Sep 30, 2026 10:49 AM PDT

  Kafka lag for Load and Truck Posting enrichment has been growing since ~5:30am PT, causing elevated posting enrichment duration. I am also seeing elevated latency from downstream services in the network domain (Company and Office service)

  Resolved
• Sep 29, 2026 6:54 AM PDT (1 day, 3 hours, and 55 minutes earlier)

  Kafka lag for Load and Truck Posting enrichment has been growing since ~5:30am PT, causing elevated posting enrichment duration. I am also seeing elevated latency from downstream services in the network domain (Company and Office service) Node Management: The team cordoned a node experiencing 100% CPU usage and spun up a new node for Emissary and the network.

  Policy Adjustments: The scaling policy for the edge off service was updated to high availability. Additionally, a 50% headroom buffer for CPU and memory was applied to Emissary (across network and freight clusters) to prevent resource starvation.

  Traffic Control: Live matching for loads was temporarily disabled to allow the system to process the backlog of messages.

  Current Status: Latency across company, office, and freight load posting services has dropped substantially. Kafka consumer lag has caught up and returned to near zero, and a PR has been created to re-enable live matching. Customers are successfully posting again.

  Monitoring