DAT Solutions Internal

Back
Oct 20, 2025 5:29 AM PDT

Shipment consumers auth is failing

Updates

  • Oct 21, 2025 9:40 AM PDT

    Seems like aws IAM does not work correctly. Consumers could not connect to Kafka brokers The incident was resolved by restarting multiple Kafka consumer pods, which restored AWS IAM authentication functionality and resolved connectivity issues to Kafka brokers. The incident status was marked as \"resolved\" following successful mitigation and monitoring showed no further issues.

    Resolved
  • Oct 21, 2025 7:22 AM PDT (2 hours and 18 minutes earlier)

    The incident has been mitigated by restarting multiple Kafka consumer pods, including those handling posting and search functionalities. This action resolved the authentication issues related to AWS IAM and restored normal system operations.

    Monitoring
  • Oct 20, 2025 8:12 AM PDT (23 hours and 9 minutes earlier)

    Seems like aws IAM does not work correctly. Consumers could not connect to Kafka brokers The incident was mitigated by restarting multiple Kafka consumer pods, including those handling posting and search functionalities. This action resolved the authentication issues related to AWS IAM and restored normal system operations. We are considering the incident mitigated at this time and will continue to monitor the environment throughout the day for any residual impact or recurrence of the original issue.

    Monitoring
  • Oct 20, 2025 8:10 AM PDT (1 minute earlier)

    Seems like aws IAM does not work correctly. Consumers could not connect to Kafka brokers The incident was mitigated by restarting multiple Kafka consumer pods, including those handling posting and search functionalities. This action resolved the authentication issues related to AWS IAM and restored normal system operations. We are considering the incident mitigated at this time and will continue to monitor the environment throughout the day for any residual impact or recurrence of the original issue.

    Monitoring
  • Oct 20, 2025 5:29 AM PDT (2 hours and 41 minutes earlier)

    Seems like aws IAM does not work correctly. Consumers could not connect to Kafka brokers The incident was mitigated by restarting multiple Kafka consumer pods, including those related to posting and search functionalities, which resolved the issues with relying on AWS IAM for authentication and restored normal operations. Additionally, the underlying DNS issue reported by AWS was fully mitigated, contributing to the overall recovery of the affected services.

    Monitoring