DAT Solutions Internal

Back
Jun 9, 2026 3:41 PM PDT

Salesforce Platform Events Devlivery Limit Exhausted

Updates

  • Jun 10, 2026 3:44 PM PDT

    Once we reach the max limit, Platform/CDC Events will no longer delivered. Resolution summary:

    • The incident was mitigated by temporarily increasing the Salesforce platform events delivery limit, allowing downstream event flow to resume. A fix was put in place to make sure a secret was in place to prevent pod restarts followed by a large amount of Salesforce API calls.
    • Salesforce was contacted to raise the limit (a Docusign quote for the limit increase was circulated; awaiting approval).
    • The team planned to wait for the limit increase and then verify that events are flowing from the replay number.
    • Once older order events are consumed, a follow-up PR would change the replay ID to -1 to ensure only live events are processed going forward.
    • The incident status was set to mitigated at 16:30 MDT on 2026-06-09.
    Resolved
  • Jun 9, 2026 3:41 PM PDT (1 day and 2 minutes earlier)

    Once we reach the max limit, Platform/CDC Events will no longer delivered. Mitigation summary

    • Primary mitigation: Raised a ticket with Salesforce to temporarily increase the Salesforce event delivery limit. A Docusign quote for the limit increase was sent and is awaiting approval. There is a small amount of remaining capacity to bridge between now and the increase.

    • Operational plan: Wait for the Salesforce limit to be increased and then verify that events are flowing again from the replay number. Once older order events are consumed, submit a follow-up PR to change the replay_id to -1 so the service restarts and processes only live events going forward.

    • Containment/visibility: Incident severity was elevated to Sev1, and a cross-team review was ongoing to confirm impact and progression. The team noted potential data inconsistency on downstream systems if the limit is still breached.

    • Downstream/workflow context: The account adapter (downstream) reads Salesforce CDC events and propagates them to two Kafka consumers; spikes in events from Salesforce were identified as a possible root contributor. There was no active development on the Salesforce account adapter since it moved from another team, and there was a hypothesis that retries on unread messages could accelerate limit consumption.

    • Next steps / follow-ups (per the incident plan):

      • Complete PIR documentation.
      • Schedule a Post-Incident Review.
      • Capture current customer impact.
      • Reconfirm after Salesforce increases the limit and replay flow is healthy, then implement the replay_id -1 PR.

    Status as of now: Waiting on Salesforce to increase the limit; plan to reconvene and verify flow once the limit is in place.

    Monitoring