DAT Solutions Internal

Updates

• Jul 9, 2026 11:27 PM PDT

  Multiple standard customer accounts are reporting inability to pay bills or add a card in the self-service billing flow. Customer-facing reports in #customer-impact indicate repeated failures with the message Card could not be added, including multiple calls and at least five affected accounts identified so far: 04095728, 04095754, 04095753, 04095759, and 0002420578.

  Datadog evidence shows a shared backend failure in payment-service during Salesforce payment-method creation:

  Error upserting payment method: Failed to create payment method in Salesforce for account 001f400001M2VpgAAF: [{"statusCode":"INVALID_FIELD","message":"No such column 'blng_BinLookupStatusc' on sobject of type blngPaymentMethod_c","fields":[]}]

  This suggests the issue is not limited to front-end user entry alone and may be caused by a Salesforce schema / field availability / deployment mismatch in the billing-payment integration path. The customer-facing Credit card could not be added message is known to be generic and can mask multiple backend failure reasons, so the Datadog error is currently the strongest technical signal. The incident was caused by an incorrect Salesforce field API name used in payment service during payment method creation, which resulted in backend failures and surfaced to customers as inability to add a card or pay bills. The issue correlated with the CAB change. Engineering corrected the field reference, pushed the fix through CI/CD, and deployed the update to production. After deployment, the Datadog error stopped reproducing, and no new customer reports were observed, so the incident has been resolved.

  Resolved
• Jul 8, 2026 10:24 PM PDT (1 day, 1 hour, and 3 minutes earlier)

  Multiple standard customer accounts are reporting inability to pay bills or add a card in the self-service billing flow. Customer-facing reports in #customer-impact indicate repeated failures with the message Card could not be added, including multiple calls and at least five affected accounts identified so far: 04095728, 04095754, 04095753, 04095759, and 0002420578.

  Datadog evidence shows a shared backend failure in payment-service during Salesforce payment-method creation:

  Error upserting payment method: Failed to create payment method in Salesforce for account 001f400001M2VpgAAF: [{"statusCode":"INVALID_FIELD","message":"No such column 'blng_BinLookupStatusc' on sobject of type blngPaymentMethod_c","fields":[]}]

  This suggests the issue is not limited to front-end user entry alone and may be caused by a Salesforce schema / field availability / deployment mismatch in the billing-payment integration path. The customer-facing Credit card could not be added message is known to be generic and can mask multiple backend failure reasons, so the Datadog error is currently the strongest technical signal. Mitigation summary:
  - The incident was mitigated by a production update implemented in response to a Salesforce schema/permission issue detected in the payment-method creation path. The Datadog errors pointed to No such column 'blng_BinLookupStatusc' on blngPaymentMethod_c, indicating a schema/permission mismatch after a recent change.
  - The timeline notes: “prod has been updated” at 23:16 MDT, which represents the applied mitigation.

  Key context:
  - The likely root cause is a Salesforce field availability/permission problem introduced by a deployment around 3 PM PT, correlated with the observed errors.
  - Immediate mitigation was the production update; investigation into the exact field existence and required permission set for the payment-service user is ongoing.

  Next steps (recommended):
  - Verify in Salesforce that the field blng_BinLookupStatusc exists on blngPaymentMethod_c and confirm the payment-service user has the necessary permissions/permission set.
  - Validate the integration path end-to-end in a staging/controlled environment before broader rollout.
  - Complete the Post-Incident Review (PIR) and document current customer impact, then communicate resolution to customers.
  - Consider adding safeguards for schema/permission changes (change control, pre-deploy checks, and more robust observability around Salesforce integration).

  Monitoring