DAT Solutions Internal
Error Spike in One Web BFF
Updates
-
Mar 2, 2026 12:03 PM PST
notice a spike of 403 errors in one web bff service https://app.datadoghq.com/logs?query=service%3Aone-web-bff%20env%3Aprod%20status%3Aerror%20%22failed%20to%20get%20metrics%20for%20postingids%22&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&messageDisplay=inline&refresh_mode=sliding&storage=hot&stream_sort=time%2Casc&viz=stream&from_ts=1771521938567&to_ts=1771522838567&live=true Root cause: an internal frontend change caused one-web-bff to call the negotiation metrics endpoint without the required permissions, triggering 403s. Resolution: a frontend fix was deployed and validated via Datadog traces/logs and Slack confirmations; error volume returned to baseline with no confirmed customer impact, and the incident was closed as resolved.
Resolved -
Feb 20, 2026 5:38 AM PST (1 week, 3 days, 6 hours, and 25 minutes earlier)
notice a spike of 403 errors in one web bff service https://app.datadoghq.com/logs?query=service%3Aone-web-bff%20env%3Aprod%20status%3Aerror%20%22failed%20to%20get%20metrics%20for%20postingids%22&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&messageDisplay=inline&refresh_mode=sliding&storage=hot&stream_sort=time%2Casc&viz=stream&from_ts=1771521938567&to_ts=1771522838567&live=true Root cause was identified as an internal service calling the wrong endpoint, and a fix was deployed to address it. Validation was done via Datadog traces/logs and Slack confirmations that the fix had been deployed and the incident mitigated, though rollout promotion across environments was still in progress.
Monitoring -
Feb 19, 2026 10:11 AM PST (19 hours and 26 minutes earlier)
notice a spike of 403 errors in one web bff service https://app.datadoghq.com/logs?query=service%3Aone-web-bff%20env%3Aprod%20status%3Aerror%20%22failed%20to%20get%20metrics%20for%20postingids%22&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&messageDisplay=inline&refresh_mode=sliding&storage=hot&stream_sort=time%2Casc&viz=stream&from_ts=1771521938567&to_ts=1771522838567&live=true
Investigating