6:00 PM PDT - 11:00 PM PDT (5h duration)
The scheduled maintenance has been completed.
6:00 PM PDT - 9:00 PM PDT (3h duration)
The scheduled maintenance has been completed.
12:00 AM PDT - 2:00 AM PDT (2h duration)
Tracking API migration from MongoDB to DocumentDB
The scheduled maintenance has been completed.
5:43 AM PDT - 10:09 AM PDT (4h26min duration)
Unable to access Power and Rateview
This incident has been resolved.
10:14 AM PDT - Jul 23 12:22 PM PDT (1d2h7min duration)
SalesForce Emails Not Delivered
This incident has been resolved.
3:53 AM PDT - 8:08 AM PDT (4h15min duration)
OnTime is unavailable after maintenance
This incident has been resolved.
12:00 AM PDT - 4:00 AM PDT (4h duration)
The scheduled maintenance has been completed.
3:09 PM PDT - 3:58 PM PDT (49min duration)
Unable to Pull Images from JFrog Artifactory via AWS
The fix action by the Cloud Enablement team has resolved the issue. Given the nature of the fix, this issue will skip monitoring and transition to resolved. Those with changes currently in may start them at their regularly scheduled time. If your change needs to be rescheduled, please contact the NOC via #noc-help.
6:00 PM PDT - 9:00 PM PDT (3h duration)
Upgrade of Production F5 load Balancers to Address Critical Security Bug
The scheduled maintenance has been completed.
1:00 PM PDT - 2:00 PM PDT (1h duration)
Temporary Shutdown of api.dat.com
The scheduled maintenance has been completed.
6:00 PM PDT - 7:00 PM PDT (1h duration)
Salesforce Fiscal Period Close
The scheduled maintenance has been completed.
6:00 PM PDT - 7:00 PM PDT (1h duration)
Deploy Rateview with a series of bug fixes and enhancements
The scheduled maintenance has been completed.
12:00 AM PDT - 6:00 AM PDT (6h duration)
The scheduled maintenance has been completed.
6:00 PM PDT - 7:00 PM PDT (1h duration)
National HMM feature and infrastructure upgrade wrap-up.
The scheduled maintenance has been completed.
11:03 AM PDT - 1:38 PM PDT (2h35min duration)
We have received no further reports of issues with SalesForce emails over the last hour and will be closing this incident at this time.
6:00 PM PDT - 7:27 PM PDT (1h27min duration)
SQL Server Service Pack updates on BVTSQL10 and BOISQL06
The scheduled maintenance has been completed.
7:20 AM PDT - Jun 26 11:03 AM PDT (7d3h43min duration)
Test Environment is Unavailable
The test environment has been stable for several days with the new equipment. This incident will be closed at this time.
6:00 PM PDT - 8:00 PM PDT (2h duration)
Test Internal Firewall Upgrade
The scheduled maintenance has been completed.
8:33 AM PDT - 8:59 AM PDT (27min duration)
Certs not flowing from DICE to Flexicapture
All available disk had been consumed. Engineering freed space and restarted the services. Certs are confirmed moving again.
5:00 PM PDT - 11:00 PM PDT (6h duration)
The scheduled maintenance has been completed.
6:00 PM PDT - 10:00 PM PDT (4h duration)
Salesforce Deploy and Data Load
The scheduled maintenance has been completed.
8:00 PM PDT - 9:30 PM PDT (1h30min duration)
Rebuild table pud_fme_data in bapprod to improve performance
The scheduled maintenance has been completed.
1:00 AM PDT - 3:00 AM PDT (2h duration)
The scheduled maintenance has been completed.
6:00 PM PDT - 9:00 PM PDT (3h duration)
The scheduled maintenance has been completed.
12:17 PM PDT - 5:24 PM PDT (5h7min duration)
Reports of HTTPS Not Working for Some Users
We have received no further reports of issues with certificates. We have verified with the engineers that the SSL certificate renewal process was followed appropriately and we were unable to reproduce the issue. At this time, this appears to be an isolated incident so we will be marking this as resolved.
6:22 AM PDT - 3:24 PM PDT (9h3min duration)
Agents are unable to process payments
We have not received any more reports over the last 8 hours of failed transactions so we will be closing this incident at this time. If any further incidents occur, please notify the NOC.
5:00 PM PDT - 6:30 PM PDT (1h30min duration)
Hot Market Maps - End of MCI Forecast Promotion
The scheduled maintenance has been completed.
12:00 AM PDT - 4:00 AM PDT (4h duration)
TrackingAPI & OnTime Web Release
The scheduled maintenance has been completed.
6:00 PM PDT - 7:00 PM PDT (1h duration)
The scheduled maintenance has been completed.
5:06 AM PDT - 5:49 AM PDT (43min duration)
Some features of On Time degraded.
All issues have been resolved.
The trouble with the SMS links was sourced to a verification tool that mimics productions behavior. No broken links were actually sent, they only occurred in the verification step.
6:00 PM PDT - 10:00 PM PDT (4h duration)
The scheduled maintenance has been completed.
6:41 AM PDT - 8:00 AM PDT (1h19min duration)
Salesforce is not processing Sales contracts
This incident has been resolved.
1:22 PM PDT - 4:58 PM PDT (3h36min duration)
OnTime tracking requests not sending via SMS
This incident has been resolved.
6:00 PM PDT - 10:00 PM PDT (4h duration)
The scheduled maintenance has been completed.
7:13 AM PDT - 7:52 AM PDT (39min duration)
Customers are unable to access load boards
This incident has been resolved.
12:53 PM PDT - 1:41 PM PDT (48min duration)
This incident is part of the larger product outage. We will be resolving this incident and continuing to track the resolution on the product outage incident.
12:07 PM PDT - 3:29 PM PDT (3h22min duration)
OnTime Tracking Requests Not Sending
Rollback has been successfully implemented and service has recovered.
12:00 AM PDT - 4:00 AM PDT (4h duration)
The scheduled maintenance has been completed.
7:00 PM PDT - May 14 12:00 AM PDT (5h duration)
The scheduled maintenance has been completed.
5:08 PM PDT - May 13 12:15 AM PDT (7h7min duration)
Slack has reported the issue as resolved and the service has remained stable for several hours.
3:48 AM PDT - 8:55 AM PDT (5h7min duration)
HLP experiencing intermittent issues
This incident has been resolved.
6:00 PM PDT - 7:00 PM PDT (1h duration)
Deploy Rateview hotfix with "Upgrade Now" button for Ratecast Users
The scheduled maintenance has been completed.
6:00 PM PDT - 8:00 PM PDT (2h duration)
Adding Confirmation for Book Now in Power
The scheduled maintenance has been completed.
6:00 PM PDT - 7:00 PM PDT (1h duration)
Deploy Rateview with Ratecast 8 Day Forecast for Combo Pro and Combo Premium Accounts
The scheduled maintenance has been completed.
7:00 PM PDT - 8:00 PM PDT (1h duration)
Migrate https://migration.dat.com to infrastructure-identity migration, in prod
The scheduled maintenance has been completed.
5:00 PM PDT - May 2 12:00 AM PDT (7h duration)
Test VM Environment ZFS Upgrade
The scheduled maintenance has been completed.