DAT Solutions Internal

Back

Incident history

Feb 2022 - Apr 2022

April 2022

Apr 29

12:33 PM PDT - 1:52 PM PDT (1h19min duration)

Delays with orders activating - Received notice of an Apex job error for ActivateOrdersFromClosedWonOppsBatch

This incident has been resolved.

2 updates
Apr 29

6:48 AM PDT - May 11 6:43 AM PDT (11d23h55min duration)

Bad request error message in Power/Rateview

Caleb Chenoweth : "We got the JIRA tickets setup for the follow on tasks to fix this issue in the apps:
- https://transcore.jira.com/browse/RV-4497
- https://transcore.jira.com/browse/RV-4500
- https://transcore.jira.com/browse/WTF-678

So marking this incident as "resolved" as the work is getting planned and underway, no point in leaving this specific incident open any more. Just adds noise when looking at the list."

3 updates
Apr 27

4:10 AM PDT - 5:43 AM PDT (1h33min duration)

Customer no able to login into Power

Notes: Power has been reverted to a previous version and is now available

3 updates
Apr 26

11:21 AM PDT - 2:41 PM PDT (3h20min duration)

Power notifications are delayed

This incident has been resolved.

3 updates
Apr 25

8:35 PM PDT - Apr 26 5:25 AM PDT (8h50min duration)

Prod freight apps and services are in a degraded state due to CAB-2775

This incident has been resolved.

2 updates
Apr 24

7:58 AM PDT - Jun 1 7:55 AM PDT (37d23h57min duration)

RateView is not billing customers for multilane lookups

This incident has been resolved.

4 updates
Apr 24

4:50 AM PDT - 11:47 AM PDT (6h57min duration)

Issues With Rate Forecasts

This incident has been resolved.

6 updates
Apr 23

2:00 PM PDT - 3:00 PM PDT (1h duration)

CSB scheduled restart

The scheduled maintenance has been completed.

4 updates
Apr 22

1:02 PM PDT - 2:40 PM PDT (1h38min duration)

Apex jobs getting hung up

This incident has been resolved.

2 updates
Apr 21

9:18 PM PDT - 11:20 PM PDT (2h2min duration)

RateView Multi-Lane cannot download results file

This incident has been resolved.

2 updates
Apr 21

6:00 PM PDT - 9:15 PM PDT (3h15min duration)

CSB Normal release

The scheduled maintenance has been completed.

4 updates
Apr 21

4:43 PM PDT - 11:37 PM PDT (6h54min duration)

Market Conditions Data Unavailable

This incident has been resolved.

2 updates
Apr 21

11:24 AM PDT - 1:29 PM PDT (2h4min duration)

Onboard Products are Unavailable

This incident has been resolved.

5 updates
Apr 21

7:17 AM PDT - 9:05 AM PDT (1h48min duration)

Onboard Won't Load

This incident has been resolved.

2 updates
Apr 18

1:11 PM PDT - 2:59 PM PDT (1h48min duration)

Nintex Assuresign Outage - Carriers are unable to sign in to nintex services to sign contracts.

This incident has been resolved.

3 updates
Apr 15

3:35 PM PDT - Apr 18 1:58 PM PDT (2d22h23min duration)

Replicable issue with CXN & Carrierwatch

This incident has been resolved.

3 updates
Apr 15

7:23 AM PDT - 9:51 AM PDT (2h29min duration)

Customers are not getting auto-refreshed loads in Power

This incident has been resolved.

3 updates
Apr 14

7:30 AM PDT - 11:51 AM PDT (4h21min duration)

problems with Market conditions Index

This incident has been resolved.

4 updates
Apr 07

1:29 PM PDT - 3:51 PM PDT (2h22min duration)

Issues Logging Into NICE

This incident has been resolved.

3 updates
Apr 07

10:22 AM PDT - 3:24 PM PDT (5h2min duration)

Power Consumers are not keeping up with alarm and sync notifications

This incident has been resolved.

4 updates

March 2022

Mar 31

5:03 PM PDT - Apr 1 8:51 AM PDT (15h48min duration)

Phone Phishing attack

This incident has been resolved.

3 updates
Mar 31

7:38 AM PDT - 3:31 PM PDT (7h54min duration)

Orders are getting stuck in draft when self reactivating.

This incident has been resolved.

6 updates
Mar 29

1:08 PM PDT - Mar 30 8:34 AM PDT (19h27min duration)

POWER - posts are not being crossed out in searches

This incident has been resolved.

3 updates
Mar 28

11:59 AM PDT - 1:20 PM PDT (1h21min duration)

Search System failing all requests to the CSB

This incident has been resolved.

4 updates
Mar 28

11:45 AM PDT - 1:27 PM PDT (1h43min duration)

TruckersEdge login failure

Duplicated of INC-246, same issue

2 updates
Mar 28

11:41 AM PDT - 1:29 PM PDT (1h48min duration)

CSB services shutdown unexpectedly

This incident has been resolved.

2 updates
Mar 24

11:15 PM PDT - Mar 25 9:31 AM PDT (10h16min duration)

Trucker API - Rates transactions are slow alert

Resolved by Tom Orth in XMatters

2 updates
Mar 24

2:05 PM PDT - 2:38 PM PDT (33min duration)

TruckersEdge login failure

Check note on INC-242, they are related

2 updates
Mar 24

2:01 PM PDT - 2:37 PM PDT (36min duration)

Discrepancies with the power load board, Pages is down

This incident has been resolved.

4 updates
Mar 22

8:34 AM PDT - 11:43 AM PDT (3h9min duration)

Hydrator (Search v1) response times are slow for One Mobile requests

This incident has been resolved.

2 updates
Mar 21

10:45 AM PDT - Mar 22 11:43 AM PDT (1d57min duration)

CSRs reporting can not hear anything in MAX

This incident has been resolved.

2 updates
Mar 21

7:20 AM PDT - Mar 22 11:44 AM PDT (1d4h23min duration)

DAT One for Android users may experience crashes when viewing load details

This incident has been resolved.

2 updates
Mar 18

3:22 PM PDT - Mar 22 11:44 AM PDT (3d20h22min duration)

cx's reporting getting a grey screen when attempting to access RV, its not redirecting to HLP

This incident has been resolved.

4 updates
Mar 18

5:27 AM PDT - 8:18 AM PDT (2h51min duration)

Nice/MAX - Unable to accept calls/transfer

This incident has been resolved.

5 updates
Mar 17

1:20 PM PDT - Mar 18 8:17 AM PDT (18h57min duration)

Some paid user of DAT One Mobile Android may be forced into "Free/Basic" mode

This incident has been resolved.

5 updates
Mar 16

6:02 AM PDT - Mar 17 4:08 PM PDT (1d10h6min duration)

Cx's are getting a forced migration window

This incident has been resolved.

4 updates
Mar 15

6:00 PM PDT - 9:00 PM PDT (3h duration)

CSB Normal release to production.

The scheduled maintenance has been completed.

4 updates
Mar 15

2:02 PM PDT - Mar 16 6:23 AM PDT (16h21min duration)

slowness across many apps, CSB ping response times are very slow

This incident has been resolved.

4 updates
Mar 10

7:28 AM PST - 10:35 AM PST (3h7min duration)

Power- posts are being deleted as soon as they're made

This incident has been resolved.

3 updates
Mar 07

7:01 AM PST - Mar 8 8:14 AM PST (1d1h13min duration)

INC-231 Power alarm and sync consumers falling behind

This incident has been resolved.

3 updates
Mar 07

7:00 AM PST - Mar 8 9:51 AM PST (1d2h51min duration)

INC-230 Visible slowness in Prod Freight cluster

This incident has been resolved.

6 updates
Mar 04

11:33 AM PST - Mar 7 8:25 AM PST (2d20h52min duration)

KP related - security related

This incident has been resolved.

2 updates
Mar 04

6:29 AM PST - 1:17 PM PST (6h48min duration)

Power - Login loop

This incident has been resolved.

3 updates
Mar 01

6:00 PM PST - 9:00 PM PST (3h duration)

CSB Normal release

The scheduled maintenance has been completed.

5 updates

February 2022

Feb 25

10:32 AM PST - 10:59 AM PST (28min duration)

DAT One Mobile unable to accept or decline tracking request

Team has determined this is a known issue, documented already.
Instructions were given to customer support for them to be able to helps the clients.

3 updates
Feb 24

5:38 AM PST - 12:19 PM PST (6h41min duration)

Ambassador service was maxed out on CPU.

This incident has been resolved.

4 updates
Feb 23

12:38 PM PST - 3:24 PM PST (2h46min duration)

Customers are getting errors when attempting to sign up using SSU and OAS

This incident has been resolved.

3 updates
Feb 23

12:23 PM PST - Feb 24 8:18 AM PST (19h55min duration)

Error message when trying to sign up after they enter in their CC info

This incident has been resolved.

3 updates
Feb 22

6:27 AM PST - 12:39 PM PST (6h12min duration)

Slack not loading/logging in

This incident has been resolved.

5 updates
Feb 21

7:14 AM PST - 5:10 PM PST (9h56min duration)

Power consumers behind

Taken from an email from Sergey Doroshenko
........ When Power users search for Loads / Trucks - they automatically get alarms set meaning Alarm events will be published to the the CSB Hyperbus (stream) when other users post Loads / Trucks that match the search criteria, or when existing postings are updated, canceled, refreshed, unposted, or deleted. When Power group users manage loads, manage searches, or interact with search matches, then "sync" events are generated for other group members and published to the Hyperbus. The hyperbus has a designated queue for Alarm events, and a separate one for Sync events.

3 updates
Feb 17

12:42 PM PST - Feb 18 9:56 AM PST (21h15min duration)

Identity domain outage

This incident has been resolved.

4 updates
Feb 16

9:07 AM PST - Feb 17 5:05 AM PST (19h58min duration)

DAT One Mobile creates multiple new active searches on the user profile including at least one entry of the "previous search"

This incident has been resolved.

3 updates
Feb 14

5:54 AM PST - Feb 15 12:43 PM PST (1d6h49min duration)

CSR Pages unreachable from office

This incident has been resolved.

10 updates
Feb 11

6:52 AM PST - 1:54 PM PST (7h1min duration)

Abnormal traffic on the CSB causing slowness

This incident has been resolved.

4 updates
Feb 10

12:29 PM PST - Feb 11 3:06 PM PST (1d2h37min duration)

Issue with load search results in Power getting out of order.

This incident has been resolved.

9 updates
Feb 10

9:20 AM PST - 10:08 AM PST (47min duration)

Customers are reporting incorrect credit store and days to pay

Jeremy Branecky
Outcome is we are 'blaming Astonia' for the bad data :old-man-yells-at-ansonia: (edited)
9:58
@galena will work with @ashleyb and Rick so we are doing a better job tracking Astonia calls, so there is more visibility into how often customers are complaining about the data quality
9:59
@galena will work with Rick to communicate this incident in the next staff meeting, to share knowledge / info about how to handle it better

4 updates
Feb 07

7:10 AM PST - Feb 8 1:44 PM PST (1d6h34min duration)

Power Notifications not keep up

This incident has been resolved.

4 updates
Feb 04

8:33 AM PST - 10:42 AM PST (2h9min duration)

When users are doing a single lane search in RV for Lakeland FL it is returning a "destination invalid error". T

This incident has been resolved.

4 updates
Feb 03

6:00 PM PST - 9:00 PM PST (3h duration)

Migrate tillamook and willapa to from RHEL5/6 to RHEL8 (supported OS)

The scheduled maintenance has been completed.

4 updates
Feb 03

5:41 AM PST - 4:32 PM PST (10h51min duration)

TE customers are clicking on Trip miles the data is taking while to load

This incident has been resolved.

4 updates
Feb 02

6:00 PM PST - 10:00 PM PST (4h duration)

Power and RateView Update

The scheduled maintenance has been completed.

4 updates
Feb 02

8:16 AM PST - 11:00 AM PST (2h45min duration)

A handful of Power users are peppering the CSB with failing login attempts

This incident has been resolved.

2 updates
Feb 02

7:41 AM PST - Feb 18 10:06 AM PST (16d2h24min duration)

Issues with Heroku API

This incident has been resolved.

5 updates