The incident has been resolved by removing the failing controller from the environment.
Posted May 18, 2023 - 17:05 CDT
Monitoring
The environment is stable after identifying a controller which had abhorrent behavior, showing no outward signs of problems, but dropping work internally.
Posted May 18, 2023 - 13:47 CDT
Identified
An issue has been identified with some of the storage fabric within the datacenter. We are working towards resolution.
Posted May 18, 2023 - 13:03 CDT
Update
We are continuing to investigate the root cause of the issue.
Posted May 18, 2023 - 12:22 CDT
Update
We are continuing to investigate the root cause of the issue.
The majority of traffic had automatically shifted to other regions to rebalance the delays, and in addition to that StreamGuys staff have overrode load balancing to shift more traffic out of the affected region.
Posted May 18, 2023 - 11:50 CDT
Update
We are continuing to investigate the root cause of the issue.
Posted May 18, 2023 - 11:21 CDT
Investigating
We are currently investigating some processing delays in the IAD datacenter.