Storage Issue

Status: Watching - on 11/11/19
Description:

We are experiencing abnormal load on our storage systems. This affects website load times and email access. We're working on resolving this incident.

Update on 11/12/19
Status changed from Investigating Identified

New information:

Work in progress.

Update on 11/13/19
Status changed from Identified Investigating

New information:

We still experiencing some difficulties with storage functionality. We keep working to regain stable performance.

Update 1 month ago
Status changed from Investigating Investigating

New information:

We keep working to regain stable performance.

Update 3 weeks, 6 days ago
Status changed from Investigating Identified

New information:

Due to storage issues over the last week that came to be more severe towards Friday, we started heavy maintenance operations during the weekend to alleviate the impact on our customers for the weeks to come. Some resource-heavy operations were stopped 7 AM today, but some of this work has spilled over into regular work hours. We are working with our storage vendors intensely to remedy this and restore the performance to normal.

Update 3 weeks, 5 days ago
Status changed from Identified Watching

New information:

The immediate issue has been resolved and we are seeing restored performance across the board meaning both website load times and email access has been restored to normal. We will keep monitoring the problem closely while continuing improvements as well as intensive maintenance work to ensure long term stability.

Please contact our support if you experience any further issues.

Update 2 weeks, 4 days ago
Status changed from Watching Investigating

New information:

Maintenance operation have impacted storage performance, we are working on lowering the maintenance impact on the performance.

Update 1 week, 5 days ago
Status changed from Investigating Watching

New information:

We are monitoring the storage platform and working on improving the current situation. Overall access and performance are returned to normal, but we're working close with the vendors to improve it further.

Update 1 week, 3 days ago
Status changed from Watching Investigating

New information:

Maintenance operation have impacted storage performance, we are working on lowering the maintenance impact on the performance.

Update 1 week, 3 days ago
Status changed from Investigating Identified

New information:

There is a very demanding backend maintenance task running on the storage, this was not anticipated by hardware vendors support and can't be force-stopped. We unfortunately have to let this finish. Issue is escalated to the highest level and we're doing our best to solve this asap. Measures are taken to entirely avoid this behavior in the future. Once the maintenance is complete we expect overall storage performance improved compared to last weeks.

Update 6 days, 13 hours ago
Status changed from Identified Identified

New information:

Maintenance operation that was started on Thursday by storage vendors support team has not finished yet, and vendors strongly recommend that we wait for it to complete - that should improve the platform stability. We are working on ways to accelerate this

Update 5 days, 11 hours ago
Status changed from Identified Identified

New information:

Maintenance operation that was started on Thursday by storage vendors support team is still running, storage is not performing at full capacity and we are aware of it and working to minimize the impact. Vendors strongly recommend that we wait for it to complete - that should improve the platform stability. Vendors developers have adjusted a firmware update to our needs that is expected to accelerate this, we will be attempting to apply the patch starting 2019-12-09 16:00. This isn't expected to cause downtime

Update 5 days, 10 hours ago
Status changed from Identified Identified

New information:

Update: The firmware patch hasn't gone through, it will be modified and we will attempt again tomorrow at 16:00

Update 3 days, 18 hours ago
Status changed from Identified Watching

New information:

Finally the background operations has finished and storage performance has normalized. We continue to monitor the situation and solving some minor issues. Please contact our support if you experience issues.

logo
Copyright © 2019 Servage GmbH