rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_red
icon_green
icon_green
icon_blue
icon_red
icon_red
icon_green
icon_green
icon_green
icon_red
icon_green
icon_red
icon_green
icon_green
icon_red
icon_green
icon_green
icon_green
icon_green
icon_red
icon_orange
icon_green
icon_green
icon_red
icon_green
icon_green
icon_green
icon_blue
icon_red
 

FS#47481 — Object Storage / Cloud Archive GRA

Attached to Project— Cloud
Incident
cloud
CLOSED
100%
We are observing an increase of latency and error rate. We are investigating.
Date:  Tuesday, 10 November 2020, 17:08PM
Reason for closing:  Done
Comment by OVH - Wednesday, 04 November 2020, 09:17AM

Latency is due to an increase of traffic on our containers servers. We are scaling up.


Comment by OVH - Wednesday, 04 November 2020, 11:58AM

We added more servers to handle the load. Since 7h45 UTC we observe better performances. We are still investigating.


Comment by OVH - Wednesday, 04 November 2020, 16:24PM

As a consequences of FS#47465 , one component of the Swift cluster was late in executing asynchronous task. With too much concurrency, these tasks were overwhelming the cluster. We are reducing concurrency to better manage the load.


Comment by OVH - Thursday, 05 November 2020, 10:44AM

The load is managed and the lag in asynchronous tasks treatment is recovering. We are monitoring closely.


Comment by OVH - Thursday, 05 November 2020, 17:24PM

Performances are back to normal since 04 November 7h45 UTC.
The listing of some containers could be inconsistent due to the lag with the asynchronous update tasks.
Now that the lag has been partially recovered, we are increasing the concurrency to recover faster and we are watching closely the performances to prevent any impact.


Comment by OVH - Friday, 06 November 2020, 12:17PM

The infra is recovering the lag of asynchronous tasks. We are monitoring infra performances closely.


Comment by OVH - Friday, 06 November 2020, 18:47PM

The consistency lag is reducing well, we keep watching the monitoring.


Comment by OVH - Saturday, 07 November 2020, 12:09PM

The consistency lag is still reducing well. At this time, 80% of asynchronous update tasks have been treated.


Comment by OVH - Sunday, 08 November 2020, 21:22PM

The consumption of asynchronous update tasks has slowed down due to an increase of traffic. We are monitoring closely the infrastructure to avoid any other impact.


Comment by OVH - Monday, 09 November 2020, 11:48AM

The consumption of asynchronous update tasks is back to normal since few hours.


Comment by OVH - Monday, 09 November 2020, 14:17PM

The consistency lag is still reducing well. At this time, 90% of asynchronous update tasks have been treated.


Comment by OVH - Tuesday, 10 November 2020, 17:08PM

All the asynchronous update tasks have been treated.