We are observing an increase of latency and error rate. We are investigating.
Update(s):
Date: 2020-11-10 16:08:00 UTC All the asynchronous update tasks have been treated.
Date: 2020-11-09 13:17:31 UTC The consistency lag is still reducing well. At this time, 90% of asynchronous update tasks have been treated.
Date: 2020-11-09 10:48:03 UTC The consumption of asynchronous update tasks is back to normal since few hours.
Date: 2020-11-08 20:22:58 UTC The consumption of asynchronous update tasks has slowed down due to an increase of traffic. We are monitoring closely the infrastructure to avoid any other impact.
Date: 2020-11-07 11:09:44 UTC The consistency lag is still reducing well. At this time, 80% of asynchronous update tasks have been treated.
Date: 2020-11-06 17:47:17 UTC The consistency lag is reducing well, we keep watching the monitoring.
Date: 2020-11-06 11:17:24 UTC The infra is recovering the lag of asynchronous tasks. We are monitoring infra performances closely.
Date: 2020-11-05 16:24:20 UTC Performances are back to normal since 04 November 7h45 UTC.
The listing of some containers could be inconsistent due to the lag with the asynchronous update tasks.
Now that the lag has been partially recovered, we are increasing the concurrency to recover faster and we are watching closely the performances to prevent any impact.
Date: 2020-11-05 09:44:59 UTC The load is managed and the lag in asynchronous tasks treatment is recovering. We are monitoring closely.
Date: 2020-11-04 15:24:08 UTC As a consequences of FS#47465 , one component of the Swift cluster was late in executing asynchronous task. With too much concurrency, these tasks were overwhelming the cluster. We are reducing concurrency to better manage the load.
Date: 2020-11-04 10:58:11 UTC We added more servers to handle the load. Since 7h45 UTC we observe better performances. We are still investigating.
Date: 2020-11-04 08:17:37 UTC Latency is due to an increase of traffic on our containers servers. We are scaling up.