rssLink RSS for all categories
 
icon_blue
icon_green
icon_orange
icon_orange
icon_red
icon_green
icon_green
icon_orange
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_red
icon_orange
icon_red
icon_green
icon_blue
icon_red
icon_red
icon_green
icon_green
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_green
icon_green
icon_red
 

FS#29462 — storage.bhs1.cloud.ovh.net

Attached to Project— Cloud
Maintenance
cloud
CLOSED
100%
Latencies are unusually high, we are investigating
Date:  Monday, 22 January 2018, 06:19AM
Reason for closing:  Done
Additional comments about closing:  Latencies and bandwidth are back to normal values
Comment by OVH - Friday, 19 January 2018, 09:17AM

We identified and are isolating a suspicious swift-proxy server


Comment by OVH - Friday, 19 January 2018, 09:26AM

Latencies are back to normal values


Comment by OVH - Friday, 19 January 2018, 09:30AM

This fault proxy server missed a kernel patch which fixes a bug in TCP stack, resulting in high load (lots of soft IRQs) : https://github.com/torvalds/linux/commit/627f3abeeac1354a4b15eaf0f22b8ffb901029cb
We upgraded its kernel and are rebooting it


Comment by OVH - Friday, 19 January 2018, 09:35AM

The upgraded proxy is back in production and behaves normally


Comment by OVH - Friday, 19 January 2018, 09:41AM

There is still an issue on bandwidth, we are adding resources in the cluster


Comment by OVH - Friday, 19 January 2018, 09:48AM

We are deploying proxy service on account/container servers and adding a spare host as an additional proxy server


Comment by OVH - Friday, 19 January 2018, 10:25AM

2 more nodes are now handling requests


Comment by OVH - Friday, 19 January 2018, 10:46AM

With the newly added capacity on swift side load balancers are now overloaded


Comment by OVH - Friday, 19 January 2018, 11:27AM

We added an additional node, bandwidth is better but latencies increased, we are ordering more hardware


Comment by OVH - Friday, 19 January 2018, 11:52AM

4 more nodes have been delivered, we are installing and preparing them for production


Comment by OVH - Friday, 19 January 2018, 14:28PM

the 4 additional nodes are now in production


Comment by OVH - Friday, 19 January 2018, 16:17PM

We managed to stabilize the latencies, but they are still high. We are ordering a new batch of servers


Comment by OVH - Friday, 19 January 2018, 17:57PM

the new batch of servers are now in production


Comment by OVH - Friday, 19 January 2018, 21:42PM

we managed to improve bandwidth on this cluster, the situation is not yet satisfying but is no more critical