rssLink RSS for all categories
 
icon_red
icon_green
icon_blue
icon_orange
icon_blue
icon_green
icon_green
icon_red
icon_green
icon_red
icon_green
icon_green
icon_green
icon_red
icon_green
icon_green
icon_green
icon_green
icon_orange
icon_orange
icon_green
icon_green
icon_blue
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_green
icon_green
icon_orange
 

FS#28243 — PCI - SBG

Attached to Project— Cloud
Incident
cloud
CLOSED
100%
Nous avons un incident électrique majeur sur SBG l'ensemble du datacentre n'est pas joignable.

Nous vous tenons informé de la suite des évènements.
Date:  Thursday, 16 November 2017, 14:33PM
Reason for closing:  Done
Comment by OVH - Thursday, 09 November 2017, 07:54AM

vous pouvez suivre les évènements ici : http://travaux.ovh.net/?do=details&id=28244


Comment by OVH - Thursday, 09 November 2017, 17:16PM

FR

Depuis ce matin nous avons remis en route les API OpenStack de SBG1 et SBG3. Il reste 50 hosts indisponibles suite à des problèmes hardware. Nous travaillons en collaboration avec les équipes au DC pour tout remettre en route.

L'ensemble des IP à SBG3 répondent aux pings.
À SBG1 il reste ~3000 ips qui sont toujours down, mais nous travaillons à les fixer

Une conséquence de cet incident est que le DHCP dans les réseau vRack ne fonctionnent pas correctement (Problématique sur les spawn d'instances).

---
EN

Since this morning we have put back online the OpenStack APIs for SBG1 and SBG3. There are still 50 hosts that remain down due to a hardware Issue. We are working in collaboration with the teams in the DC to re-enable them all.

All IPs in SBG3 region are now up.
In SBG1 there are still ~3000 ips that are still down, but we are working on fixing all of them.

One of the consequences of this incident is with the DHCP servers within a vRack network not working properly (This causes issue mainly on new instance creation within a vRack network).


Comment by OVH - Friday, 10 November 2017, 00:02AM

There are still ~1000 ips that are still down, but we are working to fix all of them as soon as possible.


Comment by OVH - Friday, 10 November 2017, 05:21AM

We lost a part of our control plane in SBG3 but everything is still working as usual.


Comment by OVH - Friday, 10 November 2017, 13:14PM

Multiple host are down on SBG1 and SBG3. DC team working on host to fix it.


Comment by OVH - Friday, 10 November 2017, 15:10PM

We have 64 hosts that are still down. We are working on the spares
to change the pieces and restart each server.


Comment by OVH - Monday, 13 November 2017, 10:46AM

We are facing an issue with our mysql cluster right now on SBG3.
API are impacted.
Instances are not impacted.