OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
NX-OS 5.2.1
Scheduled Maintenance Report for Network & Infrastructure
Completed
Suite à la mise à jour de switchs Nexus 5000
avec la derniere version 5.2.1 nous avons
rencontré pas mal de problemes aleatoires
qui ont provoqué de plusieurs pannes.

Nous avons travaillé avec les équipes de
Cisco et nous avons trouvé l'origine du
probleme. Dans la version 5.2.X Cisco a
ajouté de nouvelles fonctionalités sur
les vlan. Ainsi un vlan peut être de
2 types differents. Lors de la mise à jour
vers 5.2.1 il faut effectuer l'upgrade en
passant par une version special qui fixe
le type de vlan. Sinon, le type de vlan
n'est pas fixé et il est inconnu.

# sh vlan internal info | i UNKNOWN
sdb_vlan_type UNKNOWN(16777216, err 0x0), oper =up
sdb_vlan_type UNKNOWN(67108864, err 0x0), oper =up
sdb_vlan_type UNKNOWN(50331648, err 0x0), oper =up
sdb_vlan_type UNKNOWN(50331648, err 0x0), oper =up
sdb_vlan_type UNKNOWN(67108864, err 0x0), oper =up
sdb_vlan_type UNKNOWN(16777216, err 0x0), oper =up
sdb_vlan_type UNKNOWN(16777216, err 0x0), oper =up
sdb_vlan_type UNKNOWN(16777216, err 0x0), oper =up
sdb_vlan_type UNKNOWN(16777216, err 0x0), oper =up

Consequence: quand un robot va passer sur les
switchs pour configurer un nouveau client, tout va
sauter. C'est ce qu'il s'est passé dans la nuit de
vendredi à samedi sur un reseau HG et aujourd'hui
sur un reseau pCC:
http://travaux.ovh.net/?do=details&id=7462
http://travaux.ovh.net/?do=details&id=7469
http://travaux.ovh.net/?do=details&id=7471
http://travaux.ovh.net/?do=details&id=7470

Pour fixer:
il faut faire un
no vlan XX,XX,XX
puis reappliquer la configuration de vlan
vlan XX,XX,XX


Update(s):

Date: 2012-10-15 23:20:10 UTC
64 bytes from 172.16.XX.1: icmp_seq=59. time=0.334 ms
64 bytes from 172.16.XX.1: icmp_seq=60. time=0.433 ms

64 bytes from 172.16.XX.1: icmp_seq=66. time=0.371 ms
64 bytes from 172.16.XX.1: icmp_seq=67. time=0.344 ms
64 bytes from 172.16.XX.1: icmp_seq=68. time=0.484 ms
64 bytes from 172.16.XX.1: icmp_seq=69. time=0.337 ms
64 bytes from 172.16.XX.1: icmp_seq=70. time=0.721 ms
^C
----172.16.XX.1 PING Statistics----
95 packets transmitted, 90 packets received, 5% packet loss
round-trip (ms) min/avg/max/stddev = 0.228/0.6535/6.17/0.866
pcc-19-n5 fait.

Nous n'avons constanté aucun impact sur la production de pcc.

Tous les vlans de production des switchs de stockage de pcc ont
été corrigés du bug concernés. Nous allons effectuer ces corrections
sur le vlans de maintenance dans la journée de demain.

Date: 2012-10-15 23:13:13 UTC
64 bytes from 172.16.XX.1: icmp_seq=11. time=0.307 ms
64 bytes from 172.16.XX.1: icmp_seq=12. time=0.361 ms

64 bytes from 172.16.XX.1: icmp_seq=18. time=0.299 ms
64 bytes from 172.16.XX.1: icmp_seq=19. time=0.345 ms
64 bytes from 172.16.XX.1: icmp_seq=20. time=0.960 ms
^C
----172.16.XX.1 PING Statistics----
21 packets transmitted, 16 packets received, 23% packet loss
round-trip (ms) min/avg/max/stddev = 0.299/0.4337/0.960/0.182
pcc-118-n5 fait.

Le dernier switch pcc: pcc-119-n5

Date: 2012-10-15 23:10:13 UTC
64 bytes from 172.20.XX.1: icmp_seq=13. time=0.341 ms
64 bytes from 172.20.XX.1: icmp_seq=14. time=0.616 ms

64 bytes from 172.20.XX.1: icmp_seq=31. time=0.312 ms
64 bytes from 172.20.XX.1: icmp_seq=32. time=0.299 ms
^C
----172.20.XX.1 PING Statistics----
33 packets transmitted, 17 packets received, 48% packet loss
round-trip (ms) min/avg/max/stddev = 0.299/0.4072/0.616/0.109
pcc-113-n5 fait.

Le suivant: pcc-118-n5

Date: 2012-10-15 23:04:32 UTC
64 bytes from 172.19.XXX.1: icmp_seq=54. time=0.312 ms

64 bytes from 172.19.XXX.1: icmp_seq=66. time=0.414 ms
64 bytes from 172.19.XXX.1: icmp_seq=67. time=0.304 ms
64 bytes from 172.19.XXX.1: icmp_seq=68. time=0.273 ms
64 bytes from 172.19.XXX.1: icmp_seq=69. time=0.310 ms
64 bytes from 172.19.XXX.1: icmp_seq=70. time=0.337 ms
64 bytes from 172.19.XXX.1: icmp_seq=71. time=0.321 ms
^C
----172.19.XXX.1 PING Statistics----
72 packets transmitted, 61 packets received, 15% packet loss
round-trip (ms) min/avg/max/stddev = 0.273/0.3487/0.879/0.0768
pcc-112-n5 fait.

Le suivant: pcc-113-n5

Date: 2012-10-15 23:01:07 UTC
64 bytes from 172.19.XXX.1: icmp_seq=97. time=0.320 ms
64 bytes from 172.19.XXX.1: icmp_seq=98. time=0.339 ms

64 bytes from 172.19.XXX.1: icmp_seq=115. time=0.321 ms
64 bytes from 172.19.XXX.1: icmp_seq=116. time=0.310 ms
^C
----172.19.XXX.1 PING Statistics----
117 packets transmitted, 101 packets received, 13% packet loss
round-trip (ms) min/avg/max/stddev = 0.278/0.33181/0.538/0.04253
pcc-107-n5 fait.

Le suivant: pcc-112-n5

Date: 2012-10-15 22:51:54 UTC
64 bytes from 172.19.XX.1: icmp_seq=215. time=0.330 ms
64 bytes from 172.19.XX.1: icmp_seq=216. time=0.341 ms
64 bytes from 172.19.XX.1: icmp_seq=217. time=0.320 ms

64 bytes from 172.19.XX.1: icmp_seq=226. time=0.709 ms
64 bytes from 172.19.XX.1: icmp_seq=227. time=0.347 ms
64 bytes from 172.19.XX.1: icmp_seq=228. time=0.335 ms
64 bytes from 172.19.XX.1: icmp_seq=229. time=0.365 ms
^C
----172.19.XX.1 PING Statistics----
230 packets transmitted, 222 packets received, 3% packet loss
round-trip (ms) min/avg/max/stddev = 0.256/0.37774/1.37/0.1168
pcc-105-n5 fait.

Le suivant: pcc-107-n5

Date: 2012-10-15 22:43:57 UTC
64 bytes from 172.18.XX.1: icmp_seq=277. time=0.515 ms
64 bytes from 172.18.XX.1: icmp_seq=278. time=0.416 ms
64 bytes from 172.18.XX.1: icmp_seq=279. time=0.377 ms
64 bytes from 172.18.XX.1: icmp_seq=280. time=0.404 ms

64 bytes from 172.18.XX.1: icmp_seq=292. time=0.360 ms
64 bytes from 172.18.XX.1: icmp_seq=293. time=0.356 ms
64 bytes from 172.18.XX.1: icmp_seq=294. time=0.291 ms
64 bytes from 172.18.XX.1: icmp_seq=295. time=0.326 ms
^C
----172.18.XX.1 PING Statistics----
296 packets transmitted, 285 packets received, 3% packet loss
round-trip (ms) min/avg/max/stddev = 0.272/0.45126/2.74/0.2082
pcc-104-n5 fait.

Le suivant: pcc-105-n5

Date: 2012-10-15 22:39:01 UTC
Nous commencons avec pcc-104-n5.

Date: 2012-10-15 20:24:04 UTC
il reste les N5 de stockage de pCC
que nous allons faire cette nuit.
puis:
rps, backup, 3 couples de N5 stockage-15,27,31
qu'on fera demain matin avec toutes l'équipe
de stockage.

Date: 2012-10-15 20:19:21 UTC
178.33.122.248/24 fait

Date: 2012-10-15 19:52:25 UTC
188.165.14.0/24 fait

Date: 2012-10-15 18:36:14 UTC
188.165.15.0/24 fait
Posted Oct 15, 2012 - 17:57 UTC