OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
Upgrades ASR 9K Roubaix
Scheduled Maintenance Report for Network & Infrastructure
Completed
Nous allons upgrader les 2 routeurs \"core\" ici à Roubaix en tête du réseau afin de pouvoir ajouter de la capacité. Ces chassis sont maintenant full en terme de linecard 8x10G. Nous allons passer aux nouvelles RSP 440 qui vont nous permettre de remplacer certaines de ces linecards par les nouvelles 24x10G. La procédure est la suivante:
- upgrade software
- remplacement des RSPs (arrêt/redémarrage complet du chassis nécessaire)
- remplacement de 2 des linecards 8G par des 24g

Ces interventions auront bien entendu lieu de nuit, on démarre cette nuit (du 10 au 11) avec rbx-g1-a9 à 00:00 CET. Pas d'impact sur le trafic à anticiper car nous allons forcer le trafic par rbx-g2 pendant la durée des travaux.

Update(s):

Date: 2012-01-27 00:08:27 UTC
Les nouvelles cartes sont en prod sur rbx-g1-a9.

Date: 2012-01-26 23:20:52 UTC
Les nouvelles cartes sont en prod sur rbx-g2. Même chose maintenant pour rbx-g1.

Date: 2012-01-26 23:02:39 UTC
Nous commencons l'intervention de remplacement des cartes 8x10G par les 24x10G.
Nous allons intervenir sur rbx-g2-a9 puis rbx-g1-a9.

Date: 2012-01-26 18:50:52 UTC
Nous allons upgrader la capa entre rbx-g1/g2-a9 et les rbx-g1/g2-n7. L'upgrade sur l'ensemble g2 est en cours. Sur g1, nous devons remplacer certaines cartes 8T par des 24x10G. Nous allons réaliser cette opération cette nuit à partir de 00:00 CET.

Date: 2012-01-17 03:00:00 UTC
Nous avons finalement pu faire fonctinner un autre type d'optique. On laisse tourner cette 24x10G afin de vérifier son fonctionnement. S'il n'y a pas d'autre soucis, nous insérerons les suivantes.

Date: 2012-01-17 02:18:34 UTC
On fallback sur la 8x10G. Nous allons travailler avec cisco sur le problème de compatibilité des optiques et vérifier les implications exacts du message \"MIXED_MEMORY_ALARM\" remonté par la carte à l'insertion.

Date: 2012-01-17 02:04:29 UTC
Nos optiques sont bien reconnues par les cartes mais n'émettent pas de lumière. Nous suspectons une incompatibilité.

Date: 2012-01-17 01:22:25 UTC
RP/0/RSP0/CPU0:rbx-g2-a9#RP/0/RSP1/CPU0:Jan 17 01:21:39 UTC: dao_tmp[55]: Card Inserted 0/0/CPU0
RP/0/RSP0/CPU0:Jan 17 01:21:39 UTC: dao_tmp[55]: Card Inserted 0/0/CPU0
RP/0/RSP0/CPU0:Jan 17 01:21:50 UTC: shelfmgr[385]: %PLATFORM-SHELFMGR-3-MIXED_MEMORY_POWERED : Node 0/0/CPU0 is powered, system running mixed li
ode
RP/0/RSP0/CPU0:Jan 17 01:21:50 UTC: shelfmgr[385]: %PLATFORM-SHELFMGR-3-MIXED_MEMORY_ALARM : System enters mixed line card memory mode
RP/0/RSP0/CPU0:Jan 17 01:21:50 UTC: eem_ed_oir[190]: Messge received content : Event 0 NodeId: 0x801


Date: 2012-01-17 01:15:47 UTC
Les 2 routeurs sont stables. On remplace la carte 0 de rbx-g2 par l'une des nouvelles carte 24x10G.

Date: 2012-01-17 01:02:50 UTC
On réactive le trafic sur g2.

Date: 2012-01-17 00:39:19 UTC
On redescend la config sur g2.

Date: 2012-01-17 00:34:44 UTC
Sur ces RSPS, une nouvelle config port console est nécessaire:
line console
exec-timeout 0 0
parity none
stopbits 1
length 60
access-class ingress 10
access-class egress 10


Date: 2012-01-17 00:20:42 UTC
Nous remplacons les RSP sur rbx-g2

Date: 2012-01-17 00:20:30 UTC
Le trafic s'écoule normalement sur g1. On attaque g2.

Date: 2012-01-17 00:05:37 UTC
g1 a été configuré. on va lui repasser le trafic


Date: 2012-01-16 23:58:22 UTC
Le routeur a démarré correctement sur les nouvelles RSPs. Nous redescendons la config.

Date: 2012-01-16 23:47:31 UTC
le routeur a booté. on est en train de lui repusher la configuration

Date: 2012-01-16 23:27:23 UTC
Nous remplacons les cartes RSP.

Date: 2012-01-16 23:11:13 UTC
Nous commencons les travaux. On passe le trafic vers rbx-g2.

Date: 2012-01-16 18:46:10 UTC
Nous reprenons les travaux ce soir à partir de 00:00 CET. Les 2 routeurs sont maintenant à jour au niveau soft et nous avons installé l'IOS XR sur les nouvelles RSPS 440. Nous allons les insérer cette nuit dans les chassis rbx-g1 et g2. Si tout se déroule correctement, nous pourrons insérer les nouvelles 24x10G.

Date: 2012-01-11 03:44:48 UTC
Le trafic est réactivé. Les 2 routeurs rbx-g1 et g2 sont en fonctionnement normal.

Date: 2012-01-11 02:53:01 UTC
Le routeur est en cours de redémarrage

Date: 2012-01-11 01:35:43 UTC
on remet les anciennes RP2 et on redemarre le g1.
on va lui remettre le trafic comme avant mais en 4.2
puis mettre à jour le g2.

puis on va regarder avec cisco comment on demarre
ces cartes sans image ..

Date: 2012-01-11 01:21:36 UTC
c'est fait. le routeur est à nouveau down. on insert les nouvelles
cartes qu'on va reconfigurer from scratch puisqu'elles sont vierges

Date: 2012-01-11 01:18:39 UTC
on va maintenant sortir les RP2 du chassis g1 et
mettre en place les nouvelles RP440.

Date: 2012-01-11 01:15:10 UTC
Successfully upgraded fpga3 for A9K-RSP-4G on location 0/RSP0/CPU0 from 1.18 to 1.23
Successfully upgraded fpga3 for A9K-RSP-4G on location 0/RSP1/CPU0 from 1.18 to 1.23
FPD upgrade in progress. Max timeout remaining 89 min.
Successfully upgraded cpld1 for A9K-8T-L on location 0/4/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/4/CPU0 from 1.00 to 1.03
FPD upgrade in progress. Max timeout remaining 88 min.
FPD upgrade in progress. Max timeout remaining 87 min.
FPD upgrade in progress. Max timeout remaining 86 min.
FPD upgrade in progress. Max timeout remaining 85 min.
FPD upgrade in progress. Max timeout remaining 84 min.
Successfully upgraded fpga for A9K-8T-L on location 0/6/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/6/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/6/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/6/CPU0 from 1.00 to 1.02
FPD upgrade in progress. Max timeout remaining 83 min.
FPD upgrade in progress. Max timeout remaining 82 min.
Successfully upgraded fpga for A9K-8T-L on location 0/2/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/2/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/2/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/2/CPU0 from 1.00 to 1.02
Successfully upgraded fpga for A9K-8T-L on location 0/1/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/1/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/1/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/1/CPU0 from 1.00 to 1.02
FPD upgrade in progress. Max timeout remaining 81 min.
Successfully upgraded fpga for A9K-8T-L on location 0/5/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/5/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/5/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/5/CPU0 from 1.00 to 1.02
Successfully upgraded fpga for A9K-8T-L on location 0/0/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/0/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/0/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/0/CPU0 from 1.00 to 1.02
Successfully upgraded fpga for A9K-8T-L on location 0/7/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/7/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/7/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/7/CPU0 from 1.00 to 1.02
Successfully upgraded fpga for A9K-8T-L on location 0/3/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/3/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/3/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/3/CPU0 from 1.00 to 1.02


FPD upgrade has ended.



Date: 2012-01-11 01:06:26 UTC
RP/0/RSP0/CPU0:rbx-g1-a9(admin)#upgrade hw-module fpd all location all
Wed Jan 11 01:04:46.813 UTC

***** UPGRADE WARNING MESSAGE: *****
* This upgrade operation has a maximum timout of 90 minutes. *
* If you are executing the cmd for one specific location and *
* card in that location reloads or goes down for some reason *
* you can press CTRL-C to get back the RP's prompt. *
* If you are executing the cmd for _all_ locations and a node *
* reloads or is down please allow other nodes to finish the *
* upgrade process before pressing CTRL-C. *

% RELOAD REMINDER:
- The upgrade operation of the target module will not interrupt its normal
operation. However, for the changes to take effect, the target module
will need to be manually reloaded after the upgrade operation. This can
be accomplished with the use of \"hw-module reload\" command.
- If automatic reload operation is desired after the upgrade, please use
the \"reload\" option at the end of the upgrade command.
- The output of \"show hw-module fpd location\" command will not display
correct version information after the upgrade if the target module is
not reloaded.
NOTE: Chassis CLI will not be accessible while upgrade is in progress.
Continue? [confirm]
This can take some time for a full chassis.
Ensure that system is not power cycled during the upgrades.
Please consult the documentation for more information.
Continue ? [no]: yes


FPD upgrade in progress on some hardware, reload/configuration change
on those is not recommended as it might cause HW programming failure
and result in RMA of the hardware.



Date: 2012-01-11 01:02:37 UTC
RP/0/RSP0/CPU0:rbx-g1-a9(admin)#sh install inactive
Wed Jan 11 00:59:08.043 UTC
Secure Domain Router: Owner

Node 0/RSP0/CPU0 [RP] [SDR: Owner]
Boot Device: disk0:
Inactive Packages:
disk0:asr9k-mini-p-4.0.1
disk0:asr9k-doc-p-4.0.1
disk0:asr9k-k9sec-p-4.0.1
disk0:asr9k-mpls-p-4.0.1
disk0:asr9k-mgbl-p-4.0.1
disk0:asr9k-mcast-p-4.0.1
[...]

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#install remove ?
WORD (may include '*' or '?')
disk0:asr9K-doc-supp-4.0.1 Package to remove
disk0:asr9k-base-4.0.1 Package to remove
disk0:asr9k-cpp-4.0.1 Package to remove
[...]

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#install remove inactive synchronous
Wed Jan 11 01:00:11.094 UTC
Install operation 136 '(admin) install remove inactive synchronous' started by user 'octave' via CLI at 01:00:11 UTC Wed Jan 11 2012.
Info: This operation will remove the following packages:
Info: disk0:asr9k-mpls-p-4.0.1
Info: disk0:asr9k-mini-p-4.0.1
Info: disk0:asr9k-mgbl-p-4.0.1
Info: disk0:asr9k-mcast-p-4.0.1
Info: disk0:asr9k-k9sec-p-4.0.1
Info: disk0:asr9k-doc-p-4.0.1
Info: After this install remove the following install rollback points will no longer be reachable, as the required packages will not be present:
Info: 114, 116
Proceed with removing these packages? [confirm]


Date: 2012-01-11 00:59:12 UTC
RP/0/RSP0/CPU0:rbx-g1-a9(admin)#sh install summary
Wed Jan 11 00:58:56.412 UTC
Default Profile:
SDRs:
Owner
Active Packages:
disk0:asr9k-mini-p-4.2.0
disk0:asr9k-doc-p-4.2.0
disk0:asr9k-k9sec-p-4.2.0
disk0:asr9k-mpls-p-4.2.0
disk0:asr9k-mgbl-p-4.2.0
disk0:asr9k-mcast-p-4.2.0



Date: 2012-01-11 00:55:14 UTC
le routeur est up.

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#sh install summary
Wed Jan 11 00:53:30.008 UTC
Default Profile:
SDRs:
Owner
Active Packages:
disk0:asr9k-upgrade-p-4.2.0
disk0:asr9k-mini-p-4.2.0
disk0:asr9k-doc-p-4.2.0
disk0:asr9k-k9sec-p-4.2.0
disk0:asr9k-mpls-p-4.2.0
disk0:asr9k-mgbl-p-4.2.0
disk0:asr9k-mcast-p-4.2.0

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#install deactivate disk0:asr9k-upgrade-p-4.2.0 sync

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#install remove disk0:asr9k-upgrade-p-4.2.0 sync
Wed Jan 11 00:56:34.030 UTC
Install operation 134 '(admin) install remove disk0:asr9k-upgrade-p-4.2.0 synchronous' started by user 'octave' via CLI at 00:56:34 UTC Wed Jan 11 2012.
Info: This operation will remove the following package:
Info: disk0:asr9k-upgrade-p-4.2.0
Info: After this install remove the following install rollback point will no longer be reachable, as the required packages will not be present:
Info: 132
Proceed with removing these packages? [confirm]
Install operation 134 completed successfully at 00:56:52 UTC Wed Jan 11 2012.


Date: 2012-01-11 00:47:08 UTC
This (D)RP Node is not ready or active for login /configuration

This (D)RP Node is not ready or active for login /configuration

This (D)RP Node is not ready or active for login /configuration

*** insthelper node 0/RSP0/CPU0: rebooting in 35 seconds due to reload upgrade 132 ...
[0x79701f673bab5] Record Reboot History, reboot cause = 0x4000014, descr = Cause: Install operation. Shutdown standby node gracefully. Process: redcon
Traceback: 4c223c30 4rebooting

Selecting ROMMON Image... B
DDR in Interleaved mode
POST 1 : PASSED : code 0 : DDR2 Memory Quick Test

CPU Reset Reason = 0x000d
POST 2 : PASSED : code 0 : FPGA Flash Image CRC Checks

Loading Field Programmable Devices:
FPGA 0-B PROGRAMMED : image: 0xff500028 - 0xff576cca, et: 117ms
FPGA 1-B PROGRAMMED : image: 0xff400028 - 0xff4d1034, et: 206ms
FPGA 2-B PROGRAMMED : image: 0xff100028 - 0xff276358, et: 369ms
FPGA 3-B PROGRAMMED : image: 0xff000028 - 0xff0454a8, et: 69ms

System Bootstrap, Version 1.05(20101118:025914) [ASR9K ROMMON],
Copyright (c) 1994-2010 by Cisco Systems, Inc.
Compiled Wed 17-Nov-10 18:59 by saurabja
[...]

Date: 2012-01-11 00:42:34 UTC
Proceed with this install operation (y/n)? [y]
Info: Install Method: Parallel Reload
Info: The changes made to software configurations will not be persistent across system reloads. Use the command '(admin) install commit' to make changes persistent.
Info: Please verify that the system is consistent following the software change using the following commands:
Info: show system verify
Info: install verify packages
Install operation 132 completed successfully at 00:41:37 UTC Wed Jan 11 2012.
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#

ça y est ... il s'est rebooté

Date: 2012-01-11 00:20:51 UTC
pendant ce temps là sur le g2:

RP/0/RSP0/CPU0:rbx-g2-a9#delete disk0:ASR9K-iosxr-k9-4.1.1.tar

RP/0/RSP0/CPU0:rbx-g2-a9#sh install inactive
Wed Jan 11 00:19:21.016 UTC
Node 0/RSP0/CPU0 [RP] [SDR: Owner]
Boot Device: disk0:
Inactive Packages:
disk0:asr9k-upgrade-p-4.1.0
disk0:asr9k-mini-p-4.1.0
disk0:asr9k-doc-p-4.1.0
disk0:asr9k-k9sec-p-4.1.0
disk0:asr9k-mpls-p-4.1.0
disk0:asr9k-mgbl-p-4.1.0
disk0:asr9k-mcast-p-4.1.0
[...]
RP/0/RSP0/CPU0:rbx-g2-a9#admin
Wed Jan 11 00:19:38.546 UTC
RP/0/RSP0/CPU0:rbx-g2-a9(admin)#install remove inactive
Wed Jan 11 00:19:44.906 UTC
Install operation 26 '(admin) install remove inactive' started by user 'octave' via CLI at 00:19:45 UTC Wed Jan 11 2012.
Info: This operation will remove the following packages:
Info: disk0:asr9k-upgrade-p-4.1.0
Info: disk0:asr9k-doc-4.1.0
Info: disk0:asr9k-adv-video-4.1.0
Info: disk0:asr9k-sbc-4.1.0
Info: disk0:asr9k-k9sec-4.1.0
[...]
Proceed with removing these packages? [confirm]
The install operation will continue asynchronously.


Date: 2012-01-11 00:15:54 UTC
(admin) install add source disk0:// asr9k-doc-p.pie-4.2.0 asr9k-k9sec-p.pie-4.2.0 asr9k-mcast-p.pie-4.2.0 asr9k-mgbl-p.pie-4.2.0 asr9k-mini-p.pie-4.2.0 asr9k-mpls-p.pie-4.2.0 asr9k-upgrade-p.pie-4.2.0 synchronous

Date: 2012-01-10 23:55:59 UTC
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#show install inactive
Tue Jan 10 23:55:33.960 UTC
Secure Domain Router: Owner

There are no inactive packages on the SDR.

RP/0/RSP1/CPU0:rbx-g1-a9#sh install summary
Tue Jan 10 23:56:25.147 UTC
Active Packages:
disk0:asr9k-mini-p-4.0.1
disk0:asr9k-doc-p-4.0.1
disk0:asr9k-k9sec-p-4.0.1
disk0:asr9k-mpls-p-4.0.1
disk0:asr9k-mgbl-p-4.0.1
disk0:asr9k-mcast-p-4.0.1



Date: 2012-01-10 23:30:16 UTC
nous allons preparer le routeur à la mise en place
de la nouvelle version de IOS XR

Date: 2012-01-10 22:54:38 UTC
Nous démarrons les travaux. Nous allons rerouter une partie du trafic des transits à Londres et Amsterdam vers Paris afin de délester les liens de rbx-g1 vers ldn-1/5 et ams-1/5.
Posted Jan 10, 2012 - 11:30 UTC