OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
vss-1b-6k
Incident Report for Network & Infrastructure
Resolved
Depuis quelques jours nous avons de messages bizarres:
Sep 8 08:42:54 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (3/3),process = RFSS worker process.
-Traceback= 40B6823C 40B671A4 40B672DC 40B6984C 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4

c'est totalement bizarre car le routeur est un routeur
de failover qui ne route pas le trafic pour l'instant.



Update(s):

Date: 2011-09-08 10:00:04 UTC
c'est bon, plus de probleme.


Date: 2011-09-08 09:59:03 UTC
et le CPU est okey maintenant. donc c'est réellement la
sauvegarde sur le disk0 qui a bloqué tout le routeur !?

Sep 8 10:13:54 GMT: %IPC-SP-5-WATERMARK: 3388 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:14:31 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:15:03 GMT: %IPC-SP-5-WATERMARK: 3387 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:15:39 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:16:12 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:16:45 GMT: %IPC-SP-5-WATERMARK: 3387 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:17:31 GMT: %IPC-SP-5-WATERMARK: 1685 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:18:06 GMT: %IPC-SP-5-WATERMARK: 1696 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:18:38 GMT: %IPC-SP-5-WATERMARK: 1687 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:19:14 GMT: %IPC-SP-5-WATERMARK: 1684 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:19:45 GMT: %IPC-SP-5-WATERMARK: 3391 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:20:22 GMT: %IPC-SP-5-WATERMARK: 3397 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:20:51 GMT: %IPC-SP-5-WATERMARK: 1697 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:21:14 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (2/2),process = RFSS worker process.
-Traceback= 40B8A0E4 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4 40D66278 40D61EF4 40D628CC 40D615AC
Sep 8 10:21:29 GMT: %IPC-SP-5-WATERMARK: 3393 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:22:00 GMT: %IPC-SP-5-WATERMARK: 3399 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:22:39 GMT: %IPC-SP-5-WATERMARK: 1700 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:23:11 GMT: %IPC-SP-5-WATERMARK: 3392 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:23:41 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (3/3),process = RFSS worker process.
-Traceback= 40B673B8 40B6984C 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4 40D66278
Sep 8 10:23:57 GMT: %IPC-SP-5-WATERMARK: 3387 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:24:41 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:25:17 GMT: %IPC-SP-5-WATERMARK: 3388 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:25:48 GMT: %IPC-SP-5-WATERMARK: 1685 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:26:24 GMT: %IPC-SP-5-WATERMARK: 3392 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:27:04 GMT: %IPC-SP-5-WATERMARK: 1685 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:27:35 GMT: %IPC-SP-5-WATERMARK: 1701 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:28:07 GMT: %IPC-SP-5-WATERMARK: 3394 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:28:12 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (3/3),process = RFSS worker process.
-Traceback= 40B68864 40B69890 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4 40D66278
Sep 8 10:28:15 GMT: %FILESYS-SP-5-DEV: PCMCIA flash card removed from disk0
Sep 8 10:28:23 GMT: %FILESYS-SP-5-DEV: PCMCIA flash card inserted into disk0
Sep 8 10:28:23 GMT: %DIAG-SP-6-TEST_RUNNING: Module 5: Running TestCFRW{ID=41} ...
Sep 8 10:28:25 GMT: %DIAG-SP-6-TEST_OK: Module 5: TestCFRW{ID=41} has completed successfully
Sep 8 10:28:41 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:29:12 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:29:42 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:30:13 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:30:43 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:31:01 GMT: %SYS-SP-3-CONFIG_NV_ERR: Nonvolatile storage write error; configuration failed
Sep 8 10:31:16 GMT: %IPC-SP-5-WATERMARK: 2887 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
vss-1b-6k#



Date: 2011-09-08 09:47:36 UTC
vss-1b-6k# sh run
Unable to get configuration. Try again later.

super !

bon, allez on va marcher un peu

avant
http://www.yfrog.com/z/18ki8hvj

action:
http://www.yfrog.com/z/kho5zjxj
Sep 8 10:28:15 GMT: %FILESYS-SP-5-DEV: PCMCIA flash card removed from disk0

action:
http://www.yfrog.com/z/h759nkpuj
Sep 8 10:28:23 GMT: %FILESYS-SP-5-DEV: PCMCIA flash card inserted into disk0

c'est bon :)

vss-1b-6k#wr me
Building configuration...
Compressed configuration from 2398126 bytes to 471700 bytes
[OK]



Date: 2011-09-08 09:01:59 UTC
Sep 8 09:52:55 GMT: %IPC-SP-5-WATERMARK: 3396 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:53:33 GMT: %IPC-SP-5-WATERMARK: 1696 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:54:10 GMT: %IPC-SP-5-WATERMARK: 1695 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:54:50 GMT: %IPC-SP-5-WATERMARK: 3393 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:55:33 GMT: %FM-4-FLOWMASK_REDUCED: Features configured on interface Port-channel5 have conflicting flowmask requirements, some features may work in software
Sep 8 09:55:22 GMT: %IPC-SP-5-WATERMARK: 1688 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:55:52 GMT: %IPC-SP-5-WATERMARK: 3396 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:56:21 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (4/4),process = RFSS worker process.
-Traceback= 40B67178 40B672DC 40B6984C 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0BC 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4
Sep 8 09:56:23 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:56:53 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:57:25 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:57:53 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (3/3),process = RFSS worker process.
-Traceback= 40B673DC 40B6984C 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4 40D66278
Sep 8 09:57:59 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:58:43 GMT: %IPC-SP-5-WATERMARK: 3388 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:59:15 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:59:50 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:00:26 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:00:56 GMT: %IPC-SP-5-WATERMARK: 3388 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000


Date: 2011-09-08 08:55:58 UTC
on active le netflow et on le desactive.

vss-1b-6k(config)#mls flow ip interface-full
vss-1b-6k(config)#mls flow ipv6 interface-full
vss-1b-6k(config)#no mls flow ipv6 interface-full
% Reset for ipv6 protocol to the default flow mask type: none

vss-1b-6k(config)#no mls flow ip interface-full
vss-1b-6k(config)#

Date: 2011-09-08 08:53:00 UTC
on ne trouve pas l'origine du probleme.

ah, du nouveau ce matin:

Sep 8 09:28:37 GMT: %IPC-SP-5-WATERMARK: 1690 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:29:15 GMT: %IPC-SP-5-WATERMARK: 3411 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:29:48 GMT: %IPC-SP-5-WATERMARK: 3402 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:30:34 GMT: %IPC-SP-5-WATERMARK: 1694 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:31:11 GMT: %IPC-SP-5-WATERMARK: 3392 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:31:43 GMT: %IPC-SP-5-WATERMARK: 3408 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:32:17 GMT: %IPC-SP-5-WATERMARK: 1697 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:33:03 GMT: %IPC-SP-5-WATERMARK: 1699 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:33:32 GMT: %IPC-SP-5-WATERMARK: 1694 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:34:03 GMT: %IPC-SP-5-WATERMARK: 3410 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000

nous avons déjà eu ce probleme là et c'est sur les tables netflow

Sep 8 09:49:29 GMT: %IPC-SP-5-WATERMARK: 1732 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Posted Sep 08, 2011 - 08:48 UTC