Cisco 7600 Router Got Software Forced Crash During Booting

Symptoms:

One of Cisco7606 Routers got a software crash during booting process. Here is output from console.


System Bootstrap, Version 12.2(33r)SRD5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2009 by cisco Systems, Inc.
C7600-RSP720/SP platform with 1048576 Kbytes of main memory

Autoboot executing command: "boot bootdisk:c7600rsp72043-advipservicesk9-mz.122-33.SRD3.bin"

Initializing ATA monitor library...

Self extracting the image... [OK]
Self decompressing the image : ################################################################################################################################################################################################################################# [OK]

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706

Cisco IOS Software, c7600rsp72043_sp Software (c7600rsp72043_sp-ADVIPSERVICESK9-M), Version 12.2(33)SRD3, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 10-Sep-09 12:50 by prod_rel_team
Image text-base: 0x0800015C, data-base: 0x0C000000


*Aug 27 13:13:43.659: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Aug 27 13:13:41.851: %PFREDUN-6-ACTIVE: Initializing as ACTIVE processor

*Aug 27 13:13:43.659: %OIR-SP-6-CONSOLE: Changing console ownership to route processor





System Bootstrap, Version 12.2(33r)SRD5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2009 by cisco Systems, Inc.

Warning: monitor nvram area is corrupt ... using default values
C7600-RSP720/RP platform with 2097152 Kbytes of main memory

Download Start
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Download Completed! Booting the image.
Self decompressing the image : ##################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################### [OK]

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706



Cisco IOS Software, c7600rsp72043_rp Software (c7600rsp72043_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRD3, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 10-Sep-09 12:19 by prod_rel_team
Image text-base: 0x08000224, data-base: 0x10000000


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
[email protected].

Cisco CISCO7606-S (M8500) processor (revision 1.1) with 1835008K/131072K bytes of memory.
Processor board ID FOX1346GM8T
 BASEBOARD: RSP720
 CPU: MPC8548_E, Version: 2.0, (0x80390020)
 CORE: E500, Version: 2.0, (0x80210020)
 CPU:1200MHz, CCB:400MHz, DDR:200MHz,
 L1:    D-cache 32 kB enabled
        I-cache 32 kB enabled

Last reset from power-on
1 SIP-200 controller .
1 Virtual Ethernet interface
50 Gigabit Ethernet interfaces
3964K bytes of non-volatile configuration memory.

507024K bytes of Internal ATA PCMCIA card (Sector size 512 bytes).


Press RETURN to get started!


*Aug 27 13:14:26.503: RP: Currently running ROMMON from S (Gold) region
*Aug 27 13:14:26.847: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Aug 27 08:14:30: %SYS-6-CLOCKUPDATE: System clock has been updated from 13:14:30 UTC Thu Aug 27 2015 to 08:14:30 EDT Thu Aug 27 2015, configured from console by console.
*Aug 27 09:14:30: %SYS-6-CLOCKUPDATE: System clock has been updated from 08:14:30 EDT Thu Aug 27 2015 to 09:14:30 EDT Thu Aug 27 2015, configured from console by console.
*Aug 27 09:14:32: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 10
*Aug 27 09:14:32: %SYS-5-CONFIG_I: Configured from memory by console
*Aug 27 09:14:34: %SYS-5-RESTART: System restarted --
Cisco IOS Software, c7600rsp72043_rp Software (c7600rsp72043_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRD3, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 10-Sep-09 12:19 by prod_rel_team
*Aug 27 09:14:34: %SSH-5-ENABLED: SSH 2.0 has been enabled
*Aug 27 09:14:34: %NTP-6-RESTART: NTP process starts
*Aug 27 13:13:43.975: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

Firmware compiled 23-Jul-09 11:21 by integ Build [100]
*Aug 27 13:14:24.003: %SPANTREE-SP-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Aug 27 13:14:24.207: SP: SP: Currently running ROMMON from S (Gold) region
*Aug 27 13:14:32.119: %SW_VLAN-SP-6-VTP_DOMAIN_NAME_CHG: VTP domain name changed to CDMA_PMI.
*Aug 27 09:14:34: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 172.16.100.249 port 514 started - CLI initiated
*Aug 27 13:14:34.119: %SYS-SP-5-RESTART: System restarted --
Cisco IOS Software, c7600rsp72043_sp Software (c7600rsp72043_sp-ADVIPSERVICESK9-M), Version 12.2(33)SRD3, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 10-Sep-09 12:50 by prod_rel_team
*Aug 27 13:14:36: %OIR-SP-6-INSPS: Power supply inserted in slot 1
*Aug 27 13:14:36: %C7600_PWR-SP-2-PSFAIL: power supply 1 output failed.
*Aug 27 13:14:36: %C7600_POWER-SP-4-GLITCH: Recovered from glitch in system power supply.
*Aug 27 13:14:36: %OIR-SP-6-INSPS: Power supply inserted in slot 2
*Aug 27 09:14:36: %SNMP-5-CHASSISALARM: Chassis Alarm Trap: tmpAlarm(OFF), minorAlarm(ON), and/or majorAlarm(OFF)
*Aug 27 13:14:36: %C7600_PWR-SP-4-PSOK: power supply 2 turned on.
*Aug 27 09:14:39: %NTP-6-PEERREACH: Peer 127.127.7.1 is reachable
Aug 27 09:14:39: %NTP-5-PEERSYNC: NTP synced to peer 127.127.7.1
Aug 27 09:14:40: %SNMP-5-COLDSTART: SNMP agent on host Horner-C7606A is undergoing a cold start
*Aug 27 09:14:39: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 5.
*Aug 27 09:14:40: %FABRIC-SP-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 5 became active.

%Software-forced reload


 09:14:44 EDT Thu Aug 27 2015: Unexpected exception to CPU: vector 1500, PC = 0xA7E60E4 , LR = 0xA7E6084 
-Traceback= A7E60E4 A7E6084 A4EE088 A380674 A3C18D8 A3C1944 892CC54 892DE20 892EC00 A278880 A279000 A82F640 A82F9B8 A831120 A7DBEAC A883B54 

CPU Register Context:
MSR = 0x00029200  CR  = 0x24444024  CTR = 0x0A3CCB14  XER   = 0x00000000
R0  = 0x0A7E6084  R1  = 0x17ECF058  R2  = 0xFFFCFFFC  R3    = 0x18710E64
R4  = 0xFFFFFFFE  R5  = 0x00000000  R6  = 0x17ECF030  R7    = 0x100A0000
R8  = 0x00029200  R9  = 0x00000000  R10 = 0x111DA034  R11   = 0x18710E60
R12 = 0xA0000000  R13 = 0x04044000  R14 = 0x00000000  R15   = 0x12DB0000
R16 = 0x12DB49A4  R17 = 0x12DB48B8  R18 = 0x00000000  R19   = 0x79EB1D50
R20 = 0x00000000  R21 = 0x184FF95C  R22 = 0x00000000  R23   = 0x000000FF
R24 = 0x1872D21C  R25 = 0x00000040  R26 = 0x13DF6EEC  R27   = 0x0000001F
R28 = 0x00000001  R29 = 0x103F0000  R30 = 0x13DF6EEC  R31   = 0x00000000

Writing crashinfo to bootdisk:crashinfo_20150827-091444-EDT
1073 Unused bytes of context save space
*** System received a Software forced crash ***
signal= 0x17, code= 0x150e
*Aug 27 09:14:45: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Aug 27 09:14:43: %EARL-SP-2-PATCH_INVOCATION_LIMIT: 10 Recovery patch invocations in the last 30 secs have been attempted. Max limit reached
*Aug 27 09:14:45: %OIR-SP-6-CONSOLE: Changing console ownership to switch processor



*** System received a Software forced crash ***
signal= 0x17, code= 0x1500, context= 0xcf6d9ec
PC = 0x82b1ec0, Vector = 0x1500, SP = 0x14c86ff0

Soulution:

It seems RSP module got an issue. Fortunately, there is a standby RSP720 in warehouse. After replaced RSP720, it booted normally.

Route switch processor (RSP) is a module that is installed in one of the card slots in the router. The RSP provides switching and local and remote management for the router and also contains the uplink ports for the router. Both types of modules (supervisor engine and RSP) perform the same functions in the router.

RSP720-3CXL-GE (RSP720 with Integrated Switch Fabric/3CXL-GE)

Front Panel of RSP720-3CXL-GE Explanation

Cisco Router 7206 with RSP720-3CXL-GE

RSP720-3CXL-GE

•Two Gigabit Ethernet uplink ports: port 1 supports a 1-Gbps SFP module; port 2 is configurable with either a 1-Gbps SFP module or a 10/100/1000-Mbps RJ-45 connector •Integrated 720-Gbps switch fabric •PFC3CXL (high-capacity) and MSFC4 with 512-MB bootdisk 4-MB NVRAM, 4-MB ROMmon, and several DRAM options: –Route processor (RP): 1- to 4-GB DRAM (default 2 GB) –Switch processor (SP): 1- to 2-GB DRAM (default 1GB) •Two CompactFlash Type II slots on front panel (512 MB default with option to 1 GB) and two internal CompactFlash slots (one each for RP and SP, 512 MB default for each) •Requires larger power supplies and a high-speed fan tray •QoS port architecture (Rx/Tx): 1p1q4t/1p2q2t

Reference: 

Cisco 7600 Series Route Switch Processor 720 Data Sheet