SRX Load Rescue Configuration After Reboot

It did not happen often, but when it happened, you will need to know how to fix it.


The rescue configuration is a previously committed, valid configuration. You must have previously set the rescue configuration through the J-Web interface or the CLI.

test@fw-srx-2> request system configuration rescue save  test@fw-srx-2> show system configuration rescue

During today's work, one of SRX firewalls got a problem to load regular configuration file, and it loaded rescue configuration. Here is what the console output told me:

*** FINAL System shutdown message from admin@fw-srx-2 ***            System going down IMMEDIATELY                                                                                                                              {secondary:node1} john@fw-srx-2> Waiting (max 60 seconds) for system process `vnlru' to stop...done Waiting (max 60 seconds) for system process `vnlru_mem' to stop...done Waiting (max 60 seconds) for system process `bufdaemon' to stop...done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining...0 0 0 done syncing disks... All buffers synced. Uptime: 32m35s Rebooting... cpu_reset: Stopping other CPUs U-Boot 1.1.6-JNPR-1.7 (Build time: May  4 2010 - 06:59:58) SRX_240_HIGHMEM board revision major:1, minor:50, serial #: AAEK3334 OCTEON CN5230R-SCP pass 2.0, Core clock: 600 MHz, DDR clock: 333 MHz (666 Mhz data rate) DRAM:  1024 MB Starting Memory POST...  Checking datalines... OK Checking address lines... OK Checking 512K memory for U-Boot... OK. Running U-Boot CRC Test... OK. Flash:  4 MB USB:   scanning bus for devices...  Root Hub 0: 3 USB Device(s) found Root Hub 1: 1 USB Device(s) found        scanning bus for storage devices... 1 Storage Device(s) found Clearing DRAM........ done BIST check passed. 1:00:00.0 Vendor/Device ID = 0x811210b5 1:01:07.0 Vendor/Device ID = 0xc72414e4 Boot Media: nand-flash usb  Net:   octeth0 POST Passed Press SPACE to abort autoboot in 1 seconds ELF file is 32 bit Loading .text @ 0x8f000078 (246092 bytes) Loading .rodata @ 0x8f03c1c4 (13940 bytes) Loading .rodata.str1.4 @ 0x8f03f838 (16580 bytes) Loading set_Xcommand_set @ 0x8f0438fc (104 bytes) Loading .rodata.cst4 @ 0x8f043964 (20 bytes) Loading .data @ 0x8f044000 (5620 bytes) Loading .data.rel.ro @ 0x8f0455f4 (120 bytes) Loading .data.rel @ 0x8f04566c (136 bytes) Clearing .bss @ 0x8f0456f8 (11912 bytes) ## Starting application at 0x8f000078 ... Consoles: U-Boot console   Found compatible API, ver. 1.7 FreeBSD/MIPS U-Boot bootstrap loader, Revision 1.7 ([email protected], Tue May  4 07:15:51 UTC 2010) Memory: 1024MB [0]Booting from nand-flash slice 1 Un-Protected 1 sectors writing to flash... Protected 1 sectors Loading /boot/defaults/loader.conf  /kernel data=0xb0567c+0x134494 syms=[0x4+0x8aa50+0x4+0xc8fc6] Hit [Enter] to boot immediately, or space bar for command prompt. Booting [/kernel]...                Kernel entry at 0x801000e0 ... init regular console Primary ICache: Sets 64 Size 128 Asso 4 Primary DCache: Sets 1 Size 128 Asso 64 Secondary DCache: Sets 512 Size 128 Asso 8 GDB: debug ports: uart GDB: current port: uart KDB: debugger backends: ddb gdb KDB: current backend: ddb kld_map_v: 0x8ff80000, kld_map_p: 0x0 Copyright (c) 1996-2014, Juniper Networks, Inc. All rights reserved. Copyright (c) 1992-2006 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994         The Regents of the University of California. All rights reserved. JUNOS 12.1X44-D40.2 #0: 2014-08-28 12:20:14 UTC     [email protected]:/volume/build/junos/12.1/service/12.1X44-D40.2/obj-octeon/junos/bsd/kernels/JSRXNLE/kernel JUNOS 12.1X44-D40.2 #0: 2014-08-28 12:20:14 UTC     [email protected]:/volume/build/junos/12.1/service/12.1X44-D40.2/obj-octeon/junos/bsd/kernels/JSRXNLE/kernel real memory  = 1073741824 (1024MB) avail memory = 526438400 (502MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs Security policy loaded: JUNOS MAC/pcap (mac_pcap) Security policy loaded: JUNOS MAC/runasnonroot (mac_runasnonroot) netisr_init: !debug_mpsafenet, forcing maxthreads from 4 to 1 cpu0 on motherboard : CAVIUM's OCTEON 52XX CPU Rev. 0.8 with no FPU implemented         L1 Cache: I size 32kb(128 line), D size 8kb(128 line), sixty four way.         L2 Cache: Size 512kb, 8 way obio0 on motherboard uart0: <Octeon-16550 channel 0> on obio0 uart0: console (9600,n,8,1) twsi0 on obio0 dwc0: <Synopsis DWC OTG Controller Driver> on obio0 usb0: <USB Bus for DWC OTG Controller> on dwc0 usb0: USB revision 2.0 uhub0: vendor 0x0000 DWC OTG root hub, class 9/0, rev 2.00/1.00, addr 1 uhub0: 1 port with 1 removable, self powered uhub1: vendor 0x0409 product 0x005a, class 9/0, rev 2.00/1.00, addr 2 uhub1: single transaction translator uhub1: 3 ports with 2 removable, self powered umass0: STMicroelectronics ST72682  High Speed Mode, rev 2.00/2.10, addr 3 dwc1: <Synopsis DWC OTG Controller Driver> on obio0 usb1: <USB Bus for DWC OTG Controller> on dwc1 usb1: USB revision 2.0 uhub2: vendor 0x0000 DWC OTG root hub, class 9/0, rev 2.00/1.00, addr 1 uhub2: 1 port with 1 removable, self powered cpld0 on obio0 pcib1: <Cavium on-chip PCIe HOST bridge> on obio0 Disabling Octeon big bar support PCIe: Waiting for port 0 to finish reset PCIe: Port 0 link active, 2 lanes PCIe: Waiting for port 1 to finish reset PCIe: Port 1 link active, 1 lanes pcib1: Initialized controller pci0: <PCI bus> on pcib1 pcib2: <PCI-PCI bridge> irq 0 at device 0.0 on pci0 pci1: <PCI bus> on pcib2 pci1: <serial bus, USB> at device 2.0 (no driver attached) pci1: <network> at device 7.0 (no driver attached) pcib0: <Cavium on-chip PCIe HOST bridge> on obio0 pci2: <PCI bus> on pcib0 pci2: <processor> at device 0.0 (no driver attached) gblmem0 on obio0 octpkt0: <Octeon RGMII> on obio0 cfi0: <AMD/Fujitsu - 4MB> on obio0 Timecounter "mips" frequency 600000000 Hz quality 0 ###PCB Group initialized for udppcbgroup ###PCB Group initialized for tcppcbgroup da0 at umass-sim0 bus 0 target 0 lun 0 da0: <ST ST72682 2.10> Removable Direct Access SCSI-2 device  da0: 40.000MB/s transfers da0: 1000MB (2048000 512 byte sectors: 64H 32S/T 1000C) Trying to mount root from ufs:/dev/da0s1a Attaching /cf/packages/junos via /dev/mdctl... Mounted junos package on /dev/md0... Media check on da0 Zone 04 Block 0499 Addr 11f300 : Bad read Recovering Block Automatic reboot in progress... ** /dev/da0s1a ** Last Mounted on / ** Root file system ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 250 files, 75946 used, 73580 free (28 frags, 9194 blocks, 0.0% fragmentation) Verified junos signed by PackageProduction_12_1_0 Verified jboot signed by PackageProduction_12_1_0 Verified junos-12.1X44-D40.2-domestic signed by PackageProduction_12_1_0 Checking integrity of BSD labels:   s1: Passed   s2: Passed   s3: Passed   s4: Passed ** /dev/bo0s3e ** Last Mounted on /config ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 28 files, 52 used, 12386 free (10 frags, 1547 blocks, 0.1% fragmentation) ** /dev/bo0s3f ** Last Mounted on /cf/var ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 616 files, 98342 used, 76976 free (544 frags, 9554 blocks, 0.3% fragmentation) Checking integrity of licenses:   JUNOS137657.lic: Passed   JUNOS187398.lic: Passed   JUNOS187665.lic: Passed   JUNOS628672.lic: Passed Checking integrity of configuration:   rescue.conf.gz: Passed Loading configuration ... mgd: error: Cannot open configuration file: /config/juniper.conf mgd: warning: loading configuration from /config/rescue.conf.gz Time and ticks drifted too much,             resetting synchronization... mgd: commit complete Setting initial options: . Starting optional daemons:  usbd. Doing initial network setup:. Initial interface configuration: additional daemons: eventd. Additional routing options:kern.module_path: /boot//kernel;/boot/modules -> /boot/modules;/modules/ifpfe_drv;/modules; kld netpfe drv: ifpfed_dialer. Doing additional network setup:. Starting final network daemons:. setting ldconfig path: /usr/lib /opt/lib starting standard daemons: cron. Initial rc.mips initialization:. Local package initialization:. starting local daemons:set cores for group access . kern.securelevel: -1 -> 1 Creating JAIL MFS partition... JAIL MFS partition created boot.upgrade.uboot="0xBFC00000" boot.upgrade.loader="0xBFE00000" Boot media /dev/da0 has dual root support WARNING: JUNOS versions running on dual partitions are not same ** /dev/da0s2a ** Last Mounted on /mfs/tmp/snap-tmp.1334/mnt.1334 ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 250 files, 75914 used, 74124 free (28 frags, 9262 blocks, 0.0% fragmentation) Sun Sep 20 17:48:34 UTC 2015 fw-srx-2 (ttyu0) login:

For somehow, the regular configuration could not be loaded, and system used rescue configuration instead.

Fix is quite simple. Made a little change to configuration and commit it to generate a new configuration. Reboot and this time console showed the regular configuration loaded successfully:

Checking integrity of licenses:   JUNOS137657.lic: Passed   JUNOS187398.lic: Passed   JUNOS187665.lic: Passed   JUNOS628672.lic: Passed Checking integrity of configuration:   rescue.conf.gz: Passed Loading configuration ... mgd: commit complete Setting initial options: .