Sunday, September 20, 2015

SRX Load Rescue Configuration After Reboot

It did not happen often, but when it happened, you will need to know how to fix it.


The rescue configuration is a previously committed, valid configuration. You must have previously set the rescue configuration through the J-Web interface or the CLI.

[email protected]> request system configuration rescue save 

[email protected]> show system configuration rescue 



During today's work, one of SRX firewalls got a problem to load regular configuration file, and it loaded rescue configuration. Here is what the console output told me:



*** FINAL System shutdown message from [email protected] ***           

System going down IMMEDIATELY                                                  

                                                                          


{secondary:node1}

[email protected]> Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `vnlru_mem' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...0 0 0 done

syncing disks... All buffers synced.

Uptime: 32m35s
Rebooting...
cpu_reset: Stopping other CPUs


U-Boot 1.1.6-JNPR-1.7 (Build time: May  4 2010 - 06:59:58)


SRX_240_HIGHMEM board revision major:1, minor:50, serial #: AAEK3334

OCTEON CN5230R-SCP pass 2.0, Core clock: 600 MHz, DDR clock: 333 MHz (666 Mhz data rate)
DRAM:  1024 MB
Starting Memory POST... 
Checking datalines... OK
Checking address lines... OK
Checking 512K memory for U-Boot... OK.
Running U-Boot CRC Test... OK.
Flash:  4 MB
USB:   scanning bus for devices... 
Root Hub 0: 3 USB Device(s) found
Root Hub 1: 1 USB Device(s) found
       scanning bus for storage devices... 1 Storage Device(s) found
Clearing DRAM........ done
BIST check passed.
1:00:00.0 Vendor/Device ID = 0x811210b5
1:01:07.0 Vendor/Device ID = 0xc72414e4
Boot Media: nand-flash usb 
Net:   octeth0
POST Passed
Press SPACE to abort autoboot in 1 seconds
ELF file is 32 bit
Loading .text @ 0x8f000078 (246092 bytes)
Loading .rodata @ 0x8f03c1c4 (13940 bytes)
Loading .rodata.str1.4 @ 0x8f03f838 (16580 bytes)
Loading set_Xcommand_set @ 0x8f0438fc (104 bytes)
Loading .rodata.cst4 @ 0x8f043964 (20 bytes)
Loading .data @ 0x8f044000 (5620 bytes)
Loading .data.rel.ro @ 0x8f0455f4 (120 bytes)
Loading .data.rel @ 0x8f04566c (136 bytes)
Clearing .bss @ 0x8f0456f8 (11912 bytes)
## Starting application at 0x8f000078 ...
Consoles: U-Boot console  
Found compatible API, ver. 1.7

FreeBSD/MIPS U-Boot bootstrap loader, Revision 1.7

([email protected], Tue May  4 07:15:51 UTC 2010)
Memory: 1024MB
[0]Booting from nand-flash slice 1
Un-Protected 1 sectors
writing to flash...
Protected 1 sectors
Loading /boot/defaults/loader.conf 
/kernel data=0xb0567c+0x134494 syms=[0x4+0x8aa50+0x4+0xc8fc6]


Hit [Enter] to boot immediately, or space bar for command prompt.

Booting [/kernel]...               
Kernel entry at 0x801000e0 ...
init regular console
Primary ICache: Sets 64 Size 128 Asso 4
Primary DCache: Sets 1 Size 128 Asso 64
Secondary DCache: Sets 512 Size 128 Asso 8
GDB: debug ports: uart
GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
kld_map_v: 0x8ff80000, kld_map_p: 0x0
Copyright (c) 1996-2014, Juniper Networks, Inc.
All rights reserved.
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
JUNOS 12.1X44-D40.2 #0: 2014-08-28 12:20:14 UTC
    [email protected]:/volume/build/junos/12.1/service/12.1X44-D40.2/obj-octeon/junos/bsd/kernels/JSRXNLE/kernel
JUNOS 12.1X44-D40.2 #0: 2014-08-28 12:20:14 UTC
    [email protected]:/volume/build/junos/12.1/service/12.1X44-D40.2/obj-octeon/junos/bsd/kernels/JSRXNLE/kernel
real memory  = 1073741824 (1024MB)
avail memory = 526438400 (502MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Security policy loaded: JUNOS MAC/pcap (mac_pcap)
Security policy loaded: JUNOS MAC/runasnonroot (mac_runasnonroot)
netisr_init: !debug_mpsafenet, forcing maxthreads from 4 to 1
cpu0 on motherboard
: CAVIUM's OCTEON 52XX CPU Rev. 0.8 with no FPU implemented
        L1 Cache: I size 32kb(128 line), D size 8kb(128 line), sixty four way.
        L2 Cache: Size 512kb, 8 way
obio0 on motherboard
uart0: <Octeon-16550 channel 0> on obio0
uart0: console (9600,n,8,1)
twsi0 on obio0
dwc0: <Synopsis DWC OTG Controller Driver> on obio0
usb0: <USB Bus for DWC OTG Controller> on dwc0
usb0: USB revision 2.0
uhub0: vendor 0x0000 DWC OTG root hub, class 9/0, rev 2.00/1.00, addr 1
uhub0: 1 port with 1 removable, self powered
uhub1: vendor 0x0409 product 0x005a, class 9/0, rev 2.00/1.00, addr 2
uhub1: single transaction translator
uhub1: 3 ports with 2 removable, self powered
umass0: STMicroelectronics ST72682  High Speed Mode, rev 2.00/2.10, addr 3
dwc1: <Synopsis DWC OTG Controller Driver> on obio0
usb1: <USB Bus for DWC OTG Controller> on dwc1
usb1: USB revision 2.0
uhub2: vendor 0x0000 DWC OTG root hub, class 9/0, rev 2.00/1.00, addr 1
uhub2: 1 port with 1 removable, self powered
cpld0 on obio0
pcib1: <Cavium on-chip PCIe HOST bridge> on obio0
Disabling Octeon big bar support
PCIe: Waiting for port 0 to finish reset
PCIe: Port 0 link active, 2 lanes
PCIe: Waiting for port 1 to finish reset
PCIe: Port 1 link active, 1 lanes
pcib1: Initialized controller
pci0: <PCI bus> on pcib1
pcib2: <PCI-PCI bridge> irq 0 at device 0.0 on pci0
pci1: <PCI bus> on pcib2
pci1: <serial bus, USB> at device 2.0 (no driver attached)
pci1: <network> at device 7.0 (no driver attached)
pcib0: <Cavium on-chip PCIe HOST bridge> on obio0
pci2: <PCI bus> on pcib0
pci2: <processor> at device 0.0 (no driver attached)
gblmem0 on obio0
octpkt0: <Octeon RGMII> on obio0
cfi0: <AMD/Fujitsu - 4MB> on obio0
Timecounter "mips" frequency 600000000 Hz quality 0
###PCB Group initialized for udppcbgroup
###PCB Group initialized for tcppcbgroup
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <ST ST72682 2.10> Removable Direct Access SCSI-2 device 
da0: 40.000MB/s transfers
da0: 1000MB (2048000 512 byte sectors: 64H 32S/T 1000C)
Trying to mount root from ufs:/dev/da0s1a
Attaching /cf/packages/junos via /dev/mdctl...
Mounted junos package on /dev/md0...

Media check on da0

Zone 04 Block 0499 Addr 11f300 : Bad read
Recovering Block
Automatic reboot in progress...
** /dev/da0s1a
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
250 files, 75946 used, 73580 free (28 frags, 9194 blocks, 0.0% fragmentation)
Verified junos signed by PackageProduction_12_1_0
Verified jboot signed by PackageProduction_12_1_0
Verified junos-12.1X44-D40.2-domestic signed by PackageProduction_12_1_0
Checking integrity of BSD labels:
  s1: Passed
  s2: Passed
  s3: Passed
  s4: Passed
** /dev/bo0s3e
** Last Mounted on /config
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
28 files, 52 used, 12386 free (10 frags, 1547 blocks, 0.1% fragmentation)
** /dev/bo0s3f
** Last Mounted on /cf/var
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
616 files, 98342 used, 76976 free (544 frags, 9554 blocks, 0.3% fragmentation)
Checking integrity of licenses:
  JUNOS137657.lic: Passed
  JUNOS187398.lic: Passed
  JUNOS187665.lic: Passed
  JUNOS628672.lic: Passed
Checking integrity of configuration:
  rescue.conf.gz: Passed
Loading configuration ...
mgd: error: Cannot open configuration file: /config/juniper.conf
mgd: warning: loading configuration from /config/rescue.conf.gz
Time and ticks drifted too much,             resetting synchronization...
mgd: commit complete
Setting initial options: .
Starting optional daemons:  usbd.
Doing initial network setup:.
Initial interface configuration:
additional daemons: eventd.
Additional routing options:kern.module_path: /boot//kernel;/boot/modules -> /boot/modules;/modules/ifpfe_drv;/modules;
kld netpfe drv: ifpfed_dialer.
Doing additional network setup:.
Starting final network daemons:.
setting ldconfig path: /usr/lib /opt/lib
starting standard daemons: cron.
Initial rc.mips initialization:.
Local package initialization:.
starting local daemons:set cores for group access
.
kern.securelevel: -1 -> 1
Creating JAIL MFS partition...
JAIL MFS partition created
boot.upgrade.uboot="0xBFC00000"
boot.upgrade.loader="0xBFE00000"
Boot media /dev/da0 has dual root support
WARNING: JUNOS versions running on dual partitions are not same
** /dev/da0s2a
** Last Mounted on /mfs/tmp/snap-tmp.1334/mnt.1334
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
250 files, 75914 used, 74124 free (28 frags, 9262 blocks, 0.0% fragmentation)
Sun Sep 20 17:48:34 UTC 2015

fw-srx-2 (ttyu0)


login: 

For somehow, the regular configuration could not be loaded, and system used rescue configuration instead.

Fix is quite simple. Made a little change to configuration and commit it to generate a new configuration. Reboot and this time console showed the regular configuration loaded successfully:



Checking integrity of licenses:

  JUNOS137657.lic: Passed
  JUNOS187398.lic: Passed
  JUNOS187665.lic: Passed
  JUNOS628672.lic: Passed
Checking integrity of configuration:
  rescue.conf.gz: Passed
Loading configuration ...
mgd: commit complete
Setting initial options: .


No comments:

Post a Comment