Juniper Firewall SRX240H Crashed with Error 'nearing maxproc limit by uid 0,please see tuning(7) and login.conf(5)'

One of Juniper Firewall SRX240H had a serious crash. Manual reboot/shutdown did not work. To reset it, I would have to do a hard reset / power cycle device.

It would allow to log in from console, but you wont be able to see any configuration.

Here is outputs from this crashed Juniper SRX240H console:

{secondary:node0}
jonny@fw-1> show interfaces terse 
Interface               Admin Link Proto    Local                 Remote
fxp0                    up    up  
fxp0.0                  up    up   inet     10.9.1.11/24  
fxp1                    up    up  
fxp1.0                  up    up   inet     129.16.0.1/2    
                                   tnp      0x1100001       
fxp2                    up    up  
fxp2.0                  up    up   tnp      0x1100001       
gre                     up    up  
ipip                    up    up  
lo0                     up    up  
lo0.16384               up    up   inet     127.0.0.1           --> 0/0
lo0.16385               up    up   inet     10.0.0.1            --> 0/0
                                            10.0.0.16           --> 0/0
                                            128.0.0.1           --> 0/0
                                            128.0.0.4           --> 0/0
                                            128.0.1.16          --> 0/0
lo0.32768               up    up  
lsi                     up    up  
mtun                    up    up  
pimd                    up    up  
pime                    up    up  
tap                     up    up  
                                        
{secondary:node0}
jonny@fw-1> show configuration 
nearing maxproc limit by uid 0, please see tuning(7) and login.conf(5).
Process with Most Children- 0:swapper - Children - 60

Process with Most Children- 1:init - Children - 82
nearing maxproc limit by uid 0, please see tuning(7) and login.conf(5).
Process with Most Children- 1:init - Children - 82
nearing maxproc limit by uid 0, please see tuning(7) and login.conf(5).
Process with Most Children- 1:init - Children - 82
init died (signal 4, exit 0)
panic: Going nowhere without my init!
cpuid = 0
KDB: stack backtrace:
0x4afb64+0x20 (0x6,0,0x3f7eef10,0x4bef40) ra 0x4afb2c sz 0
0x4afaf0+0x3c (0x6,0,0x3f7eef10,0x4bef40) ra 0x4ae444 sz 32
0x4ae3c0+0x84 (0x6,0,0x3f7eef10,0x4bef40) ra 0x4453d4 sz 56
0x445360+0x74 (0x6,0,0x3f7eef10,0x4bef40) ra 0x445450 sz 40
0x445360+0xf0 (0x6,0,0x3f7eef10,0x4bef40) ra 0x44659c sz 40
0x446514+0x88 (0x6,0,0x3f7eef10,0x4bef40) ra 0x446cb8 sz 64
0x446c84+0x34 (0x6,0,0x3f7eef10,0x4bef40) ra 0x4b0724 sz 32
0x4b06e4+0x40 (0x6,0,0x3f7eef10,0x4bef40) ra 0x4b09e8 sz 40
0x4b0908+0xe0 (0x6,0,0x3f7eef10,0x4bef40) ra 0x4929d4 sz 32
0x492950+0x84 (0x6,0,0x3f7eef10,0x4bef40) ra 0x48d9d4 sz 48
0x48d8a8+0x12c (0x6,0,0x3f7eef10,0x4bef40) ra 0x4039e4 sz 3512
0x4039a8+0x3c (0x6,0x4d2608,0x3f7f0060,0x4bef40) ra 0x403e80 sz 40
0x403da0+0xe0 (0x6,0x4d2608,0x3f7f0060,0x4bef40) ra 0x3ffeefe0 sz 32
VA 0x3ffdefdc: not in user area or heuristics failed
_start+0xbfeeef00 (0x6,0x4d2608,0x3f7f0060,0x4bef40) ra 0 sz 0
pid 1, process: init
Uptime: 13m56s
Cannot dump. No dump device defined.
Ignoring watchdog timeout during boot/reboot
Ignoring watchdog timeout during boot/reboot
Ignoring watchdog timeout during boot/reboot
Ignoring watchdog timeout during boot/reboot
panic: Hardware watchdog timeout
cpuid = 0
Uptime: 16m19s
Cannot dump. No dump device defined.


NMI Exception on core:0
Watchdog status, core 0: 0xfffe6bffffb
FPA INT Summery: 0x0
Err EPC: 0x807c6d58
Trapframe Register Dump:
zero: 0000000000000000  at: fffffffffffffffe  v0: 0000000000000001  v1: 000000000000000e
  a0: 00000000000003e8  a1: 0000000000000001  a2: 00000000ffff8010  a3: 0000000010000010
  t0: 00000000508008e1  t1: 0000000000000000  t2: 0000000004200029  t3: 0000000010000588
 ta0: 0000000002000000 ta1: 0000000000000004 ta2: ffffffffc1cc3640 ta3: 0000000000000001
  t8: 0000000023c34600  t9: 0000000008507580  s0: 000000000004f823  s1: 0000000038247ad4
  s2: 00000000000927c0  s3: ffffffffc1cd0680  s4: ffffffff80c20000  s5: ffffffffd66a6ee8
  s6: fffffffffffffffe  s7: ffffffff80ae2d9c  k0: 1a00000080c099e8  k1: 808042a80000000a
  gp: ffffffff80c197b0  sp: ffffffffd66a6e78  s8: 0000000000000000  ra: ffffffff807c6d60
  sr: 0000000050c808e5 mullo: 0000000005a0d200    mulhi: 0000000009600000
  pc: ffffffff80a40bd8 cause: 0000000040008400 badvaddr: ffffffffc1d1a4d8
ErrPC: 0000000000000840
Current ticks/softticks 920517/824600, curproc [1] init
Core0: CacheErr(I/D: current: 0x7f7f0000000000/0x1130)

PCPU dump:
cpuid        = 0
curthread    = 0xc1ce0420: pid 1 "init"
ipis         = 0x0
cpuid        = 1
curthread    = 0xc1ce5210: pid 21 "idle: cpu1"
ipis         = 0x0
cpuid        = 2
curthread    = 0xc1ce5000: pid 20 "idle: cpu2"
ipis         = 0x0
cpuid        = 3
curthread    = 0xc1ce1c60: pid 19 "idle: cpu3"
ipis         = 0x0
cpuid        = 4
curthread    = none
ipis         = 0x0
cpuid        = 5
curthread    = none
ipis         = 0x0
cpuid        = 6
curthread    = none
ipis         = 0x0
cpuid        = 7
curthread    = none
ipis         = 0x0
cpuid        = 8
curthread    = none
ipis         = 0x0
cpuid        = 9
curthread    = none
ipis         = 0x0
cpuid        = 10
curthread    = none
ipis         = 0x0
cpuid        = 11
curthread    = none
ipis         = 0x0
Memory dump of 1024 words starting at 0x80000000
0x80000000: 082905e3 401a4000 00000000 800580e4 
0x80000010: 80058148 800767f4 aaaaaaaa aaaaaaaa 
0x80000020: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000030: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000040: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000050: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000060: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000070: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000080: 082905e3 401a4000 00000000 aaaaaaaa 
0x80000090: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800000a0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800000b0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800000c0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800000d0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800000e0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800000f0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000100: 3c1b80d5 277bae68 7c1a003b 001ad0c0 
0x80000110: 035bd821 403ad801 ff7a0000 401a6000 
0x80000120: 335a0002 17400005 00000000 3c1a80a4 
0x80000130: 275a2af0 03400008 00000000 3c1a807e 
0x80000140: 275a70b0 03400008 00000000 1000ffff 
0x80000150: 00000000 42000018 aaaaaaaa aaaaaaaa 
0x80000160: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000170: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000180: 401a6000 401b6800 335a0010 001ad0c0 
0x80000190: 337b007c 037ad825 3c1a80c0 275a5130 
0x800001a0: 035bd021 8f5a0000 00000000 03400008 
0x800001b0: 00000000 aaaaaaaa aaaaaaaa aaaaaaaa 
0x800001c0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800001d0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800001e0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800001f0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000200: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000210: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000220: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000230: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000240: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000250: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000260: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000270: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000280: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000290: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800002a0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800002b0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800002c0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800002d0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800002e0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800002f0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000300: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000310: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000320: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000330: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000340: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000350: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000360: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000370: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000380: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x80000390: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800003a0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800003b0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800003c0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800003d0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800003e0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
0x800003f0: aaaaaaaa aaaaaaaa aaaaaaaa aaaaaaaa 
Stack trace:
DELAY+0x4c (0x3e8,0x1,0xffff8010,0x10000010) ra 0x80118764 sz 32
xpt_polled_action+0x64 (0x3e8,0x1,0xffff8010,0x10000010) ra 0x8011c93c sz 48
dashutdown+0xa0 (0x3e8,0x1,0xffff8010,0x10000010) ra 0x8023ac58 sz 664
boot+0xd48 (0x3e8,0x1,0xffff8010,0x10000010) ra 0x8023b910 sz 64
panic+0x8a8 (0x3e8,0x80d44dc8,0xffff8010,0x508008e1) ra 0x807df914 sz 72
panic_on_watchdog_timeout+0x78 (0x3e8,0x80d44dc8,0xffff8010,0x508008e1) ra 0x80804e2c sz 32
re_srxsme_watchdog_intr+0x158 (0x3e8,0x80d44dc8,0xffff8010,0x508008e1) ra 0x807b5068 sz 24
mips_handle_this_interrupt+0x8c (0x3e8,0x80d44dc8,0xffff8010,0x508008e1) ra 0x807b50fc sz 40
mips_handle_interrupts+0x60 (0x3e8,0x80d44dc8,0xffff8010,0x508008e1) ra 0x807b5528 sz 48
mips_interrupt+0x22c (0x3e8,0x80d44dc8,0xffff8010,0x508008e1) ra 0x80a420c4 sz 32
MipsKernIntr+0x140 (0x3e8,0x1,0xffff8010,0x10000010) ra 0x807c6d60 sz 368
DELAY+0x54 (0x3e8,0x1,0xffff8010,0x10000010) ra 0x80118764 sz 32
xpt_polled_action+0x64 (0x3e8,0x1,0xffff8010,0x10000010) ra 0x8011c93c sz 48
dashutdown+0xa0 (0x3e8,0x1,0xffff8010,0x10000010) ra 0x8023ac58 sz 664
boot+0xd48 (0x3e8,0x1,0xffff8010,0x10000010) ra 0x8023b910 sz 64
panic+0x8a8 (0x3e8,0x1,0xffff8010,0x3) ra 0x801f9930 sz 72
exit1+0x3dc (0x3e8,0x1,0xffff8010,0x3) ra 0x80245bf8 sz 80
sigexit+0x1814 (0x3e8,0x1,0xffff8010,0x3) ra 0x807c8aa8 sz 496
sendsig+0x51c (0x3e8,0x1,0xffff8010,0x3) ra 0x80246268 sz 528
sigexit+0x1e84 (0x3e8,0x1,0xffff8010,0x3) ra 0x4d2a84 sz 496
PC 0x4d2a84: not in kernel
uart_z8530_class+0x4d2a84 (0x3e8,0x1,0xffff8010,0x3) ra 0 sz 0
pid 1, process: init
Resetting the  system now...
cpu_reset: Stopping other CPUs
timeout stopping cpus


U-Boot 1.1.6-JNPR-2.4 (Build time: Aug 31 2012 - 12:15:03)

SRX_240_HIGHMEM board revision major:2, minor:56, serial #: ACKF8991
OCTEON CN5230R-SCP pass 2.0, Core clock: 600 MHz, DDR clock: 333 MHz (666 Mhz data rate)
DRAM:  1024 MB
Starting Memory POST... 
Checking datalines... OK
Checking address lines... OK
Checking 512K memory for U-Boot... OK.
Running U-Boot CRC Test... OK.
Flash:  4 MB
USB:   scanning bus for devices... 
Root Hub 0: 4 USB Device(s) found
Root Hub 1: 1 USB Device(s) found
       scanning bus for storage devices... 2 Storage Device(s) found
Clearing DRAM........ done
BIST check passed.
1:00:00.0 Vendor/Device ID = 0x811210b5
1:01:07.0 Vendor/Device ID = 0xc72414e4
Boot Media: nand-flash usb 
Net:   octeth0
POST Passed
Press SPACE to abort autoboot in 1 seconds
ELF file is 32 bit
Loading .text @ 0x8f000078 (246924 bytes)
Loading .rodata @ 0x8f03c504 (13944 bytes)
Loading .rodata.str1.4 @ 0x8f03fb7c (16776 bytes)
Loading set_Xcommand_set @ 0x8f043d04 (100 bytes)
Loading .rodata.cst4 @ 0x8f043d68 (20 bytes)
Loading .data @ 0x8f044000 (5608 bytes)
Loading .data.rel.ro @ 0x8f0455e8 (120 bytes)
Loading .data.rel @ 0x8f045660 (136 bytes)
Clearing .bss @ 0x8f0456e8 (11656 bytes)
## Starting application at 0x8f000078 ...
Consoles: U-Boot console  
Found compatible API, ver. 2.4

FreeBSD/MIPS U-Boot bootstrap loader, Revision 2.4
([email protected], Fri Aug 31 12:18:02 UTC 2012)
Memory: 1024MB
[0]Booting from nand-flash slice 2
Un-Protected 1 sectors
writing to flash...
Protected 1 sectors
Loading /boot/defaults/loader.conf 
/kernel data=0xb16d5c+0x134b2c syms=[0x4+0x8bbd0+0x4+0xcadc3]


Hit [Enter] to boot immediately, or space bar for command prompt.
Booting [/kernel]...               
Kernel entry at 0x801000e0 ...
init regular console
Primary ICache: Sets 64 Size 128 Asso 4
Primary DCache: Sets 1 Size 128 Asso 64
Secondary DCache: Sets 512 Size 128 Asso 8
GDB: debug ports: uart
GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
kld_map_v: 0x8ff80000, kld_map_p: 0x0
Copyright (c) 1996-2016, Juniper Networks, Inc.
All rights reserved.
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
JUNOS 12.1X46-D55.3 #0: 2016-07-08 18:46:54 UTC
    [email protected]:/volume/build/junos/12.1/service/12.1X46-D55.3/obj-octeon/junos/bsd/kernels/JSRXNLE/kernel
JUNOS 12.1X46-D55.3 #0: 2016-07-08 18:46:54 UTC
    [email protected]:/volume/build/junos/12.1/service/12.1X46-D55.3/obj-octeon/junos/bsd/kernels/JSRXNLE/kernel
real memory  = 1073741824 (1024MB)
avail memory = 509661184 (486MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Security policy loaded: JUNOS MAC/pcap (mac_pcap)
Security policy loaded: JUNOS MAC/runasnonroot (mac_runasnonroot)
netisr_init: !debug_mpsafenet, forcing maxthreads from 4 to 1
cpu0 on motherboard
: CAVIUM's OCTEON 52XX CPU Rev. 0.8 with no FPU implemented
        L1 Cache: I size 32kb(128 line), D size 8kb(128 line), sixty four way.
        L2 Cache: Size 512kb, 8 way
obio0 on motherboard
uart0: <Octeon-16550 channel 0> on obio0
uart0: console (9600,n,8,1)
twsi0 on obio0
dwc0: <Synopsis DWC OTG Controller Driver> on obio0
usb0: <USB Bus for DWC OTG Controller> on dwc0
usb0: USB revision 2.0
uhub0: vendor 0x0000 DWC OTG root hub, class 9/0, rev 2.00/1.00, addr 1
uhub0: 1 port with 1 removable, self powered
uhub1: vendor 0x0409 product 0x005a, class 9/0, rev 2.00/1.00, addr 2
uhub1: single transaction translator
uhub1: 3 ports with 2 removable, self powered
umass0: STMicroelectronics ST72682  High Speed Mode, rev 2.00/2.10, addr 3
umass1: Kingston DataTraveler G3, rev 2.00/1.00, addr 4
dwc1: <Synopsis DWC OTG Controller Driver> on obio0
usb1: <USB Bus for DWC OTG Controller> on dwc1
usb1: USB revision 2.0
uhub2: vendor 0x0000 DWC OTG root hub, class 9/0, rev 2.00/1.00, addr 1
uhub2: 1 port with 1 removable, self powered
cpld0 on obio0
pcib1: <Cavium on-chip PCIe HOST bridge> on obio0
Disabling Octeon big bar support
PCIe: Waiting for port 0 to finish reset
PCIe: Port 0 link active, 2 lanes
PCIe: Waiting for port 1 to finish reset
PCIe: Port 1 link active, 1 lanes
pcib1: Initialized controller
pci0: <PCI bus> on pcib1
pcib2: <PCI-PCI bridge> irq 0 at device 0.0 on pci0
pci1: <PCI bus> on pcib2
pci1: <serial bus, USB> at device 2.0 (no driver attached)
pci1: <serial bus, USB> at device 2.1 (no driver attached)
pci1: <network> at device 7.0 (no driver attached)
pcib0: <Cavium on-chip PCIe HOST bridge> on obio0
pci2: <PCI bus> on pcib0
pci2: <processor> at device 0.0 (no driver attached)
gblmem0 on obio0
octpkt0: <Octeon RGMII> on obio0
cfi0: <AMD/Fujitsu - 4MB> on obio0
Timecounter "mips" frequency 600000000 Hz quality 0
###PCB Group initialized for udppcbgroup
###PCB Group initialized for tcppcbgroup
da1 at umass-sim1 bus 1 target 0 lun 0
da1: <Kingston DataTraveler G3 1.00> Removable Direct Access SCSI-0 device 
da1: 40.000MB/s transfers
da1: 7639MB (15644912 512 byte sectors: 255H 63S/T 973C)
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <ST ST72682 2.10> Removable Direct Access SCSI-2 device 
da0: 40.000MB/s transfers
da0: 1000MB (2048000 512 byte sectors: 64H 32S/T 1000C)
Trying to mount root from ufs:/dev/da0s2a
WARNING: / was not properly dismounted
MFSINIT: Initialising MFSROOT 
WARNING: / was not properly dismounted
Process-1 beginning MFSROOT initialization...
Creating MFSROOT...
/dev/md0: 20.0MB (40956 sectors) block size 16384, fragment size 2048
        using 4 cylinder groups of 5.00MB, 320 blks, 640 inodes.
super-block backups (for fsck -b #) at:
 32, 10272, 20512, 30752
Populating MFSROOT...
Creating symlinks...
Setting up mounts...
Continuing boot from MFSROOT...
Attaching /cf/packages/junos via /dev/mdctl...
Mounted junos package on /dev/md1...
N
WARNING: R/W mount of /cf/var denied.  Filesystem is not clean - run fsck
mount/dev/bo0s3f : Operation not permitted
chflags: /var/packages/*: No such file or directory
umount: /dev/bo0s3f: unknown file system
Media check on da0
Automatic reboot in progress...
** /dev/da0s2a (NO WRITE)
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
500 files, 78381 used, 71657 free (17 frags, 8955 blocks, 0.0% fragmentation)
mount reload of '/' failed: Operation not supported 

Verified junos signed by PackageProductionEc_2016 method ECDSA
Verified jboot signed by PackageProductionEc_2016 method ECDSA
Verified junos-12.1X46-D55.3-domestic signed by PackageProductionEc_2016 method ECDSA
Checking integrity of BSD labels:
  s1: Passed
  s2: Passed
  s3: Passed
  s4: Passed
** /dev/bo0s3e
** Last Mounted on /config
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
17 files, 52 used, 12386 free (26 frags, 1545 blocks, 0.2% fragmentation)

***** FILE SYSTEM MARKED CLEAN *****
** /dev/bo0s3f
** Last Mounted on /cf/var
** Phase 1 - Check Blocks and Sizes
PARTIALLY TRUNCATED INODE I=141
SALVAGE? yes

** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
UNREF FILE I=22702  OWNER=0 MODE=100660
SIZE=245770 MTIME=Nov 21 16:32 2016 
CLEAR? yes

UNREF FILE I=22715  OWNER=0 MODE=100660
SIZE=262144 MTIME=Nov 21 16:32 2016 
CLEAR? yes

** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? yes

SUMMARY INFORMATION BAD
SALVAGE? yes

BLK(S) MISSING IN BIT MAPS
SALVAGE? yes

2251 files, 109315 used, 66003 free (379 frags, 8203 blocks, 0.2% fragmentation)

***** FILE SYSTEM MARKED CLEAN *****

***** FILE SYSTEM WAS MODIFIED *****
Checking integrity of licenses:
  JUNOS476910.lic: Passed
  JUNOS476950.lic: Passed
  JUNOS638415.lic: Passed
  JUNOS649298.lic: Passed
  JUNOS859875.lic: No recovery data
Checking integrity of configuration:
  rescue.conf.gz: Passed
cd: can't cd to /etc/db/pkg
Loading configuration ...
Time and ticks drifted too much,                        resetting synchronization...
IDP policy daemon: [edit security idp idp-policy Space-IPS-Policy rulebase-ips rule 1 match]
IDP policy daemon:   'attacks'
IDP policy daemon:     Please install the Signature Database
IDP policy daemon: 
mgd: error: configuration check-out failed
Warning: Commit failed, activating partial configuration.
Warning: Edit the router configuration to fix these errors.
Setting initial options: .
Starting optional daemons:  usbd.
Doing initial network setup:.
Initial interface configuration:
additional daemons: eventd.
Additional routing options:kern.module_path: /boot//kernel;/boot/modules -> /boot/modules;/modules/ifpfe_drv;/modules;
kld netpfe drv: ifpfed_dialer ipsec kld.
Doing additional network setup:.
Starting final network daemons:.
setting ldconfig path: /usr/lib /opt/lib
ldconfig: warning: /opt/lib: No such file or directory
starting standard daemons: cron.
Initial rc.mips initialization:.
Local package initialization:.
starting local daemons:set cores for group access
.
kern.securelevel: -1 -> 1
Creating JAIL MFS partition...
JAIL MFS partition created
boot.upgrade.uboot="0xBFC00000"
boot.upgrade.loader="0xBFE00000"
Boot media /dev/da0 has dual root support
ERROR: cannot mount /dev/da0s2a
** /dev/da0s1a
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 71145 free (89 frags, 8882 blocks, 0.1% fragmentation)
Mon Nov 21 16:39:14 UTC 2016

fw-1 (ttyu0)

login: jonny
Password:

--- JUNOS 12.1X46-D55.3 built 2016-07-08 18:46:54 UTC
could not open user interface connection: management daemon not responding
Retry connection attempts ? [yes,no] (yes) 
CLI Output END

After system crashed, it will do a reboot by itself, but it will get back into Bad_Page_Fault error then reboot itself again.

BAD_PAGE_FAULT: pid 1 (init), uid 0: pc 0x48d8dc got a write fault at 0x3f7ee3d0
Trapframe Register Dump:
zero: 0000000000000000  at: 0000000000000001  v0: 000000000051d748  v1: 000000000051d730
  a0: 0000000000000000  a1: 00000000004cb298  a2: 000000003f7ef19c  a3: 00000000004bef40
  t0: 0000000000000009  t1: 0000000000000000  t2: 0000000000000004  t3: 0000000000000000
 ta0: 0000000001ab2600 ta1: 0000000000000009 ta2: 0000000000000020 ta3: 000000000056f042
  t8: 000000000056f034  t9: 000000000048d8a8  s0: 0000000000000006  s1: 00000000004d2a84
  s2: 000000000000096f  s3: 000000003f7f0268  s4: 0000000000000000  s5: 000000003f7f02ec
  s6: 0000000000000019  s7: 0000000000000009  k0: 0000000000000000  k1: 0000000000000000
  gp: 0000000000544f80  sp: 000000003f7ee3b8  s8: 0000000000564a80  ra: 00000000004039e4
  sr: 0000000050808cf3 mullo: 0000000066666667    mulhi: 0000000000000000
  pc: 000000000048d8dc cause: 000000000000000c badvaddr: 000000003f7ee3d0
Page table info for pc address 0x48d8dc: pte = 0x4032b45a
Dumping 4 words starting at pc address 0x48d8dc: 
afbc0018 00808021 00a09821 2402fc00
cpuid = 0
BAD_PAGE_FAULT: pid 1 (init), uid 0: pc 0x48d8dc got a write fault at 0x3f7ee3d0
Trapframe Register Dump:
zero: 0000000000000000  at: 0000000000000001  v0: 000000000051d748  v1: 000000000051d730
  a0: 0000000000000000  a1: 00000000004cb298  a2: 000000003f7ef19c  a3: 00000000004bef40
  t0: 0000000000000009  t1: 0000000000000000  t2: 0000000000000004  t3: 0000000000000000
 ta0: 0000000001ab2600 ta1: 0000000000000009 ta2: 0000000000000020 ta3: 000000000056f042
  t8: 000000000056f034  t9: 000000000048d8a8  s0: 0000000000000006  s1: 00000000004d2a84
  s2: 000000000000096f  s3: 000000003f7f0268  s4: 0000000000000000  s5: 000000003f7f02ec
  s6: 0000000000000019  s7: 0000000000000009  k0: 0000000000000000  k1: 0000000000000000
  gp: 0000000000544f80  sp: 000000003f7ee3b8  s8: 0000000000564a80  ra: 00000000004039e4
  sr: 0000000050808cf3 mullo: 0000000066666667    mulhi: 0000000000000000
  pc: 000000000048d8dc cause: 000000000000000c badvaddr: 000000003f7ee3d0
Page table info for pc address 0x48d8dc: pte = 0x4032b45a
Dumping 4 words starting at pc address 0x48d8dc: 
afbc0018 00808021 00a09821 2402fc00
cpuid = 0

From another cluster member which is working normal, you will see some cluster status. 

{primary:node1}
admin@fw-2> show chassis cluster status    
Monitor Failure codes:
    CS  Cold Sync monitoring        FL  Fabric Connection monitoring
    GR  GRES monitoring             HW  Hardware monitoring
    IF  Interface monitoring        IP  IP monitoring
    LB  Loopback monitoring         MB  Mbuf monitoring
    NH  Nexthop monitoring          NP  NPC monitoring              
    SP  SPU monitoring              SM  Schedule monitoring
 
Cluster ID: 1
Node   Priority Status         Preempt Manual   Monitor-failures

Redundancy group: 0 , Failover count: 1
node0  200      secondary      no      no       None           
node1  100      primary        no      no       None           

Redundancy group: 1 , Failover count: 1
node0  0        hold           yes     no       IF CS          
node1  0        primary        yes     no       CS             

{primary:node1}
admin@fw-2> 

After show it to JTAC, it was RMA-ed.