JunOS SRX Cluster Upgrade Failed - NETSEC

Latest

Learning, Sharing, Creating

Cybersecurity Memo

Wednesday, August 10, 2016

JunOS SRX Cluster Upgrade Failed


For SRX1400, SRX3400, SRX3600, SRX5600, and SRX5800 devices, command introduced in Junos OS Release 9.6 and support for reboot as a required parameter added in Junos OS Release 11.2R2. For SRX100, SRX210, SRX220, SRX240, and SRX650 devices, command introduced in Junos OS Release 11.2R2. For SRX5400 devices, the command is introduced in Junos OS Release 12.1X46-D20.

Symptoms: 

Symptom 1: "tar: Archive contains obsolescent base-64 headers"

root@fw-1> request system software add no-copy /var/tmp/junos-srxsme-12.1X44-D40.2-domestic.tgz no-validate 
Formatting alternate root (/dev/da0s2a)...
/dev/da0s2a: 627.4MB (1284940 sectors) block size 16384, fragment size 2048
        using 4 cylinder groups of 156.86MB, 10039 blks, 20096 inodes.
super-block backups (for fsck -b #) at:
 32, 321280, 642528, 963776
Extracting /var/tmp/junos-srxsme-12.1X44-D40.2-domestic.tgz ...
tar: Skipping to next header
tar: Archive contains obsolescent base-64 headers

gzip: stdin: invalid compressed data--format violated
tar: Child returned status 1
tar: Error exit delayed from previous errors
ERROR: Failed to extract /var/tmp/junos-srxsme-12.1X44-D40.2-domestic.tgz to /altroot/cf/packages/install-tmp/junos-12.1X44-D40.2-domestic


Cause:

This error usually occurs when a file is transferred over to the server in ASCII mode or ftp server software itself issue. To resolve the issue, transfer the file over to the server again; this time in the binary mode. The best practice is to verify file integrity after uploaded the package to the device.

Solution:
using WinSCP 

Using ftp bin mode:

root@fw-1% ftp 10.9.1.109
Connected to 10.9.1.109.
220 Welcome to Quick 'n Easy FTP Server
Name (10.9.1.109:root): test
331 Password required for test
Password:
230 User successfully logged in.
Remote system type is UNIX.
ftp> bin
200 Type set to BINARY
ftp> get junos-srxsme-12.1X44-D40.2-domestic.tgz
local: junos-srxsme-12.1X44-D40.2-domestic.tgz remote: junos-srxsme-12.1X44-D40.2-domestic.tgz
200 Port command successful.
150 Opening BINARY mode data connection for file transfer.
 29% |**************                                    | 42615 KB    07:26 ETA^                            |  1964 KB    19:34 ETAC



Symptom 2: ISSU in progress error

I am having a hiccup when upgrading JunOS on SRX1400. ISSU upgrade failed because of not enough space on CF folder. Here is the output of failed upgrade process:



root@fw-1-2> request system software in-service-upgrade /var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz reboot no-copy    
Chassis ISSU Started
node0:
--------------------------------------------------------------------------
Chassis ISSU Started
ISSU: Validating Image
Initiating in-service-upgrade

node0:
--------------------------------------------------------------------------
Initiating in-service-upgrade
Checking compatibility with configuration
Initializing...
Verified manifest signed by PackageProduction_12_1_0
Verified junos-12.1X44-D40.2-domestic signed by PackageProduction_12_1_0
Using /var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz
Checking junos requirements on /
Available space: 289258 require: 24128
Saving boot file package in /var/sw/pkg/junos-boot-srx1k3k-12.1X46-D25.7.tgz
Verified manifest signed by PackageProduction_12_1_0
Hardware Database regeneration succeeded
Validating against /config/juniper.conf.gz
Usage: license-check -f "<features>" -m -p -q -M -u -U -V
  -V verify if release based licenses are present
FIPS daemon: warning: request security internal-security-association refresh to take effect
FIPS daemon:
mgd: commit complete
Validation succeeded
Validating against /config/rescue.conf.gz
FIPS daemon: warning: request security internal-security-association refresh to take effect
FIPS daemon:
mgd: commit complete
Validation succeeded
failover all RG 1+ groups to node 1 
Initiated manual failover for all redundancy-groups to node1
Redundancy-groups-0 will not failover and the primaryship remains unchanged.
ISSU: Preparing Backup RE
Pushing bundle to node0
Installing package '/var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz' ...
Verified junos-boot-srx1k3k-12.1X46-D25.7.tgz signed by PackageProduction_12_1_0
Verified junos-srx1k3k-12.1X46-D25.7-domestic signed by PackageProduction_12_1_0
Available space: 289258 require: 297584

WARNING: The /cf filesystem is low on free disk space.
WARNING: This package requires 297584k free, but there
WARNING: is only 289258k available.

WARNING: This installation attempt will be aborted.
ERROR: junos-12.1X46-D25.7-domestic fails requirements check
Installation failed for package '/var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz'
error: Failed to install image on secondary node (error-code: 3.1)
error: ISSU Aborted! Backup node maybe in inconsistent state, Please restore backup node



ISSU aborted. But, both nodes are in ISSU window.
 Please do the following:
 1. Rollback the node with the newer image using rollback command
    Note: use the 'node' option in the rollback command
          otherwise, images on both nodes will be rolled back
 2. Make sure that both nodes (will) have the same image
 3. Ensure the node with older image is primary for all RGs
 4. Abort ISSU on both nodes
 5. Reboot the rolled back node

Solutions: 

1. Clear system to get enough space for upgrading process


{secondary:node0}
root@fw-1-1> request system software delete-backup 
Delete backup system software package [yes,no] (no) yes


{secondary:node0}
root@fw-1-1> show system storage node 0 
node0:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             891M       537M       282M       66%  /
devfs                   1.0K       1.0K         0B      100%  /dev
/dev/md0                523M       523M         0B      100%  /junos
/cf                     891M       537M       282M       66%  /junos/cf
devfs                   1.0K       1.0K         0B      100%  /junos/dev/
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad0s1e              99M       102K        91M        0%  /config
/dev/ad2s1f              12G       461M        10G        4%  /var
/dev/md1               1006M       1.0M       924M        0%  /mfs
/var/jail                12G       461M        10G        4%  /jail/var
/var/log                 12G       461M        10G        4%  /jail/var/log
devfs                   1.0K       1.0K         0B      100%  /jail/dev

{secondary:node0}
root@fw-1-1> file delete /cf/packages/?
Possible completions:
  <[Enter]>            Execute this command
  <path>               Path to delete
  /cf/packages/junos   Size: 280224023, Last changed: Aug 28 15:07:56
  /cf/packages/junos-12.1X44-D40.2-domestic  Size: 280224023, Last changed: Aug 28 15:07:56
  /cf/packages/junos-12.1X44-D40.2-domestic.certs  Size: 7153, Last changed: Aug 28 10:15:58
  /cf/packages/junos-12.1X44-D40.2-domestic.sha1  Size: 41, Last changed: Aug 28 15:08:54
  /cf/packages/junos-12.1X44-D40.2-domestic.sig  Size: 525, Last changed: Aug 28 15:09:27
  /cf/packages/junos-srx1k3k-12.1X44-D25.5-domestic  Size: 279916118, Last changed: May 13 2014
  /cf/packages/junos.old  Size: 279916118, Last changed: May 13 2014
{secondary:node0}
root@fw-1-1> file delete /cf/packages/junos-srx1k3k-12.1X44-D25.5-domestic 

{secondary:node0}
root@fw-1-1> show system storage detail 
node0:
--------------------------------------------------------------------------
Filesystem       1024-blocks       Used      Avail   Capacity   Mounted on
/dev/ad0s1a           912232     276540     562714        33%  /
devfs                      1          1          0       100%  /dev
/dev/md0              535230     535230          0       100%  /junos
/cf                   912232     276540     562714        33%  /junos/cf
devfs                      1          1          0       100%  /junos/dev/
procfs                     4          4          0       100%  /proc
/dev/ad0s1e           101202        102      93004         0%  /config
/dev/ad2s1f         12432412     472166   10965654         4%  /var
/dev/md1             1029676       1046     946256         0%  /mfs
/var/jail           12432412     472166   10965654         4%  /jail/var
/var/log            12432412     472166   10965654         4%  /jail/var/log
devfs                      1          1          0       100%  /jail/dev

node1:
--------------------------------------------------------------------------
Filesystem       1024-blocks       Used      Avail   Capacity   Mounted on
/dev/ad0s1a           912232     276540     562714        33%  /
devfs                      1          1          0       100%  /dev
/dev/md0              535230     535230          0       100%  /junos
/cf                   912232     276540     562714        33%  /junos/cf
devfs                      1          1          0       100%  /junos/dev/
procfs                     4          4          0       100%  /proc
/dev/ad0s1e           101202        102      93004         0%  /config
/dev/ad2s1f         12432412     830654   10607166         7%  /var
/dev/md1             1029676      11190     936112         1%  /mfs
/var/jail           12432412     830654   10607166         7%  /jail/var
/var/log            12432412     830654   10607166         7%  /jail/var/log
devfs                      1          1          0       100%  /jail/dev
SW1#


2. Clear ISSU in progress error

After got enough space on both cluster members, I tried another ISSU upgrade but there is another warning message:

root@fw-1-2> request system software in-service-upgrade /var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz reboot no-copy 
warning: ISSU in progress


///Then another one for abort command
root@fw-1-2> request system software abort in-service-upgrade 
error: command is not valid on the srx1400


///Finally got the right command for SRX1400
root@fw-1-2> request chassis cluster in-service-upgrade abort 
Exiting in-service-upgrade window
Chassis ISSU Aborted



3. Successfully upgraded SRX1400 to 12.1X46-D25.7



root@fw-1-2> request system software in-service-upgrade /var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz reboot no-copy    
Chassis ISSU Started
Chassis ISSU Started
ISSU: Validating Image
Initiating in-service-upgrade
Initiating in-service-upgrade
Checking compatibility with configuration
Initializing...
Verified manifest signed by PackageProduction_12_1_0
Verified junos-12.1X44-D40.2-domestic signed by PackageProduction_12_1_0
Using /var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz
Checking junos requirements on /
Available space: 562714 require: 297584
Saving boot file package in /var/sw/pkg/junos-boot-srx1k3k-12.1X46-D25.7.tgz
Verified manifest signed by PackageProduction_12_1_0
Hardware Database regeneration succeeded
Validating against /config/juniper.conf.gz
Usage: license-check -f "<features>" -m -p -q -M -u -U -V
  -V verify if release based licenses are present
FIPS daemon: warning: request security internal-security-association refresh to take effect
FIPS daemon:
mgd: commit complete
Validation succeeded
Validating against /config/rescue.conf.gz
FIPS daemon: warning: request security internal-security-association refresh to take effect
FIPS daemon:
mgd: commit complete
Validation succeeded
failover all RG 1+ groups to node 1 
Initiated manual failover for all redundancy-groups to node1
Redundancy-groups-0 will not failover and the primaryship remains unchanged.
ISSU: Preparing Backup RE
Pushing bundle to node0
Installing package '/var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz' ...
Verified junos-boot-srx1k3k-12.1X46-D25.7.tgz signed by PackageProduction_12_1_0
Verified junos-srx1k3k-12.1X46-D25.7-domestic signed by PackageProduction_12_1_0
Available space: 562714 require: 297584
Saving boot file package in /var/sw/pkg/junos-boot-srx1k3k-12.1X46-D25.7.tgz
JUNOS 12.1X46-D25.7 will become active at next reboot
WARNING: A reboot is required to load this software correctly
WARNING:     Use the 'request system reboot' command
WARNING:         when software installation is complete
Saving state for rollback ...
Finished upgrading secondary node node0
Rebooting Secondary Node
Shutdown NOW!
[pid 90847]
ISSU: Backup RE Prepare Done
Waiting for node0 to reboot.
node0 booted up.
Waiting for node0 to become secondary
node0 became secondary.
Waiting for node0 to be ready for failover
ISSU: Preparing Daemons
Secondary node0 ready for failover.
Installing package '/var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz' ...
Verified junos-boot-srx1k3k-12.1X46-D25.7.tgz signed by PackageProduction_12_1_0
Verified junos-srx1k3k-12.1X46-D25.7-domestic signed by PackageProduction_12_1_0
Available space: 562714 require: 297584
Saving boot file package in /var/sw/pkg/junos-boot-srx1k3k-12.1X46-D25.7.tgz
JUNOS 12.1X46-D25.7 will become active at next reboot
WARNING: A reboot is required to load this software correctly
WARNING:     Use the 'request system reboot' command
WARNING:         when software installation is complete
Saving state for rollback ...
Failing over all redundancy-groups to node0
ISSU: Preparing for Switchover
Initiated failover for all the redundancy groups to node0
Waiting for node0 take over all redundancy groups


{secondary:node1}
john@fw-1-2> show chassis fpc pic-status    
node0:
--------------------------------------------------------------------------
Slot 0   Online       SRX1k 10GE SYSIO
  PIC 0  Online       6x 1GE RJ45 3x 1GE SFP 3x 10GE SFP+
Slot 1   Online       SRX1k Dual Wide NPC+SPC Support Card
  PIC 0  Online       SPU Cp-Flow
Slot 3   Online       BUILTIN NPC
  PIC 0  Online       NPC PIC

node1:
--------------------------------------------------------------------------
Slot 0   Online       SRX1k 10GE SYSIO
  PIC 0  Online       6x 1GE RJ45 3x 1GE SFP 3x 10GE SFP+
Slot 1   Present      SRX1k Dual Wide NPC+SPC Support Card
Slot 3   Online       BUILTIN NPC
  PIC 0  Online       NPC PIC

{secondary:node1}
john@fw-1-2> show chassis fpc pic-status    
node0:
--------------------------------------------------------------------------
Slot 0   Online       SRX1k 10GE SYSIO
  PIC 0  Online       6x 1GE RJ45 3x 1GE SFP 3x 10GE SFP+
Slot 1   Online       SRX1k Dual Wide NPC+SPC Support Card
  PIC 0  Online       SPU Cp-Flow
Slot 3   Online       BUILTIN NPC
  PIC 0  Online       NPC PIC

node1:
--------------------------------------------------------------------------
Slot 0   Online       SRX1k 10GE SYSIO
  PIC 0  Online       6x 1GE RJ45 3x 1GE SFP 3x 10GE SFP+
Slot 1   Online       SRX1k Dual Wide NPC+SPC Support Card
  PIC 0  Offline     
Slot 3   Online       BUILTIN NPC
  PIC 0  Online       NPC PIC

john@fw-1-2> show chassis fpc pic-status    
node0:
--------------------------------------------------------------------------
Slot 0   Online       SRX1k 10GE SYSIO
  PIC 0  Online       6x 1GE RJ45 3x 1GE SFP 3x 10GE SFP+
Slot 1   Online       SRX1k Dual Wide NPC+SPC Support Card
  PIC 0  Online       SPU Cp-Flow
Slot 3   Online       BUILTIN NPC
  PIC 0  Online       NPC PIC

node1:
--------------------------------------------------------------------------
Slot 0   Online       SRX1k 10GE SYSIO
  PIC 0  Online       6x 1GE RJ45 3x 1GE SFP 3x 10GE SFP+
Slot 1   Online       SRX1k Dual Wide NPC+SPC Support Card
  PIC 0  Online       SPU Cp-Flow
Slot 3   Online       BUILTIN NPC
  PIC 0  Online       NPC PIC



Symptom 3: ISSU not allowed


ohn@fw-com1-1> request system software in-service-upgrade /var/tmp/junos-srxsme-15.1X49-D50.3-domestic.tgz no-sync 
WARNING: Not enabled dual root partition on secondary node
         ISSU not allowed

Solution:

Using regular upgrade method.

Reference:

1 comment: