Install PAS (Privileged Account Security) Vault Cluster High Availability

In Windows 2012 R2 and Windows 2016, high availability is provided by the CyberArk Digital Cluster Vault Server, a group of two independent Vault Servers that share access to common networks and storage. To all other CyberArk components, the two Vault Servers in the cluster can be viewed as a single system, which allows high availability of the Vault services and allows for the loss of one Vault server without service disruption.

This post is for High Availability Cluster set up. For Disaster Service set up, please check this post:

• CyberArk Vault Disaster Recovery Installation

Topology

Server Name       IP 
Vault01a          10.0.10.1
Vault01b          10.0.10.2 
Storage           10.0.0.1
Cluster VIP       10.0.12.1 
DC (NTP Src)      10.0.0.2

Install Vault in Cluster Mode

Following steps are instruction how to install the Vault in cluster mode using above infrastructure.:

• Set up the Shared Storage
• Vault Hardening - Preparing Network Interfaces
• Install the Vault on Node A
• Configure the Cluster on Node A
• Install the Vault on Node B

Set up the Shared Storage
1. Launch the iSCSI Initiator application on your Vault A server.
2. Enter 10.0.0.1 in the Target Field and click Quick Connect.
3. A target with name, Vault01 should appear. Click Done.
4. On the Volumes and Devices tab, click Auto Configure. Two Volumes should appear in the Volume List. Click OK.
5. Repeat steps 1-4 on Vault Node B
6. On Vault Node A, right-click on the Start button and click Disk Management
7. Right-click and bring both Disk 1 and Disk 2 on line.
8. Right-click on each Disk again and select Initialize Disk. Accept the default settings, including selecting MBR. The disks may not appear in the same order shown. Remember that the 1 GB drive is for the Quorum disk and the 5 GB drive is for Vault data.
9. Right-click the Unallocated space on the 1 GB drive and choose New Simple Volume.
10. Click Next, then click Next again to accept the default volume size, in MB’s.
11. Assign drive E as the drive letter for the 1 GB disk and then click Next.
12. Choose to Format the volume using NFTS, Default Allocation unit size. Enter Quorum as the Volume label. Select Perform a quick format, then click Next, then Finish.
13. Repeat the steps to create a partition on the 5 GB drive, using the following settings:
Drive letter
F Volume Label Vault Data
14. For both disks, click Cancel on the dialog box that pops up asking to format the newly-created disks.
15. Using File Explorer, verify that a 1 GB E: drive and a 5 GB F: drive exist on the system.



Vault Hardening - Preparing Network Interfaces
Each node includes 2 network interface cards: Public and Private.
1. Right-click on the Network icon in the taskbar, then click Open Network and Sharing Center.
2. On Vault01A for both NICs labeled Public and Private, configure the network card properties by unchecking all options, leaving only Internet Protocol Version 6 and Internet Protocol Version 4 selected.
a. Confirm IPV4 properties for both NICS show a static IP address and no DNS.
3. Repeat steps 1 and 2 of this procedure on Vault01B.


Install the Vault on Node A
Note: The installation of a Vault in Cluster mode differs from installing a stand-alone Vault in a number of ways, so it is very important that you follow the installation steps as described here very carefully.
1. Browse to C:\CyberArkInstallationFiles\Server. Right click on setup.exe, and choose “Run as administrator”
2. Click Install if any prerequisites that need to be installed, then press Next > to continue.
3. Press Yes to accept the license agreement, then enter CyberArk Demo as the Name and Company and click Next.
4. Press the Cluster-node Vault Installation button to install the Vault as part of a cluster.
5. Press Next to accept the default installation location.
6. Press Browse to change the default Safes location.
7. Change the path to F:\PrivateArk\Safes then click OK, then click Next.
8. Click Browse to select a custom license file path of C:\CyberArkInstallationFiles\License and Operator Keys\License, click on OK, then Next.
9. Press Browse to select the custom Operator CD path of C:\CyberArkInstallationFiles\License and Operator Keys\Operator CD, click OK, then Next.
10. At the “Configuring the Remote Control Agent” window, enter the IP addresses of your two Component servers separated by comma (e.g. 10.0.20.1,10.0.21.1) in the Remote Terminal IP Address field and Cyberark1 in the password fields and press Next.
11. Press Next to allow the machine to be hardened.
12. Press Next to accept the default Program Folder
13. The Performing Vault Server Machine Hardening window will appear. This will take a few minutes. After the hardening process ends the installation will begin.
Important! Do not continue the installation, i.e. setting Master and Administrator passwords, until you have completed the following configuration steps.


Very Important Step:
Using iSCSI to connect to shared storage requires configuring a firewall rule on the vault server, forcing us to interrupt the Vault Server Installer. If this step is missed or ignored, after the Vault Server restart, the Vault will be unable to connect to the shared storage and will not start. In a customer’s production environment, storage should be connected to a Storage Area Network via Fiber Channel, and this step will not be necessary.
14. Browse to C:\Program Files (x86)\PrivateArk\Server\Conf. Edit dbparm.ini and add the following line to the [MAIN] section, or a new section can be added as shown:
AllowNonStandardFWAddresses=[10.0.0.1],Yes,3260:outbound/tcp,3260:inbound/tcp
Make sure the line is entered exactly as shown. To ensure accuracy you can copy the line from Advanced-Firewall-Rules.txt on the desktop and paste it into the dbparm.ini file as shown below.
WARNING: Enter your changes carefully and double check them for accuracy. If this is not done correctly your installation will fail and you may be required to start over.
15. Save your changes and close the file. Return to the installer.
16. Enter Cyberark1 in all of the Password fields and press Next.
17. Choose No, I will restart my computer later and press Finish.



Configure the Cluster on Node A

In this step we will configure the ClusterVault.ini.
1. Open a command prompt as Administrator in C:\Program Files (x86)\PrivateArk\ Server\ClusterVault\. Run the following command:
StorageManager.exe –qE -sF

2. Next, open C:\Program Files (x86)\PrivateArk\ Server\ClusterVault\Conf\ClusterVault.ini in NotePad and make the changes highlighted in yellow below.
Note: Make sure to use the IP addresses of your environment. Use IPCONFIG /ALL to check the configuration on your server. LocalNode is Vault Server you are logged into currently. When signed in to the Vault01A console, Vault01A is the LocalNode and Vault01B is the PeerNode. When signed in to the Vault01B console, Vault01B is the LocalNode and Vault01A is the PeerNode. Ensure there are no trailing spaces when entering values.

3. Save the file, and proceed to “Install the PrivateArk Client”



Install the PrivateArk Client
1. Go to C:\CyberArk Install Files\Client. Right click on setup.exe and select “Run as administrator”. Accept the default options in each of the next six windows, entering your company name (E.g. CyberArk) on the User Information screen if needed.

2. Press OK to define your first PrivateArk Vault.
3. Enter the following and press OK:
„h ClusterVault as the Server Name.
„h Enter the IP address of your Vault Virtual IP (e.g. 10.0.12.1) as the Server Address.
Enter administrator as the Default User Name or leave it blank. Leaving the user field blank will cause the client to display the last user to login. Select OK, to create a shortcut to your Vault within the PrivateArk Client.
4. You may receive a message regarding your Internet proxy. This is normal for our lab environment. Press OK to acknowledge that message.
5. Select Yes, I want to restart my computer now and press Finish.
6. After the server has restarted, log in as Administrator / Cyberark1.
a. Open Cluster Vault Management on the Desktop and verify that all Vault services have been restarted successfully.
b. Validate all information displayed in the Cluster Vault Management window. Vault01a should be displayed on the left side of the window. Validate IP addresses are displayed accurately including the VaultCluster VIP at the top of the window, and next to each server name. Validate the shared storage. The Quorum drive should be reserved, and the shared storage (F:) must be online. All services should be started. If any values are incorrect, edit the ClusterVault.ini and make the necessary corrections.
7. Important files need to be made available to Vault01B.
a. Create a ¡¥Keys¡¦ folder on the drive F:\
b. Copy the following files from Node A to the shared storage folder F:\keys:
„h C:\CyberArkInstallationFiles\License and Operator Keys\Operator CD\Backup.key
„h C:\CyberArkInstallationFiles\License and Operator Keys\Operator CD\ReplicationUser.pass
„h C:\CyberArkInstallationFiles\License and Operator Keys\Operator CD\VaultEmergency.pass
„h C:\CyberArkInstallationFiles\License and Operator Keys\Operator CD\VaultUser.pass
„h C:\Program Files (x86)\PrivateArk\Server\Conf\DBParm.ini
„h C:\Program Files (x86)\PrivateArk\Server\ClusterVault\Conf\ClusterVault.ini
„h C:\Program Files (x86)\PrivateArk\Server\Database\my.ini
8. Stop the services on Node A using the CVM Console (click on the Stop icon) and ensure that all Vault services are stopped and that the shared disks (Quorum and Vault Data) are both offline in Node A. The following message will appear in the ClusterVaultConsole.log : ¡¨CVMCS137I ClusterVault shut down completed successfully.¡¨



Install the Vault on Node B
1. Log on to your Vault Node B Server and run Disk Management. Switch the shared disks resources to Online, making sure to bring the 1 GB Quorum disk online first.
Note: Make sure the drives have been assigned with the same drive letters as on Node A: Quorum on E: and Vault Data on F:
2. Copy the backup.key and password (.pass) files from the shared storage to C:\CyberArkInstallationFiles\License and Operator Keys\Operator CD\ on Node B.
3. Repeat the steps for Installing the Vault. Note the only significant difference installing the Vault Server on Vault01B, is that you will not be prompted to enter the Master and Administrator passwords.
a. Return here after selecting “No, I will restart my computer later” on the Setup Complete screen and clicking Finish at the end of the installation, and continue to the next step in the procedure.
4. Edit C:\Program Files (x86)\PrivateArk\Server\dbparm.ini to:
a. Copy the VaultId parameter from from Node A dbparm.ini to Node B dbparm.ini.
b. Add the firewall exception line for the shared storage
AllowNonStandardFWAddresses=[10.0.0.1],Yes,3260:outbound/tcp,3260:inbound/tcp
5. Edit C:\Program Files (x86)\PrivateArk\Server\Database\my.ini and copy the server-id from Node A to Node B:
server-id=1470569852
6. Repeat steps 1 and 2 of Cluster Configuration by running StorageManager.exe and editing the ClusterVault.ini file. Make sure you enter the correct information for Node B (it should be a mirror image of the file on Node A, LocalNode is now Vault Node B; PeerNode is the remote computer, Vault Node A).
7. Repeat the procedure to Install the PrivateArk Client and restart node B.
8. After the server has restarted verify that all Vault services have been restarted successfully using the CVM console.
9. Start the Cluster Vault Manager service on Node A (using the CVM console). Ensure that Vault01A is reported to be online in the CVM.
10. Perform a manual switch over from Node B (the local node is the always the one on the left):
Congratulations! You have now completed High Availability Vault setup!

Important Note: From now on, before restarting a Vault machine with the Cluster running on it, it is recommended to stop the node from the Cluster Vault Management utility in order to make sure that all resources are shut down properly. You may also turn off one of the Vault servers and work with only one server in order to simplify the exercises from now on.

References

• Install the Digital Cluster Vault Server in an HA Environment on Windows 2012 R2 or 2016