Latest Posts

CyberArk PAS HA, Backup, Failover and Failback Process

PAM Solution High Availability Design Sample
CyberArk Privileged Session Management (PSM) is a popular central control point that helps to protect target systems that are accessed by privileged accounts users throughout your data center. It usually works in conjunction with the CyberArk market-leading Privileged Identity Management Suite, an enterprise and a policy based solution that enforces, manages and secures workflows and procedures for all shared and privileged accounts in data centers.



High Availability or Load Balancing

For PVWA - HA / Load Balancing
PVWA is using IIS. The easiest way to do load balancing for PVWA is using DNS round robin method as show in following screenshot:

To redirect iis homepage, set following error code redirect configuration based on your PVWA url:

For CPM - Manual Load Balancing
You can have multiple CPM installed in a distributed environment, unfortunately it does not support high availability. It can be configured load balancing manually, which means you can use one CPM to manage certain amounts safes or accounts, and another CPM can handle other amount of safes and accounts. Typical implementation is one CPM handles Windows accounts, another CPM handles *NIX accounts.

For PSM - HA / Load Balancing
You can install multiple PSMs. For example, PSM1 and PSM2. You can find out your PSM server names from PVWA - Administration - Options -  Privileged Session Management - Configured PSM Servers 

1. Manual PSM failover.
Change your platform's settings to use different PSM server.
PVWA - Administration - Platform Management - <Platform Name> - UI & Workflows - Privileged Session Management - ID

2. Auto PSM Loadbalancing
First, you might need to configure your loadbalacer with one virutal PSM dns name to use your multiple PSM servers.


Go to "PVWA - Administration - Options -  Privileged Session Management - Configured PSM Servers "
Copy existing PSM server and paste as a new PSM server and change it to your new virtual PSM farm server name

Expand PSM-Farm. Select Connection Details > Server and change the IP address to that of your PSM Farm virtual hostname, PSM-Farm.51sec.local. Click on Apply and OK to save the changes.

Edit all target platforms to change the PSM ID to PSM-Farm.



Note: There is a key step relating to RDP service certificate. You will need to assign a certificate to the Remote Desktop Services deployment in support of the PSM Farm virtual hostname. Here are the steps:
1. Sign in to PSM Server Comp01c or Comp01d.
2. Open Server Manager and select Remote Desktop Services in the left navigation pane.
3. In Deployment Overview select Tasks > Edit Deployment Properties. In the Configure the deployment window, select Certificates > Select existing certificates > Choose a different certificate. Browse to C:\CyberArkInstallationFiles.

4. Select the pre-generated cert file with the .pfx extension and click Open. In the Password: field, enter Cyberark1, select the box to “Allow the certificate to be added to the Trusted Root Certification Authorities…” and select OK to close the Deployment Properties window.

For Vault - HA (DR)

Failover from primary Vault server to DR vault server
1. Make sure your active vault server DR user is enabled and password has been changed, for example, changed to Cyberark1
2. Install PADR software on secondary (DR) vault server. Before this, Vault Server and Vault Client  should has been installed. DR vault server has been manually stopped.
3. During installing PADR, it will ask active vault server's ip, username (DR) and password to be used to do replication.
4. Stop active vault server to simulate a failure to enable automatically failover. It will take 5 minutes for DR server PADR service to detect this failure (5 times).
5. DR vault server should launch it by PADR service.

====================================================================

Failback from DR vault server to primary vault server:


1. Make sure your active DR vault server's DR user is enabled and password has been reset to Cyberark1.
2. If there is no PADR installed before, install PADR software. Primary vault server should be still in the stopped status. It will create user.ini for DR account during PADR installation. Reboot Primary vault server.

Note: If PADR installed, before start the service, use createcredfile.exe reset user.ini DR password to Cyberark1.

3. Start PADR service, verify padr.log file to check all changes have been replicated over. Your primary vault PADR service will use DR account to verify the connectivity to DR site. If it is successful, it will replicate DR database to primary vault. If it failed, it will try five times in five minutes, after that, it will start failover process to start Vault server. We do not want this happen. We want PADR service replicate database from DR vault. In this case, since DR vault server is up and running, it must be DR user account password issue. You will need to reset DR user password on DR Vault and recreate user.ini file on Primary Vault using CreateCredFile.exe.
4. Once verified all replication succeed, Edit PADR.ini. At this moment, Primary Vault Server is still stopped.
a. Set EnableFailover=No
b. Add the following line: ActivateManualFailover=Yes . Save and exit the file.
5. Restart CyberArk Disaster Recovery Service on the primary server. This service will bring Vault server up then it will stop itself. Verify vault server started successfully.
6. At this moment, both Primary Vault and DR vault server services are up.
7. Log into DR server to edit PADR.ini file
a. Change Failover mode from Yes to No. This will stop Vault Server to start.
b. Delete the last two lines (log number and timestamp of the last successful replication) in the file.
c. Save and exit the file.
8. On DR vault server, open the PrivateArk Server GUI and stop the PrivateArk Server service, by clicking the stoplight. Exit the PrivateArk Server GUI. Change DR user password on DR Vault Server using CreateCredfile.exe to change password in user.ini at C:\Program Files (x86)\PrivateArk\PADR\Conf
9. On DR vault, open Windows Services and Start the CyberArk Vault Disaster Recovery service. This service is going to monitor your primary vault server's status. Once detected failure five times, it will start DR Vault Server. You can check padr.log to verify data has been fully replicated once service started. Powershell command to monitor/tail padr.log: “Get-Content .\logs\padr.log –wait”

Backup - PAReplicate


Backup.cmd File at C:\Program Files (x86)\PrivateArk\Replicate

PAReplicate.exe vault.ini /logonFromFile user.ini /fullbackup /tsparmfile tsparm.ini
















DR Failover


Scenario:
 - PROD Vault is down
 - DR Vault has started

Pre-configuration:
Both PVWA has configured to use PROD Vault and DR Vault. It will automatically to detect alive vault by record order and make a connection to it.
On DR PVWA, first record for valut is DR vault. On Prod PVWA, first record is PROD vault.

Make sure CPM and PSM, vault.ini file has been changed as well.

Failover procedure:
1. Navigate to DR PVWA UI - 10.1.7.18/PasswordVault
2. Login as Admin2 (ie.)

3. Browse to System Configuration -> Platform Management -> Platform Name -> Edit
 - Edit UI& Workflows -> Privileged Session Management:



Change ID to PSMServer object name (As defined in Options -> Privileged Session Management -> Configured PSM Servers

Prod Failback

Please refer to following CyberArk article:
How to perform a manual DR Failover (Backup Link)

Failback to prod PVWA and PSM procedures:

1.  Start the PROD Vault using PrivateArkServer Console on the desktop of the Vault


2.  Stop the DR VAult server using PrivateArkServer Console on the desktop of the DR VAult


3.  Open c:\Program files(x86)\PrivateArk\PADR\conf\padr.ini and edit the file:




FailoverMode=Yes  ->  Change Yes to No
NextBinaryLogNumberToStartAt=0 - Remove this line
LastDataReplicationTimestamp=1570820901835879 -> remove this line

Save the file.

3.  Start the Cyberark Disaster Recovery Service on the DR VAult.


4.  Confirm replication by navigating to c:\Program files(x86)\PrivateArk\PADR\logs\padr.log.  Open this file to confirm:
[11/10/2019   15:37:22.532136]    ::    PADR0010I Replicate ended.
[11/10/2019   15:37:23.534770]    ::    PADR0099I Metadata Replication is running successfully.

Above two lines appears at the end of the padr.log file



5. log into primary pvwa UI and edit the platforms to change the UI & Workflows-> Privileged Session Management ID to the PROD PSM server (PSMServer)




No comments