CyberArk PAS Configuration Issues and Troubleshooting (Vault)

 This post is created to collect some issues or error messages I met and solutions I resolved them. 

• Safe PSMRecording is out of space
• DR Replication Error
• DR site changed to 'inactive' due to lack of replication activity
• PADR.ini EnableFailover=No
• Vault Patching
• Vault External Firewall Rules

Related posts:

• CyberArk PAS Solution Issues and Troubleshooting (PVWA)
• CyberArk PAS Solution Issues and Troubleshooting (PSM)
• CyberArk PAS Configuration Issues and Troubleshooting (CPM)
• CyberArk PAS Configuration Issues and Troubleshooting (Vault)

Safe PSMRecording is out of space

Safe PSMRecordings is out of space.

Log in as administrator - Safe - PSMRecordings - Open - Properties ->

Default size is 51200MB, changed to 251200MB

DR Replication Error

[Distributed Vault] - error 1236 master has purged binary logs containing GTIDs that the slave requires

[07/08/2020   12:56:02.412963]    ::    GetPADRWorkingDirectory returned [C:\Program Files (x86)\PrivateArk\PADR\Conf] 
[07/08/2020   12:56:02.412992]    ::    GetPADRWorkingDirectory returned [C:\Program Files (x86)\PrivateArk\PADR\Conf] 
[07/08/2020   12:56:02.417949]    ::    PADR0010I Replicate ended. 
[07/08/2020   13:00:52.663872]    ::    PADR0102E Metadata Replication encountered an error - Restart replication will be triggered. See Diagnostic Info below. 
[07/08/2020   13:00:52.663911]    ::     Last SQL Error Code: 0 
[07/08/2020   13:00:52.663930]    ::     Last SQL Error: "" 
[07/08/2020   13:00:52.663946]    ::     Last IO Error Code: 1236 
[07/08/2020   13:00:52.663969]    ::     Last IO Error: "Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.'" 
[07/08/2020   13:00:52.664041]    ::     SQL Thread Running State: "Yes" 
[07/08/2020   13:00:52.664062]    ::     IO Thread Running State: "No" 
[07/08/2020   13:00:52.679336]    ::    PADR0009I Replicate started. 
[07/08/2020   13:00:52.679637]    ::    PADR0095I Refreshing Vault configuration files. 
[07/08/2020   13:00:52.746654]    ::    PADR0097I Refreshing Vault configuration files completed successfully. 
[07/08/2020   13:00:52.751424]    ::    GetPADRWorkingDirectory returned [C:\Program Files (x86)\PrivateArk\PADR\Conf] 
[07/08/2020   13:00:52.751455]    ::    GetPADRWorkingDirectory returned [C:\Program Files (x86)\PrivateArk\PADR\Conf] 
[07/08/2020   13:00:52.753812]    ::    PADR0010I Replicate ended. 
[07/08/2020   13:05:52.028154]    ::    PADR0102E Metadata Replication encountered an error - Restart replication will be triggered. See Diagnostic Info below. 
[07/08/2020   13:05:52.028201]    ::     Last SQL Error Code: 0 
[07/08/2020   13:05:52.028222]    ::     Last SQL Error: "" 
[07/08/2020   13:05:52.028241]    ::     Last IO Error Code: 1236 
[07/08/2020   13:05:52.028267]    ::     Last IO Error: "Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.'" 
[07/08/2020   13:05:52.028288]    ::     SQL Thread Running State: "Yes" 
[07/08/2020   13:05:52.028307]    ::     IO Thread Running State: "No" 
[07/08/2020   13:05:52.043512]    ::    PADR0009I Replicate started. 
[07/08/2020   13:05:52.043814]    ::    PADR0095I Refreshing Vault configuration files. 
[07/08/2020   13:05:52.122958]    ::    PADR0097I Refreshing Vault configuration files completed successfully. 
[07/08/2020   13:05:52.127879]    ::    GetPADRWorkingDirectory returned [C:\Program Files (x86)\PrivateArk\PADR\Conf] 
[07/08/2020   13:05:52.127915]    ::    GetPADRWorkingDirectory returned [C:\Program Files (x86)\PrivateArk\PADR\Conf] 
[07/08/2020   13:05:52.130296]    ::    PADR0010I Replicate ended. 

Cause 

The DR service has not run for a long time. Once you start the DR service, it is not able to find the transaction logs it requires to do the metadata replication.

Resolution 

Do a full replication from the Primary Vault / Master Vault.

Reset replication in the padr.ini:

1. Delete the following lines in padr.ini: 

NextBinaryLogNumberToStartAt
LastDataReplicationTimestamp

2. Restart CyberArk Vault Disaster Recovery service

DR site changed to 'inactive' due to lack of replication activity

ITATP052W The status of the DR site (Username = DR) changed to 'inactive' due to lack of replication activity.

DR Account Activities

Once PADR issue has been resolved and replication activity was successful, this kind of message should be gone. 

Set PADR.ini EnableFailover=No

For manual failover, not auto, in case there were network outage which you are not able to control, set PADR.ini EnableFailover=No

[MAIN]
ReplicateLogonFromFile="C:\Program Files (x86)\PrivateArk\PADR\Conf\user.ini"
EnableCheck=Yes
EnableReplicate=Yes
EnableFailover=No
EnableDbsync=Yes
CheckInterval=60
CheckRetriesCount=5
CheckRetriesInterval=30
ReplicateInterval=3600
ReplicateRetriesInterval=300
AccessVaultForInactivity=Yes
FailoverMode=No

NextBinaryLogNumberToStartAt=0
LastDataReplicationTimestamp=1596827567642853 

Vault Patching

For Windows Updates - just follow the standard vault patching instructions....

Stop the vault, enable Windows Update & Windows Module Installer, Install the KB/Patch, disable the services & start the vault.

Vault External Firewall Rules

[HSM]

AllowNonStandardFWAddresses=[2.2.2.2],Yes,1792:inbound/tcp,1792:outbound/tcp

[NTP]

AllowNonStandardFWAddresses=[1.1.1.1],Yes,123:outbound/udp

[RDP]

AllowNonStandardFWAddresses=[3.3.3.3],Yes,3389:inbound/tcp,3389:outbound/tcp

AllowNonStandardFWAddresses=[3.3.3.3],Yes,3389:inbound/udp,3389:outbound/udp

[Syslog]

AllowNonStandardFWAddresses=[4.4.4.4],Yes,514:outbound/udp

[Windows Update]

Some Vault Warning Messages

1. ITADB:399 : Using encryption algorithms: Advanced Encryption Standard (AES), 256bit, RSA (2048 bit).....

• Change to CA signed Certificate
  

2. ITATP044W Security warning - Vault certificate is self-signed. It 's recommended to use a CA signed certificate with this vault's configuration

• Change to CA signed certificate

3. ITATP056W Security warning - "Warning - Some or all of your LDAP Connections are not secured, It is recommended to use a secured LDAP connection (LDAPS) with this Vaults configuration"

• https://docs.cyberark.com/Product-Doc/OnlineHelp/PAS/Latest/en/Content/PAS%20INST/Configuring-Transparent-User-Management.htm
• https://docs.cyberark.com/Product-Doc/OnlineHelp/PAS/Latest/en/Content/PASREF/LDAP%20Integration%20-%20Introduction.htm
• Go into the LDAP configuration menu via PVWA, turn on SSL for all domains and for each domain controller host object. Then restart the Vault service.

1. Go to "LDAP Integration" --> "Directories"
2. Select <Directory Name>
3. Look for "SSLConnect" parameter at the right window
4. Change the value to "Yes" and save.
5. Restart vault

 

This should resolve the issue considering all other prerequisites [install LDAPS cert, hosts file entry] are successfully completed.

Don't forget to update LDAPS port configuraiton. 

Vault Server Temporarily Unavailable

Issue 1:

ITACM020S The server could not complete the operation because the vault was temporarily unavailable. If this error recurs, please logoff from the vault, logon again and retry the operation.
(Diagnostic information: 520,513,10054) 

The same can be seen in the PALog.



In the vault trace.dx log we can see the following error for the user logon operation:

ITAPE287E System error (Code: 287, Diagnostic information: userid, 1, 0).

Workaround:

Try a PrivateArk authentication without "Use pre-authentication secured session" selected in the Advanced PA client authentication settings:



Possible Resolutions:

- Ensure the vault certificate is valid (if using a CA signed certificate)
- Ensure the PA Client version matches the vault version

If you are using a Self-Signed Certificate on your Vault Server, please do the following to update the self-signed certificate:

Open an administrative command prompt at the /Server location on the vault server. Then run the following command:

CACert.exe Uninstall

This will uninstall the old self-signed certificate and install a new one.
Note: After the command has been run, you will need to restart the PrivateArk Vault service.

https://cyberark.my.site.com/s/article/PrivateArk-Client-ITACM020S-The-server-could-not-complete-the-operation-because-the-vault-was-temporarily-unavailable-If-this-error-recurs-please-logoff-from-the-vault-logon-again-and-retry-the-operation-Diagnostic-information-520-513-10054

References

• [Distributed Vault] - error 1236 master has purged binary logs containing GTIDs that the slave requires