CyberArk PAS Solution Issues and Troubleshooting (PSM)

This is my CyberArk troubleshooting post to record those issues I met during working on CyberArk PAS (Privileged Account Security) Solutions . This post is focus on PSM. I have another two posts are for PVWA and CPM.

• PSM: This app has been blocked
• Issue: Network Level Authentication Disabled
• Issue: RDS Installation - Collection Role failed to create
• Issue: Remote Desktop Licensing mode is not configured
• Issue: SSH through PSM failed
• Issue: RDP Remote through PSM failed using local admin account
• PSM Session Failed Login - Username and Password is incorrect. 
• PSMSR196E PSM is not enabled or not defined for policy
• Error: The privileged session could not be established securely.
• Remote Connection from PSM to Target Server Error
• PSMSR196E PSM is not enabled or not defined for policy
• PSMSR308E Selected component does not contain the target settins definitions in policy 
• PSMConnect Credential does not work
• The referenced account is currently locked out
• Unable to load private key (file format error)
• more...

Related posts:

• CyberArk PAS Solution Issues and Troubleshooting (PVWA)
• CyberArk PAS Solution Issues and Troubleshooting (PSM)
• CyberArk PAS Configuration Issues and Troubleshooting (CPM)
• CyberArk PAS Configuration Issues and Troubleshooting (Vault)

This app has been blocked 

1. Using PSM SSH to connect to Remote Site but got an error
"This app has been blocked by your system administrator."

Resolution:

Reference: https://cyberark-customers.force.com/s/article/00004458

Network Level Authentication Disabled

2. NLA Enabled on PSM servers

Resolution:
You can use domain group policy to fix this.

RDS Installation - Collection Role failed to create

When install RDS role on PSM server, you might meet RDS Collection Role Creation Failed error.

Resolution:
Group Policy related. Move PSM servers out of regular Domain OU to a new OU without any group policy on it except default domain group policy.

Remote Desktop Licensing mode is not configured

RDS License issue
Remote Desktop Licensing mode is not configured. Remote Desktop Services will stop working in 123 days. On the RD Connection Broker server, use Server Manager to specify the Remote Desktop Server.

Resolution:
You will need to add license before it is expired.

SSH through PSM failed

Symptoms:
Trying to a remote ssh through PSM, but got following failed message. RDP to same network's server was fine.

Cause and Solution:
It has been caused by global policy removed PSMShadowusers access locally.

RDP Remote through PSM failed using local admin account

Trying to log in remote server through PSM using local admin account, failed with following error.

Resolution:
It is network connectivity issue between PSM and Remote Destination. If you met this error, try to RDP directly from PSM server to see if you will meet this issue or not.

PSM Session Failed Login - Username and Password is incorrect.

Using PVWA to connect to remote RDP servers, but failed log into PSM server before PSM can launch remote server's RDP session. It gives out an error "The username and password is incorrect".

This usually relates to PSM server's local accounts:
1. PSMCONNECT - for RDP session to log into PSM servers.
2. PSMADMINCONNECT - for auditor monitoring to use

The password for those two accounts might lost sync to the vault. You can just use PVWA to show password then copy it to PSM server local user. Basically it is to change PSM server's psmconnect and psmadminconnect account's password to match vault's password.

PSMSR196E PSM is not enabled or not defined for policy

It happened when PSM was just installed and tried to use PVWA to test PSM with connect button.
PSM has been registered with PVWA. Confirmed Option setting for PSM to use ActiveX set to never.  Mostly it is because of delay of system, I am guessing. It went away after a while.

Error with Network Level Authentication and CredSSP encryption oracle remediation

All following error messages are same issue. The error is relating to Remote Desktop settings. Uncheck "Allow connections only from computer running Remote Desktop with Network Level Authentication (Recommended)" from all of your PSM servers, and target servers.

An authentication error has occurred. The function requested is not supported. Remote computer: <computer name or IP>. This could be due to CredSSP encryption oracle remediation. For more information, see https://go.microsoft.com/fwlink/?linkid=866660.

PSMRD001E User was disconnected from remote machine. Reason: [An internal error has occurred.] (Code: 519)

There are some solutions from :  https://support.microsoft.com/en-au/help/4295591/credssp-encryption-oracle-remediation-error-when-to-rdp-to-azure-vm

Another Configuration for your environment if the setting has been disabled by your domain group policy, you might try from Server Manager - Remote Desktop Services - Collections - <RDP Server> - Tasks - Edit Properties - Security - Uncheck NLA settings.

Error: The privileged session could not be established securely. Contact your system administrator.

Most likely your PSM service is down. You can confirm that from services.msc or CyberArk PVWA System health page.

Remote Connection From PSM to Targets Error (Code: 516)

This error is actually a standard RDP error. PSMRD001E, Code:516

Mostly it is caused by remote target server's RDP service not up or network connection broken between PSM and targets.


https://cyberark-customers.force.com/s/article/00004491

PSM is not enabled or not defined

PSMSR196E Privileged Session Management is not enabled or not defined for policy

Add PSM to your platform. Restart your PSM service to take this change into effect.

PSMSR308E Selected component does not contain the target settins definitions in policy

Usually it is caused by connection component target settings. Check LogonURL or Form Settings such as UserID, Password ID, Click/Button.

PSMRD001E An internal error has occurred (Code: 3335)

Usually it caused by a disabled / locked account.

PSMConnect Credential does not work

From Event Viewer - Security log, it has 4625 audit failure log to show log in failed.

It usually caused by out-of-sync PSMConnect account. It has been happening randomly in my environment, mostly over VPN. Not sure real cause still. It also might be relating to RDCMan, based on CyberArk support.

Warning: If you have multiple sessions opened at the same time and this issue happened, it might get your PSMConnect locked out.

The referenced account is currently locked out

I met this during lost my vpn connection and tried to reconnect back using RDCM.

Cause 1: Log back into your PSM server using other account or from other PSM server, uncheck your PSMConnect's locked out option.

Cause 2: It might relate to RDCM. For some reason, RDCM did not pass session id properly to PSM server. PSM log error:

[22/06/2020 | 09:36:09.605467] | {pid= 1348} | {tid=     3912} | class CPSMBaseException *           | PSM\PSMServer\PSMBaseException.cpp(68)                       |  ::  | PSMSR009I Privileged Session Manager exception occurred. PSMSR268E An error occurred while waiting for the recorder process to shutdown (Code: 1073807364). (Codes: -1, -1) [22/06/2020 | 09:36:06.144982] | {pid= 1348} | {tid= 3920} | class CPSMBaseException * | PSM\PSMServer\PSMBaseException.cpp(68) | :: | PSMSR009I Privileged Session Manager exception occurred. PSMSR975W Session keeper was already closed with successful exit code: 0 (Codes: -1, -1) [22/06/2020 | 09:36:06.213461] | {pid= 1348} | {tid= 3944} | class CPSMConfigurationManager * | PSM\PSMServer\PSMConfigurationManager.cpp(569) | :: | PSMSR174I Starting configuration management operation (operation: lock configuration manager for read) [22/06/2020 | 09:36:06.213606] | {pid= 1348} | {tid= 3944} | class CPSMConfigurationManager * | PSM\PSMServer\PSMConfigurationManager.cpp(574) | :: | PSMSR175I Configuration management operation ended successfully (operation: lock configuration manager for read) [22/06/2020 | 09:36:06.214990] | {pid= 1348} | {tid= 3944} | class CPSMConfigurationManager * | PSM\PSMServer\PSMConfigurationManager.cpp(585) | :: | PSMSR174I Starting configuration management operation (operation: release configuration manager lock (read)) [22/06/2020 | 09:36:06.215075] | {pid= 1348} | {tid= 3944} | class CPSMConfigurationManager * | PSM\PSMServer\PSMConfigurationManager.cpp(587) | :: | PSMSR175I Configuration management operation ended successfully (operation: release configuration manager lock (read)) [22/06/2020 | 09:36:06.215199] | {pid= 1348} | {tid= 3920} | class CPSMLiveSessionsList * | PSM\PSMServer\PSMLiveSessionsList.cpp(198) | :: | PSMSR631I A session UUID [4a2b722a-2a7d-4687-b5ea-1d606a543577] was unregistered from the sessions list [22/06/2020 | 09:36:06.215348] | {pid= 1348} | {tid= 3520} | class CPSMUpdateLiveSessionsListJob * | PSM\PSMServer\PSMUpdateLiveSessionsListJob.cpp(198) | :: | PSMSR673I Updating live sessions list [22/06/2020 | 09:36:06.215626] | {pid= 1348} | {tid= 3520} | class CPSMVaultLiveSessionsList * | PSM\PSMServer\PSMVaultLiveSessionsList.cpp(78) | :: | PSMSR655I PSM current live sessions number is: 2 [22/06/2020 | 09:36:09.421734] | {pid= 1348} | {tid= 3912} | class CPSMBaseException * | PSM\PSMServer\PSMBaseException.cpp(68) | :: | PSMSR009I Privileged Session Manager exception occurred. PSMSR278I [0cc33c01-aec8-4add-aa95-09f828883ee1] Session component [SessionKeeper] has stopped. Ending session. (Codes: -1, -1) [22/06/2020 | 09:36:09.424413] | {pid= 1348} | {tid= 4212} | class PipeIPCChannel * | PSM\(310) | :: | PSMIC008I IPC Pipe Channel was instructed to stop while waiting for data [22/06/2020 | 09:36:09.424612] | {pid= 1348} | {tid= 4212} | class CIPCChannelExceptionBase * | PSM\(39) | :: | PSMIC001I IPC Channel exception was thrown. PSMIC013I IPC Pipe Channel received stop event. (Codes: -1, 0) [22/06/2020 | 09:36:09.424824] | {pid= 1348} | {tid= 4212} | class CPSMProtocolExceptionBase * | PSM\(35) | :: | PSMPR001I Protocol exception occurred. PSMPR062I Stop event received, terminating gracefully. (Codes: -1, -1) [22/06/2020 | 09:36:09.605467] | {pid= 1348} | {tid= 3912} | class CPSMBaseException * | PSM\PSMServer\PSMBaseException.cpp(68) | :: | PSMSR009I Privileged Session Manager exception occurred. PSMSR268E An error occurred while waiting for the recorder process to shutdown (Code: 1073807364). (Codes: -1, -1) [22/06/2020 | 09:36:09.605882] | {pid= 1348} | {tid= 3912} | class CPSMBaseException * | PSM\PSMServer\PSMBaseException.cpp(68) | :: | PSMSR009I Privileged Session Manager exception occurred. PSMSR975W Session keeper was already closed with successful exit code: 0 (Codes: -1, -1) [22/06/2020 | 09:36:09.666256] | {pid= 1348} | {tid= 3944} | class CPSMConfigurationManager * | PSM\PSMServer\PSMConfigurationManager.cpp(569) | :: | PSMSR174I Starting configuration management operation (operation: lock configuration manager for read) [22/06/2020 | 09:36:09.666415] | {pid= 1348} | {tid= 3944} | class CPSMConfigurationManager * | PSM\PSMServer\PSMConfigurationManager.cpp(574) | :: | PSMSR175I Configuration management operation ended successfully (operation: lock configuration manager for read) [22/06/2020 | 09:36:09.667838] | {pid= 1348} | {tid= 3912} | class CPSMLiveSessionsList * | PSM\PSMServer\PSMLiveSessionsList.cpp(198) | :: | PSMSR631I A session UUID [0cc33c01-aec8-4add-aa95-09f828883ee1] was unregistered from the sessions list [22/06/2020 | 09:36:09.667931] | {pid= 1348} | {tid= 3520} | class CPSMUpdateLiveSessionsListJob * | PSM\PSMServer\PSMUpdateLiveSessionsListJob.cpp(198) | :: | PSMSR673I Updating live sessions list [22/06/2020 | 09:36:09.668232] | {pid= 1348} | {tid= 3520} | class CPSMVaultLiveSessionsList * | PSM\PSMServer\PSMVaultLiveSessionsList.cpp(78) | :: | PSMSR655I PSM current live sessions number is: 1 [22/06/2020 | 09:36:09.668308] | {pid= 1348} | {tid= 3944} | class CPSMConfigurationManager * | PSM\PSMServer\PSMConfigurationManager.cpp(585) | :: | PSMSR174I Starting configuration management operation (operation: release configuration manager lock (read)) [22/06/2020 | 09:36:09.668401] | {pid= 1348} | {tid= 3944} | class CPSMConfigurationManager * | PSM\PSMServer\PSMConfigurationManager.cpp(587) | :: | PSMSR175I Configuration management operation ended successfully (operation: release configuration manager lock (read)) [22/06/2020 | 09:36:11.762269] | {pid= 1348} | {tid= 3924} | class CPSMBaseException * | PSM\PSMServer\PSMBaseException.cpp(68) | :: | PSMSR009I Privileged Session Manager exception occurred. PSMSR951E Session uuid was not found in session synchronizer. Diagnostic information: 1, (Session UUID: 71d4d036-d24c-4d8e-bde6-3788e6ba16df) (Codes: -1, -1) [22/06/2020 | 09:36:11.762555] | {pid= 1348} | {tid= 3924} | class CPSMBaseException * | PSM\PSMServer\PSMBaseException.cpp(68) | :: | PSMSR009I Privileged Session Manager exception occurred. PSMSR952E Failed to remove the session parameters from the session synchronizer (Session UUID: 71d4d036-d24c-4d8e-bde6-3788e6ba16df). Reason: PSMSR951E Session uuid was not found in session synchronizer. Diagnostic information: 1, (Session UUID: 71d4d036-d24c-4d8e-bde6-3788e6ba16df) (Codes: -1, -1) [22/06/2020 | 09:36:11.762719] | {pid= 1348} | {tid= 3924} | class CPSMBaseException * | PSM\PSMServer\PSMBaseException.cpp(68) | :: | PSMSR009I Privileged Session Manager exception occurred. PSMSR1105I The Vault session associated with session UUID [71d4d036-d24c-4d8e-bde6-3788e6ba16df] does not exist. (Codes: -1, -1) [22/06/2020 | 09:36:11.762867] | {pid= 1348} | {tid= 3924} | class CPSMBaseException * | PSM\PSMServer\PSMBaseException.cpp(68) | :: | PSMSR009I Privileged Session Manager exception occurred. PSMSR603I [71d4d036-d24c-4d8e-bde6-3788e6ba16df] Termination event [Session event] received (Codes: -1, -1) [22/06/2020 | 09:36:11.764384] | {pid= 1348} | {tid= 4264} | class PipeIPCChannel * | PSM\(310) | :: | PSMIC008I IPC Pipe Channel was instructed to stop while waiting for data [22/06/2020 | 09:36:11.764825] | {pid= 1348} | {tid= 4264} | class CIPCChannelExceptionBase * | PSM\(39) | :: | PSMIC001I IPC Channel exception was thrown. PSMIC013I IPC Pipe Channel received stop event. (Codes: -1, 0) [22/06/2020 | 09:36:11.765266] | {pid= 1348} | {tid= 4264} | class CPSMProtocolExceptionBase * | PSM\(35) | :: | PSMPR001I Protocol exception occurred. PSMPR062I Stop event received, terminating gracefully. (Codes: -1, -1) [22/06/2020 | 09:36:12.223458] | {pid= 1348} | {tid= 3924} | class CPSMBaseException * | PSM\PSMServer\PSMBaseException.cpp(68) | :: | PSMSR009I Privileged Session Manager exception occurred. PSMSR975W Session keeper was already closed with successful exit code: 0 (Codes: -1, -1) [22/06/2020 | 09:36:12.276163] | {pid= 1348} | {tid= 3944} | class CPSMConfigurationManager * | PSM\PSMServer\PSMConfigurationManager.cpp(569) | :: | PSMSR174I Starting configuration management operation (operation: lock configuration manager for read) [22/06/2020 | 09:36:12.276293] | {pid= 1348} | {tid= 3944} | class CPSMConfigurationManager * | PSM\PSMServer\PSMConfigurationManager.cpp(574) | :: | PSMSR175I Configuration management operation ended successfully (operation: lock configuration manager for read) [22/06/2020 | 09:36:12.277563] | {pid= 1348} | {tid= 3924} | class CPSMLiveSessionsList * | PSM\PSMServer\PSMLiveSessionsList.cpp(198) | :: | PSMSR631I A session UUID [71d4d036-d24c-4d8e-bde6-3788e6ba16df] was unregistered from the sessions list

Unable to load private key (file format error)

It is mostly because the private key is protected by passphrase. You will have to remove passphrase, and upload key to CybaerArk vault account again. 

Connect Button is greyed out

1. Make sure enabled isolation on Session Management settings in Master Policy (active, no exception for your platform)

2. In your platform, enabled Connector

3. Account level, make sure something put into address. Else the connect button will be greyed out.

Parameter BrowsePath is invalid

https://cyberark.my.site.com/s/article/Failed-to-initialize-web-browser-The-selected-browser-was-not-found

When tried to launch Azure Portal Connection, it failed with this message. 

By default it is using x86 32b chrome path. If you installed 64b chrome, you will need to make browser path change:

Failed to retrieve password object properties. Safe: VaultInternal

modify the permission on a specific safe through PA Client. Select safe-> right click and open -> right click and go to properties -> Sharing tab -> Check the box "Enable access to partially impersonated users"

RDS License Not Been Used

Customer has installed RDS license, but the PSM servers does not consume those licenses for some reasons. As you can see, Built-in overused licesne has been used four in the following screenshot. Purchased 500 user license shows 0 issued. 

Root Cause:

This is 2019 server. But RDS license is for 2022. Not same version which cause issue. 

Solution:

Generate right OS license from https://admin.microsoft.com/ - Billing - Your products :

Install new license key generated from previous step , issue resolved right away.

References