Thycotic Secret Server Troubleshooting Tips and Tricks

 This post is to summarize some common troubleshooting cases during working on Thycotic Secret Server.

Can not search secret's certain custom field

Although in secret template, the custom field has been set to searchable, users still could not search those custom fields.

Change Index Mode from Standard to Extend. Standard will need to type full internal code to search. Extend will use partial searching. 

After change, review and run index in secret server to take changes into effect. 

Incorrect Role Syncing to Secret Server

Default, new user role always is assigned to user. For most use cases, default user role set to user is enough. For large organization, the best practice is to set to <None>, then use Role Assignment to assign correct role based on group. 

If we create a role assignment for a group or user, that user or group will not only get default user group role which by default it is user role, also it will get this new created role. That is why best practice for default user role is to set to None. 

Summary:

• Roles, Groups and Users should be reviewed regularly
• Use Event Subscriptions to alert on any changes made to basic configurations
• Always review default settings and confirm if they can be customized
• Using the Hybrid approach (default user role and role assignment) will minimize consequences if users are incorrectly synced to Secret server

Custom Launcher Process Not Found

Summary:

• Launchers can be customized to work with any command-line-started application
• Always confirm applications are mapped properly for all client machines that will be leveraging custom launchers
• Don't forget to add the program folder in the PATH environment variable
• Each custom launcher will have unique requirements -review the support portal for most up to date configuration steps

Troubleshooting Discovery

Discovery Application pool Error:

Exception: Retrieving the COM class factory for remote component with CLISID from <machine> failed due to error:80070005

It usually means you do not have proper permission to scan the machine.

Enter error code into support portal to find out related documents. It might tell you what kind of account permission you will need:

• Make the account be able to log on as service
• Grant the account read, write and execute privileges to the entire distributed engine install directory and sub-folders
• Add the account to the administrators group on each computer that will be scanned.

Steps:

• Delete Distributed Engine
• Stop Distributed Engine service
• Change the service properties to add the account which will do discovery , to start the service. 
• Start distributed engine Service
• Make sure discovery account is same as the account starting the DE service
• The new engine will need to be verify and it will be assigned to the site. 

Troubleshooting - Remote Password Changing

Change Password Failed: Check Out is enabled on associated Secret.

Report Schedule: Secret with Failed Password Change

Admin -> Event Subscription

Troubleshooting - Session Recording 

Scenario 1 - Node Capacity Limits

Error!

Max Concurrent Session Per Web Node Reached

Identify the issue(s):

Review system logs

Review Secret Audit Trail

Review Reports

Find the solution(s):

Review System requirements

Increase concurrent session per node

Add an additional node

• Add this report into widget into your home dashboard. 
• inetpub - wwwroot - SecretServer - web-appSettings
• add key for PrefetchCount.CovertVideoMessage value into it to set the concurrent number. 
• iisreset

Troubleshooting - Auditing and Reporting

By default, secret server does not delete any audit data

Data deletion occurs automatically at 2 AM EST every Sunday

Do not configure auto record deletion for compliance or other important data

Unlimited admin role does not include audit data retention management. There are two roles relating to retention. 

Troubleshooting - Distributed Engine

Identify issues:

Compatible feature logs

Distributed engine logs

System logs

Solutions:

Enabling debugging on server where engine is installed on. 

Check Logs

• Admin - Distributed Engine - Managed Sites - Click the site DE installed o - Review the logs
• Program Files - Thycotic Software Ltd - Distributed Engine - Log - SSDE log file

Enable DE Debugging

•  Notepad to open Thycotic.DistributedEngine.Service.exe
• Search log4net, which under the </startup>
• Change first two log level value  from 'info' to 'debug' in multiple places
• iisreset
• Also, you can create verbos to replace debug to get more details.

Troubleshooting - Upgrade

Review service account status before performing an upgrade - running an IIS reset is a good suggestion  before a running upgrade.

Secret Server： Secret Server Launcher Stopped

The Secret Server Launcher stopped because the following Secret Server URL is not approved for launch:

Solution: 

There is a file called "SSUA.dat" in the following location:

C:\Users\<Username Here>\AppData\Roaming\Thycotic

Move that file somewhere else, like the desktop or somewhere, and then try to launch a new session you should see a popup like this and make sure to click "Yes" are you then allowed to launch into a session? Usually, it's because a user has accidentally selected "No" on that pop-up. 

References