Thycotic Secret Server Upgrade Process Document Example

This process document has been developed based on Thycotic SS Upgrade process post:

• https://blog.51sec.org/2021/09/thycotic-secret-server-upgrade-methods.html

It might give you an idea what the TSS upgrading looks like. But based on your environment, current version, new version, servers, DR, and down time requirement, the whole process will be completely different.

Regular Upgrading Process

Method 1:

following the checklist, making sure you backup the application and DB. Then, in the program you can run the automatic upgrade on one node and then copy the application files from the inner zip file of the server files zip onto the other SS node (https://updates.thycotic.net/secretserver/getlatestversion.aspx?alwayslatest=true). This should handle all of your DEs and SQL automatically. 

Method 2:

Otherwise, you can upgrade each node manually by unzipping the inner zip's files into both nodes' Secret Server's IIS folders and manually running the DB upgrade script. You might also have to redownload the DEs through the Admin -> Distributed Engine -> Add process. If you do, uninstall the engines, delete the application folder for the DEs, and then run the freshly-downloaded DE installers. Then, assign them to the correct site and activate them.

Upgrade Checklist:

1. Check if DPAPI is enabled. If yes, disable it from Admin > Configuration > Security > Decrypt Key using DPAPI
2. Full back up of Web Application files for each individual Node
3. Full Back up of Secret Server Database
4. Environment is within in System Requirements of Secret Server (https://docs.thycotic.com/ss/11.0.0/secret-server-setup/system-requirements/index.md)
    - .Net 4.8 or newer  
    - SQL Server 2012 - 2019 (Note: SQL Express should only be used for POC environments)
    - Collation SQL_Latin1_General_CP1_CI_AS
5. A user that can login via SQL Management Studio (SSMS) and have write access to the Secret Server Database
6. Downloaded zipped Application files from: https://updates.thycotic.net/secretserver/getlatestversion.aspx?alwayslatest=true placed on each of the Web Server and extract ss_update.zip on a Temporary location (i.e. Desktop or Temp Folder) 
7. Ensure SQL upgrade Script is on the SQL server (only if support has previously given this to you.)
8. Microsoft Monitoring Agent in Programs and Features should not be installed.
9. Provide support the number of DEs you have currently installed
10. Check if you have IWA enabled. If yes, make sure that only Windows Authentication is enabled and Forms is disabled. (https://docs.thycotic.com/ss/10.9.0/authentication/integrated-windows-authentication/configuring-iwa/index.md)
11. Make sure you have enough Disk space on your Database Server (on the drive the database and transaction files are stored)
12. If previously directed by support truncate tables - ONLY AS DIRECTED.
13. If your environment has SQL replication, please review https://updates.thycotic.net/secretserver/documents/SS_SQLReplication.pdf
14. Make sure that you don't use a custom theme in Admin > Config

Please ensure that a DBA with full access to the SQL database is on hand as well as a user with full admin access to every web node  that Secret Server is on.  We may also require access to distributed engines and RabbitMQ if installed.

Backup

Backup application folder manually through TSS Web Gui. Application backup folder :D:\backup\secretserver	On main site TSS server and DR site TSS server	Completed
Take DBs Database Backup	On both main site and DR site DBs
Take Snapshot of all servers	On all servers listing in the scope
Application folder backup to ’s VDI machine	On all TSS servers

.Net Framework 4.5 Upgrade to 4.8

Download .Net and save it to local folder on SS’s temp folder c:\temp\upgrade	On all servers except DB servers	Completed
Install .Net Framework 4.8 and related security patches	/  On all servers except DB servers
Restart Servers	/  On all servers except DB servers

Version and Configuration Verification

TSS Version verification	All three servers verified. Version is “10.8.000000 - Platinum Edition”	Completed
DE Version verification	All DE server verified. Version is: 10.8.000000	Completed
Database version verification	On both sites’ DB: 14.0.3281.6	Completed
Check Collation value: SQL_Latin1_General_CP1_CL_AS	On both sites’ DB servers	Completed
For Secret Server folder permission, add “Everyone” and provide full control for the folder. Modify permission from full control to just modified after upgrade activity.	On all TSS servers	Optional until issue happened
ASP.net🡪.NET trust level should be full in both levels (Server and Secret Server)	On all TSS servers	Optional until issue happened
IIS🡪Request Filtering (Server and SecretServer), check the first three options at both levels (Server and SecretServer)	On all TSS servers	Optional until issue happened
Minimum requirement check: https://docs.thycotic.com/ss/11.0.0/secret-server-setup/system-requirements/index.md	On All TSS servers	Completed
No Microsoft Monitor Agent	On All TSS related servers.	Completed

Others

Thycotic Support Availability	Brian from Thycotic will support us on Sep 15 from 7pm	Completed
CAB
Copy your own admin password from Thycotic Secret Server which will be used later during upgrading.	, , , ,
Turn off alerts from monitoring system

Main Site Manual Upgrade Process Steps - Primary Thycotic Secret Server

Confirm all pre-requisites completed
Download latest 11.0.000007 version Application files (Not Installation EXE File)		Completed
Stop SS Application Pool in IIS
DBA run Upgrade script which was provided by Thycotic. Please see DB upgrade process steps.
Extract downloaded zip application file to a temporary location C:\temp
Extract the ss_update.zip file
Create a zip file of existing SS application folder and send it to the desktop. (Another backup)
Ctrl+a select all files in step 6. Copy and paste the contents contained in the newly extracted ss_update folder to SS’s application folder over the top of the existing application files. Replace all files with the same name.
Once completed, start the SS application pool
Open an administrative command prompt and perform an “iisreset” command

Main Site DB Upgrade Process

Open SQL Management Studio and connect to the SQL Server database engine that hosts the Secret Server database
Expand Databases on the right
Right-click on the Secret Server database and select New Query. Paste the script.
Confirm there is no error on the query. If so, uncomment “COMMIT TRAN”.
Click the Execute button, Ctrl+E, or hit F5
Close SSMS

Secondary Thycotic Secret Server

Confirm Primary SS Server upgraded and works.
Download latest 11.0.000007 version Application files (Not Installation EXE File)		Completed
Stop SS Application Pool in IIS
Extract downloaded zip application file to a temporary location C:\temp
Extract the ss_update.zip file
Create a zip file of existing SS application folder and send it to the desktop. (Another backup)
Ctrl+a select all files in step 6. Copy and paste the contents contained in the newly extracted ss_update folder to SS’s application folder over the top of the existing application files. Replace all files with the same name.
Once completed, start the SS application pool
Open an administrative command prompt and perform an iisreset command

DR Site Manual Upgrade Process Steps - DR Thycotic Secret Server

Confirm all pre-requisites completed
Download latest 11.0.000007 version Application files (Not Installation EXE File)		Completed
Stop SS Application Pool in IIS
DBA run Upgrade script which was provided by Thycotic. Please see DB upgrade process steps.		(Optional, decided by )
Extract downloaded zip application file to a temporary location C:\temp
Extract the ss_update.zip file
Create a zip file of existing SS application folder and send it to the desktop. (Another backup)
Ctrl+a select all files in step 6. Copy and paste the contents contained in the newly extracted ss_update folder to SS’s application folder over the top of the existing application files. Replace all files with the same name.
Once completed, start the SS application pool
Open an administrative command prompt and perform an iisreset command

DR DB Upgrade Process (Optional)

If DR DB will be synchronized automatically from Main site since all of them are in AlwaysOn group, this step can be omitted. 

Confirm main SS upgrade works
Stop DR SS application pool in IIS
Copy Web application folder from primary SS in main site to DR SS server, without database.config and encryption.config. Replace the content of the existing web application folder with the new.
Confirm there is no error on the query. If so, uncomment “COMMIT TRAN”.
Once completed, start the SS application pool
Do verification

Test and verification

For each upgrade, please do following testing and verification. 

Log into Secret Server. Check the version of Secret Server in the application files by visiting https://<server host name>/SecreteServer
Check the database
Check system and DE health
Test RDP/SSH from SS web browser protocol handler	/  /  /
Testing customized launcher
Test Connection Manager	/  /  /
Check system logs
Check DR Server
Check DE version
Check Recording	/
Privilege Manager Authentication Testing	/

Post Upgrade

There are e pending tasks which will need to be resolved later after upgrade, not same day as upgrading day. 

Web Password handler upgrade. Current configuration disabled auto-upgrade.	TBD
Remove VM Snapshots
DE Upgrade
Protocol handler upgrade

Sometimes, you might need to change following settings if you are getting problem to access website: