Testing the upgrade in a duplicated environment

Prior to upgrading your production system, the best practice is to first test the upgrade using a copy of the production database in another environment.

To do so, you would duplicate your production environment, and then perform some post-copy configuration changes to ensure that the duplicate environment does not conflict with your production environment. You would then upgrade the test environment and run some functional tests to test the upgrade. The steps in this topic cover the high-level steps that you must perform; the specifics will likely vary, depending on the database in use, the operating systems involved, and the particular environment details.

Step 1: Review the prerequisites

Before moving or replicating your TrueSight Server Automation environment, note the following potential issues:

You might need to temporarily stop your TrueSight Server Automation Application Server to create a backup of your database and file server, depending on the backup mechanisms being used. This will ensure that both backups are synchronized to the same date and time. Notify any TrueSight Server Automation users that the system will be down during this process. The amount of time that the application server will be down depends on how quickly you are able to copy your file server and database. This time period can vary, depending on how much data is contained in each of these infrastructure pieces.

Ensure that your environment includes the following elements:

  • A server in which to install the TrueSight Server Automation Application Server
  • A server in which to install the RSCD Agent for the File Server (which may also be the same server as the Application Server)
  • Enough space on the File Server to match the space for the current Production File Server
  • A database of the same type and version as the current Production database in which to import the exported database

Step 2: Duplicate the installation infrastructure

First, decide 

what you are going to

move or replicate. In some cases, you might want to move only the file server. In other cases, you might want to replicate your entire infrastructure. The following steps assume that you are moving or replicating your entire environment, so if a particular piece does not apply to what you are moving or replicating, skip that step.

Note

If you are using TrueSight Smart Reporting for Server Automation, it is recommended that you replicate that environment as well. See Moving the TrueSight Server Automation - Data Warehouse environment.

Moving the Database

To move the database, perform the following steps:
  1. In the new database server, create the TrueSight Server Automation database or tablespaces and the TrueSight Server Automation database user.
  2. Export the database/schema from the old TrueSight Server Automation Database.  There are many utilities that can do this, and several examples are noted below.
    • Oracle: Use the Oracle 'data pump' utility, as in the following example.
      expdp [sys username]/[password]@[sid] schemas=[bladelogic schema] directory=[test_dir] dumpfile=[bladelogic schema].dmpdp logfile=expdp_[bladelogic schema]_[date].log flashback_time="\"to_timestamp(to_char(sysdate,'mm-dd-yyyy hh24:mi:ss'), 'mm-dd-yyyy hh24:mi:ss')\"" 
    • MSSQL
      1. Shut down all the Application Servers.
      2. Set the database to single-user mode. You can choose one of the following methods to do this.
        • Through the SQL Management Studio:
          1. In Object Explorer, connect to an instance of the SQL Server Database Engine, and then expand that instance.
          2. Right-click the database to change, and then click Properties.
          3. In the Database Properties dialog box, click the Options page.
          4. From the Restrict Access option, select Single.
          5. If other users are connected to the database, an Open Connections message appears. To change the property and close all other connections, click Yes.
        • By running a SQL query:
          USE master;
          GO
          ALTER DATABASE BLADELOGIC
          SET SINGLE_USER
          WITH ROLLBACK BLADELOGIC;
          GO
          ALTER DATABASE BLADELOGIC
          SET READ_ONLY;
          GO
          ALTER DATABASE BLADELOGIC
          SET MULTI_USER;
          GO
      3. Right click the database and select Tasks and then Back Up, and perform a full backup.
  3. Import the core database into the new database server w/ the corresponding import tool.  An example is shown below.
    NoteIf the target test environment already has working Application Servers and you want to preserve their configurations, then prior to the database import, export the contents of the application_server_deployments and service_config_property_value tables from the test database. These tables store the configuration data for the application servers.
    • Oracle: Use the Oracle 'data pump' utility, as in the following example.
      impdp [sys username]/[password]@[sid] dumpfile=[bladelogic schema].dmpdp logfile=impdp_[bladelogic schema]_[date].log transform=oid:n full=n schemas=[old schema name] directory=[test_dir] table_exists_action=replace remap_tablespace=[old bladelogic tablespace:new bladelogic tablespace],[old bladelogic index tablespace:new bladelogic index tablespace]
      remap_schema=[old bladelogic schema:new bladelogic schema]  
    • MSSQL: Import or restore the backup that was taken from the production database.

Moving the File Server

To move the file server, perform the following steps:
  1. Install an RSCD agent on the server that will function as a File Server.
    NoteIf you are using a non-Windows environment, you can skip this step, as the RSCD Agent is included in the Application Server installation that will be performed in the next section.
  2. Copy the file server storage location from the old location to the new location.  This can be done with a variety of utilities such as dsync, rsync and NAS or SAN replication tools.  The file server replication should be paired to the database replication.
  3. If the application server is not being moved then follow the Changing-the-location-of-the-file-server to update the application server configuration and database with the location of the new file server, otherwise wait to perform this step when the new application server is online.

Using the Application Server(s) in the new environment

To use the application server(s) in the new environment with the imported database, perform the following steps:
  1. If the target environment already has one or more TrueSight Server Automation Application Servers, perform one of the following steps (a or b):
    1. Import the test application server configurations that were exported in step 3 of Moving the database above.
      1. After the database import, delete the contents of the application_server_deployments and service_config_property_value tables.
      2. Delete any deployment directories in NSH/br/deployments except _launcher and _template.
      3. Import or insert the table export data from the application_server_deployments and service_config_property_value tables that you exported just prior to the database import.
    2. Create new deployments in the newly imported database.
      1. Delete any deployment directories in NSH/br/deployments except _launcher and _template on the test environment application servers.
      2. Run a single blasadmin command against template on each application server to trigger the registration of the application server with the newly imported database
        blasadmin -s _template set Database DriverClassoracle.jdbc.driver.OracleDriver (or the appropriate driver class)
        For more information about using the blasadmin commands, see Using blasadmin commands.
      3. Create the desired instances with the blasadmin create command.
  2. If this is not an upgrade related to a Disaster Recovery site, soft delete the old Application Servers from the imported database.
    Important: Adding the relevant backup date and timeWhen you run the following SQL statements, modify the 'bkup08232019_' to the relevant backup date and time.
    (Oracle only)
    update application_server set is_deleted = 1;
    COMMIT;
    update application_server_model set is_deleted = 1;
    COMMIT;

    UPDATE APPLICATION_SERVER t1
    SET APP_SERVER_NAME = 'bkup08232019_' || (SELECT APP_SERVER_NAME
    FROM APPLICATION_SERVER t2
    WHERE t1.APPLICATION_SERVER_ID = t2.APPLICATION_SERVER_ID
    AND t2.IS_DELETED = 1)
    WHERE EXISTS (SELECT 1
    FROM APPLICATION_SERVER t2
    WHERE t1.APPLICATION_SERVER_ID = t2.APPLICATION_SERVER_ID
    AND t2.IS_DELETED = 1
    );
    COMMIT;


    UPDATE APPLICATION_SERVER_MODEL t1
    SET NAME = 'bkup08232019_' || (SELECT NAME
    FROM APPLICATION_SERVER_MODEL t2
    WHERE t1.APPLICATION_SERVER_MODEL_ID = t2.APPLICATION_SERVER_MODEL_ID
    AND t2.IS_DELETED = 1)
    WHERE EXISTS (
    SELECT 1
    FROM APPLICATION_SERVER_MODEL t2
    WHERE t1.APPLICATION_SERVER_MODEL_ID = t2.APPLICATION_SERVER_MODEL_ID
    AND t2.IS_DELETED = 1);
    COMMIT;
    (SQL Server only)
    update application_server set is_deleted = 1;
    GO
    update application_server_model set is_deleted = 1;
    GO
    UPDATE APPLICATION_SERVER
    SET APP_SERVER_NAME = 'bkup08232019_' + (select t2.APP_SERVER_NAME
    from APPLICATION_SERVER t2
    where t2.APPLICATION_SERVER_ID = t1.APPLICATION_SERVER_ID
    and t2.IS_DELETED = 1
    )
    FROM APPLICATION_SERVER t1;
    GO
    UPDATE t1
    SET t1.NAME = 'bkup08232019_' + (select t2.NAME
    from APPLICATION_SERVER_MODEL t2
    where t2.APPLICATION_SERVER_MODEL_ID = t1.APPLICATION_SERVER_MODEL_ID
    and t2.IS_DELETED = 1
    )
    FROM APPLICATION_SERVER_MODEL t1;
    GO
    NoteIn the code example, the substring bkup08232019_ must reflect the current date and be unique. You might receive an error message if same sql query is run twice on the same database.
  3. If the target environment has Application Servers of a version earlier than 8.6, and the database is already version 8.6 or later, run the configurator utility before upgrading, in order to persist configuration data to the database. This is necessary only when upgrading a Disaster Recovery environment from a release earlier than 8.6 to 8.6 or later and the Disaster Recovery Application Servers are registered in the application_server and application_server_model tables.
  4. If the target environment does not have any Application servers and the database is already present, perform the following steps
    1. Install the Application Server software on the new Application Server(s) using the individual component installers, not the unified product installer. 
    2. Using the blasadmin utility, configure the Database section parameters noted below.  These settings will be for the new database server.blasadmin -a set database ConnectionString <ConnectionString>blasadmin -a set database DriverClass <DriverClass>blasadmin -a set database UserId <user_name>blasadmin -a set database Password <Password>For more information about using the blasadmin commands, see Using blasadmin commands.
  5. Copy the files listed in the Files section of the Backup and Restore of the TrueSight Server Automation Environment document from the old application server to the new application server(s) except the bladelogic.keystore file.  Leave the bladelogic.keystore that we created during install on the new application server.
  6. Follow the Changing-the-location-of-the-file-server to update the application server configuration and database with the location of the new file server.

Provisioning

To move the provisioning environment, perform the following steps:
  1. If replicating your provisioning environment, install the TrueSight Server Automation PXE and TFTP server.
  2. Copy your provisioning data store to the new server.
  3. Replicate any custom application server settings by running <bl_install_dir>/br/blasadmin.
  4. Update the datastore locations to point to the new locations in the new environment.

Step 3: Perform post-copy configuration changes

After your environment has been moved, there are some final steps that you will want to make to ensure that there are no conflicts with your existing environment.

Updating Extended Object Locations

If there are Extended Objects that reference the old file server or other location these must be updated as well using the update_extended_object_script_location included in the external-files.zip for the release.

Updating Remote Depot Object Locations

If there are Depot objects whose payload is stored on a remote NSH path and not the file server, and those payloads are also moved in the new environment, the depot_object_location.remote_path column will need to be updated as well.

Preventing access from the new environment to the existing Application Servers

There are a few ways to accomplish this. One way to handle this is to generate a self-signed certificate for an Application Server in the new environment. Another way is to restrict network access between the two environments; this will prevent accidentally connecting to the original environment.

Removing Job Schedules

If the new environment is being used for testing you will want to disable scheduled jobs so they will not run against the registered servers.  This can be done with a simple query:(Oracle only)
update schedule set is_deleted = 1;
update job_run set progress_status_id = 7 where progress_status_id in(1, 2, 3, 5, 6);
commit; 
(SQL Server only)update schedule set is_deleted=1;
GO
update job_run set progress_status_id = 7 where progress_status_id in(1, 2, 3, 5, 6);
GO
Setting the is_deleted flag is a non-recoverable operation, the schedules will be permanently disabled in the new environment. This is typically desirable for testing environments.

Update Catalog Locations (Optional)

If you need to make use of the patch catalogs in the testing environment then you can also update the catalog location settings with a database query:(Oracle only)update job_option_value set option_value = replace (option_value, 'OLDNAME','NEWNAME') where option_value like '%OLDNAME%';commit;(SQL Server only)update job_option_value set option_value = replace (option_value,@OLDNAME,@NEWNAME) where option_value like '%' + OLDNAME + '%'Where OLDNAME is the old path to the catalog and NEWNAME is the path to the new location of the catalog.

Removing Notifications

If the new environment is being used for testing you will want to disable any job notifications to avoid confusing when testing jobs in the new environment. This can be done with a simple query:update notification set email_to='test123@example.com';commit; (Oracle Only)

Removing Routing Rules

If the new environment is being used for testing or the existing repeaters, appservers and SOCKs proxies will no longer be needed, then the routing rules can be removed with a few queries:Delete SOCKS Proxy assigned in Network routing rule
update routing_policy set is_deleted = 1 where routing_policy_type_id = 1;Delete Job Server assigned in job routing rule.update routing_policy set is_deleted = 1 where routing_policy_type_id = 2Delete Repeater assigned in Repeater routing rule.
update routing_policy set is_deleted =1 where routing_policy_type_id = 3;commit; (Oracle only)

Preventing access from the new environment to existing managed servers

If the new environment is being used for testing you may want to prevent the new application servers from talking to your existing targets.  This is possible by pushing out an exports file from the original appserver/environment to all target servers that allows connections from only the original application servers and any NSH client systems you need to allow connections from.

Decommissioning Servers (optional)

  1. Decommission any servers that are in the development environment that you will not manage. This will prevent any confusion for anyone that might log into the duplicated environment.
    NoteIf you have an 8.x environment AND have the Licensing Portal configured, make sure that you configure the Licensing Portal to not re-claim any licenses when you decommission servers. If you do, you will need to re-license your servers in the original environment that you copied from.
  2. Ensure that only the servers that are expected to be registered in the duplicated environment remain.

Step 4: Upgrade the test system

Review the pre-requisites for upgrade:

Step 5: Run some tests in the duplicate environment

After completing the steps in the new environment, perform some representative tests, such as the following:

  • Start up the Application Server services:
    • On Windows, from the Start menu, select Settings > Control Panel. Double-click Administrative Tools, and double-click Services. Right-click TrueSight Server Automation Application Server and select Start from the pop-up menu.
    • To start the Application Server, run the following command from the directory where TrueSight Server Automation is installed, if you are using a Linux system with systemd support: 
      systemctl start blappserv
      If you are using a Linux system without systemd support, run the following command:
      /etc/init.d/blappserv start
  • Log on to the TrueSight Server Automation Console from the Start menu. Choose Programs > BMC Software > BladeLogic Server Automation Suite > Server Automation Console<version>.
  • Test the provisioning process by executing a Provisioning Job and checking the results.
    • In the Jobs folder, select the Provision Job, right-click, and select  Execute .
    • Select a Provision Job, right-click, and select  Show Results .
  • Test the patch process by executing a Patch Job and checking the results.
    • In the Jobs folder, select the Patch Job, right-click, and select  Execute .
    • Select a Patch Job, right-click, and select  Show Results .

Step 6: Make required changes in the production environment

If the functional tests reveal issues, you may need to make some changes in the production environment prior to upgrading.

For example, you may discover from your testing that you need to make some database remediations, which you could apply at the beginning of the production upgrade outage.

Step 7: Upgrade the production environment

Follow the steps in Walkthrough-Upgrading-to-the-latest-version-for-Linux or Walkthrough-Upgrading-to-the-latest-version-for-Microsoft-Windows.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*