Testing the upgrade in a duplicated environment

Prior to upgrading your production system, the best practice is to first test the upgrade using a copy of the production database in another environment.

To do so, you would duplicate your production environment, and then perform some post-copy configuration changes to ensure that the duplicate environment does not conflict with your production environment. You would then upgrade the test environment and run some functional tests to test the upgrade. The steps in this topic cover the high-level steps that you must perform; the specifics will likely vary, depending on the database in use, the operating systems involved, and the particular environment details.

This topic includes the following sections:

Step 1: Review the prerequisites

Before moving or replicating your BMC Server Automation environment, note the following potential issues:

You might need to temporarily stop your BMC Server Automation Application Server to create a backup of your database and file server, depending on the backup mechanisms being used. This will ensure that both backups are synchronized to the same date and time. Notify any BMC Server Automation users that the system will be down during this process. The amount of time that the application server will be down depends on how quickly you are able to copy your file server and database. This time period can vary, depending on how much data is contained in each of these infrastructure pieces.

Ensure that your environment includes the following elements:

A server in which to install the BMC Server Automation Application Server
A server in which to install the RSCD Agent for the File Server (which may also be the same server as the Application Server)
Enough space on the File Server to match the space for the current Production File Server
A database of the same type and version as the current Production database in which to import the exported database

Step 2: Duplicate the installation infrastructure

First, decide what you are going to move or replicate. In some cases, you might want to move only the file server. In other cases, you might want to replicate your entire infrastructure. The following steps assume that you are moving or replicating your entire environment, so if a particular piece does not apply to what you are moving or replicating, skip that step.

Note

If you are using BMC Decision Support for Server Automation, it is recommended that you replicate that environment as well. See Moving the BMC Decision Support for Server Automation environment.

Moving the Database

To move the database, perform the following steps:

In the new database server, create the BladeLogic database or tablespaces and the bladelogic database user.
1. Oracle
2. MSSQL
Export the database/schema from the old BladeLogic Database. There are many utilities that can do this, a couple examples are noted below.
1. Oracle: Use the Oracle 'data pump' utility, as in the following example.
  expdp [sys username]/[password]@[sid] schemas=[bladelogic schema] directory=[test_dir] dumpfile=[bladelogic schema].dmpdp logfile=expdp_[bladelogic schema]_[date].log flashback_time="\"to_timestamp(to_char(sysdate,'mm-dd-yyyy hh24:mi:ss'), 'mm-dd-yyyy hh24:mi:ss')\""
2. MSSQL
Import the core database into the new database server w/ the corresponding import tool. An example is shown below.
Note

If the target environment already has working Application Servers at version 8.6 or higher and you want to preserve their configurations, then prior to the database import, export the contents of the application_server_deployments and service_config_property_value tables. These tables store the configuration data for the application servers.
1. Oracle: Use the Oracle 'data pump' utility, as in the following example.
  impdp [sys username]/[password]@[sid] dumpfile=[bladelogic schema].dmpdp logfile=impdp_[bladelogic schema]_[date].log transform=oid:n full=n schemas=[old schema name] directory=[test_dir] table_exists_action=replace remap_tablespace=[old bladelogic tablespace:new bladelogic tablespace],[old bladelogic index tablespace:new bladelogic index tablespace]
  remap_schema=[old bladelogic schema:new bladelogic schema]
2. MSSQL

Moving the File Server

To move the file server, perform the following steps:

Install an RSCD agent on the server that will function as a RSCD Agent.
Note

If you are using a non-Windows environment, you can skip this step, as the RSCD Agent is included in the Application Server installation that will be performed in the next section.
Copy the file server storage location from the old location to the new location. This can be done with a variety of utilities such as dsync, rsync and NAS or SAN replication tools. The file server replication should be paired to the database replication.
If the application server is not being moved then follow the Changing the location of the file server to update the application server configuration and database with the location of the new file server, otherwise wait to perform this step when the new application server is online.

Using the Application Server(s) in the new environment

To use the application server(s) in the new environment with the imported database, perform the following steps:

If the target environment already has one or more BMC Server Automation Application Servers at version 8.6 or later, perform the following steps
1. After the database import, delete the contents of the application_server_deployments and service_config_property_value tables.
2. Import or insert the table export data from the application_server_deployments and service_config_property_value tables that you exported just prior to the database import.
If the target environment has Application Servers of a version earlier than 8.6, and the database is already version 8.6 or later, run the configurator utility before upgrading, in order to persist configuration data to the database. This is necessary only when upgrading a Disaster Recovery environment from a release earlier than 8.6 to 8.6 or later.
If the target environment does not have any Application servers and the database is already 8.6 or later, perform the following steps
1. Install the Application Server software on the new Application Server(s) using the individual component installers, not the unified product installer.
2. Using the blasadmin utility, configure the Database section parameters noted below. These settings will be for the new database server.blasadmin -a set database ConnectionString <ConnectionString>blasadmin -a set database DriverClass <DriverClass>blasadmin -a set database UserId <user_name>blasadmin -a set database Password <Password>
Soft delete the old Application Servers from the imported database:
update application_server_model set is_deleted = 1;
commit; (Oracle only)
Copy the files listed in the Files section of the Backup and Restore of the BladeLogic Environment document from the old application server to the new application server(s) except the bladelogic.keystore file. Leave the bladelogic.keystore that we created during install on the new application server.
Follow the Changing the location of the file server to update the application server configuration and database with the location of the new file server

Provisioning

To move the provisioning environment, perform the following steps:

If replicating your provisioning environment, install the BladeLogic PXE and TFTP server.
Copy your provisioning data store to the new server.
Replicate any custom application server settings by running <bl_install_dir>/br/blasadmin.
Update the datastore locations to point to the new locations in the new environment.

Step 3: Perform post-copy configuration changes

After your environment has been moved, there are some final steps that you will want to make to ensure that there are no conflicts with your existing environment.

Updating Extended Object Locations

If there are Extended Objects that reference the old file server or other location these must be updated as well using the update_extended_object_script_location included in the external-files.zip for the release.

Updating Remote Depot Object Locations

If there are Depot objects whose payload is stored on a remote NSH path and not the file server, and those payloads are also moved in the new environment, the depot_object_location.remote_path column will need to be updated as well.

Preventing access from the new environment to the existing Application Servers

There are a few ways to accomplish this. One way to handle this is to generate a self-signed certificate for an Application Server in the new environment. Another way is to restrict network access between the two environments; this will prevent accidentally connecting to the original environment.

Removing Job Schedules

If the new environment is being used for testing you will want to disable scheduled jobs so they will not run against the registered servers. This can be done with a simple query:

update schedule set is_deleted = 1;

commit; (Oracle only)

Setting the is_deleted flag is a non-recoverable operation, the schedules will be permanently disabled in the new environment. This is typically desirable for testing environments.

Removing Notifications

If the new environment is being used for testing you will want to disable any job notifications to avoid confusing when testing jobs in the new environment. This can be done with a simple query:

update notification set email_to='test123@example.com';

commit; (Oracle Only)

Removing Routing Rules

If the new environment is being used for testing or the existing repeaters, appservers and SOCKs proxies will no longer be needed, then the routing rules can be removed with a few queries:

Delete SOCKS Proxy assigned in Network routing rule
update routing_policy set is_deleted = 1 where routing_policy_type_id = 1;

Delete Job Server assigned in job routing rule.

update routing_policy set is_deleted = 1 where routing_policy_type_id = 2

Delete Repeater assigned in Repeater routing rule.
update routing_policy set is_deleted =1 where routing_policy_type_id = 3;

commit; (Oracle only)

Preventing access from the new environment to existing managed servers

If the new environment is being used for testing you may want to prevent the new application servers from talking to your existing targets. This is possible by pushing out an exports file from the original appserver/environment to all target servers that allows connections from only the original application servers and any NSH client systems you need to allow connections from.

Decommissioning Servers (optional)

Decommission any servers that are in the development environment that you will not manage. This will prevent any confusion for anyone that might log into the duplicated environment.
Note

If you have an 8.x environment AND have the Licensing Portal configured, make sure that you configure the Licensing Portal to not re-claim any licenses when you decommission servers. If you do, you will need to re-license your servers in the original environment that you copied from.
Ensure that only the servers that are expected to be registered in the duplicated environment remain.

Step 4: Upgrade the test system

Review the pre-requisites for upgrade:

Follow the steps in the upgrade walkthrough topic that is appropriate for your environment:

Step 5: Run some tests in the duplicate environment

After completing the steps in the new environment, perform some representative tests, such as the following:

Start up the Application Server services:
- On Windows, from the Start menu, select Settings > Control Panel. Double-click Administrative Tools, and double-click Services. Right-click BladeLogic Application Server and select Start from the pop-up menu.
- On a UNIX system, from the directory where BMC Server Automation is installed, enter the following:
  /etc/init.d/blappserv start
Log on to the BMC Server Automation Console.
- From the Start menu, choose Programs > BMC Software > BladeLogic Server Automation Suite > Server Automation Console<version>.
- Start Network Shell, and then start BMC Server Automation by running the following command:
  <installDir>/CM/rcp/bsaclient.sh
Test the provisioning process by executing a Provisioning Job and checking the results.
- In the Jobs folder, select the Provision Job, right-click, and select Execute.
- Select a Provision Job, right-click, and select Show Results.
Test the patch process by executing a Patch Job and checking the results.
- In the Jobs folder, select the Patch Job, right-click, and select Execute.
- Select a Patch Job, right-click, and select Show Results.

Step 6: Make required changes in the production environment

If the functional tests reveal issues, you may need to make some changes in the production environment prior to upgrading.

For example, you may discover from your testing that you need to make some database remediations, which you could apply at the beginning of the production upgrade outage.

Step 7: Upgrade the production environment