Troubleshooting UPI Upgrade issues

A Truesight Server Automation upgrade using the Unified Product Installer (UPI) is failing with errors. 

This topic helps you locate and review the appropriate logs to determine the root cause of the problem and either help you identify and resolve the issue or create a BMC Customer Support case.

Issue symptoms

A Truesight Server Automation (TSSA) upgrade using the Unified Product Installer (UPI) is failing with errors. The failures might be occurring in any of the following stages:

  1. UPI initialization
  2. Pre-upgrade environment discovery
  3. Pre-upgrade prerequisites validation
  4. Database upgrade/migration
  5. File Server upgrade
  6. Application Server(s) upgrade
  7. Post-upgrade environment discovery
  8. Post-upgrade content installation

Issue scope

This topic includes the upgrade issues using the Unified Product Installer (UPI) in both Windows and Linux environments.

The fresh install issues or upgrade issues using the Configurator approach are out of scope.

Diagnosing and reporting an issue

TaskActionStepsReference
1

Determine the step when the UPI upgrade was failed.

    1. UPI initialization
    2. Pre-upgrade environment discovery
    3. Pre-upgrade prerequisites validation
    4. Database upgrade/migration
    5. File Server upgrade
    6. Application Server(s) upgrade
    7. Post-upgrade environment discovery
    8. Post-upgrade content installation

Preparing for a Windows upgrade using the UPI

UPI Walkthrough on Windows

Preparing for a Linux upgrade using the UPI

UPI Walkthrough on Linux

2

Identify and locate the appropriate logs.

  • Review the bsainstallupgrade.log log file for all the UPI failures.
    This log file is stored at the following path on the Application Server where the UPI was launched:
    • (Windows) %TEMP%
    • (Linux) /tmp
  • Review the following additional log files depending on the specific step when the UPI upgrade failed. These additional log files contain detailed error messages:
    • Database upgrade failures:
      <installation directory>/br/blmigration.log
    • File Server upgrade failures:
      • (Windows) %TEMP%/MSI*.LOG
      • (Linux) /tmp/BladeLogic_Install
      The file is located on the FileServer.
    • Application Server failures:
      • (Windows) %TEMP%/OM-install.log
      • (Linux) /tmp/BladeLogic_Install
    • BLContent failures:
      <installation directory>/br/blcontent.log
    • Post-migration:
      <installation directory>/br/postmigration.log

    • Deployment migration:
      <installation directory>/br/deployment_upgrade.log
3

Review the logs to identify the issue.

  1. Review the error message displayed in the UPI console along with the error message in the bsainstallupgrade.log file.
    The UPI appends entries to the bsainstallupgrade.log file on each successive run.
  2. Note down the time of failure to review the events from the latest run.
    • If the failure is during the pre-upgrade or post-upgrade steps (1, 2, 6, and 7 specified in task 1), review only the bsainstallupgrade.log file.
    • If the failure is during the database, File Server, or Application Server(s) steps (3, 4, or 5 specified in task 1), review the appropriate log file mentioned in step 2 for detailed error messages.

See the Resolutions for common issues section to review the common errors that can result in the UPI upgrade failure and how they can typically be resolved.

If you are unable to identify and resolve the problem, see Step 4 to create a BMC Support case.


4

Create a BMC Support case

Provide the following information and log files when creating a case with BMC Customer Support:

  • Screenshot of error message or issue from the UPI console
  • Date and time of the most recent UPI failure
  • The Application Server OS vendor and version
  • The Application Server product version (Beginning and target versions)
  • bsainstallupgrade.log
  • Additional log files as described in step 2


Resolutions for common issues

Failing UPI StepSymptomActionReference
UPI Initialization

The RSCD agent bundle is not found at the required location. The rscd directory is missing from the UPI Disk1/files/installers/ directory.

  1. Verify that the RSCDAgent.zip" file is extracted and the "rscd" directory is placed in the "files\installers" directory.
  2. Follow the steps in the referenced Knowledge Article.

KA 000372077 Open link

Pre-upgrade environment discovery

Failed to perform infrastructure discovery. Unable to launch the NSH or BLCLI commands required to discover the Application Servers in the environment.

  1. Verify that all the upgrade prerequisites are met before upgrading the environment.
  2. Follow the steps in the referenced Knowledge Article.

KA 000350706 Open link

Pre-upgrade prerequisite validation 

Database user does not have sufficient privileges for running database migration. The UPI installer did not flag any issue with the Bladelogic DB user rights to perform the upgrade.

Follow the steps in the referenced Knowledge Article.

KA 000309964 Open link

Pre-upgrade prerequisite validation 

The following prerequisite has failed for server (server01.bmc.com):

The RSCD agent is configured in a way that prevents the remote communication with the server. The RSCD agent on the specified server is either down or the ACLs are restricted.

Possible reasons could be:

  • The user is not part of root/Administrators group.
  • The user does not have sufficient permissions for read and write operations.
  • The RSCD agent is not licensed.
  • The RSCD agent is configured to request client side certificates for authentication/authorization.
  • The Network Shell is configured to run in the proxy mode and always communicate with the Network Shell Proxy Server.
  1. Verify that the RSCD agent is up and running and that the RSCD ACLs allow the OS user running the UPI access and provide a mapping to the root or a local administrator.
  2. From the server where the UPI is being run, test access by launching NSH as the same OS user running the UPI and then type "agentinfo serverX" to confirm access and mapping.
  3. Follow the steps in the referenced Knowledge Article.


KA 000276185 Open link


Pre-upgrade prerequisite validation
The File Server reported that the RSCD Agent is not reachable.
  1. Verify that the RSCD agent is up and running and that the RSCD ACLs allow the OS user running the UPI access and provide a mapping to the root or a local administrator.
  2. From the server where the UPI is being run, test access by launching NSH as the same OS user running the UPI and then type "agentinfo serverX" to confirm access and mapping.
  3. Follow the steps in the referenced Knowledge Article.


KA 000276185 Open link

Pre-upgrade prerequisite validation 

The following prerequisites failed for server (server01.bmc.com):

User {0} does not have privileges to perform the upgrade.

  1. Configure the read and write privileges to access the computer using the RSCD agent. 
  2. On a Linux Application Server, run the UPI as a root user.
  3. Follow the steps in the referenced Knowledge Article.

KA 000229630 Open link

Pre-upgrade prerequisite validation 

Unable to retrieve the database version after querying the system_property.

The database is unable to connect using the DB connection information stored in the _template deployment. 

Follow the steps in the referenced Knowledge Article.

KA 000349313 Open link

Pre-upgrade prerequisite validation 

Specified directory has one or more directories which do not have a short name 8dot3.

  1. Ensure that the directory path you specify has a short name (8dot3) generated for all the directories in the path.
  2. Follow the steps in the referenced Knowledge Article.

KA 000135478 Open link

Pre-upgrade prerequisite validation

Network Shell Client is configured in a way that prevents remote communication with other hosts.

This error indicates that the server where the UPI is being run is configured to run either in the Network Shell client mode or to use client side certificates with authentication/authorization.

Follow the steps in the referenced Knowledge Article.

KA 000308776 Open link

Database upgrade

Failed to migrate Truesight Server Automation database.

This error message indicates that the database upgrade/migration has failed.

As described in the "Step 3. Review the logs to identify the error" section, the bsainstallupgrade.log file is likely to contain a high-level error message such as:

THROWABLE EVENT {Description=[Database migration failed.],Detail=[Database migration failed.]},

LOG EVENT {Description=[Validation error],Detail=[Failed to migrate TrueSight Server Automation database.<br><b>Remediation steps:</b><br>1. Please refer blmigration.log file located in the <Installation Directory>/NSH/br directory.<br>2. Please refer to the installation log file bsainstallupgrade.log located in the host computer's temporary directory to understand the cause for the database migration failure.<br>3. If you are not able to determine the problem from the installation log file, please contact BMC support.

CONFIGURATOR_EXECUTION_FAILED]}

  1. Review the blmigration.log file located in the NSH/br directory.
    The detailed database error is usually found directly under this heading:
    DISPLAYING ALL THE ERRORS
  2. Review the bsainstallupgrade.log file located in the host computer's temporary directory to understand the cause for the database migration failure.
  3. If you are not able to determine the problem from the installation log file, please contact BMC support.





I



Database upgrade

The blmigration.log shows the following error:

com.bladelogic.mig.framework.common.MigException: Insufficient privilege. Please run migration setup script in upgrade directory. No system privileges.



In this case, the missing privileges were not detected during the UPI prerequisite validation step, and the UPI proceeded to the database upgrade/migration.

The insufficient privileges were then detected during the SB Migration and logged to blmigration.log.

Do the following:

  1. Make sure the required privileges are assigned directly to the bladelogic DB user and not via a role.
  2. Follow the steps in the referenced Knowledge Article.

KA 000309964 Open link

Database upgrade

The blmigration.log shows the following error:

[18 Jan 2017 21:57:49,602] [main] [INFO] [::] [] [1] SingleStepMigrationResult: TSK NM=mig_chng_contnts_tbl_98005 ; TSK SCR NM= ; TO=8.5.00 ;

TSK MSG=ORA-30036: unable to extend segment by 8 in undo tablespace 'UNDOTBS'
[2] SingleStepMigrationResult: TSK NM=mig_chng_contnts_tbl_98005 ; TSK SCR NM= ; TO=8.5.00 ; TSK MSG=Upgrade halted. mig_chng_contnts_tbl_98005 did not complete step change contents of
table at point 0. Contact BladeLogic Support before resuming.



  1. Identify the problematic tablespace and provide this error message to the site Oracle DBA.

    Some Oracle tablespaces cannot be extended.

    In this example, it is the UNDO tablespace but that could vary. For example, TEMP, DATA, INDEX, and so on.

  2. Once the tablespace is extended, rerun the upgrade and it should continue from the point of failure.

If a different error is encountered after resolving the tablespace issue, contact BMC Support as manual intervention or a DB restore may be required to get the database back to a state where the upgrade can be re-run.


Database upgrade

The blmigration.log shows the following error:

- [1] SingleStepMigrationResult: TSK NM=Alter Schema task ; TSK SCR NM= ; TO=8.9.04.003 ; TSK MSG=Error The transaction log for database 'BBSA' is full due to 'LOG_BACKUP'.
[2] SingleStepMigrationResult: TSK NM=Alter Schema task ; TSK SCR NM= ; TO=8.9.04.003 ; TSK MSG=Upgrade halted. Alter Schema task did not complete step Alter Schema. Contact BladeLogic Support before resuming.

The MS SQL Server transaction log is full.

Note the exact error message and provide this to the site MS SQL Server DBA.

Once the problem has been corrected, rerun the upgrade and it should continue from the point of failure.

If a different error is encountered after resolving the transaction log issue, contact BMC Support as manual intervention or a DB restore may be required to get the database back to a state where the upgrade can be re-run.


Database upgrade

The blmigration.log shows the following error:

SingleStepMigrationResult: TSK NM=alter_schema ; TSK SCR NM= ; TO=8.9.03.001 ; TSK MSG=Error in alter_schema
Error:ORA-12801: error signaled in parallel query server P001, instance dbrac1:BLADECR1 (1)
ORA-12853: insufficient memory for PX buffers: current 348784K, max needed 668304K
ORA-04031: unable to allocate 65560 bytes of shared memory

This error is the Oracle database-side memory error.

Contact the site Oracle DBA to resolve this issue by increasing the memory settings, such as SGA.

The method that is used to resolve the error varies for different Oracle environments and depends on best practices and the Oracle Memory Management method that is in use, such as AMM or ASMM.


Database upgrade

The blmigration.log shows the following error:

com.bladelogic.mig.framework.common.MigException: Error when tried to createTableSpaceORA-06550: line 0, column 0:
PLS-00907: cannot load library unit BLADELOGIC.MIG_CRT_TABLESPACE_TBL (referenced by)

The stored procedure is not complied or failed to compile due to dependencies or shared memory.
This issue is usually an intermittent DB side issue and simply rerunning the UPI may resolve the issue.

Follow the steps in the referenced Knowledge Article.

KA 000157389 Open link

Database upgrade

The blmigration.log shows the following error:

Example: Upgrade to 8.9.04.003

2020-08-18 16:10:36,028 [Thread-400 -- Task: INFO com.bladelogic.mig.framework.common.MigLogger -
EXCEPTION WHILE RUNNING SINGLE STEP MIGRATION FROM 8.9.04.002 TO 8.9.04.003
=======================================================
[1] ERROR: TSK NM=Alter DBM Schema ; TSK SCR NM= ; TO=8.9.04.003 ; TSK MSG=Error occurred: Error message:ORA-01418: specified index does not exist



  • If the following error is encountered during the 8.9.04.003 upgrade, check the referenced KA:
    "ORA-01418: specified index does not exist"
  • If the following error is encountered during any other upgrade step, contact BMC Customer Support and provide the information specified in step 4.
    "ORA-01418: specified index does not exist"



KA 000355276 Open link

Database upgrade

The blmigration.log shows the following error:

[1] SingleStepMigrationResult: TSK NM=alter_schema ; TSK SCR NM= ; TO=8.9.03.001 ; TSK MSG=Error in alter_schema
Error:ORA-00439: feature not enabled: Online Index Build



The online rebuilding of Oracle indexes is a feature for Oracle Enterprise Edition databases. It cannot be used for Oracle Standard Edition databases.

The database upgrade in the 8.9.03 patch 1 release had a defect (DRBLG-116418), which relied on this feature.

Follow the steps in the referenced Knowledge Article.

KA 000282840 Open link

Database upgrade

The blmigration.log shows the following error:

SingleStepMigrationResult: TSK NM=Alter 85 Constraints ;
TSK SCR NM= ; TO=8.6.00 ; TSK MSG=Error in Alter 85 Constraints
Error:ORA-02270: no matching unique or primary key for this column-list




This error message is a data-specific issue.

If this error is encountered in a test environment, which was cloned from the production environment to test the upgrade, it may be due to an invalid database export or import.

See the referenced documentation on how to correctly clone a TrueSight Server Automationenvironment especially the use of "flashback_time" flag when exporting the database.

If the database export is successful or the environment is not a cloned, contact BMC Customer Support with the data as specified in step 4.

Moving or duplicating a TrueSight Server Automation environment
Database upgrade

The blmigration.log shows the following error:

SingleStepMigrationResult: TSK NM=Alter Schema task ; TSK SCR NM= ; TO=8.9.04.003 ;
TSK MSG=Error ALTER TABLE ALTER COLUMN entry_id failed because one or more objects access this column.
[2] SingleStepMigrationResult: TSK NM=Alter Schema task ; TSK SCR NM= ; TO=8.9.04.003 ; TSK MSG=Upgrade halted. Alter Schema task did not complete step Alter Schema. Contact BladeLogic Support before resuming.




This error message is a data-specific issue.

If this error is encountered in a test environment, which was cloned from the production environment to test the upgrade, it may be due to an invalid database export or import.

See the referenced documentation on how to correctly clone a TrueSight Server Automationenvironment especially the use of "flashback_time" flag when exporting the database.

If the database export is successful or the environment is not a cloned, contact BMC Customer Support with the data as specified in step 4.

Moving or duplicating a TrueSight Server Automation environment
Database upgrade

The blmigration.log shows the following error:

com.bladelogic.mig.framework.common.MigLogger - EXCEPTION WHILE RUNNING SINGLE STEP MIGRATION FROM 8.5.00,8.5.01 TO 8.6.00
[1] ERROR: TSK NM=Install Content Schema ; TSK SCR NM= ; TO=8.6.00 ; TSK MSG=Error in Install Content Schema
Error:ORA-01430: column being added already exists in table



This error message is a data-specific issue.

If this error is encountered in a test environment, which was cloned from the production environment to test the upgrade, it may be due to an invalid database export or import.

See the referenced documentation on how to correctly clone a TrueSight Server Automationenvironment especially the use of "flashback_time" flag when exporting the database.

If the database export is successful or the environment is not a cloned, contact BMC Customer Support with the data as specified in step 4.

Moving or duplicating a TrueSight Server Automation environment
Database upgrade

The blmigration.log shows the following error:

[1] SingleStepMigrationResult: TSK NM=alter_schema ; TSK SCR NM= ; TO=8.9.02.000 ; TSK MSG=Error in alter_schema Error:ORA-01408: such column list already indexed

This error message is a data-specific issue.

If this error is encountered in a test environment, which was cloned from the production environment to test the upgrade, it may be due to an invalid database export or import.

See the referenced documentation on how to correctly clone a TrueSight Server Automationenvironment especially the use of "flashback_time" flag when exporting the database.

If the database export is successful or the environment is not a cloned, contact BMC Customer Support with the data as specified in step 4.

Moving or duplicating a TrueSight Server Automation environment
Appserver or File Server or RSCD Agent Upgrade

The RSCD Agent upgrade failed on blfs. Refer to the log file located at /tmp/BladeLogic_Install on the server where the upgrade failed.

Some possible causes and steps for remediation:

Check if Version.new file is present in TrueSight Server Automation installation directory.



  1. Check whether the Version.new file is present in TrueSight Server Automation installation directory.
    If present, rename it as version and run the installation program again. Before re-attempting, make sure the RSCD Agent service is up and running on each server participated in upgrade.
  2. Review the appropriate log on the specified Server:
    • Linux: /tmp/BladeLogic_Install
    • Windows: %TEMP%/MSI*.LOG or %TEMP%/OM-install.log
  3. See the referenced KAs for specific examples of RSCD Agent upgrade issues, which can trigger this error.

KA 000322980 Open link

KA 000239795 Open link


Application Server(s) upgrade

The Application Server upgrade fails with the following error:

Failed to upgrade Application/Pxe Server],Detail=[Upgrade of APPSERVER failed on XXXX. Check the log file %TEMP%\OM-Install.log

Review the appropriate log on the specified Server:

  • (Linux) /tmp/BladeLogic_Install
  • (Windows) %TEMP%/OM-install.log

Post-upgrade 

environment discovery

The following error is displayed:

TrueSight Server Automation Application Server failed: Failed to retrieve upgrade state of the infrastructure components. 

This issue occurs because the Application Server upgrade was not successful and the service is down.

Specified Application Server(s) may not be restarting after upgrade.

Review the appserver.log and/or console.log on the specified Application Servers(s) to determine if they are currently down and, if so, why.

Once Application Server(s) have been restarted, rerun the UPI to resume.


Post-upgrade content installation

Post Upgrade content import was not completed successfully. Please refer to the installation log (bsainstallupgrade.log) located in the temporary directory to rectify the problem.

As of TrueSight Server Automation 20.02, the UPI installs the following list of content items after the upgrade:

  1. Health and Value Dashboard and Import Report Job and activate Dashboard
  2. blcontent, a collection of sample prepackaged content
  3. Zipkits
  4. Agent installers, agent bundles, and Agent Installer Jobs
  5. SCAP compliance
  6. Quick start page
  7. The Application Server installers copied to the File Server for the Add Node functionality

If any of these steps fails, review the bsainstallupgrade.log file for details.

An error during the post-upgrade content step is not fatal to the upgrade. You can skip the error if all the preceding steps were successful.

Usually, a problem during the post-upgrade content step occurs due to conflicting objects already present in the TrueSight Server Automation environment or a problem launching the BLCLI.

After you resolve the problem, you can run the individual content items by referring the specified documentation page.

Manually importing out-of-box content



Was this page helpful? Yes No Submitting... Thank you

Comments