Error: Invalid spaceKey on retrieving a related space config.

Troubleshooting installer failure during upgrade

This topic provides the following information:

Resolving installer failure

After launching the installer, if you receive the following error:

Ensure that you have set the BMC_JAVA_HOME variable to point to the relevant Java path.

Restoring background processes

During upgrade, a group of background processes are disabled until the upgrade completes. See Processes automated during upgrade.

This is handled by the Operating-Mode server setting. During upgrade, the value of Operating-Mode is set to 1 or 2 based on whether the installer is running on the primary or secondary server, respectively. This is applicable to all Remedy installers. However, if the installer crashes or aborts unexpectedly, the background processes may not be restored to their pre-upgrade settings. This requires that the Operating-Mode is set to value 0, the normal mode of operating. 

To restore the background processes to their pre-upgrade values:

  1. Double-click the javadriver.bat (Windows) and javadriver.sh (Unix) file. The file is located at the following location: 
    • (Windows) <InstallDirectory>\ARSystem\Arserver\api\lib
    • (UNIX) <InstallDirectory>/api/lib

      Note

      Ensure that the value of JAVA_HOME is set correctly. If you do not set the JAVA_HOME correctly, you may not be able to run the BMC Remedy Configuration Check utility outside of the installer to perform pre-upgrade and configuration checks.

      Use Java 8 update 45 or later.

  2. To initialize, enter the init command.
  3. To log on, enter the log command and provide details such as user name, password, and server name. 
  4. If you are not using the port mapper, enter the ssp (Set Server Port) command and then enter the server port number.
  5. Enter 0 or a blank for Using private socket
  6. Enter the ver command to verify the login information.
  7. Enter the ssi(Set Server Info) command and perform the following: 
    1. Enter 1 for the Number of server info operations that you want to perform.
    2. Enter 463 as the Operation number to set the server to the operating mode.
    3. Enter 2 for integer as the Datatype
    4. Enter 0 or 1 as the Integer Value.
    Command: ssi
    SET SERVER INFO  
    	Number of server info operations (0): 1   
    		Operation (1-605) (1):463
    Datatype Null/Key/Int/Real/Char/DiaryList/Enum/Time/Bitmask/Byte   
    		Decimal/attach/currency/date/timeofday/join/trim/control/Table/Column/ulong/
    		coords/view/display (0 - 14, 30-34, 40-43) (0): 2
    Integer Value (0): 0
    Set Server Information Status
    ReturnCode: OK
    Status List : 0 items

The ar.cfg is populated with pre-upgrade values.

Frequently asked questions

This section lists some of the scenarios you might come across while performing the zero-downtime upgrade of the platform components:

 Upgrade status of the platform components is pending even after upgrading or one of the servers is not upgraded but the upgrade status is done

If you upgrade the platform components on some of the servers of a server group, the AR System Administration > Server Information form, Platform tab displays the Upgrade status as pending.

From 9.1.04 onward, when you perform upgrade for the platform components, you must upgrade AR, Atrium, and Atrium Integrator on all the servers of a server group. The AR System Administration > Server Information form, Platform tab displays the Upgrade status as Done after you upgrade the platform components on all the servers of a server group. ITSM installer cannot perform upgrade unless the platform components are upgraded on all the primary servers.

After upgrading the platform components on all the servers of a server group, if you still see the upgrade status as pending, check the following:

  • Check the platform component version information on AR System Server Group Operation Ranking from. The following fields on the AR System Server Group Operation Ranking from display 9.1.04 for all the upgraded servers:
    • AR Server Version
    • CMDB Version
    • AI Version
  • If you have any inactive servers in the server group (either a server is down or entries are there in the ranking form for a deprecated server), server waits for 48 hours and it updates the status as done.
  • Check the AR error log if there are any errors related to [Post Upgrade]. If there are any errors, resolve them. 
  • If you find multiple entries in the ft_pending table, the post-upgrade activities take time. Wait till the process is completed. Check the status in the AR error log.

Exampe for inactive servers: If you have 3 servers in your server group, Platform components are upgraded on server 1 and server 2. Server 3 is down for more than 48hrs. After 48hrs, the post upgrade activity is triggered and the upgrade status is displayed as 'Done'. Due to this reason, you will not be able to start the server 3 which is not upgraded. If you want to bring up the server 3, you must upgrade it. During upgrade, if any of the platform components fails, filesystem is rolled back but the server will not come up because of DB version mismatch. See the DB mismatch version error in the AR error logs.

Example for active server but not upgraded: If you have 3 servers in your server group. Platform components are upgraded on server 1 and server 2. Server 3 is up but not yet upgraded. Even after 48hrs, the post upgrade activity is not triggered on server 3 and upgrade status will be 'Pending'.

On the server 3 (last server), if you completed AR installation, ensure that Atrium Core and AI upgrade is completed within 48 hrs. If you do not upgrade Atrium Core or AI within 48hrs, the post upgrade activity is triggered and the upgrade status is marked as Done. Post that, if you upgrade, Atrium Core and AI and if the upgrade fails, only the file system of AR, Atrium Core, and AI is rolled back but the server will not come up because of DB version mismatch. See the DB mismatch version error in the AR error logs.

If you want to change the default 48hrs wait time, add ZDT-Upgrade-Max-Wait-Hour-For-Inactive-Server as a CCS shared parameter and add a higher value (> 48hrs). The maximum value that you can set is 336hrs.


 While upgrading applications on the primary server, should the secondary servers be down?

Zero-downtime upgrade is not supported for the applications.

Before upgrading the application components on the primary server, ensure that the AR server is down on all the secondary servers. If you do not shut down the secondary servers, you might come across with some issues. For example, delay in the upgrade time and deadlocks.

 Can I perform zero-downtime upgrade if I have only AR and no CMDB and AI?

Yes, you can perform the zero-downtime upgrade for AR even if you do not have CMDB and AI. After upgrading the AR on all the servers of a server group, AR System Administration > Server Information form, Platform tab displays the Upgrade status as Done.

For a single server environment also, you can perform the zero-downtime upgrade. Few of the operations may not be available during upgrade.

 Is the backup folder deleted automatically?

After upgrading the platform components successfully, the installer deletes the backup folders. If the backup folders are not deleted automatically, you have to manually run the cleanup utility. For information about backup folder, see https://communities.bmc.com/docs/DOC-108153

 What happens if I cancel the zero-downtime upgrade while it is in progress? Will the platform components be rolled back to the earlier version?

BMC recommends not to cancel the installation after you click Install. However, if you cancel the installation, platform components are not rolled back to the earlier version automatically. You must manually run the Rollback utility to roll back the platform components and the file system to the earlier version. To rollback a component, go to the component installation directory and run the following command:

rollback.bat "AR Server Name" "AR Admin User Name" "AR Admin password" "ComponentName"

Example (to rollback a particular component): If you cancel the installer while performing the zero-downtime upgrade for Atrium Core, you must manually run the rollback utility from the Atrium Core installation folder to roll back CMDB to the older version. You have to pass the AtriumCore parameter through the rollback utility in the following manner:

rollback.bat "<AR Server Name>" "<AR Admin User Name>" "<AR Admin password>" "AtriumCore"

[Example to rollback only AtriumCore] rollback.bat "server1" "Demo" "unknown" "AtriumCore"

After reverting Atrium Core, run the installer again to upgrade Atrium Core. Check the Upgrade Summary screen for the status of the upgrade. 

Similarly you can roll back the following components:

  • Atrium Integrator: Run the rollback command from the Atrium Integrator installation directory and specify AtriumIntegrator as the ComponentName.
  • AR System server: Run the rollback command from the AR System server installation directory. Do not specify any ComponentName.

For more details, see Rollback mechanism

 After upgrading the platform and application components on the primary server, if I start upgrading the secondary servers, will the upgrade fail?

Yes, upgrade fails. BMC recommends that you upgrade the platform components (AR, CMDB, and AI) on all the servers of a server group, and then start upgrading the application components.

 What if I upgrade only AR in a server group? What is the impact? Is AR only upgrade supported?

AR only upgrade is not supported. You must upgrade CMDB and AI also as part of platform upgrade.

 How do I access zero-downtime installer?

There is no separate zero-downtime installer. It is the regular installer. By default, the installer runs in zero-downtime mode. For more information, see Preparing for zero downtime upgrade of the platform

 What is a primary server? Is it the server ranked #1 for administration operation?

Yes. Normally it is the first server that was installed in the server group, but correct definition is any server with rank=1 for Administration operation is the primary server. That is the first server to be upgraded in a server group. 

 What is mixed-server-version mode?

In a server group environment, if the platform components are upgraded to the latest version (for example, 9.1.04) and couple of servers are yet to be upgraded (still on old version), it is called mixed-server-mode.

 Will SmartIT/DWP work in mixed-server-version AR environment?

Yes. Any client such as SmartIT/DWP that uses AR API to connect to the AR server continues to work in case of mixed-server-versions mode. Since all APIs and functionality of the server is backward compatible, receiving a call from a client such as SmartIT using the same API to any version of server is served with the same expected response.

 Most of the mid-tier users do not run their mid-tiers in a true clustered environment with session replication. Will the users come across with 9201 session timeout or invalid errors?

If users do not use a true cluster, there will not be seamless failover. So if that mid tier is taken out of load balancer, users will see session timeout. If they need to use zero-downtime, they need to have true cluster with failover configured.

 When upgrading mid tier, will it retain other component agents such as RSSO, additional web application filters, and customized properties files?

Only the backup and restore functionality is changed in Mid Tier installer 9.1.04. The Mid Tier installer continues to work in the same manner as it was in the versions prior to 9.1.04. 

 Is there a possibility to continue failed installation without re running from 'START'. For example, AR and CMDB are upgraded successfully. However AI upgrade failed. Is there a possibility to continue with the AI re install rather than going back to AR and CMDB?

No. There are dependencies in AR, CMDB and AI - so AI will rollback AI+CMDB+AR and it has to start again from AR. 

If AI installation is cancelled, run the rollback utility manually to rollback only AI and reinstall AI.

 Does the installer take folder backup if Installation fails in second attempt?

Yes. Installer always takes backup of latest file system everytime you start upgrade.

 Though rollback is available for zero-downtime from 9.x to 9.1.04, are VM/DB Backups still recommended as a fallback?

No backups of VM/DB. There is live data coming into the database, so it cannot be restored. If upgrade fails, fix the issues that casused failure and then start with the upgrade. Restoring database is not recommended.

 Is it suggested to keep Mid Tier, AR, CMDB, and AI on seperate machines and upgrade them? If upgrade fails, only the failed component is rolled and not all the platform components?

In a production environment, Mid Tier is usually installed on a different machine than AR. AR, CMDB and AI are rolled back together as they have to be on the same version at any time. If AR is upgraded successfully and CMDB upgrade fails, users may decide to stop the upgrade and resume it after a week. In that case, all the platform components (AR, CMDB, and AI) have to be on the same version so that users can continue to use the older version.

 Can non-primary ARServers be upgraded in parallel after upgrading the primary AR System Server?

It can be done technically. However, for zero-downtime upgrade, it is not recommended. As users are still accessing the system and only one primary server may not be able to take up the user load.

 When I perform zero-downtime upgrade through scripts or job, I see that AR and CMDB backup folders are not removed though the upgrade is successful. What should I do?

After performing the zero-downtime upgrade through a script or through a job ( for example, Jenkis job) successfully for the platform components, you may notice the following scenarios:

  • The AI installer cleans up the AI backup folder but is unable to clean up the CMDB and AR backup folders.
  • If Atrium Core (CMDB) upgrade fails, the CMDB installer cleans up the Atrium Core backup folder but is unable to clean up the AR backup folder.
  • If AR upgrade fails, AR installer cleans up the AR backup folder.

Workaround: Before starting the zero-downtime upgrade through a script or a job on a Windows or UNIX environment, ensure that the environment variable is correctly set for all the 3 platform components - AR, CMDB, and AI.

 BMC Atrium Core Web Services are not resuming on external Tomcat and rollback is failing during zero-downtime upgrade

If the BMC Atrium Core Web Services are installed on external Tomcat 7.0.58 or lower, rollback mechanism does not work. BMC recommends that you upgrade your external tomcat to 7.0.59 or above.

 Upgrade process is stuck. Log event details mention the unavailability of required free space.

The upgrade process gets stuck if the back up directory path contains a soft link. Instead of providing the soft link for backup path, point to the directory where you would like to create a backup folder. Make sure that the directory intended for the backup contains the required space.


 How do I check the server group status?

If you are upgrading the Remedy platform components in a server group, the installer triggers the post installation tasks after successfully upgrading all the secondary servers. However, no message is displayed indicating all the secondary servers are upgraded or the post install tasks are triggered. 

  • To view the upgrade status of a server group, go to AR System Administration > Server Information form, Platform tab. The Server Group Upgrade Status field displays the status Done if all the servers of a server group are upgraded successfully.
  • To verify if the post upgrade activities are triggered, check the arerror.log located at <InstallDirectory>\BMC Software\ARSystem\Arserver\Db.

 After automatic rollback of AR, the Application License list in the User form displays an error.

After automatic rollback of AR, when you click the Applicaction License list on the User form, the following error is displayed:

The specified menu is invalid. (ARERR 9372)

As a workaround, manually enter the license details in the Application License field.


Rollback mechanism 

When zero-downtime upgrade fails, platform components and file system are rolled back to the older version either automatically or manually. 

  • Automatic rollback: The installer (AR, Atrium Core, and AI) triggers rollback whenever upgrade fails.
  • Manual rollback: Every installer provides rollback utility. You have to run it manully if the automatic rollback fails. 

The database remains upgraded even if the upgrade fails. As database is upgraded, some of the forms may display additional fields introduced in 9.1.04 but do not function.  

The graphic below shows the platform components that are rolled back at different stages of zero-downtime upgrade:


Recommendation

While performing zero-downtime upgrade, BMC recommends that you do not cancel the installer after clicking Install. If you cancel the installer, you have to manually run the rollback utility and use the scripts to rollback the respective platform component to the older version. For example, after performing zero-downtime upgrade successfully for AR and CMDB, if you cancel the AI installer, you have to run the rollback utility and use the script to roll back AI to the older version.

Note: Rollback utility works if the zdt backup folder is available. If the backup folder is deleted, manual rollback does not work.

Do not delete the backup folder. The installer deletes the backup folder after successfully upgrading the platform components on an upgraded server. For example, if you have AR, Atrium Core, and AI system, only after completing the AI upgrade, the AI installer deletes the backup folder of AR, Atrium Core, and AI. If the automatic cleanup of backup fails, you must run the cleanup utility from the installation folder. 

Running the rollback utility manually

You must run the rollback utility manually only when one of the following conditions arises:

  • Zero-downtime upgrade of Remedy platform failed and automatic rollback failed. For example, automatic rollback did not copy some of the files or folders.
  • Zero-downtime upgrade of Remedy platform is cancelled explicitly or abruptly.

When automatic rollback fails on a server in a server group, run the rollback utility on the server where automatic rollback failed.

Note

BMC suggests that you do not run the rollback utility when all servers of a server group are successfully upgraded as couple of tasks that get executed after the upgrade are not reversible through rollback.

To run the rollback utility

The table below lists the actions to be performed if automatic rollback fails:

ScenarioActionAdditional information
  • AR upgrade and automatic rollback of AR failed
Run the rollback.sh (UNIX) or rollback.bat (Windows) utility 
located at
<AR installation directory >\AR\installcompletionutility\
rollback.sh
or rollback.bat.
  • Depending on the platform component, you must run the rollback utility from respective installer folder only. For example, if the AI upgrade fails and you run the rollback utility from the Atrium Core folder, AI is not rolled back to the older version.

  • If you have to run the rollback utility for CMDB and AI manually, ensure that AR System Server is up and running. If you run the rollback utility manually when the AR System Server is down, CMDB and AI are not rolled back to the older version.
    If the AR System Server is down, start it manually before running the rollback utility manually for CMDB and AI.
  • AR upgrade is successful
  • Atrium Core (CMDB) upgrade and automatic rollback failed
  • Run the rollback.sh (UNIX) or rollback.bat (Windows) utility located at <Atrium Core installation
    directory>\
    AtriumCore\
    installcompletionutility\
    rollback.sh
    or rollback.bat
  • When you run the rollback utility placed the Atrium Core folder, both Atrium Core
    and AR are rolled back to their older version.
  • If you want to roll back only Atrium Core and not AR, you can pass the Atrium Core parameter while running the rollback utility.



  • AR and Atrium Core (CMDB) upgrade is successful
  • Atrium Integrator upgrade and automatic rollback failed
Run the rollback.sh (UNIX) or rollback.bat (Windows) utility located at <AI installation directory>\AI\installcompletionutility\ rollback.sh or rollback.bat.
Cancel the installer for a platform component

Run the rollback utlity from the respective installation directory and pass on the required parameter.

For example, while performing the zero-downtime upgrade for Atrium Core, if you cancel the installer, you must run the rollback utility from the Atrium Core installation folder manually to roll back CMDB to the older version. You have to pass on the AtriumCore parameter through the rollback utility in the following manner:

<Atrium Core 
installation 
folder>\
AtriumCore\
installcompletionutility\
rollback.bat

[Usage] rollback.bat 
"<AR Server Name>" 
"<AR Admin User
Name>" "<AR Admin password>" 
[AtriumCore]

[Example to rollback only 
AtriumCore] rollback.bat
"server1" "Demo" 
"unknown" "AtriumCore"

For more information, see What happens if I cancel the zero-downtime upgrade while it is in progress? Will the platform components be rolled back to the earlier version? in the FAQ section.


For more information regarding the Rollback utility, see   https://communities.bmc.com/thread/176116

Was this page helpful? Yes No Submitting... Thank you

Comments

  1. Kelly Logan

    In the section - "What happens if I cancel the zero-downtime upgrade while it is in progress? Will the platform components be rolled back to the earlier version?" - the example is difficult to follow. Here is the command line given: [Usage] rollback.bat "" "" "" " This seems to be missing at the least a final double-quote after the tag, and it is not clear what value should be set to. Maybe give an actual example with values filled in to demonstrate.

    Jul 02, 2019 09:40
    1. Manash Baruah

      Hello Kelly,

      Thank you for the feedback. We have updated the example.


      The format is:

      rollback.bat "<AR Server Name>" "<AR Admin User Name>" "<AR Admin password>" "ComponentName"

      Example (to roll back Atrium Core):

      rollback.bat "server1" "Demo" "unknown" "AtriumCore"


      Hope that helps!


      Thanks,

      Manash 

      Jul 02, 2019 11:27
      1. Kelly Logan

        That does help, thank you Manash! It would also be helpful to have a list of valid component names to use with rollback (or point to where they are).

        Jul 03, 2019 09:15
        1. Manash Baruah

          Sure, Kelly. I am checking with the SMEs. I'll update as soon as I get a confirmation.

          Jul 05, 2019 02:23
          1. Manash Baruah

            Hi Kelly, we have updated the topic. Please see What happens if I cancel the zero-downtime upgrade while it is in progress? Will the platform components be rolled back to the earlier version? in the FAQ section.


            Thanks,

            Manash

            Sep 23, 2019 12:01
  2. Kelly Logan

    Here's a bigger issue - on version 19.02 installs it appears that the directory listed does not exist. Here are the directions: Run the rollback.sh (UNIX) or rollback.bat (Windows) utility located at \AR\installcompletionutility\rollback.sh or rollback.bat. In the Linux install, there is no \AR\ directory.

    What does exist in a Linux AR install is /UninstallBMCARSystem/uninstall.bin This file seems to be the executable called by the shell script alluded to. (Maybe if the install had completed the other directory would have been created with the shell script to call this file?)

    Sep 25, 2019 08:39