Default language.

Troubleshooting upgrade startup, connection, and timeout issues


When you observe startup, connection, and timeout errors while upgrading Remedy AR Server, use this guide for appropriate logging and troubleshooting steps to either resolve the problem or create a BMC Support case.


Related topics

Contacting Support

Symptoms

  • Upgrade gets stuck when restarting AR Server services.
  • Upgrade fails with RIKJ errors.
  • Upgrade fails when you import definition files.
  • Upgrade shows timeouts.
  • Upgrade fails when the Pluginsvrclient utility fails to execute 

Scope

  • One or more users experience the problem. 
  • In a server group environment, this could affect one or more servers.
  • This affects any server in the group that is currently being upgraded.
  • These issues can occur with primary or secondary servers.

Resolution

Step

Task

Description

1

Review the server group configuration

You can find the Server-Connect-Name setting for each member of the server group in the server's ar.cfg/ar.conf file in the <ARSystemInstallDir>/conf directory.

ar.cfg.png

  1. Search the AR System Server Group Operation Ranking form for Server Name field values and perform the following:
    1. Compare all the Server Name field values with the values of the Server-Connect-Name settings of the members of the server group. (Ensure that you search with the correct case as these values are case-sensitive.)
    2. If any values are different, see Symptom 1.
  2. Review the Centralized Configuration settings.
    1. Check if the Server-Connect-Name value is used for all components and there are no incorrect or duplicated entries. (Ensure that you search with the correct case as these values are case-sensitive).
    2. If any values are different, see Symptom 2.
    3. Search the AR System Configuration Component Setting form and verify that the values for the Component Name field are consistent with the Server Name values in the environment.

  3. Run the following queries using a database client:

    • select serverName from servgrp_board
    • select serverName from servgrp_config
    • select serverName from servgrp_resources

    1. Check if the correct Server-Connect-Name value is displayed under the serverName column.
    2. If any values are different, see Symptom 3.

  4. Run the following queries on the control table by using a database client to find values in dbVersion or currDbVersion columns:

    • For Remedy version 9.1.03 and earlier:  select dbVersion from control;
    • For Remedy version 9.1.04 and later:  select currDbVersion from control;
      (% style="color: rgb(51,51,51);" %)
    1. Validate these values with the DBVERSION and API Protocol Numbers section in this article: BMC Remedy Release Version Strings. 
    2. If any values are different, see Symptom 4.

  5. Review the ar.cfg/ar.conf file and verify that the following parameters are set according to the recommendations in the following table. If any values are different, take a backup of this file (add an extension as .bak) and then make the following recommended changes:

Parameter

Recommend Setting

Recommended Changes

Server-Connect-Name

Fully Qualified Domain Name (FQDN) of the specific server.

If the FQDN is incorrect, replace with a Fully Qualified Name that uniquely resolves to this host.

Configuration-Name

Fully Qualified Domain Name (FQDN) of the specific server. This is usually the same as the Server-Connect-Name.

Replace with a Fully Qualified Name that uniquely identifies this host.

Operating-Mode

0 (default value)

If this value is not 0, replace it with 0.

Server-Name

Short server name of the server alias.
If a load balancer is used, this will be the short name of the load balancer. If required, also use Domain-Name so that Server-Name + “.” + Domain-Name resolves to the load balancer.

Replace with the short server name that resolves to the load balancer.

Server-Group-Member

T

Replace with 'T' if the value is 'F'.

Disable-Admin-Ops

F on primary server.

Replace with 'F' on the primary server if the value is 'T'.

6. Edit the ARSystemInstalledConfiguration.xml file in a text editor and search for the following parameters. If any values are different, take a backup of this file and make the following recommended changes. Restart AR Server Services before running the upgrade.

ArsysInstalled.png

Default file location:

(Windows) : \BMC Software\ARSystem

(Unix) : /opt/bmc/ARSystem

Parameter

Recommended Setting

Recommended Changes

BMC_AR_SERVER_HOST_NAME_PRIMARY

On secondary server, this should point to the Admin server.

Replace the hostname with the hostname of the Admin server.

BMC_IS_SECONDARY_SERVER

This value should be set to False for standalone servers or Server Group Administrator (primary server).

For primary server, ensure that the value is set to "False". For secondary servers, ensure that the value is set to "True".

BMC_AR_SERVER_HOST_NAME

Current AR Server-Connect-Name.

Compare with the value in the ar.cfg/ar.conf file.

Replace the value with the Server-Connect-Name from ar.cfg/ar.conf file.

BMC_DATABASE_HOST

Current AR Server Database Host Name.

Compare with the value in the ar.cfg/ar.conf file.

Replace the value with the Db-Server-Host value from ar.cfg/ar.conf file.

DATABASE

Review all parameters having the DATABASE parameter.

Verify the values and replace as necessary.

For Oracle Database:

Parameter

Recommend Setting

BMC_DATABASE_ORACLE_SID_OR_SERVICE

This value should be 'False' when database is using Service Name instead of SID.
(From Remedy version 9.1.0x, we recommend to use Service Name)

BMC_DATABASE_ORACLE_CONNECTION_IDENTIFIER

Service Name

BMC_DATABASE_INSTANCE

Service Name

7. Ensure that no monitoring tools are active during the upgrade. These tools can cause a significant impact on the upgrade time. These tools are sometimes added as Java parameters in the armonitor.cfg/armonitor.conf file.

2

Restart Services

Restart the AR System server after verifying the configuration and making any changes.

3

Perform the Upgrade

Perform the upgrade after making all the recommended modifications.

If the upgrade continues to fail, perform the following steps 4 and 5.

4

Collect the logs

Use the AR System Maintenance Tool to gather the logs and configuration files modified during the upgrade. You can find the instructions for using the tool in the following article:

How to collect configuration files and Remedy logs for BMC Support

5

Create a BMC Support Case

Collect and send logs and detailed information when creating a case with BMC Support:

  1. Provide the following information as part of your case:

    • Operating system and database versions.
    • Are you working with a Server Group?
      How many servers are in the group?
      Is this a Primary or Secondary server?
    • Environment: PROD/QA/TEST/DEV
    • Did you review the results of the Configuration Checker utility before starting the upgrade? Were there any errors or warnings displayed?

2. Collect the following files:


    • Maintenance Tool Zip Log File

3. Zip up the file with logs collected which you can attach to a case (up to 2 GB) or transfer the files to BMC via FTP.
For more information, see How to use BMC's Managed File   Transfer (MFT) to send log files to BMC Support to comply with GDPR policies. 


Cause and resolution for error messages


Symptom

Cause

Solution

Reference

1

Incorrect or duplicate Server Connect Name entries are present in the AR System Server Group Operation Ranking form.

Database was copied from another instance.

Server name changed due to policy or host name change.

  • Remove any incorrect entries from the AR System Server Group Operation Ranking form  and save the form.
  • At the database level, clear the servgrp_board table by running the following query:  

      delete * from servgrp_board; 

  • Restart all the AR Servers of the group and review the form.

2

Incorrect or duplicate Server Connect Name entries are present in the AR System Configuration Generic UI form.

Database was copied from another instance.

Server name changed due to a policy or host name change.

  • Take a backup of the existing configuration and perform a search on the AR System Configuration Component form.
  • Delete all incorrect entries only.

Server restart not required.

3

servgrp_** database tables shows incorrect or duplicate Server Connect Name entries under serverName column.

Database was copied from another instance.

Server name changed due to a policy or host name change.

At the database level, run the following SQL statements:

  • delete * from servgrp_board where serverName = 'add incorrect server name'
  • delete * from servgrp_config where serverName = 'add incorrect server name'
  • delete * from servgrp_resources where serverName = 'add incorrect server name'


4

dbVersion and/or currDbVersion columns shows incorrect API Protocol Numbers.

Database was copied from another instance.

Database was not rolled back post a failed upgrade attempt.

Ask the Database Admin (DBA) to roll back Remedy server database to a point in time before the upgrade.

5

Could not start AR Server in a timely manner.

  1. java.lang.OutOfMemoryError: unable to create new native thread

  2. java.lang.NullPointerException: null at com.bmc.arsys.domain.group.GroupImpl.getFloatingWriteLicense

  3. SQL Server Authentication
  1. When the problem is only with secondary servers, the cause might be that the ARMonitor.properties file is not being set properly.



  1. Increase the RAM memory.

  2. Update the Floating Licenses from null to 0 (for all Groups).
  1. Verify that the SQL-Secure-Connection parameter is set correctly and that the ARSystem services is running as the correct domain account.

  2. Follow these steps:

    • Stop AR Server services before starting the upgrade.
    • Run the upgrade in PAUSE mode.
    • After the second pause, click OK to continue the upgrade but immediately perform the next steps:
    • Modify the ARMonitor.properties file created under <ARInstallDir>/ARSystem/bin. 
      Make sure this parameter is set to false:
      com.bmc.arsys.armonitor.serverpatchmode=false
    • Restart the AR System server
    • The upgrade should continue successfully.

6

When you run the RIKJ command, PING resulted in Return code 90

This error points to a connection issue during the upgrade, usually caused when the AR Server services are not able to start.

Review suggestion from Task 5 to validate the details.


7

Exception from RIKJ command Definition import failed, code 2

The system is not responding to the Admin Operations.

  • Review the  Disable-Admin-Ops  or  Disable-Admin-Ops-Global parameters in the Centralized Configuration form. These parameters should be set to F or the setting should not be present.

  • Review the following values of the BMC_IS_SECONDARY_SERVER and BMC_AR_SERVER_HOST_NAME_PRIMARY parameters in the ARSystemInstalledConfiguration.xml  file.

    • BMC_IS_SECONDARY_SERVER should be 'false'.
    • BMC_AR_SERVER_HOST_NAME_PRIMARY should be the hostname of the primary server.
    • If either of these are wrong, change the value as required and continue with the upgrade.
  • Remove bad server references in the Server Group tables or in the AR System Server Group Operation Ranking form.
    See Symptom 3 in this table.

8

Timeout during database update

The 92 errors are related to a busy server and most likely related to a configuration issue.

Review the following configurations according to BMC recommendations:

9

Remedy AR System server stops working after you upgrade Remedy ITSM Suite.

This failure occurs because of the missing parameters in the arserverd.conf file.

Perform the following steps:

  1. Upgrade Remedy AR System.
  2. Open the arserverd.conf file in the installation folder.
  3. Add the following parameters to the arserverd.conf file:
    1. jvm.option.15=-XX:+UseConcMarkSweepGC
    2. jvm.option.16=-XX:+UseParNewGC
    3. jvm.option.17=-XX:+UseCompressedOops
    4. jvm.option.18=.XX:NewRatio=2
  4. Upgrade the ITSM Suite components.


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*