Troubleshooting package deployment issues

This topic lists the issues that you might encounter while deploying a package. See the guidelines in this topic to obtain the appropriate logging and troubleshooting steps.

Symptoms

The AR System Deployment Management Console displays error after you deploy a package.

Scope

  • One or more users experience the problem. 
  • In a Server Group environment, this could affect one or more servers

Resolution

Perform the following steps to troubleshoot issues while deploying packages:

Step

Task

Description

1

Review the Configuration

  1. Ensure that the packages can communicate with the ARMigrate plugin:
    1. Ensure that the ar.cfg file contains an entry for Server-Plugin-Alias: ARSYS.ARF.ARMIGRATE.
      ar-cfg.png
    2. Go to ARSystemInstallDirectory/BMC Software/ARSystem/pluginsvr/ and ensure that the pluginsvr_config.xml file contains definitions for the ARSYS.ARF.ARMIGRATE plugin and all required paths. All JAR files must be present at the defined locations.
      pluginsvr-config.png

    1. Ensure that the arfiledeployer plugin is defined in the armonitor.cfg (Windows) or armonitor.conf (UNIX) file along with correct JAR file version.

      arfiledeployer.png

    2. Ensure that the AR System Monitor form contains correct hostname entries for each monitor type that is a part of the environment: AR System server, Mid Tier, and Smart IT server. Delete any incorrect or duplicate host entries from this form.
      arMonitor.png
    3. If you have made any changes as per above steps, you must restart the AR Java plugin server. See How to restart the AR Java plugin server.
    4. Compare monitor ID for each monitor type between AR System monitor form and the monitor-<application>-guid.properties file. The file locations are:
      monitor-ARServer-guid.properties: ARSystem_InstallDirectory
      monitor-Midtier-guid.properties: MidTier_InstallDirectory\filedeployer
      monitor-SmartIt-guid.properties: SmartIt_InstallDirectory\filedeployer

      If the Monitor IDs do not match, perform the following steps to generate the file with new GUID and update the AR System monitor form:
      1. Delete the monitor-<application>-guid.properties file.
      2. Delete the entry from the AR System monitor form.
      3. Depending on the monitor type, restart the AR System service, MidTier file deployer service, or the Smart IT file deployer service.
  2. Enable arjavaplugin.log logs in DEBUG mode. The logs are generated in the ARSystemInstallDirectory/ARSystem/dbdirectory.
    See How to enable DEBUG Java plugin server logging.
  3. After the package is imported, view the state and status reason fields. For a successful import, the status reason field displays Success:ARMIGRATE.

2

Collect the logs

Identify and record any related errors in following logs in ARSystemInstallDirectory/ARSystem/db directory.

Enable the following logs before importing the package and reproduce the issue. Copy the arjavaplugin.log logs to another location where you can review them so that they do not get overwritten. Note that a few logs may be large in size depending on use case performed.

3

Disable logs

Disable the arjavaplugin.log by changing the log level to INFO.

4

Create a BMC Support Case

Collect and send logs and detailed information when creating a case with BMC Support:

  1. Provide the following information as part of your case:
    • Name of the package
    • Package status and error message
    • Screenshot of error
  2. Collect following files:
  3. Create a zip file of the collected logs and either attach the zip file to the case (up to 2 GB) or send the file to BMC by using an FTP.

Cause and resolution for error messages

Once you have determined a specific symptom or error message, use the following table to analyze the issue:

Issue

Cause and resolution

The payload deployment does not progress after displaying the Waiting For Utility Run status   



Possible mismatch between the File Deployer property and the value of the AR System monitor local deployment required flag.

Solution:

When you change the filedeployer.localDeploymentRequired property in the deployconfig.properties file from True (manual deployment) to False(automatic deployment), this change is synced with the AR System server. This sync requires 10 minutes. During the sync process, if the AR System server restarts, the sync may fail because the File Deployer cannot connect with the AR System Server.  In this scenario, the payload is deployed neither manually nor automatically.

Perform the following steps:

Note: Avoid the possibilities of AR Server restart while performing these steps:

  1. In the deployconfig.properties file, change the filedeployer.localDeploymentRequired property to True.
  2. Restart the File Deployer by using the
    ./filedeployer.sh start command.
  3. Change the filedeployer.localDeploymentRequired property to False.

AR System Deployment Management Console displays the Init Error.

Package is not uploaded completely after import.

The package is imported but the package contents are not uploaded to the AR System forms.

Solution:

  1. Deploy a package only when the state of the package is Deployable and the Deploy operation is enabled on the console.
  2. If the issue persists, delete the existing package and import the package again.
  3. Check the arjavaplugin.log file for more details.

AR System Deployment Management Console displays the following error:

Init Pending Deploy


Solution:

Open the arjavaplugin.log file and search for errors related to:

  • ARSYS.ARF.ARMIGRATE plugin
  • 'RDA:* forms and fields

AR System Deployment Management Console displays the following error:

Application is not able to connect with Plugin.

Please contact your Administrator. (ARNOTE 10000)


Solution:

  1. Check the AR System Monitor form and ensure that the Java plugin server has server rank 1.
  2. After the issue is resolved, rollback the package and start the deployment process again.
    For more information, see Applying-a-patch.

The arjavaplugin.log file contains the following error for some files:

Access is denied.

No permission to access the BMC_DEPLOYMENT_PACKAGE folder.

If there is no permission to access the BMC_DEPLOYMENT_PACKAGE folder, the deploy operation fails.

During the package deployment process, the package attachments are downloaded from the AR System forms but the directory is inaccessible to copy the package contents to the local system.

The Deployment Management Console does not show any error message related to this. Relevant error message is logged in the arjavaplugin.log file.

Solution: Provide Read/Write permission to the BMC_DEPLOYMENT_PACKAGE folder and its parent folders. The BMC_DEPLOYMENT_PACKAGE folder is available at the following location:

  • (Windows) \AR_Install_Dir\Arserver\Db
  • (UNIX) /opt/bmc/ARSystem/db

AR System Deployment Management Console displays the Unable to communicate with the plugin server error.

Target server or the plugin server is down.

If the target server or the plugin server is down, package deployment fails. If the server and plugin server was running when the operation was initiated, but later crashed, the Deployment Management Console shows Processing Error.

If the plugin server only crashed but the commands had been taken over by the target server, then there is a chance that the package operation will be completed. 

Solution: It is recommended that you roll back the package and start the deployment operation again. See Applying-a-patch.

After you roll back the package, before you start the deployment again, perform the following to clean up the package trace:

  • Delete the package from Deployment Management Console.
  • Delete the existing payload entry from AR System Single Point Deployment Payload form. 
  • Delete the BMC_DEPLOYMENT_PACKAGES folder available at the following folder location:
    (Windows) \AR_Install_Dir\Arserver\Db
    (UNIX) /opt/bmc/ARSystem/db

The AR System Deployment Management Console displays the [ARERR]:10000 error.

Precheck validation failure

A user created package can contain prechecks to validate the package and its contents. For example, a package creator can define a precheck to ensure that a package containing SRM must get installed on a system that has SRM installed on it and a custom error message must be displayed if SRM is not installed on that system. If this condition is met, the package deployment fails displaying the custom error message.

If the precheck is incorrect, you will see the custom error message that is defined during precheck definition.

Solution: 

  1. View the package and select Content Pre-Check to obtain more details about the failure.
  2. For custom packages, the package creator must review and fix the issue.
  3. After the issue is resolved, rollback the previously imported package and import the corrected package.

The arjavaplugin.log file contains the following error message:[main] WARN  com.bmc.arsys.filedeployer.
PayloadProcessor -
Ignoring destination %BMC_REMEDY_ITSM_SUITE_HOME%
\Logs, 
Environment variable BMC_REMEDY_ITSM_SUITE_HOME
is not found

Environment variable BMC_REMEDY_ITSM_SUITE_HOME is not found.

Some of the payload entries got skipped because the ITSM Home environment variable was not found.

Solution: After deployment of ITSM latest deployment package, verify that all expected payload entries are successfully deployed.

  1. Verify that the armonitor.cfg file contains entry for the environment variable BMC_REMEDY_ITSM_SUITE_HOME pointing correctly to the <ITSM_Install_Home> directory.
  2. Restart AR System services.
  3. Deploy the skipped payloads again for successful completion by clicking Retry Deployment on the AR System Single Point Deployment Payload Status Monitor Join form.
  4. Run the arpayloaddeploymentutil.bat (Windows) or arpayloaddeploymentutil.sh (UNIX) utility.
    The arpayloaddeploymentutil.bat (Windows) or arpayloaddeploymentutil.sh (UNIX) utility is available at the following location:
    • (Windows): <installDirectory>\Program Files\BMC Software\ARSystem
    • (UNIX): <installDirectory>/ARSystem/bin

The package deployment is stuck even though the payload deployment utility has completed its run.

The package deployment becomes unresponsive because file deployer service is not running or not responding. Ensure that the file deployer service is running. Perform the following:

(Linux)

  • Check whether the file deployer service is running by using the following command:
ps-ef | grep filedeployer
  • If the file deployer service is not responding, stop the service by using the following command:
kill -9 <processID_FileDeployer>

(Windows)

  • Open Windows Task Manager, select the relevant file deployer process, and end the task.

    filedeployer_end.png

  • Open the armonitor.cfg file and uncomment the following file deployer process entry:

    armonitor_filedeployer_entry.png

ARMonitor starts the file deployer service automatically.


Related topics

Applying-a-patch

Collecting-logs-to-troubleshoot-patch-deployment-issues


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*