Troubleshooting Smart IT D2P package deployment

When you try to access the Smart IT application from the browser, you might see some errors or face issues with logs. Use this information to understand and resolve the problem or to create a BMC Support case.

Symptoms

  • You cannot see the package after the .zip file import is completed in the Deployment Console.
  • The package import process fails with the following error message: 
    Init Error
  • The package fails to upload the .zip file, and the status gets stuck in the following stage:
    Importing
  • The package import process fails with the following error message:
    Error in plugin :ARSYS.ARF.ARMIGRATE
  • An error message is displayed when you run the arpayloaddeploymentutil.bat file.
  • The status of the deployment payload entries are stuck in the following stage:
    Waiting For Utility Run

Scope

  • One or more users experience the same problem.
  • In a server group environment, this can affect one or more servers.


Resolution

Perform the following tasks to troubleshoot the D2P package deployment issues:

Task

Action

Steps

1

Review the configuration check.

D2P interacts with the ARMigrate plug-in when you initiate a build, validate, deploy, or rollback process. For more information, see AR Migrate plug-in configuration.

  1. Confirm that one Server-Plugin-Alias entry for ARSYS.ARF.ARMIGRATE plug-in is defined in the ar.cfg file.
    1 ar-cfg.png
  1. Confirm that one entry of the ARSYS.ARF.ARMIGRATE plug-in along with all the required path elements are defined in the arpluginsvr_config.xml file. Also confirm that all required .jar binary files are present at the defined location.
    2 pluginsvr-config.png

  1. Confirm that one entry with the correct hostname for each monitor type (for example, Smart IT servers) which is a part of the environment is defined in the AR System Monitor form. Delete the incorrect or duplicate host entries from this form.
    Important:  Restart the AR Java Plug-in server after you make changes to the preceding files. For more information, see How to restart the Java Plugin Server.
    3 arMonitor.png

  2. Compare the Monitor ID for each Monitor Type from the AR System Monitor form with the monitor-SmartIt-guid.properties file.
    1. For Smart ITMonitor Type, the monitor-SmartIt-guid.properties file is located under ..\Smart IT\filedeployer directory.
      If the Monitor ID does not match, perform the following steps. This generates the file with a new GUID and updates the record in the AR System Monitor form:
      1. Delete the monitor-<application>-guid.properties file.
      2. Delete the entry from the AR System Monitor form.
      3. Restart the BMC Remedy Smart IT File Deployer Service application server.
  3. Before importing the package, check the AR plug-in server heap size defined in the armonitor.cfg file. Set the heap size to minimum 3 GB (For example, 3072 m). This helps to avoid the performance issue or the heap space issues when you import the package.
    Example: "%BMC_JAVA_HOME%\java" -Xmx3072m -classpath "C:\Program Files\BMC Software\ARSystem\pluginsvr;
  4. Check if the Tomcat server used by Mid-Tier in the server group has the sufficient JAVA heap memory size to process the D2P package.
  1. Open the SHARE:Application_Properties form, and make sure that the version for Property Value is set as Remedy Smart IT.

  1. Review the following checkpoints:


    1. Review the best practices recommendations by the Performance Team. For more information, see REMEDY 9.x/18xx/19xx/20xx Release - CONFIGURATION CHECKLIST from the BMC R&D PERFORMANCE TEAM.

    2. Compare the downloaded package zip file size with the original file on the server.
    3. Enable the following logs. These logs are generated in the ..ARSystem/ARServer/db directory:
      1. ard2pplugin.log: This log file is generated by default from version 18.08 and later.

      2. (Only for versions 18.02 and 18.05) arjavaplugin.log in the DEBUG Log level. For more information, see Remedy AR Server - How to enable DEBUG Java Plugin Server Logging in AR System in the arjavaplugin.log file in 9.x.

  2. Delete the existing failed package.
  3. Delete *** <ss> from Deployment form.
  4. Reimport the D2P package.
  5. Once the package is imported, always check View, and review the details in the State and Status Reason fields. You should see Success:ARMIGRATE under the Status Reason field which confirms that the package is imported successfully.
  6. If the package fails to import, check task 2.

2

Collect the logs

  1. Identify and record the related errors in the logs specified in step 3.
  2. Enable these logs before importing the package, and reproduce the issue. 
  3. Copy these logs to another location where you can review them so that they do not get overwritten. A few logs may be large in size depending on the use case performed.
    1. Smart IT logs location : ..\Smart_IT\Smart_IT\filedeployer\Logs
      1. arfiledeployer.log.
      2. armonitor.log.
    2. Remedy AR System server logs : ..ARSystem/ARServer/db directory.
      1. ard2pplugin.log.
      2. (Only for the versions 18.02 and 18.05) arjavaplugin.log in DEBUG Log level.


Resolutions for common issues

Issue

Cause

Solution

Package import, or deployment failed without any specific reason on the AR System Deployment Management console, but the log might contain the OutOfMemory error.

Insufficient physical memory on the system where the plug-in server is running.

During the import process, the deployment plug-in extracts the information from the deployment package. However, due to insufficient physical memory, the package extraction fails. That is why, the package is not imported.

Increase the physical memory on your system.

Check the arjavaplugin.log file for the information about the operation that resulted in import failure. For example, if the package size is 300 MB, and if there is no free space to create the ZIP file structure after extraction, the package is not imported.

The following error is logged:

ARMigratePlugin Initializing the thread pool completed

Insufficient plug-in heap size

Typically, the size of the packages with the payload content is 250 to 300 MB. It needs a higher heap size for the package import to succeed.

The AR System Deployment Management console does not show any error message related to this problem. Check the arjavaplugin.log file for information about the cause of failure. If you see this error message in the log file, it means that the ARMigrate plug-in has been restarted. Always check for the timestamp for this message when you do the debugging.

If the heap size is less than 1024 MB, increase the heap size on all the servers in the server group. If you have multiple plug-in servers, update the heap size on the plug-in server where the ARMigrate plug-in is deployed.

To increase the heap size, open the armonitor.conf file and add -Xmx1024m value to the Java parameter, and then restart the plug-in server.

For example:

"%BMC_JAVA_HOME%\java" -Xmx1024m 
"<AR_Install_Dir>\ARSystem\pluginsvr;
<AR_Install_Dir>\ARSystem\pluginsvr
\arpluginsvr<version>.jar;
<AR_Install_Dir>\ARSystem\arserver
\api\lib\arcmnapp<version>.jar" 
com.bmc.arsys.pluginsvr.ARPluginServerMain
-x <serverName> -i 
"<AR_Install_Dir>\ARSystem" -m

If you face deployment error, as a best practice, make sure that the heap size is three times more than the package size.

An error message might appear when the following files are not found in the deployment package:

    • _items.zip
    • _meta.arx
    • _package_guid.txt

Error message:

ARMigrate:java.io.FileNotFoundException: C:\BMCData\ARSystem\ARServer\Db
\BMC_DEPLOYMENT_PACKAGES\
IDGI08B9ID0GZAP4DWE4P3GZ5RLSHG\
_validateItems.md5 
(The system cannot find the file specified)


Package is empty.

Sometimes, due to network issues, the package is not downloaded completely. When you try to import a package that was not fully downloaded, the extraction of the ZIP file fails. If the ZIP file is not extracted correctly, the md5 checksum file and signing information is also not read, and exceptions are logged.

The Deployment Management console does not show any error message related to this.

  1. Ensure that the package is downloaded completely before you start importing it. You can compare the file sizes of the package before and after downloading.
  2. Check arjavaplugin.log for more details.
  3. Check the Java plug-in server heap size in the armonitor.cfg (Windows) or armonitor.conf (UNIX) file.
    For more information, see Review the configuration task in (step 1) in Resolution in this topic.

The AR System Deployment Management console displays the following message:

Unable to import the package

Remedy Mid Tier is unable to upload the package.

This issue occurs when the deployment package size is large (for example, more than 300 MB), and Remedy Mid Tiercannot upload the package and is unable to communicate with the plug-in server. Therefore, the import operation is never completed. 

  1. Ensure that the Tomcat heap size is three times bigger than the size of the deployment package.
  2. Use the command-line interface only for importing the package. Use the AR System Deployment Management console for deploying the imported package.

For more information about importing a package from the command line, see Using a command-line interface to manage a package  (AR System documentation).

The AR System Deployment Management console displays the following error message:

Error in plugin :ARSYS.ARF.ARMIGRATE (ARERR 8753)

When you import the D2P package, ARSYS.ARF.ARMIGRATE plug-in is called. If the plug-in entry is missing in the configuration files/CCS due to this issue, the import operation displays an error.

  1. Ensure that the ar.cfg (Windows) or ar.conf (UNIX) contains an entry for the ARSYS.ARF.ARMIGRATE  plug-in.
  2. Check the arjavaplugin.log file for more details.

For more information, see the KA #DOC-109135 on BMC communities.

The deployment package is imported until 95% and then the package becomes unavailable. The Mid-Tier log contains the following message:

ERROR in preload, aborting preload for server <server-name>
 ARERR [91] RPC call failed : <server-name> Java heap space

Remedy Mid Tier is unable to preload the package.

This issue occurs due to the insufficient heap size for the Remedy Mid Tier server.

  1. Navigate to the Tomcat installation directory and run the tomcat8w.exe file.
  2. Increase the INITIAL & MAX Memory Pool size to 4 GB.
  3. Restart the Tomcat server.

The status of the hotfix does not change from the Deployment Pending status. Despite of running the ardeployment utility multiple times, the status of the payload still displays Waiting for Deploy message or the payload status displays the Waiting For Utility Run message.


Filedeployer fails to start or stop the service of Remedy with Smart IT, which cannot be confirmed from the armonitor.log or arfiledeployer.log files.

The deployment package stays in the Pending Deploy status if the generated log file is large.







  1. Verify if the Monitor ID is matching with the monitor-SmartIT-guid.properties file on the specific server.
  2. Review the armonitor.log file and confirm if the logging has stopped logging after Sent PAYLOAD_DEPLOYMENT_UTILITY signal to file deployer from deployment utility status.
  3. Confirm if the arfiledeployer plug-in process is running on the specific Remedy with Smart IT server.
  4. Review arfiledeployer.log for more details.


Payload Status change to Rollback Failed during the rollback process.


  1. To understand the cause, review the arfiledeployer.log file and search for the following log messages:
    "Update the status of the payload as DEPLOYMENT FAILED"

    "Update the status of the payload as PENDING ROLLBACK"

  2. Find the 'PayloadProcessor' which failed before the status update message DEPLOYMENT FAILED ("BMC:SmartITWebServer") and look for the following error message in the logs that are present on the Remedy with Smart IT D2P server.
    Error Snippet :

    Arfiledeployer.log
    2020-10-07 20:07:41,749 [main] ERROR com.bmc.arsys.filedeployer.PayloadProcessor  - Error while deploying payload with DEPLOYMENT ID IDGAA5V0FH896AP50RGAP43TC9ZEPZ : Errors while stopping processes. [Failed to stop processes {BMC:SmartITWebServer=[BMC:SmartITWebServer]}]


    Armonitor.log (Smart IT) :
    <MNTR> <TNAME: RMI TCP Connection(6)-<IP-Adrress>> <ERROR> <ProcessController                  > <   ProcessController.java:1219      > /* Fri Jul 24 2020 10:54:51.0227  */ Unable to execute command Process[BMC:SmartITWebServer] Command[net stop "SmartITTomcat8"]
  3. Confirm Remedy with Smart IT is running on a specific server.
  1. Check the known issue with the installed JAVA version which is used by Remedy with Smart IT Tomcat server in the KA #000371426.

  2. Check the scenarios that are seen in the secured environment where users have lots of rule set on the file download or permission restrictions. In this case, run the following the command to confirm if Remedy with Smart IT can start and stop by using command prompt.
    You can find the "Service-name" value from the armonitor.cfg under ../Smart_IT\Smart_IT\filedeployer\conf folder.
    Command : net stop/start "<service-name>"

    Example:

    net stop "SmartITTomcat8"
    net start "SmartITTomcat8"
  3. Perform the following steps to check if Remedy with Smart IT service is being able to stop and start is by using the  ARMonitor_Admin.bat utility which D2P uses to start/stop the service:
    1. Open the command prompt and navigate to Remedy with Smart IT Installation Directory (...\Smart_IT\Smart_IT\filedeployer)
    2. Run the ARMonitor_Admin.bat utility. It asks for the arguments. Enter the following arguments:
      1. startprocess BMC:SmartITWebServer
      2. stopprocess BMC:SmartITWebServer
  4. If Remedy with Smart IT service does not start or stop by using the preceding methods then this means that this is an environmental issue. You need to connect with your network and system administrators to get more details on this issue.


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*