Troubleshooting BMC Digital Workplace Catalog
The topic provides information about how to troubleshoot the most common issues related to BMC Digital Workplace Catalog.
Issue scope
This topic covers the most common issues with the following items:
- Installation and upgrade
- Startup
- Stopping and starting BMC Digital Workplace Catalog
- RSSO logs and diagnostics
- Web server (Jetty)
- Connector logging
- Process logging
- Catalog bundle log
- logback_server.xml debug
- Approvals and Remedy IT Service Management (Remedy ITSM) integration
- BMC Digital Workplace Catalog debug
- Remedy ITSM Integration
- Connector refresh
- Catalog file system
- OSGI console
- BMC Digital Workplace controller scripts
- Browser tools
Diagnosing and reporting an issue
Instructions: After you identify the symptoms and scope of the issue, use this troubleshooting guide to diagnose and resolve the issue or to create a BMC Support case. To see the contents of BMC Digital Workplace Catalog files, see https://docs.bmc.com/docs/digitalworkplaceadvanced/2002/contents-of-the-logs-908224224.html
Issues | Steps | Reference |
---|---|---|
Installation and upgrade | The installation script informs you if there are any problems. Full details are available in the arsystem_install log, which is located in the /tmp directory. To obtain a live feed on installation progress, run the tail command on the log: [root@server tmp]# tail -1000 /tmp/arsystem_install_log.txt The following extract is an example of a successful installation. (Dec 15 2016 03:54:59.677 PM +0530),INFO,com.bmc.install.utility.logging.Log, -------------------------------------------------------- BMC Remedy Action Request System 9.5.00 install succeeded. -------------------------------------------------------- (Dec 15 2016 03:54:59.697 PM +0530),INFO,com.bmc.install.product.base.installer.InstallCompletionInstallationTask, PROGRESS EVENT {Description=[installCompletion.installCompletion.description],Progress=[100],Detail=[installCompletion.installCompletion.complete]} (Dec 15 2016 03:54:59.917 PM +0530),CONFIG,com.bmc.install.product.base.project.runner.ProjectRunner, LOG EVENT {Description=[COMPLETED InstallationTask],Detail=[com.bmc.install.product.base.installer.InstallCompletionInstallationTask]} | If the installer fails and the arsystem log doesn't identify the root cause, run the uninstall script (make sure you also remove any BMC Digital Workplace Catalog database references, if present) and run the script again in verbose mode as follows: bash -x ./install-myit-sb.sh This command generates a detailed log to the terminal. In the event of a failure, the output will stop allowing you to scroll up and identify what was happening at the time. Back up the BMC Digital Workplace Catalogdatabases and file system before upgrading. If you encounter a problem, you can roll back and send the logs to BMC Technical Support. If you receive error or warning messages during installation, see Troubleshooting-installation-and-upgrade-issues-in-BMC-Digital-Workplace. |
Startup | Navigate to the $AR_SYSTEM_HOME/sb/rxscripts/bin directory and uncomment the following line from the setenv.sh script: #export rx_trace="--trace-ascii ./trace.log" unset rx_trace | Stop and start BMC Digital Workplace Catalog again and check the trace log for errors. For more information about starting and stopping the BMC Digital Workplace Catalog server, see Starting-and-stopping-the-BMC-Digital-Workplace-Catalog-server. |
Stopping and starting BMC Digital Workplace Catalog | Use the dwpcontroller script from the $AR_SYSTEM_HOME directory to restart BMC Digital Workplace Catalog. (This script also restarts the platform.) [root@server digitalworkplace]#./dwpcontroller [-u $user_name] [-p $password] <stop|start|restart> | The start option calls /sb/start.sh, which does the following:
|
RSSO logs and diagnostics | RSSO debug can be written to the bundle log by adding the following lines under the existing bundle section. <logger name="com.bmc.rsso" level="DEBUG" additivity="false"> <appender-ref ref="BUNDLE" /> </logger> To enable RSSO logging:
Starting with 19.02, a diagnostic tool checks whether the RSSO configuration is correct. (This tool is located in the $AR_SYSTEM_HOME/sb/configure_rsso directory.) To run open the run_diagnostic.sh script and make sure the paths and diagnostic file name are correct, run the following command: #!/bin/bash # Path to Catalog directory. #catalog_home=/opt/bmc/digitalworkplace catalog_home=/opt/bmc/digitalworkplace DIAGNOSTIC_JAR=rsso-diagnostic-1.0-SNAPSHOT.jar if [ $# -eq 0 ]; then params=("${catalog_home}") else params="$@" fi if [ "X${JAVA_HOME}" = "X" ]; then JAVA_EXEC="java" else JAVA_EXEC="${JAVA_HOME}/bin/java" fi ${JAVA_EXEC} -jar ${DIAGNOSTIC_JAR} ${params[*]} ~ | After the run_diagnostic.sh script is run, the output should look something like this: [root@clm-aus-123 configure_rsso]# ./run_diagnostic.sh 05:13:11.164 - Common validations: 05:13:11.167 - All RSSO agent jar files are in place in '/opt/bmc/digitalworkplace/deploy' and '/opt/bmc/digitalworkplace/pluginsvr' 05:13:11.167 - All RSSO agent config files are in place in '/opt/bmc/digitalworkplace/conf' 05:13:11.167 - Your RSSO agent version is '9.1.03.001'. 05:13:11.167 - It is recommended to have RSSO agent of version '19.5'. 05:13:11.167 - Your RSSO server version is '18.05.00'. 05:13:11.167 - It is recommended to have RSSO server and agent of same version. 05:13:11.167 - Validating 'rsso-agent.properties' file in '/opt/bmc/digitalworkplace/conf': 05:13:11.168 - agent-id is dwpid 05:13:11.168 - Agent id should be the same on all DWP-Catalog nodes 05:13:11.168 - sso-external-url is http://<servername>:8080/rsso 05:13:11.168 - HTTPS is recommended for external URL 05:13:11.168 - sso-service-url is http://<servername>:8080/rsso 05:13:11.168 - http://<servername>:8080/rsso is accessible 05:13:11.169 - Validating 'rsso.cfg' file in '/opt/bmc/digitalworkplace/conf': 05:13:11.169 - rsso.cfg file is configured correctly |
Web server (Jetty) | Jetty logging can be very useful to capture HTTP traffic. To enable Jetty logging, uncomment the following section and change the log level:<!-- Jetty Logging - uncomment block if jetty logging is required --> <appender name="JettyLog" class="com.bmc.arsys.logging.ReconfigurableRollingFileAppender"> <file>${com.bmc.arsys.homedir}/${com.bmc.arsys.server.dbdir}/jetty.log </file> <append>true</append> <rollingPolicy class="com.bmc.arsys.logging.StoredRollingPolicy"> <fileNamePattern>${com.bmc.arsys.homedir}/${com.bmc.arsys.server.dbdir}/jetty.log.%i </fileNamePattern> <minIndex>1</minIndex> <maxIndex>8</maxIndex> </rollingPolicy> <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy"> <maxFileSize>128MB</maxFileSize> </triggeringPolicy> <needPreLoading>true</needPreLoading> <encoder> <pattern>%d{EEE MMM dd HH:mm:ss.SSS yyyy} %m%nopex%n</pattern> </encoder> </appender> <logger name="org.eclipse.jetty.server.Server" level="all" additivity="false"> | If connection issues are observed in a server group, run the following command to analyze the traffic (where ens192 is your network adapter): tcpdump -A -i ens192 port 8008 |
Connector logging | Connector logging is available from version 19.02. To enable connector logging, change the log level to all: </appender> <logger name="com.bmc.myservice.connect.remedy" level="all" additivity="false"> <appender-ref ref="connect-remedy" /> </logger> <logger name="com.bmc.myservice.connect.commons-ar" level="all" additivity="false"> <appender-ref ref="connect-remedy" /> </logger> | You can also test the connector from BMC Digital Workplace Catalog. The following screenshot indicates that the BMC Digital Workplace Advanced ITSM Integration patch needs to be updated. |
Process logging | Back up logback_server.xml and add the following to the end of the file: <!-- Process --> <appender name="ProcessLogToForm" class="com.bmc.arsys.logging.SubscribableAppender"> </appender> <appender name="ProcessLog" class="com.bmc.arsys.logging.SiftingTenantAppender"> <appender class="com.bmc.arsys.logging.SuspendableRollingFileAppender"> <file>${com.bmc.arsys.homedir}/${com.bmc.arsys.server.dbdir}/arprocess.log </file> <!-- <param name="Encoding" value="UTF-8" /> --> <append>true</append> <enableThreadLogging>true</enableThreadLogging> <rollingPolicy class="com.bmc.arsys.logging.StoredRollingPolicy"> <fileNamePattern>${com.bmc.arsys.homedir}/${com.bmc.arsys.server.dbdir}/arprocess${threadId}.log.%i </fileNamePattern> <minIndex>1</minIndex> <maxIndex>10</maxIndex> </rollingPolicy> <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy"> <maxFileSize>99MB</maxFileSize> </triggeringPolicy> <encoder> <pattern>%m%n</pattern> </encoder> </appender> </appender> <logger name="com.bmc.arsys.rx.services.process" level="debug" additivity="false"> <appender-ref ref="ProcessLog" /> <appender-ref ref="AlwaysOnLog" /> <appender-ref ref="ProcessLogToForm" /> </logger> <logger name="com.bmc.arsys.server.rx.services.process" level="debug" additivity="false"> <appender-ref ref="ProcessLog" /> <appender-ref ref="AlwaysOnLog" /> <appender-ref ref="ProcessLogToForm" /> </logger> <logger name="com.bmc.arsys.server.rx.services.action.DefaultCustomActionExecutor" level="debug" additivity="false"> <appender-ref ref="ProcessLog" /> <appender-ref ref="AlwaysOnLog" /> <appender-ref ref="ProcessLogToForm" /> </logger> <logger name="com.bmc.arsys.server.rx.services.action.ActionServiceImpl" level="debug" additivity="false"> <appender-ref ref="ProcessLog" /> <appender-ref ref="AlwaysOnLog" /> <appender-ref ref="ProcessLogToForm" /> </logger> <logger name="com.bmc.arsys.server.rx.services.action.DefaultCustomActionManager" level="debug" additivity="false"> <appender-ref ref="ProcessLog" /> <appender-ref ref="AlwaysOnLog" /> <appender-ref ref="ProcessLogToForm" /> </logger> <logger name="com.bmc.arsys.server.rx.services.connector.ConnectorCustomActionManager" level="debug" additivity="false"> <appender-ref ref="ProcessLog" /> <appender-ref ref="AlwaysOnLog" /> <appender-ref ref="ProcessLogToForm" /> </logger> Then log into the mid-tier (Catalog) as dwpadmin and navigate to Sever Information > Log Files and tick "Process Log". You need to perform a "dwpcontroller stop/start" for the changes to take effect. Please remember to turn off this logging if it's no longer required. | dwpc_process.log $AR_SYSTEM_HOME/db/tenant/dragonfly/coffeecom/db |
BMC Digital Workplace bundle log | Almost every troubleshooting scenario will require analysis of the bundle log. Here's an example of what a "healthy" bundle log looks like. 05-05 15:56:26.419 DEBUG com.bmc.myservice.connectors.service.ConnectorSrvImpl testing URI [http://localhost:8008/api/sbe/connect/remedy ] 05-05 15:56:26.520 DEBUG com.bmc.myservice.connectors.service.ConnectorSrvImpl connector URIs [http://localhost:8008/api/sbe/connect/baopoc ] com.bmc.myservice.connectors.service.ConnectorSrvImpl connector URIs [http://localhost:8008/api/sbe/connect/ad ] In this example, DEBUG is enabled and it's constantly looping to check each connector's status. The following example demonstrates how you would use the bundle log to troubleshoot a typical problem. In the following screenshot, Allen submitted a Service Request in BMC Digital Workplace, the Status is Fault. | To get a live feed of what's happening in the log, open a session in the $AR_SYSTEM_HOME/db directory, and run a tail on the log: [root@clm-aus-0123 db]# tail -f bundle.log In the following example, the service request is faulting because a required field is empty: at sun.reflect.GeneratedMethodAccessor636.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_95] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_95] at com.bmc.myservice.common.util.SrvInterceptor.invoke(SrvInterceptor.java:23) ~[com.bmc.myservice.common-1.0.00-SNAPSHOT.jar:1.0.00-SNAPSHOT] ... 94 common frames omitted Caused by: com.bmc.arsys.domain.etc.ARException: ERROR (100): Required Entry ID is empty.; form: rxn:/myit-sb/ServiceRequest at com.bmc.arsys.server.domain.validation.EntryValidatorImpl.validateEntryId(EntryValidatorImpl.java:335) ~[na:na] at com.bmc.arsys.server.domain.validation.EntryValidatorImpl.validateGetEntry(EntryValidatorImpl.java:323) ~[na:na] at com.bmc.arsys.server.domain.validation.handler.EntryValidationHandler.validateGetEntry(EntryValidationHandler.java:79) ~[na:na] at com.bmc.arsys.server.domain.service.impl.EntryServiceImpl.getEntry(EntryServiceImpl.java:366) ~[na:na] at com.bmc.arsys.server.facade.impl.EntryFacadeImpl.getEntry_aroundBody10(EntryFacadeImpl.java:143) ~[na:na] at com.bmc.arsys.server.facade.impl.EntryFacadeImpl$AjcClosure11.run(EntryFacadeImpl.java:1) ~[na:na] In the following example, another catalog request is submitted from the BMC Digital Workplace end user console, and the bundle log shows that a Work Order Manager should be defined in Remedy ITSM: Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_95] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_95] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_95] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_95] at com.bmc.arsys.rx.business.action.impl.ActionServiceImpl.invokeMethod(ActionServiceImpl.java:710) ~[na:na] ... 80 common frames omitted Caused by: com.bmc.myservice.connect.exception.ConnectorException: ERROR (1440853): ; No work order manager group could be found. Manually select a group from the menus. If no group with an individual in the functional role of work order manager is defined, notify your System Administrator. at com.bmc.myservice.connect.ar.api.ArApiManager.createEntry(ArApiManager.java:448) ~[na:na] at com.bmc.myservice.connect.ar.api.ArApiManager.createEntry(ArApiManager.java:411) ~[na:na] at com.bmc.myservice.connect.ar.api.ArApiManager.createEntry(ArApiManager.java:400) ~[na:na] at com.bmc.myservice.connect.ar.common.services.WorkOrderService.createWorkOrder(WorkOrderService.java:14 Most errors are now visible by simply opening the Request from the Console. To learn more about how to configure the logs and troubleshoot, see Configuring-logs and Troubleshooting-common-BMC-Digital-Workplace-Catalog-issues-by-using-log-files. |
logback_server.xml debug | Logging level is controlled within logback_server.xml, which is located in the $AR_SYSTEM_HOME/conf directory. Note: Before making any changes, back up logback_server.xml. XML mistakes may impact BMC Digital Workplace Catalog on startup. The best level for debugging is level=all, but it can impact performance. <appender name="BUNDLE" class="com.bmc.arsys.logging.ReconfigurableRollingFileAppender"> <file>${com.bmc.arsys.homedir}/${com.bmc.arsys.server.dbdir}/bundle.log</file> <append>true</append> <rollingPolicy> <fileNamePattern>${com.bmc.arsys.homedir}/${com.bmc.arsys.server.dbdir}/bundle.log.%i</fileNamePattern> <minIndex>1</minIndex> <maxIndex>10</maxIndex> </rollingPolicy> <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy"> ** <maxFileSize>99MB</maxFileSize>**\\ </triggeringPolicy> <!-- Extra param to say log file creation at startup --> <needPreLoading>true</needPreLoading> <encoder> <pattern>%d{MM-dd HH:mm:ss.SSS} %-4le %c %m%n</pattern> </encoder> </appender> | In 19.02 and later versions, you can control BMC Digital Workplace Catalog logging from the Application Settings panel. |
Approvals and Remedy ITSM integration | The MyITSB- Approval process is responsible for sending the approval request from BMC Digital Workplace Catalog to the Remedy ITSM server. When a service request is raised from the BMC Digital Workplace end user console:
| If the correct approval status is not reflected in BMC Digital Workplace, enable filter/API logging on the Remedy ITSM server before reproducing the problem. Then, you can see the relevant filters triggering the approval change and sending it back to BMC Digital Workplace Catalog. The next step would be to open the SB:ServiceRequestStub form on the Remedy ITSM erver and select the Error Message check box. The following example shows a timeout error reaching BMC Digital Workplace Catalog:
Make sure the SB:LocalApprovalConfiguration and SB:RemoteApprovalConfiguration Approval forms on the Remedy ITSM servers have the correct information. Additionally, with the remoteaction log file (enabled in the logback_remoteaction file), you can track the status transmission back to BMC Digital Workplace Catalog.
|
BMC Digital Workplace Catalog debug | Open the logback-dwp.xml file from the /opt/apache/tomcat8.x/external-conf directory and change these two extracts: <!--Logging of DWP Catalog --> <property name="LOG_DWP_CATALOG" value="true"/> <!--Optional DWP Catalog logger--> <if condition='${LOG_DWP_CATALOG}'> <then> <logger name="com.bmc.bsm.myit.service.sb" level="debug" > <appender-ref ref="DWP_CATALOG_FILE"/> </logger> <logger name="com.bmc.bsm.myit.rest.v2.assistance" level="debug" > <appender-ref ref="DWP_CATALOG_FILE"/> </logger> <logger name="com.bmc.bsm.myit.service.social.activity_stream" level="debug" > <appender-ref ref="DWP_CATALOG_FILE"/> </logger> </then> </if> After the scan period of 60 seconds a new log file (dwp-catalog.log) will be created in the DWP/Logs directory. |
ITSM Integration
The following illustration depicts the process flow when submitting a request from BMC Digital Workplace. Approvals and Remedy ITSM use a combination of the remoteaction.xml and associated JAR files in combination with Remedy workflow, so you can use the same troubleshooting methodology as described above for approvals.
Catalog File System
Most of the BMC Digital Workplace Catalog files are contained within the digitalworkplace directory. Some of the more important directories and files are highlighted in the following illustration:
OSGI Console
BMC Digital Workplace Catalog is run through a variety of bundles, which are loaded after the dwpcontroller script is run as part of a platform restart. Unlike the BMC Digital Workplace client, you cannot restart the individual components of BMC Digital Workplace Catalog (without negative consequence); however, you can check the bundle status of BMC Digital Workplace Catalog by using the Eclipse Open Services Gateway Initiative ( OSGI) Console.
The logging level for OSGI is controlled in the $AR_SYSTEM_HOME/conf/config.properties file:
To connect to the console, use the telnet command to connect to the BMC Digital Workplace Catalog host and execute the ss command:
You can identify important bundles related to BMC Digital Workplace Catalog by looking for the ones with myservice at the beginning:
If you notice any bundles in a RESOLVED state, you can attempt to restart with the following commands:
If the BMC Digital Workplace Catalog login page becomes unavailable for all users, restart com.bmc.myservice.uiservice-war_0.0.1 and try again. If you are still unable to log in, contact BMC Support. (Enter disconnect to end the session.)
The connectors and bundles are located in the $AR_SYSTEM_HOME/deployedsmart bundles directory.
com.bmc.myservice.connect.ad-19.5.0-SNAPSHOT.514-dev.jar
DWP Controller Scripts
The rxscripts directory and subdirectories contain scripts that BMC Digital Workplace Catalogcalls. In some cases, you may also run them manually.
Script | Description |
---|---|
/bin/check_migrations.sh | Checks migrations |
/bin/create_schema.sh | Gets called by the post install script |
/bin/current-user.sh | Checks the current user who’s in the current session |
/bin/tenant/find.sh | Lists all the tenants |
/bin/tenant/delete.sh | Deletes a tenant |
/bin/connector/refresh.sh | Refreshes the available connectors |
/bin/healthcheck_core.sh | Checks if the Catalog is up or down |
/bin/login.sh | Used to log into Remedy from the command line |
/bin/run_migrations.sh | Gets called when DB migrations are required |
/bin/setenv.sh | Sets up the environmental variables to allow the API endpoints |
/bin/tenant_seed.sh | Gets called by the post install script |
/bin/version.sh | Gets current version by running |
/bin/service/search.sh | Lists all catalog services visible by the current DWP user |
/bin/process_def | Remove and find workflow |
To run these scripts, set up your environmental variables by sourcing the setenv.sh script:
After you source the setenv script, log in to BMC Digital Workplace Catalog from the command line and generate an authentication cookie. After the mycookies key is generated, subsequent scripts (provided they are in the same directory as the mycookie.txt) run.
In the following example, the find.sh script is run to obtain information about a problematic tenant. You can also delete and re-create a tenant by using the post-install scripts (be careful deleting files).
Browser Tools
You can enable Developer Mode (F12) in a browser. Then, open the Network tab response and look for any exceptions. The following screenshot illustrates how the root cause of a problem was captured by simply clicking the REST call and selecting the response.
Ensure BMC Digital Workplace Catalog is reachable and the fully qualified domain name is being used.
Don't forget that the users must exist on the User form on the BMC Digital Workplace Catalog server (which is populated via the sync script) and the CTM:People form in Remedy ITSM.