High CPU utilization and unresponsive server

If multiple users access the system simultaneously, CPU utilization can reach 100 percent and the server might stop responding. This might happen if you have installed the BMC Release Process Management (BRPM)product by using the installer. The installer automatically configures the system for an average use of approximately 10 concurrent users with a maximum of 10 steps per request. You can decrease the CPU load by tuning your deployment to handle more requests simultaneously. For more information about tuning your deployment to handle a higher load, see Performance tuning.

If the issue of high CPU utilization persists, even after tuning your deployment, we must analyze the reasons behind this defect. Perform the following steps to help BMC Support troubleshoot the issue:

1. Create the garbage collector log
   The garbage collector log will help to understand the performance of the garbage collector and understand the reason for high CPU utilization. To create the garbage collector log, perform the following steps:

   1. Add the following parameters inside the <RPM_HOME>/bin/start.sh file:

      export RPM_LOG_TIMESTAMP=`date
      +'%m-%d-%Y-%H-%M-%S'`
      export
      RPM_GC_LOG_LOCATION=/tmp/gclog-$RPM_LOG_TIMESTAMP.log
      export
      JAVA_OPTS="$JAVA_OPTS -Xms8192m -Xmx8192m -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps
      -verbose:gc -XX:+PrintGCDetails -Xloggc:$RPM_GC_LOG_LOCATION
      -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M"

   2. From the /tmp/gclog-08-31-2020-17-49-45.log.0.current location, capture the file with the following format in the file name:

      gclog-<dd-mm-yyyy-hh-mm-ss>.log.0.current

      This file contains the garbage collector activity.

      Note

      The date and time stamp on all file names correspond to the date and time the BRPM service was started.

2. Enable the thread dump
   The thread dump records all the data required for the analysis of the high CPU utilization issue. Perform the following steps to enable the thread dump:
   1. Update the BRPM <RPM_HOME>/bin/start.sh file on all the nodes from 

      $EXECUTABLE" -c standalone-ha.xml -b $NODE_IP -Djboss.node.name=$NODE_NAME -Djboss.logging.level=$LOG_LEVEL >   /dev/null 2>&1 &

      to

      export RPM_LOG_TIMESTAMP=`date +'%m-%d-%Y-%H-%M-%S'`"$EXECUTABLE" -c standalone-ha.xml -b $NODE_IP -Djboss.node.name=$NODE_NAME -Djboss.logging.level=$LOG_LEVEL >> /tmp/brpm-console-$RPM_LOG_TIMESTAMP.log 2>&1 &

   2. Update the BRPM <JBOSS_HOME>/standalone/configuration/standalone-full.xml OR standalone-full-ha.xml file on all the nodes from

      <subsystem xmlns="urn:jboss:domain:logging:8.0">

          <console-handler name="CONSOLE">

           <level name="${jboss.logging.level:INFO}"/>

            --

          </console-handler>
      to

      <subsystem xmlns="urn:jboss:domain:logging:8.0">

          <console-handler name="CONSOLE">

           <level name="OFF"/>

            --

          </console-handler>

   3. Restart all the nodes.


   4. A file, named thread-dump.sh, with the following script is generated:

      #!/bin/sh
      #
      # Takes the JBoss PID as an argument. 
      #
      # Captures cpu by light weight thread and thread dumps a   specified number of
      # times and INTERVAL. Thread dumps will be in the file where stdout was
      # redirected or in console output.
      #
      # Usage: sh ./thread-dump.sh <JBOSS_PID>
      #
      
      LOOP=6
      
      # Interval in seconds between data points
      INTERVAL=20
      
      FILE_TIMESTAMP=`date +'%m-%d-%Y-%H-%M-%S'`
      
      for (( i = 1; i <= $LOOP; i++ )); do
        echo "Iteration - $i" >> high-cpu-$FILE_TIMESTAMP.out
        date >> high-cpu-$FILE_TIMESTAMP.out
        top -b -n 1 -H -p $1 >> high-cpu-$FILE_TIMESTAMP.out
        kill -3  $1
        echo "thread dump #" $i
        if [[ $i -lt $LOOP ]]; then
          echo "Sleeping..."
          sleep $INTERVAL
        fi
      
        echo -en '\n\n' >> high-cpu-$FILE_TIMESTAMP.out
      done

      Copy this file and save it on the BRPM hosts folder.

   5. Use the command ps -ef | grep jboss or ps aux | grep java to display the process ID for the BRPM JVM. It is displayed as follows:
      

   6. Run the script with the process ID.

      cd
      <folder containing the thread-dump.sh>
      sh
      ./thread-dump.sh <JBOSS_PID>

      Example

      sh
      ./thread-dump.sh 31583

   7. Capture the following two files:

      1. From the location /tmp/brpm-console-08-31-2020-17-49-45.log, capture the file brpm-console-<dd-mm-yyyy-hh-mm-ss>.log.

         This file contains six thread dump when the thread-dump.sh script is executed.

      2. From the location /root/RPM-thread-dump/high-cpu-08-31-2020-16-50-00.out, where the thread dump is located, capture the file high-cpu-<dd-mm-yyyy-hh-mm-ss>.out. 

         This file contains 6 output of threads created by the BRPM process with CPU and memory.

3. Send these files to BMC Support.