Setting up BMC Deployment Engine on a server with internet access

Set up BMC Deployment Engine to deploy BMC Helix Service Management applications.

BMC Deployment Engine is set up on a fresh single virtual machine.

BMC Deployment Engine consists of the following tools:

  • Jenkins—Jenkins is the primary deployment tool.
  • Ansible—The Jenkins pipeline automation is written by using Ansible and shell scripts.
  • Git—Git is used to store the deployment artifacts.
  • kubectl—is the Kubernetes client that is used to run the Kubernetes commands.
  • Helm—Helm is the package manager used to deploy applications in Kubernetes.

These tools are used by the Jenkins pipelines to deploy BMC Helix Service Management applications.

 

Related topics

System-requirements

BMC-Deployment-Engine-overview

FAQ

Skills required

  • Using Jenkins
  • Using Linux

See Recommended-skill-set-and-trainings

The steps to set up BMC Deployment Engine are shown in the following figure:

NewDESetup.png

Perform these steps when you perform a fresh installation of BMC Helix Service Management.

Before you begin

Important

  • To login to the BMC Deployment Engine server, use SSH clients, such as PuTTY, MobaXterm, mRemoteNG, or equivalent tool supported by HOST/GUEST operating system. This server is referred to as Jenkins server in the rest of the document.
  • While configuring BMC Deployment Engine, use a user with sudo access. The sudo access permission is required to install application packages such as Jenkins.
  • Make sure that BMC Deployment Engine server is registered with the Red Hat subscription manager and a valid subscription pool ID.
  • Unless specified, the user refers to non root user for this setup.
  • BMC Deployment Engine uses the following two users:
    • git—Owner of git repositories where deployment code is present. This user requires sudo access.
    • jenkins—Jenkins server runs as a Jenkins user. This is a non-interactive user.
  • Make sure that the user has a default shell as sh or bash.
  • Make sure that the Jenkins server has access to the internet to access the online Jenkins repository for OS software update or upgrade as well as install tools required to set up Deployment Engine and supporting packages.
  • The nomenclature used in pipeline development is as follows:
    • The Home directory of non root user is referred to as <HOME> in all subsequent steps. By default, it is /home/git.
    • <user> refers to the git repo owner. Jenkins user owns the Jenkins setup. The git user is a member of the git and Jenkins groups.
      The Jenkins user is member of the Jenkins and git groups.
    • <hostname> is the host name/IP address of the Jenkins server, which is resolvable by all participating entities, such as databases and clusters in this environment.
    • <GIT_REPO_DIR> is the location where the entire Deployment Engine git code is copied by the installer for setting up the pipelines. The recommended size of this directory must be 10GB or more.
    • <Library_REPO_DIR> is the location where the installer copies the required libraries that need to be configured manually after installation.

Ensure that your environment meets the following requirements:

  • Virtual machine—A fresh and dedicated physical server or virtual machine with the following configuration for the Jenkins server:

    Component

    Node

    vCPU

    Operating System

    RAM (GB)

    Disk (GB)

    Jenkins server

    1

    2

    • Cent OS 7.x and 8.x are not supported.
    • RHEL 8.6 or later, RHEL 9.x, Rocky Linux 8.x and 9.x are certified.
    • If you are using a proxy server, make sure that you specify the values for the proxy parameters.

    Minimum 8 GB

     

    100

    Make sure that the /home, /var, /usr mount points have a minimum of 15 GB free space and /tmp has a minimum of 5 GB free space.

    Make that the virtual machine has a minimum of 1.5 GB boot partition.

  • Proxy parameters—If you are using a proxy server, make sure that you specify the values for the following parameters:
    • Parameters in the /etc/environment file:
      • export http_proxy
        Example: export http_proxy="http:<proxy host name or IP>:<proxy port>"
      • export https_proxy
        Example: export https_proxy="https:<proxy host name or IP>:<proxy port>"

    • Parameters in the /etc/rhsm/rhsm.conf file:

      Important

      These parameters are applicable only for RHEL 8 and RHEL 9 Red Hat Network access for installing packages.

      • proxy_hostname
        Example: proxy_hostname = <Proxy host name or IP>
      • proxy_scheme
        Example: proxy_scheme = http
      • proxy_port
        Example: proxy_port = <Proxy port>
    • Parameter in the /etc/yum.conf file:
      • proxy
        Example: proxy=http:<proxy host name or IP>:<proxy port>
    • Make sure that you relogin to the BMC Deployment Engine server so that the proxy changes are reflected.
  • Kubernetes or OpenShift cluster—Kubernetes cluster is accessible from the Deployment Engine server. 
    For Kubernetes cluster requirements, see System Requirements.

  • Yum or DNF utility—is installed, configured, and working fine. 
    Make sure that the yum client is installed and accessible, and you can install certified repos stable versions of applications. 
    Run the following commands by using a user with sudo access to update your system libraries and packages to the latest available version.

    sudo yum update -y
    sudo yum upgrade -y
    sudo yum clean all

  • Unzip utility—is installed.
    Use the following command to install the unzip utility:

    sudo yum install unzip

  • Perl—Perl (version 5.x +) is installed and is accessible from shell prompt.
    Use the following command to set up Perl:

    sudo yum install perl -y

  • Perl-Data-Dumper package—Set up Perl-Data-Dumper package by using the following commands:

    sudo yum makecache
    sudo yum -y install perl-Data-Dumper
    • Make sure that /etc/resolv.conf has valid DNS entries.

    • From any remote machine run the following commands:

      nslookup <jenkins-server-name> 
      nslookup <jenkins-server-ip>

      both the commands should resolve via DNS from all servers where clusters are setup.

    • from /etc/hosts make sure to have only relevant entries in this file. If an entry is not correct, delete the entry and make sure that you clean up all the stale entries in the file.
    • Remove the ivp6 entry for the localhost and ensure that the hostname resolves to the correct DNS IP address.
    • Run the yum repolist or dnf repolist command to view all accessible and OS matching repositories.  The yum or dnf commands should be able to install the required repository software. 
  •  Download the BMC_Helix_Innovation_Suite_And_Service_Management_Apps_Version_23.3.04.006.zip file from EPD to the Jenkins server.
    Use the BMC Helix Innovation Suite & Service Management Apps Version 23.3.04.006 option to download the file. This file contains the following files:
    • BMC_Remedy_Deployment_Manager_Configuration_Release_23.3.04.006.zip - This file contains the repositories that go into git.
    • BMC_Remedy_Deployment_Engine_Setup_23.3.04.006.zip - This file contains the BMC Deployment Engine automation script.

For details, see Downloading-the-installation-files.

Back to top

To create and configure users

BMC Deployment Engine uses the following two users:

  • git—Owner of git repositories where deployment code is present. This user requires sudo access.
  • jenkins—Jenkins server runs as jenkins user. This is a non-interactive user.

You must create and configure these users.

To add users

Perform the following steps to add the users:

Important: Make sure that you do not use special characters in the git user password.

sudo useradd git -m
sudo passwd git
sudo useradd jenkins -m
sudo passwd jenkins

To configure passwordless sudo access for the git user

  1. Log in to BMC Deployment Engine as a root user or a user with permission to run visudo and to create a sudoers override file at the /etc/sudoers.d location.
  2. Create a sudoers override file such as /<your path to>/git_sudoers.

  3. Add the following details to the file:

    git ALL=(ALL) NOPASSWD: \
       /usr/sbin/alternatives, \
       /usr/bin/cat, \
       /usr/bin/chmod, \
       /usr/bin/chown, \
       /usr/bin/cp, \
       /usr/bin/curl, \
       /usr/bin/dnf, \
       /usr/bin/dos2unix, \
       /usr/bin/grep, \
       /usr/bin/java, \
       /usr/bin/ln, \
       /usr/bin/ls, \
       /usr/bin/mkdir, \
       /usr/bin/mv, \
       /usr/bin/netstat, \
       /usr/bin/rpm, \
       /usr/bin/sed, \
       /usr/bin/su, \
       /usr/sbin/subscription-manager, \
       /usr/bin/systemctl, \
       /usr/bin/unzip, \
       /usr/bin/crb, \
       /usr/sbin/update-alternatives, \
       /usr/bin/wget, \
       /usr/bin/yum

  4. Verify that the file has no errors by running the following command:

    visudo -c -f /<your path to>/git_sudoers

    If the output is parsed OK, the file is valid.

  5. Identify the sudoers override file location from visudo by running the following command:

    visudo

    Search for the #includedir parameter in the command output:

    ## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
    #includedir /etc/sudoers.d

    In the example, /etc/sudoers.d is the location where you must copy the sudoers override file.

  6. Copy the sudoers override file to the location found in visudo by using the following command:

    cp <override file> <#includedir location>

    Example:

    cp /path/git_sudoers /etc/sudoers.d/git_sudoers

  7. Modify access to sudoers file by using the following command:

    chmod 440  <#includedir location> 

    Example:

    chmod 440 /etc/sudoers.d/git_sudoers

  8. Switch the user to git user, and validate passwordless sudo access by running the following command:

    sudo ls /root

    The command works without prompting for the password.

To copy the ssh keys

  1. Log in to BMC Deployment Engine as a git user.
  1. Configure the SSH keys for the git user to enable pipeline access to Git repositories.

    1. Generate the SSH key by running the following command and accept all the defaults.

      ssh-keygen

    2. Copy the ID by running the following command:

      ssh-copy-id git@<jenkins_server_name>

    3. Verify that the passwordless ssh login works from git to git user by using the following command:

      ssh git@<jenkins_server_name>
  2. Log in to BMC Deployment Engine as a jenkins user.
  3. Configure jenkins ssh keys to git so that the pipeline dry-run step is successful.

    1. Generate the SSH key by running the following command and accept all the defaults.

      ssh-keygen

    2. Copy the ID by running the following command:

      ssh-copy-id git@<jenkins_server_name>

    3. Verify that you are logged in as git user and the passwordless ssh login works by using the following command:

      ssh git@<jenkins_server_name>

Back to top

To run the BMC Deployment Engine automation script

The BMC Deployment Engine automation script installs the following software:

  • Ansible
    Ansible 2.15 is certified.
  • Python
  • Git
  • kubectl
  • Helm
  • Java
  • PostgreSQL client
  • Docker
  • additional packages, such as MS SQL tools, unixODBC, xmlstarlet, dos2unix, jq

Along with the software and Jenkins, the BMC Deployment Engine automation script installs the plug-ins listed in the plugins.txtfile.

Perform the following steps to run the BMC Deployment Engine automation script:

  1. Log in as a git user:

    ssh git@<hostname>

    or switch to the git user by using the following command:

    su - git
  2. Copy the BMC_Helix_Innovation_Suite_And_Service_Management_Apps_Version_23.3.04.004.zip file downloaded from EPD and extract the files to the git user home directory or any other directory owned by the git user.
    • BMC_Remedy_Deployment_Manager_Configuration_Release_23.3.04.004.zip
    • BMC_Remedy_Deployment_Engine_Setup_23.3.04.004.zip

  3. Change the ownership of these files to git user by using the following command: 

    sudo chown -R git:git <filename>
  4. Unzip BMC_Remedy_Deployment_Engine_Setup_23.3.04.004.zip file. 
  5. Change the directory to DE1.0.
  6. Update build.properties and customize the following parameters according to your requirements:
    • ITSM_REPO_GIT_ZIP=</path/to/BMC_Remedy_Deployment_Manager_Configuration_Release_23.3.04.004.zip>

    • JENKINS_CONFIG_FILES_ZIP_PATH=<path to Jenkins_Config_Files.zip file>
    • LIBRARY_REPO_ZIP_PATH=<path to LIBRARY_REPO.zip>
    • DB_TYPE=<Database type>
      Specify the value as postgres, mssql, or oracle.

    • JENKINS_HOSTNAME=<fully-qualified-hostname-or-ip-address-where-DE-setup-is-planned>

    • KUBERNETES_VERSION=<Kubernetes cluster version>

      Important

      Specify the entire version such as KUBERNETES_VERSION=1.25.13

      For the supported Kubernetes version, see System-requirements. Make sure that you specify the correct Kubernetes version value. See Kubernetes version in Kubernetes documentation.

    • POSTGRES_VERSION=<PostgreSQL version>
      Specify the value of your external PostgreSQL server used for BMC Helix Service Management, such as 13 or 15. Specify only the major version. The default value is 15.
    • GIT_REPO_USER=git
    • GIT_USER_HOME_DIR=~git
    • JENKINS_USER=jenkins

    • HELM_VERSION=<Helm version>

      Important

      Specify the version HELM_VERSION value in the following format major.minor.patch such as 3.13.0

       

    • JENKINS_INSTALL_DIR=/var/lib/jenkins
    • HTTP_PROTOCOL=http
    • JENKINS_PORT=8080
    • ANSIBLE_NODE_ROOT_DIR=~/ansible_node_root_dir
    • JENKINS_NODE_ROOT_DIR=~/jenkins_node_root_dir
      This is the path that the Jenkins_node needs for various operations that are internal to Jenkins.
    • GIT_REPO_DIR=~/git_repo/
      This is the location where the entire Deployment Engine git code is copied by the installer for setting up the pipelines.
    • JENKINS_CONFIG_FILES_DIR=~/Jenkins_Config_Files
      The Jenkins job pipeline uses this location to store its pipeline jobs.

    • Proxy support parameters
      Specify the value for the following parameters if you are using a proxy. Else, leave the values blank.

      Important

      The HTTP_PROXY_HOST and HTTP_PROXY_PORT are commented by default. If you using proxy, uncomment these parameters and specify the values.
      The PROXY_USER and PROXY_USER_PWD parameters are not supported.



      • HTTP_PROXY_HOST=<Proxy host name>
        Make sure the proxy host name is resolvable from the Jenkins server.
      • HTTP_PROXY_PORT=<Proxy port number>
      • NO_PROXY_HOST=<List of hosts for which proxy configuration is not required>
      • PROXY_TEST_URL=<Test proxy URL>
        Specify the test proxy URL for configuring the Test Proxy URL setting in the Jenkins configuration.

    • ANSIBLE_SUPPORTED_VERSION=2.15

      Important

      Do not modify the values of the JENKINS_INSTALL_DIR, ANSIBLE_SUPPORTED_VERSION, HTTP_PROTOCOL, and JENKINS_PORT parameters.

      For information about the supported Kubernetes, OpenShift, and Helm versions, see System-requirements.

  2. After updating the build.properties file, run the BMC Deployment Engine automation script to set up the Jenkins job pipeline framework.
    The log file is generated in the Git user home directory and it can be used for debugging setup related issues.
     

    Important

    Make sure that you use the git user to run the script.

    Do not use a sudo user or the root user.

    perl setup-Helix-ITSM-onPrem.pl  2>&1 | tee ~/BMC-HELIX-DE-AUTO.log.$$
    Important

    Due to a recent change in the Jenkins server, the deployment scripts may incorrectly report the Jenkins server status as failed, even though it is installed and functioning correctly. You can safely ignore this status if you are able to log in to the Jenkins server UI and verify that it is running as expected.

Back to top

To perform post-installation configurations

Complete the following configurations:

No.

Action

Steps

1

Install plug-ins

  1. Log in to the Jenkins server by using the following URL:
    http://<Jenkins server host name>:<Jenkins port>

    The default password is available in $<JENKINS_INSTALL_DIR>/secrets/initialAdminPasswordFor example: /var/lib/jenkins/secrets/initialAdminPassword

  2. On the Jenkins User Interface, select Install Suggested Plugins.
  3. In the Admin User Creation wizard, provide a preferred username and password and run the wizard.
  4. Verify that ssh works fine with passwordless login for the git and jenkins users:

    1. Verify the git user login to git by using the following commands:

      ssh git@<jenkins_server_name>

      git to git passwordless is required for the pipeline jobs to access git repository.

    2. Log in to BMC Deployment Engine as a jenkins user and verify login to git by using the following commands:

      ssh git@<jenkins_server_name>

      jenkins to git passwordless access is required for saving pipeline configuration and dry-run to work.

  2.  Add the kubeconfig file on the Jenkins server.
    1. Download the kubeconfig.yaml file to the $<git user home> directory.

    2. Copy the kubeconfig.yaml file to $<git user home>/.kube/config directory.

      cp <kubeconfig complete file path> $<git user home>/.kube/config

2

Update the Jenkins credentials

  1. Login to the Jenkins server by using the following URL:
    http://<Jenkins server host name>:<Jenkins port>/credentials
  2. Add the credentials in the ID: kubeconfig file.
    1. Select kubeconfig and click the name of the kubeconfig credential.
      JenkinsCredentials.png
    2. Click Update.
    3. Select the Replace checkbox.
    4. Click Choose File and select the kubeconfig file
    5. Click Save.
  3. Add the credentials in the ID: github account.
    1. Select github and click the name of the github credential.
    2. Click Update.
    3. Click Change Password and enter the git user password.
    4. Click Save.
  4. Add the credentials in the ID: ansible_host account.
    1. Select ansible_host and click the name of the ansible_host credential
    2. Click Update.
    3. Click Change Password and enter the git user password.
    4. Click Save.
  5. Add the credentials in the ID: ansible file.
    1. Select ansible and click the name of the ansible credential.
    2. Click Update.
    3. Click Change Password and enter the git user password. 
    4. Click Save.
  6. Add credentials in the ID: git account.
    1. Select git and click the name of the git credential.
    2. Click Update.
    3. Click Change Password and enter the git user password.
    4. Click Save.
  7. (For cloud environments) Update git user private key credentials
    1. Navigate to http://<Jenkins server host name>:<Jenkins port>/credentials/store/system/domain/_/
    2. Click Add Credentials.  

    3. Enter the following details:

      Kind

      SSH Username with private key

      ID

      git_pk

      Username

      git

      Private Key

      1. Click Enter.
      2. Paste the contents of git user private key value (/home/git/.ssh/id_rsa)
    4. Save the details.

Depending on your environment, you can choose to update the node configuration in a cloud or non cloud environment.

3
 

Update the node configuration in a cloud environment

  1. Navigate to http://<Jenkins server host name>:<Jenkins port>/computer.
  2. Perform the following steps for both nodes, the one with the actual hostname and another with the git hostname.
    1. Click the node name. 
    2. Click Configure.
    3. Under Credentials, select the git user ssh key credential and click Save.
    4. Click Launch Agent.
    5. Verify that the launch agent is successful.

Update the node configuration in a non cloud environment

  1. Navigate to http://<Jenkins server host name>:<Jenkins port>/computer.
  2. Perform the following steps for both nodes, the one with actual hostname and another with git-hostname.
    1. Click the node name.
    2. Click Launch Agent.
      Important: If the status of the node is already 'online', then skip Step b.
    3. Verify that the launch agent is successful.

4

 Add the Jenkins libraries

Perform the following steps to add the pipeline-framework library:

  1. On the Jenkins home page, click Manage Jenkins.
  2. Select System.
  3. In Global Trusted Pipeline Libraries, add the pipeline-framework library as shown in the following figure:
  4. In the Project Repository field, specify the complete path of pipeline-framework.git according to the environment.
    <GIT_REPO_DIR> value can be fetched from build.properties file.
    pipelineframework_1.png
  5. Make sure you provide exact location of pipeline-framework.git according to the environment.
  6. Add JENKINS-27413-workaround-library as shown in the following figure:
  7. In the Project Repository field, specify the complete path of JENKINS-27413-workaround-library.git according to the environment.
  8. Select Load implicitly.
    JenkinsLibrary_1.png
  9. Save the changes.

5

Run the deployment pipelines in dry-run mode

Dry-run is a mandatory step to update the pipeline configuration for any changes to the BMC Helix Service Management installer.

  1. Navigate to Jenkins Dashboards to view all the pipelines required for deployment.
  2. Select each pipeline and click Build with Parameters.
  3. In the AGENT parameter, provide the value of the node that has the name git-<hostname>.
    The AGENT parameter value is the Jenkins agent that runs the pipeline.
  4. Click Build.
    The build job will fail, which is expected.

Important:

  • The agent-add-pipeline and HELIX_DR pipelines do not require a dry-run.
  • For the HELIX_ONPREM_DEPLOYMENT pipeline, enter the value of the AGENT parameter, and do not select any pipelines in the PRODUCT_DEPLOY section.
  • For the other pipelines, run Build with Parameters with default values present in the pipeline.

Back to top

Troubleshooting

Refer to the troubleshooting information if you encounter any of the following issues while setting up BMC Deployment Engine.

Important

When logging a case with BMC Support for any issues that occur while setting up the BMC Deployment Engine, we recommend including the BMC-HELIX-DE-AUTO.log file and build.properties file to facilitate debugging the issues.

Issue symptom

During deployment engine script execution, you might receive the following message:
This system is not registered with an entitlement server. You can use subscription-manager to register.
         Log: Query :  Please, suggest should we disable subscription manager and proceed with setting up prereq for ITSM onprem deployment,  Y(yes) or N(no) ? :

Resolution

  • If you are using Rocky Linux 8 or Rock Linux 9 operating system, enter yes and proceed.
  • If you are using RHEL 8 or RHEL 9 operating system, enter no and proceed. In this case, register your system with Red Hat for a valid subscription.

Issue symptom

The installation script fails or aborts for some reason.

Resolution

  1. Review the BMC-HELIX-DE-AUTO.log file to check for errors.
  2. Fix the errors.
  3. Stop Jenkins by using the systemctl command.
  4. Check if port 8080 is free by using netstat -anp | grep 8080 command and then rerun automation script command.
  5. Make sure to save log files.

Issue symptom

HELIX_ONPREM_DEPLOYMENT pipeline fails with the following scripts not permitted error.

method hudson.model.Run getLog
method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild

Resolution

  1. Right click the error message link and open it in a new tab.
  2. Click Approve.
    KnownIssues1.png
  3. Click the latest failed job on the HELIX_ONPREM_DEPLOYMENT pipeline tab.
    FailedJob.png
  1. Click Rebuild.
    Rebuild.png
  2. Enter the required parameters and click Rebuild.
    EnterParameters.png
  3. You might get other scripts not permitted error. Repeat the same steps to approve the script execution and rebuild HELIX_ONPREM_DEPLOYMENT jobs.

For other issues, see Troubleshooting-deployment-pipeline-failure-issues.

For additional questions about BMC Deployment Engine, see FAQ.

Where to go from here

Next task

Back to process

If you are finished installing BMC Helix Platform services, return to the appropriate installation or upgrade process:

 

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*