Setting up BMC Deployment Engine in an air-gapped environment

Set up BMC Deployment Engine to deploy BMC Helix Service Management in an air-gapped environment. An air-gapped environment is an environment that is disconnected or physically isolated from public internet and unsecured networks.

BMC Deployment Engine consists of the following tools:

  • Jenkins—Jenkins is the primary deployment tool.
  • Ansible—The Jenkins pipeline automation is written by using Ansible and shell scripts.
  • Git—Git is used to store the deployment artifacts.
  • kubectl—kubectl is the Kubernetes client used to run the Kubernetes commands.
  • Helm—Helm is the package manager used to deploy applications in Kubernetes.

These tools are used by the Jenkins pipelines to deploy BMC Helix Service Management applications.

Related topics

Setting-up-BMC-Deployment-Engine

Installing

Before you begin

  • Make sure that you create the following virtual machines by using the same operating system image or template:
    • Offline server—Virtual machine without internet access to set up BMC Deployment Engine in an air-gapped environment.
    • Online server—Virtual machine with internet access to download artifacts required to set up air-gapped BMC Deployment Engine required for the offline server.
      • The online server must be an exact replica of the offline server.
      • The online server must have a nonroot user with sudo access to install and con­figure repositories and to download artifacts from the RHN, Jenkins, Red Hat OpenShift, Ansible, EPEL repositories, and Python modules.
      • The online server must be registered with the Red Hat subscription manager and a valid subscription pool ID.

  • Make sure that the offline and online virtual machines meet the following specifications:

    Component

    Node

    vCPU

    Operating System

    RAM (GB)

    Disk (GB)

    Jenkins server

    1

    2

    • RHEL 8.6 or later
    • RHEL 9.x
    • Rocky Linux 9.x

    Minimum 8 GB

     

    100
    Make sure that the /home, /var, /usr mount points have a minimum of 15 GB free space and /tmp has a minimum of 5 GB free space.

The following image shows the tasks to set up BMC Deployment Engine in an air-gapped environment:

Air gapped deployment.png

Task 1: To set up BMC Deployment Engine on an online server

  1. Log in to your online server.
    1. Log in to your online server by using a user with sudo access.

    2. Create the git user and provide sudo access to the git user by using the following commands:
      Important: Make sure that you do not use special characters in the git user password.

      sudo useradd git -m
      sudo passwd git
    3. Log in to your online server by using the git user.
  2. Register the RHEL repository.
    Make sure that the subscription manager is registered.

  3. Configure passwordless sudo access for the git user.

    1. Log in to BMC Deployment Engine as a root user or a user with permission to run visudo and to create a sudoers override file at the /etc/sudoers.d location.
    2. Create a sudoers override file such as /<your path to>/git_sudoers.

    3. Add the following details to the file:

      git ALL=(ALL) NOPASSWD: \
         /usr/sbin/alternatives, \
         /usr/bin/cat, \
         /usr/bin/chmod, \
         /usr/bin/chown, \
         /usr/bin/cp, \
         /usr/bin/curl, \
         /usr/bin/dnf, \
         /usr/bin/dos2unix, \
         /usr/bin/firewall-cmd, \
         /usr/bin/grep, \
         /usr/bin/java, \
         /usr/bin/ln, \
         /usr/bin/ls, \
         /usr/bin/mkdir, \
         /usr/bin/mv, \
         /usr/bin/netstat, \
         /usr/bin/rpm, \
         /usr/bin/sed, \
         /usr/bin/su, \
         /usr/sbin/subscription-manager, \
         /usr/bin/systemctl, \
         /usr/bin/unzip, \
         /usr/sbin/update-alternatives, \
         /usr/bin/wget, \
         /usr/bin/yum, \
         /usr/bin/crb, \
         /usr/bin/perl, \
         /usr/bin/updatedb, \
         /usr/bin/ansible-config , \
        /usr/bin/crb, \
        /usr/bin/echo, \
        /usr/bin/find, \
        /usr/bin/yumdownloader , \
        /usr/bin/sh, \
        /usr/bin/updatedb, \
        /usr/bin/locate

    4. Verify that the file has no errors by running the following command:

      visudo -c -f /<your path to>/git_sudoers

      If the output is parsed OK, the file is valid.

    5. Identify the sudoers override file location from visudo by running the following command:

      visudo

      Search for the #includedir parameter in the command output:

      ## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
      #includedir /etc/sudoers.d

      In the example, /etc/sudoers.d is the location where you must copy the sudoers override file.

    6. Copy the sudoers override file to the location found in visudo by using the following command:

      cp <override file> <#includedir location>

      Example:

      cp /path/git_sudoers /etc/sudoers.d/git_sudoers

    7. Modify access to sudoers file by using the following command:

      chmod 440  <#includedir location> 

      Example:

      chmod 440 /etc/sudoers.d/git_sudoers

    8. Switch the user to git user, and validate passwordless sudo access by running the following command:

      sudo ls /root

      The command works without prompting for the password.

  4. Download and extract the installer.
  5. Download the BMC_Helix_Innovation_Suite_And_Service_Management_Apps_Version_25.2.01.zip files from EPD to the server.
    See Downloading-the-installation-files.
  6. Extract the BMC_Helix_Innovation_Suite_And_Service_Management_Apps_Version_25.2.01.zip file.
    This file contains the following files:
    • BMC_Remedy_Deployment_Manager_Configuration_Release_25.2.01.zip - This file contains the repositories that go in to git.
    • BMC_Remedy_Deployment_Engine_Setup_25.2.01.zip

  7. Extract the BMC_Remedy_Deployment_Engine_Setup_25.2.01.zip file.
    This file is required to set up BMC Deployment Engine.

    The ZIP file contains following files:
  8. Create a working directory and copy files:

    1. Create folders by using the following commands:

      mkdir -p airgap_setup/build     
      mkdir -p airgap_setup/jenkins_plugins/jpi     
    2. Copy the following files to the airgap_setup/build folder:
      • Jenkins_Config_Files.zip
      • LIBRARY_REPO.zip
      • BMC_Remedy_Deployment_Manager_Configuration_Release_25.2.01.zip
    3. Copy all the files from the AIRGAP folder to the airgap_setup folder.
      The following files will be available in the AIRGAP folder:
      • online-airgapped-setup-Helix-ITSM-onPrem.pl
      • plugins.txt
      • offline-airgapped-setup-Helix-ITSM-onPrem.pl
      • rhel8.airgapped
      • rhel9.airgapped
      • airgapped-build.properties
      • rocky9.airgapped

  9. Navigate to the airgap_setup folder and download Perl by using the following command:

    sudo dnf download --resolve --downloaddir=. perl
    sudo dnf download --resolve --downloaddir=. perl-Data-Dumper
  10. Download the kubeconfig file and copy the file to the airgap_setup folder.
    Contact your Kubernetes cluster administrator to download the kubeconfig file.
  11. Run the BMC Deployment Engine online setup script:

    1. Navigate to the airgap_setup folder and provide permissions by using the following commands:

      sudo chmod -R 755 *
      sudo chown -R git:git /home/git/airgap_setup

    2. In the airgapped-build.properties file, update the following parameters:

      Parameter

      Description

      ITSM_REPO_GIT_ZIP

      Specify the path name to the BMC_Remedy_Deployment_Manager_Configuration_Release_25.2.01.zip file.

      JENKINS_OFFLINE_PLUGIN_PATH

      Specify the path name to the airgap_setup/jenkins_plugins/jpi folder.

      JENKINS_CONFIG_FILES_ZIP_PATH

      Specify the path name to the Jenkins_Config_Files.zip file.

      LIBRARY_REPO_ZIP_PATH

      Specify the path name to the LIBRARY_REPO.zip file.

      KUBERNETES_VERSION

      Specify the Kubernetes cluster version.
      Important:       Specify the entire version such as KUBERNETES_VERSION=1.25.13
      For the supported Kubernetes version, see System-requirements. 
      Make sure that you specify the correct Kubernetes version value. See Kubernetes version in Kubernetes documentation.

      POSTGRES_VERSIONSpecify the value of your external PostgreSQL server used for BMC Helix Service Management, such as 13 or 15. Specify only the major version.

      GIT_REPO_USER

      Specify the Git repository user such git.

      GIT_USER_HOME_DIR

      Specify the Git repository user home directory such as ~git

      HELM_VERSION

      Specify the Helm version.

      Important: Specify the version HELM_VERSION value as 3.13.0

      JENKINS_INSTALL_DIR

      Specify value as /var/lib/jenkins

      HTTP_PROTOCOL

      Specify the value as http

      JENKINS_HOSTNAME

      Specify the fully qualified host name or IP address where online server setup is planned.

      JENKINS_PORT

      Specify the value as 8080

      JENKINS_USER

      Specify the Jenkins user as jenkins.

      Important: Do not specify any other user name.

      ANSIBLE_NODE_ROOT_DIR

      Specify the path name to the Ansible node root directory such as ~/<ansible_node_root_dir>

      JENKINS_NODE_ROOT_DIR

      Specify the path name to the Jenkins node root directory such as ~/<jenkins_node_root_dir>
      The Jenkins_node uses this path name for various operations that are internal to Jenkins.

      GIT_REPO_DIR

      Specify the path name to the Git repository directory such as ~/git_repo/
      The entire Deployment Engine git code is copied by the installer to this location.

      JENKINS_CONFIG_FILES_DIR

      Specify the path name to the Jenkins configuration files directory such as ~/Jenkins_Config_Files
      The Jenkins pipeline uses this location to store its pipeline jobs.

      DB_TYPE

      Specify the type of database to be used during the deployment. Specify the value such as postgres, mssql, or oracle.

      ANSIBLE_SUPPORTED_VERSIONIndicates the supported version of Ansible that is compatible with the current deployment. 

      Important

      Do not modify the values of the JENKINS_INSTALL_DIR,  ANSIBLE_SUPPORTED_VERSION, HTTP_PROTOCOL, and JENKINS_PORT parameters.

      Proxy parameters are not applicable for air-gapped environments.

    3. Run the script by using the following commands:

      perl online-airgapped-setup-Helix-ITSM-onPrem.pl  2>&1 | tee ~/OnlineDElogs.log.$$

      The packages required for the BMC Deployment Engine components are downloaded and a completion message is displayed.

    4. To ensure that all files are readable, run the following command:

      sudo chmod a+r *

Task 2: To set up BMC Deployment Engine on an offline server

  1. Log in to your offline server by using a user with sudo access.
  2. Create git and jenkins users by using the following commands:sudo useradd git -m
    sudo passwd git
    sudo useradd jenkins -m
    sudo passwd jenkins

    Important: Make sure that you do not use special characters in the git user password.

  3. Configure passwordless sudo access for the git user.

    1. Log in to BMC Deployment Engine as a root user or a user with permission to run visudo and to create a sudoers override file at the /etc/sudoers.d location.
    2. Create a sudoers override file such as /<your path to>/git_sudoers.

    3. Add the following details to the file:

      git ALL=(ALL) NOPASSWD: \
         /usr/sbin/alternatives, \
         /usr/bin/cat, \
         /usr/bin/chmod, \
         /usr/bin/chown, \
         /usr/bin/cp, \
         /usr/bin/curl, \
         /usr/bin/dnf, \
         /usr/bin/dos2unix, \
         /usr/bin/firewall-cmd, \
         /usr/bin/grep, \
         /usr/bin/java, \
         /usr/bin/ln, \
         /usr/bin/ls, \
         /usr/bin/mkdir, \
         /usr/bin/mv, \
         /usr/bin/netstat, \
         /usr/bin/rpm, \
         /usr/bin/sed, \
         /usr/bin/su, \
         /usr/sbin/subscription-manager, \
         /usr/bin/systemctl, \
         /usr/bin/unzip, \
         /usr/sbin/update-alternatives, \
         /usr/bin/wget, \
         /usr/bin/yum, \
         /usr/bin/crb, \
         /usr/bin/perl, \
         /usr/bin/updatedb, \
         /usr/bin/ansible-config , \
        /usr/bin/crb, \
        /usr/bin/echo, \
        /usr/bin/find, \
        /usr/bin/yumdownloader , \
        /usr/bin/sh, \
        /usr/bin/updatedb, \
        /usr/bin/locate

    4. Verify that the file has no errors by running the following command:

      visudo -c -f /<your path to>/git_sudoers

      If the output is parsed OK, the file is valid.

    5. Identify the sudoers override file location from visudo by running the following command:

      visudo

      Search for the #includedir parameter in the command output:

      ## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
      #includedir /etc/sudoers.d

      In the example, /etc/sudoers.d is the location where you must copy the sudoers override file.

    6. Copy the sudoers override file to the location found in visudo by using the following command:

      cp <override file> <#includedir location>

      Example:

      cp /path/git_sudoers /etc/sudoers.d/git_sudoers

    7. Modify access to sudoers file by using the following command:

      chmod 440  <#includedir location> 

      Example:

      chmod 440 /etc/sudoers.d/git_sudoers

    8. Switch the user to git user, and validate passwordless sudo access by running the following command:

      sudo ls /root

      The command works without prompting for the password.

  4. Copy the ssh keys.

    1. Log in to BMC Deployment Engine as a git user.

    2. Configure the SSH keys for the git user to enable pipeline access to Git repositories.

      1. Generate the SSH key by running the following command and accept all the defaults.

        ssh-keygen

      2. Copy the ID by running the following command:

        ssh-copy-id git@<jenkins_server_name>

      3. Verify that the passwordless ssh login works from git to git user by using the following command:

        ssh git@<jenkins_server_name>
    3. Log in to BMC Deployment Engine as a jenkins user.
    4. Configure jenkins ssh keys to git so that the pipeline dry-run step is successful.

      1. Generate the SSH key by running the following command and accept all the defaults.

        ssh-keygen

      2. Copy the ID by running the following command:

        ssh-copy-id git@<jenkins_server_name>

      3. Verify that you are logged in as git user and the passwordless ssh login works by using the following command:

        ssh git@<jenkins_server_name>
  5. Copy the airgap_setup folder from the online server to offline server.
    1. Log in to the offline server as a git user.

    2. In the git user home directory, create the airgap_setup folder by using the following command:

      mkdir -p /home/git/airgap_setup

    3. Copy the airgap_setup folder from online server to offline server.
      Example commands to copy the airgap_setup folder:

      cd /home/git/airgap_setup
      scp -r <git user>@<online server>:<git user home dir path>/airgap_setup/* .
  6. Run the BMC Deployment Engine offline setup script:

    1. Provide permissions to the airgap_setup folder by using the following commands:

      sudo chmod -R 755 <airgap_setup folder location>
      sudo chown -R git:git <airgap_setup folder location>

    2. In the airgapped-build.properties file, in the JENKINS_HOSTNAME parameter, specify the fully qualified host name or IP address where offline server setup is planned.

    3. Run the BMC Deployment Engine offline setup script by using the following commands:

      Important

      Do not run the command with full sudo access. Make sure that you run the command by using a non-root user.

      cd <airgap_setup folder location>
      perl offline-airgapped-setup-Helix-ITSM-onPrem.pl  2>&1 | tee ~/OfflineDElogs.log.$$

      Make sure that you type YES to accept the license terms. This command creates the nohup.out file in the current folder.

    4. Review the logs for any errors.

Task 3: To verify BMC Deployment Engine components

  1. Verify that the following components are installed:

    Component

    Command to verify

    Ansible

    ansible --version

    Java

    java --version

    Helm

    helm version

    kubectl

    kubectl version

  2. Verify the Python version.

    1. Check the Python version that Ansible refers by using the following command:
      anisble --version

      Example command output:
      ansible [core 2.15.3]
        config file = /etc/ansible/ansible.cfg
        configured module search path = ['/home/git/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
        ansible python module location = /usr/lib/python3.11/site-packages/ansible
        ansible collection location = /home/git/.ansible/collections:/usr/share/ansible/collections
        executable location = /usr/bin/ansible
      python version = 3.11.11 (main, Dec  9 2024, 15:32:27) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] (/usr/bin/python3.11)
        jinja version = 3.1.2
        libyaml = True
       
    2. Check the Python version by using the following command:
      python --version

    3. If the Python versions in step 1 and step 2 do not match, update the Python version to the version that Ansible refers by using the following command:
      sudo update-alternatives --config python

      Example command output:
       Selection    Command
      -----------------------------------------------
      *  1           /usr/libexec/no-python
        2           /usr/bin/python3
      + 3           /usr/bin/python3.11
      Enter to keep the current election[+], or type selection number:
      For example, to select Python 3.11, enter 3.

Task 4: To perform postinstallation configurations

Complete the following configurations:

No.

Actions

Steps

1

Log in to the Jenkins sever

  1. Log in to Jenkins server by using the following URL:
    http://<Jenkins server host name>:<Jenkins port>
    The default password is available in $JENKINS_INSTALL_DIR/secrets/initialAdminPassword file.
  2. Click Skip Plugins Installations.
  3. In the Admin User Creation wizard, provide a preferred user name and password and then run the wizard.
  4. Click Save and Finish.
    Jenkins is ready for use.

2

Install plug-ins

  1. Log in to Jenkins server by using the following URL:
    http://<Jenkins server host name>:<Jenkins port>
  2. On the Jenkins User Interface, click Skip Plugins Installations.

  3. Copy the kubeconfig.yaml file to <Git user home directory>/.kube/config.

    cp <kubeconfig complete file path> $HOME/.kube/config

3

To update the Jenkins credentials

  1. Log in to the Jenkins server by using the following URL:
    http://<Jenkins server host name>:<Jenkins port>/credentials
  2. Add the credentials in the ID: kubeconfig file.
    1. Select kubeconfig and click the name of the kubeconfig credential.
      JenkinsCredentials.png
    2. Click Update.
    3. Select the Replace checkbox.
    4. Click Choose File and select the kubeconfig file
    5. Click Save.
  3. Add the credentials in the ID: github account.
    1. Select github and click the name of the github credential.
    2. Click Update.
    3. Click Change Password and enter the git user password.
    4. Click Save.
  4. Add the credentials in the ID: ansible_host account.
    1. Select ansible_host and click the name of the ansible_host credential
    2. Click Update.
    3. Click Change Password and enter the git user password.
    4. Click Save.
  5. Add the credentials in the ID: ansible file.
    1. Select ansible and click the name of the ansible credential.
    2. Click Update.
    3. Click Change Password and enter the git user password. 
    4. Click Save.
  6. Add credentials in the ID: git account.
    1. Select git and click the name of the git credential.
    2. Click Update.
    3. Click Change Password and enter the git user password.
    4. Click Save.
  7. (For cloud environments) Update git user private key credentials
    1. Navigate to http://<Jenkins server host name>:<Jenkins port>/credentials/store/system/domain/_/
    2. Click Add Credentials.  
    3. Enter the following details:
      • Kind—SSH user name with private key
      • ID—git_pk
      • Username—git
      • Private Key—Click Enter and paste the contents of git user private key value (/home/git/.ssh/id_rsa)
    4. Save the details.

4

Update the node configuration in a cloud environment

  1. Navigate to http://<Jenkins server host name>:<Jenkins port>/computer.
  2. Perform the following steps on both nodes, the one with the actual host name and another with the git host name.
    1. Click the node name. 
    2. Click Configure.
    3. Under Credentials, select the git user ssh key credential and click Save.
    4. Click Launch Agent.

Update the node configuration in a non cloud environment

  1. Navigate to http://<Jenkins server host name>:<Jenkins port>/computer.
  2. Perform the following steps on both nodes, the one with actual hostname and another with git-hostname.
    1. Click the node name.
      Important: If the status of the node is already 'online', then skip step b.
    2. Click Launch Agent.

5

Add the Jenkins libraries

Perform the following steps to add the pipeline-framework library:

  1. On the Jenkins home page, click Manage Jenkins.
  2. Select Configure System.
  3. In Global Pipeline Libraries, add the pipeline-framework library as shown in the following figure:pipelineframework_1.png
  4. In the Project Repository field, specify the complete path of pipeline-framework.git according to the environment.
    Fetch the <GIT_REPO_DIR> value from the build.properties file.
  5. Make sure you provide the exact location of pipeline-framework.git according to the environment.
  6. Add JENKINS-27413-workaround-library as shown in the following figure:JenkinsLibrary_1.png
    1. In the Project Repository field, specify the complete path of JENKINS-27413-workaround-library.git according to the environment.
    2. Select Load implicitly.
  7. Save the changes.

Important: After you complete the BMC Deployment Engine configurations, remove the sudo access of the Git user by using the following command:

gpasswd --delete git wheel

6

Run the deployment pipelines in dry-run mode

Dry-run is a mandatory step to update the pipeline configuration for any changes to the BMC Helix Innovation Suite and Service Management Installer.

Important: Even though you see the Build with Parameters option for all the parameters, you have to still perform a dry-run.

  1. Navigate to Jenkins Dashboards to view all the pipelines required for deployment.
  2. Select each pipeline and click Build with Parameters.
  3. In the AGENT parameter, provide the value of the node that has the name git-<hostname>.
    The AGENT parameter value is the Jenkins agent that runs the pipeline.
  4. Click Build.
    The build job will fail, which is expected.

Important:

  • The agent-add-pipeline and HELIX_DR pipelines do not require a dry-run.
  • For HELIX_ONPREM_DEPLOYMENT pipeline, enter the value of the AGENT parameter.
  • Do not select any pipelines in the PRODUCT_DEPLOY section.
  • For the other pipelines, run Build with Parameters with default values present in the pipeline.

Troubleshooting

Refer to the troubleshooting information if you encounter any of the following issues while setting up BMC Deployment Engine.

Issue symptom

During deployment engine script execution, you might receive the following message:

Container startup failed 
java.io.FileNotFoundException: /var/cache/jenkins/war/META-INF/MANIFEST.MF (No such file or directory)

Example:

Feb 10 09:33:12 clm-aus-wnmmad jenkins[272829]: #011at executable.Main.main(Main.java:335)
Feb 10 09:33:12 clm-aus-wnmmad jenkins[272829]: 2025-02-10 07:33:12.465+0000 [id=1]#011SEVERE#011winstone.Logger#logInternal: Container startup failed
Feb 10 09:33:12 clm-aus-wnmmad jenkins[272829]: java.io.FileNotFoundException: /var/cache/jenkins/war/META-INF/MANIFEST.MF (No such file or directory)
Feb 10 09:33:12 clm-aus-wnmmad jenkins[272829]: #011at java.base/java.io.FileOutputStream.open0(Native Method)
Feb 10 09:33:12 clm-aus-wnmmad jenkins[272829]: #011at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293)
Feb 10 09:33:12 clm-aus-wnmmad jenkins[272829]: #011at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:235)

Resolution

Run the following commands:

mkdir -p /var/cache/jenkins/war
chown -R jenkins:jenkins  /var/cache/jenkins
 iptables -F
systemctl restart NetworkManager

systemctl restart jenkins

Where to go from here

Next task

Back to process

If you are finished installing BMC Helix Platform services, return to the appropriate installation or upgrade process:

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*