Control-M for Azure Data Factory

Azure Data Factory is a cloud-based ETL and data integration service that allows you to create data-driven workflows to automate the movement and transformation of data. 

Control-M for Azure Data Factory enables you to do the following:

• Connect to any Azure Data Factory pipeline from a single computer with secure login, which eliminates the need to provide authentication.
• Integrate Azure Data Factory jobs with other Control-M jobs into a single scheduling environment.
• Monitor the pipeline status and view the results in the Monitoring domain.
• Attach an SLA job to your entire Azure data service.
• Introduce all Control-M capabilities to Azure Data Factory, including advanced scheduling criteria, complex dependencies, quantitative and control resources, and variables.
• Run 50 Azure Data Factory jobs simultaneously per Control-M/Agent.

Control-M can connect to Azure Data Factory using one of the following authentication methods:

• Managed Identity: Enables you to access other Azure AD-protected resources. The identity is managed by the Azure platform and does not require you to provide additional credentials within Control-M. Use this option if the Agent is installed on an Azure virtual machine that has an assigned Managed Identity with the required permissions.
• Service Principal: An Azure service principal, also known as App Registration, is an identity created for use with applications, hosted services, and automated tools to access Azure resources. This access is restricted by the roles assigned to the service principal, which gives the Azure Administrator control over which resources can be accessed and at which level. You can use this option if the Agent is installed on-premises, on an Azure virtual machine, or any other cloud vendor.
  

Setting up Control-M for Azure Data Factory

This procedure describes how to install the Azure Data Factory plug-in, create a connection profile, and define an Azure Data Factory job in Helix Control-M and Automation API.

Before you Begin

• Verify that Automation API is installed, as described in Setting up the API.
• Verify that Agent version 9.0.21.080 or later is installed.

Begin

1. On the Agent host, set the Java environment variable by running one of the following commands through a command line:
   
   • Linux:
     • Bourne shell/bash: export BMC_INST_JAVA_HOME=<java_11_directory>
     • csh/tcsh: setenv BMC_INST_JAVA_HOME <java_11_directory>
   • Windows: set BMC_INST_JAVA_HOME="<java_11_directory>"
2. Run one of the following API commands:
   • For a fresh installation, use the provision image command:
     • Linux: ctm provision image ADF_plugin.Linux
     • Windows: ctm provision image ADF_plugin.Windows
   • For an upgrade, use the following command:
     ctm provision agent::update
3. Create an Azure Data Factory connection profile in Helix Control-M or Automation API, as follows:
   • Helix Control-M: Creating a Centralized Connection Profile with Azure Data Factory Connection Profile Parameters
   • Automation API:  ConnectionProfile:ADF
4. Define an Azure Data Factory job in Helix Control-M or Automation API, as follows:
   • Helix Control-M: Create a Job and then define specific ADF parameters in Azure Data Factory Job parameters.
   • Automation API: Job:ADF