Control-M for Azure Data Factory

Azure Data Factory is a cloud-based extract, transform, load (ETL) and data integration service that allows you to create data-driven workflows to automate the movement and transformation of data. 

Control-M for Azure Data Factory enables you to do the following:

• Execute Azure Data Factory jobs.
• Manage Azure Data Factory credentials in a secure connection profile.
• Connect to any Azure Data Factory endpoint.
• Integrate Azure Data Factory jobs with other Control-M jobs into a single scheduling environment.
• Monitor the status, results, and output of Azure Data Factory jobs in the Monitoring domain.
• Attach an SLA job to your Azure Data Factory jobs.
• Introduce all Control-M capabilities to Control-M for Azure Data Factory including advanced scheduling criteria, complex dependencies, Resource Pools, Lock Resources, and variables.
• Run 50 Azure Data Factory jobs simultaneously per Agent.

Control-M for Azure Data Factory Compatibility

The following table lists the prerequisites that are required to use the Azure Data Factory plug-in, each with its minimum required version.

Component	Version
Control-M/EM	9.0.20.200
Control-M/Agent	9.0.20.200
Control-M Application Integrator	9.0.20.201
Control-M Automation API	9.0.20.200

Control-M for Azure Data Factory is supported on Control-M Web and Control-M Automation API, but not on Control-M client.

To download the required installation files for each prerequisite, see Obtaining-Control-M-Installation-Files.

Setting up Control-M for Azure Data Factory

This procedure describes how to deploy the Azure Data Factory plug-in, create a connection profile, and define an Azure Data Factory job in Control-M Web and Automation API.

Before You Begin

Verify that Automation API is installed, as described in Automation API Installation.

Begin

1. Create a temporary directory to save the downloaded files.
2. Download the Azure Data Factory plug-in from the Control-M for Azure Data Factory download page in the EPD site.
3. Install the Azure Data Factory plug-in via one of the following methods:
   • (9.0.21 or higher) Use the Automation API Provision service:
     1. Log in to the Control-M/EM Server machine as an Administrator and store the downloaded zip file in the one of the following locations (within several minutes, the job type appears in Control-M Web):
        • Linux: $HOME/ctm_em/AUTO_DEPLOY
        • Windows: <EM_HOME>\AUTO_DEPLOY
     2. Log in to the Control-M/Agent machine and run the provision image command, as follows:
        • Linux: ctm provision image ADF_plugin.Linux
        • Windows: ctm provision image ADF_plugin.Windows
   • (9.0.20.200 or lower) Use the Automation API Deploy service, as described in deploy jobtype.
4. Create an Azure Data Factory connection profile in Control-M Web or Automation API, as follows:
   • Web: Creating a Centralized Connection Profile with Azure Data Factory Connection Profile Parameters
   • Automation API: ConnectionProfile:ADF
5. Define an Azure Data Factory job in Control-M Web or Automation API, as follows:
   • Web: Creating a Job with Azure Data Factory Job parameters
   • Automation API: Job:ADF

Change Log

The following table provides details about changes that were introduced in new versions of this plug-in:

Plug-in Version	Details
1.0.02	New job icon
1.0.03	Fixes in the abort operation for REST API steps.
1.0.04	Failure Tolerance parameter added. Connection timeout override applied to all execution steps. Polling interval default value changed to 45.
1.0.05	Added resolve on rerun for job parameters.