Control-M for Azure Data Factory

Azure Data Factory is a cloud-based extract, transform, load (ETL) and data integration service that allows you to create data-driven workflows to automate the movement and transformation of data. 

Control-M for Azure Data Factory enables you to do the following:

• Execute Azure Data Factory jobs.
• Manage Azure Data Factory credentials in a secure connection profile.
• Connect to any Azure Data Factory endpoint.
• Integrate Azure Data Factory jobs with other Control-M jobs into a single scheduling environment.
• Monitor the status, results, and output of Azure Data Factory jobs in the Monitoring domain.
• Attach an SLA job to your Azure Data Factory jobs.
• Introduce all Control-M capabilities to Control-M for Azure Data Factory including advanced scheduling criteria, complex dependencies, Resource Pools, Lock Resources, and variables.
• Run 50 Azure Data Factory jobs simultaneously per Agent.

Setting up Control-M for Azure Data Factory

This procedure describes how to install the Azure Data Factory plug-in, create a connection profile, and define an Azure Data Factory job in Helix Control-M and Automation API.

Before you Begin

• Verify that Automation API is installed, as described in Setting up the API.
• Verify that Agent version 9.0.21.080 or later is installed.

Begin

1. On the Agent host, set the Java environment variable by running one of the following commands through a command line:
   
   • Linux:
     • Bourne shell/bash: export BMC_INST_JAVA_HOME=<java_11_directory>
     • csh/tcsh: setenv BMC_INST_JAVA_HOME <java_11_directory>
   • Windows: set BMC_INST_JAVA_HOME="<java_11_directory>"
2. Run one of the following API commands:
   • For a fresh installation, use the provision image command:
     • Linux: ctm provision image ADF_plugin.Linux
     • Windows: ctm provision image ADF_plugin.Windows
   • For an upgrade, use the following command:
     ctm provision agent::update
3. Create an Azure Data Factory connection profile in Helix Control-M or Automation API, as follows:
   • Helix Control-M: Creating a Centralized Connection Profile with Azure Data Factory Connection Profile Parameters
   • Automation API: ConnectionProfile:ADF
4. Define an Azure Data Factory job in Helix Control-M or Automation API, as follows:
   • Helix Control-M: Creating a Job with Azure Data Factory Job parameters
   • Automation API: Job:ADF

Change Log

The following table provides details about changes that were introduced in new versions of this plug-in:

Plug-in Version	Details
1.0.02	New job icon
1.0.03	Fixes in the abort operation for REST API steps.
1.0.04	Failure Tolerance parameter added. Connection timeout override applied to all execution steps. Polling interval default value changed to 45.
1.0.05	Added resolve on rerun for job parameters.