Control-M for AWS Data Pipeline

AWS Data Pipeline is a cloud-based ETL service that you can use to automate the transfer, processing, and storage of your data.

Control-M for AWS Data Pipeline enables you to do the following:

• Connect to any AWS Data Pipeline endpoint from a single computer with secure login, which eliminates the need to provide authentication.
• Integrate AWS Data Pipeline  jobs with other Control-M jobs into a single scheduling environment.
• Trigger AWS Data Pipeline jobs based on ETL processing.
• Attach an SLA job to your entire pipeline.
• Introduce all Control-M capabilities to AWS Data Pipeline, including advanced scheduling criteria, complex dependencies, quantitative and control resources, and variables.
• Run 50 AWS Data Pipeline jobs simultaneously per Control-M/Agent.

Control-M for AWS Data Pipeline Compatibility

The following table lists the prerequisites that are required to use the AWS Data Pipeline plug-in, each with its minimum required version.

Control-M for AWS Data Pipeline is supported on Control-M Web and Control-M Automation API, but not on Control-M client.

To download the required installation files for each prerequisite, see Obtaining Control-M Installation Files.

Setting up Control-M for AWS Data Pipeline

This procedure describes how to deploy the AWS Data Pipeline plug-in, create a connection profile, and define an AWS Data Pipeline job in Control-M Web and Automation API.

NOTE: Integration plug-ins released by BMC require an Application Integrator installation at your site. However, these plug-ins are not editable and you cannot import them into Application Integrator. To deploy these integrations to your Control-M environment, you import them directly into Control-M using Control-M Automation API.

Before you Begin

Verify that Automation API is installed, as described in Automation API Installation.

Begin

1. Create a temporary directory to save the downloaded files.

2. Click http://www.bmc.com/available/epd and follow the instructions on the EPD site to download the AWS Data Pipeline plug-in from the Control-M Integrations page, or go directly to the Control-M for AWS Data Pipeline download page.

3. Deploy the AWS Data Pipeline job via Automation API, as described in deploy jobtype.
4. Create an AWS Data Pipeline connection profile in Control-M Web or Automation API, as follows:
   • Web: Creating a Centralized Connection Profile with AWS Data Pipeline Connection Profile Parameters
   • Automation API: ConnectionProfile:AWS Data Pipeline
5. Define an AWS Data Pipeline job in Control-M Web or Automation API, as follows:
   • Web: Create a Job and then define specific AWS Data Pipeline parameters in AWS Data Pipeline Job parameters.
   • Automation API: Job:AWS Data Pipeline