Skip to Content

Control-M for Azure Databricks

Azure Databricks is a cloud-based data analytics platform that enables you to process large workloads of data.

Control-M for Azure Databricks enables you to do the following:

  • Execute Azure Databricks jobs.
  • Manage Azure Databricks credentials in a secure connection profile.
  • Connect to any Azure Databricks endpoint.
  • Integrate Azure Databricks jobs with other Control-M jobs into a single scheduling environment.
  • Monitor the status, results, and output of Azure Databricks jobs in the Monitoring domain.
  • Attach an SLA job to your Azure Databricks jobs.
  • Introduce all Control-M capabilities to Control-M for Azure Databricks including advanced scheduling criteria, complex dependencies, Resource Pools, Lock Resources, and variables.
  • Run 50 Azure Databricks jobs simultaneously per Agent.

Control-M for Azure Databricks Compatibility

The following table lists the prerequisites that are required to use the Azure Databricks plug-in, each with its minimum required version.

Component

Version

Control-M/EM

9.0.20.200

Control-M/Agent

9.0.20.200

Control-M Application Integrator

9.0.20.201

Control-M Automation API

9.0.20.235

Control-M for Azure Databricks is supported on Control-M Web and Control-M Automation API, but not on Control-M client.

To download the required installation files for each prerequisite, see Obtaining-Control-M-Installation-Files.

Setting up Control-M for Azure Databricks

This procedure describes how to deploy the Azure Databricks plug-in, create a connection profile, and define an Azure Databricks job in Control-M Web and Automation API.

Warning

Note

Integration plug-ins released by BMC require an Application Integrator installation. However, these plug-ins are not editable and you cannot import them into Application Integrator. To deploy these integrations to your Control-M environment, import them directly into Control-M using Control-M Automation API.

Before You Begin

Verify that Automation API is installed, as described in Automation API Installation.

Begin

  1. Create a temporary directory to save the downloaded files.
  2. Download the Azure Databricks plug-in from the Control-M for Azure Databricks download page in the EPD site.
  3. Install the Azure Databricks plug-in via one of the following methods:
    • (9.0.21 or higher) Use the Automation API Provision service:
      1. Log in to the Control-M/EM Server machine as an Administrator and store the downloaded zip file in the one of the following locations (within several minutes, the job type appears in Control-M Web):
        • Linux: $HOME/ctm_em/AUTO_DEPLOY
        • Windows: <EM_HOME>\AUTO_DEPLOY
      2. Log in to the Control-M/Agent machine and run the provision image command, as follows:
        • Linux: ctm provision image ZDX_plugin.Linux
        • Windows: ctm provision image ZDX_plugin.Windows
    • (9.0.20.200 or lower) Use the Automation API Deploy service, as described in deploy jobtype.
  4. Create an Azure Databricks connection profile in Control-M Web or Automation API, as follows:
  5. Define an Azure Databricks job in Control-M Web or Automation API, as follows:
Warning

Note

To remove this plug-in from an Agent, see Removing a Plug-in. The plug-in ID is ZDX112021.

Change Log

The following table provides details about changes that were introduced in new versions of this plug-in:

Plug-in Version

Details

1.0.00

Initial release

1.0.01

Multiple task enhancement

1.0.02

Idempotency enhancement

1.0.03

New job icon.

1.0.04

Removal of the Job Name attribute.

1.0.05

Failure Tolerance job parameter added.

1.0.06

Managed Identity authentication added.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*

Control-M