Job types

The following series of sections provide details about the available types of jobs that you can define and the parameters that you use to define each type of job.

Job:Command

The following example shows how to use the Job:Command to run operating system commands.

	"JobName": {
		"Type" : "Job:Command",
    	"Command" : "echo hello",
        "PreCommand": "echo before running main command",
        "PostCommand": "echo after running main command",
    	"Host" : "myhost.mycomp.com",
    	"RunAs" : "user1"  
	}

_{Back to top}

Job:Script

 The following example shows how to use Job:Script to run a script from a specified script file.

    "JobWithPreAndPost": {
        "Type" : "Job:Script",
        "FileName" : "task1123.sh",
        "FilePath" : "/home/user1/scripts",
        "PreCommand": "echo before running script",
        "PostCommand": "echo after running script",
        "Host" : "myhost.mycomp.com",
        "RunAs" : "user1"   
    }

_{Back to top}

Job:EmbeddedScript

The following example shows how to use Job:EmbeddedScript to run a script that you include in the JSON code. Control-M deploys this script to the Agent host during job submission. This eliminates the need to deploy scripts as part of your application stack.

    "EmbeddedScriptJob":{
        "Type":"Job:EmbeddedScript",
        "Script":"#!/bin/bash\\necho \"Hello world\"",
        "Host":"myhost.mycomp.com",
        "RunAs":"user1",
        "FileName":"myscript.sh",
        "PreCommand": "echo before running script",
        "PostCommand": "echo after running script"
    }

_{Back to top}

Job:FileTransfer

The following example shows a Job:FileTransfer.

{
  "FileTransferFolder" :
  {
	"Type" : "Folder",
	"Application" : "aft",
	"TransferFromLocalToSFTP" :
	{
		"Type" : "Job:FileTransfer",
		"ConnectionProfileSrc" : "LocalConn",
		"ConnectionProfileDest" : "SftpConn",
		"Host": "AgentHost",
		"FileTransfers" :
		[
			{
				"Src" : "/home/controlm/file1",
				"Dest" : "/home/controlm/file2",
				"TransferType": "Binary",
				"TransferOption": "SrcToDest"
			},
			{
				"Src" : "/home/controlm/otherFile1",
				"Dest" : "/home/controlm/otherFile2",
				"TransferOption": "DestToSrc"
			}
		]
	}
  }
}

Where:

The following example presents a File Transfer job in which the transferred file is watched using the File Watcher utility:

{
  "FileTransferFolder" :
  {
	"Type" : "Folder",
	"Application" : "aft",
	"TransferFromLocalToSFTPBasedOnEvent" :
	{
		"Type" : "Job:FileTransfer",
		"Host" : "AgentHost",
		"ConnectionProfileSrc" : "LocalConn",
		"ConnectionProfileDest" : "SftpConn",
		"FileTransfers" :
		[
			{
				"Src" : "/home/sftp/file1",
				"Dest" : "/home/sftp/file2",
				"TransferType": "Binary",
				"TransferOption" : "SrcToDestFileWatcher",
				"PreCommandDest" :
				{
					"action" : "rm",
					"arg1" : "/home/sftp/file2"
				},
				"PostCommandDest" :
				{
					"action" : "chmod",
					"arg1" : "700",
					"arg2" : "/home/sftp/file2"
				},
				"FileWatcherOptions":
				{
					"MinDetectedSizeInBytes" : "200",
					"TimeLimitPolicy" : "WaitUntil",
					"TimeLimitValue" : "2000",
					"MinFileAge" : "3Min",
					"MaxFileAge" : "10Min",
					"AssignFileNameToVariable" : "FileNameEvent",
					"TransferAllMatchingFiles" : true
				}
			}
		]
	}
  }
}

This example contains the following additional optional parameters: 

_{Back to top}

Job:FileWatcher

A File Watcher job enables you to detect the successful completion of a file transfer activity that creates or deletes a file. The following example shows how to manage File Watcher jobs using Job:FileWatcher:Create and Job:FileWatcher:Delete. 

    "FWJobCreate" : {
	    "Type" : "Job:FileWatcher:Create",
		"RunAs":"controlm",
 	    "Path" : "C:/path*.txt",
	    "SearchInterval" : "45",
	    "TimeLimit" : "22",
	    "StartTime" : "201705041535",
	    "StopTime" : "201805041535",
	    "MinimumSize" : "10B",
	    "WildCard" : true,
	    "MinimalAge" : "1Y",
	    "MaximalAge" : "1D2H4MIN"
    },
    "FWJobDelete" : {
        "Type" : "Job:FileWatcher:Delete",
        "RunAs":"controlm",
        "Path" : "C:/path.txt",
        "SearchInterval" : "45",
        "TimeLimit" : "22",
        "StartTime" : "201805041535",
        "StopTime" : "201905041535"
    }

This example contains the following parameters:

_{Back to top}

Job:Database

The following types of database jobs are available:

• SQL Script job, using Job:Database:SQLScript
• Stored Procedure job, using Job:Database:StoredProcedure

Job:Database:SQLScript

The following example shows how to create a database job that runs a SQL script from a file system.

{
	"OracleDBFolder": {
		"Type": "Folder",
		"testOracle": {
			"Type": "Job:Database:SQLScript",
			"Host": "AgentHost",
			"SQLScript": "/home/controlm/sqlscripts/selectOrclParm.sql",
			"ConnectionProfile": "ORACLE_CONNECTION_PROFILE",
			"Parameters": [
				{"firstParamName": "firstParamValue"},
				{"secondParamName": "secondParamValue"}
			],
			"Autocommit": "N",
			"OutputExcecutionLog": "Y",
			"OutputSQLOutput": "Y",
			"SQLOutputFormat": "XML"
		}
	}
}

This example contains the following parameters:

Another example:

{
	"OracleDBFolder": {
		"Type": "Folder",
		"testOracle": {
			"Type": "Job:Database:SQLScript",
			"Host": "app-redhat",
			"SQLScript": "/home/controlm/sqlscripts/selectOrclParm.sql",
			"ConnectionProfile": "ORACLE_CONNECTION_PROFILE"
		}
	}
}

Job:Database:StoredProcedure

The following example shows how to create a database job that runs a program that is stored on the database.

{
	"storeFolder": {
		"Type": "Folder",
		"jobStoredProcedure": {
			"Type": "Job:Database:StoredProcedure",
			"Host": "myhost.mycomp.com",
			"StoredProcedure": "myProcedure",
			"Parameters": [ "value1","variable1",["value2","variable2"]],
			"ReturnValue":"RV",
			"Schema": "public",
			"ConnectionProfile": "DB-PG-CON",
			"Autocommit": "N",
			"OutputExcecutionLog": "Y",
			"OutputSQLOutput": "Y",
			"SQLOutputFormat": "XML"
		}
	}
}

This example contains the following parameters:

_{Back to top}

Job:Hadoop

Various types of Hadoop jobs are available for you to define using the Job:Hadoop objects:

• Spark Python
• Spark Scala or Java
• Pig
• Sqoop
• Hive
• DistCp (distributed copy)
• HDFS commands
• HDFS File Watcher
• Oozie
• MapReduce
• MapReduce Streaming

Job:Hadoop:Spark:Python

The following example shows how to use Job:Hadoop:Spark:Python to run a Spark Python program.

    "ProcessData": {
        "Type": "Job:Hadoop:Spark:Python",
        "Host" : "edgenode",
        "ConnectionProfile": "DEV_CLUSTER",

        "SparkScript": "/home/user/processData.py"
    }

Optional parameters:

 "ProcessData1": {
    "Type": "Job:Hadoop:Spark:Python",
    "Host" : "edgenode",
    "ConnectionProfile": "DEV_CLUSTER",
    "SparkScript": "/home/user/processData.py",            
    "Arguments": [
        "1000",
        "120"
    ],            

    "PreCommands": {
        "FailJobOnCommandFailure" :false,
        "Commands" : [
            {"get" : "hdfs://nn.example.com/user/hadoop/file localfile"},
            {"rm"  : "hdfs://nn.example.com/file /user/hadoop/emptydir"}
        ]
    },

    "PostCommands": {
        "FailJobOnCommandFailure" :true,
        "Commands" : [
            {"put" : "localfile hdfs://nn.example.com/user/hadoop/file"}
        ]
    },

    "SparkOptions": [
        {"--master": "yarn"},
        {"--num":"-executors 50"}
    ]
 }

_{Back to top}

Job:Hadoop:Spark:ScalaJava

The following example shows how to use Job:Hadoop:Spark:ScalaJava to run a Spark Java or Scala program.

 "ProcessData": {
  	"Type": "Job:Hadoop:Spark:ScalaJava",
    "Host" : "edgenode",
    "ConnectionProfile": "DEV_CLUSTER",

    "ProgramJar": "/home/user/ScalaProgram.jar",
	"MainClass" : "com.mycomp.sparkScalaProgramName.mainClassName" 
}

Optional parameters:

 "ProcessData1": {
  	"Type": "Job:Hadoop:Spark:ScalaJava",
    "Host" : "edgenode",
    "ConnectionProfile": "DEV_CLUSTER",

    "ProgramJar": "/home/user/ScalaProgram.jar"
	"MainClass" : "com.mycomp.sparkScalaProgramName.mainClassName",

    "Arguments": [
        "1000",
        "120"
    ],            

    "PreCommands": {
        "FailJobOnCommandFailure" :false,
        "Commands" : [
            {"get" : "hdfs://nn.example.com/user/hadoop/file localfile"},
            {"rm"  : "hdfs://nn.example.com/file /user/hadoop/emptydir"}
        ]
    },

    "PostCommands": {
        "FailJobOnCommandFailure" :true,
        "Commands" : [
            {"put" : "localfile hdfs://nn.example.com/user/hadoop/file"}
        ]
    },

    "SparkOptions": [
        {"--master": "yarn"},
        {"--num":"-executors 50"}
    ]
 }

_{Back to top}

Job:Hadoop:Pig

The following example shows how to use Job:Hadoop:Pig to run a Pig script.

"ProcessDataPig": {
    "Type" : "Job:Hadoop:Pig",
    "Host" : "edgenode",
    "ConnectionProfile": "DEV_CLUSTER",

    "PigScript" : "/home/user/script.pig" 
}

Optional parameters:

    "ProcessDataPig1": {
        "Type" : "Job:Hadoop:Pig",
        "ConnectionProfile": "DEV_CLUSTER",
        "PigScript" : "/home/user/script.pig",            
        "Host" : "edgenode",
        "Parameters" : [
            {"amount":"1000"},
            {"volume":"120"}
        ],            
        "PreCommands": {
            "FailJobOnCommandFailure" :false,
            "Commands" : [
                {"get" : "hdfs://nn.example.com/user/hadoop/file localfile"},
                {"rm"  : "hdfs://nn.example.com/file /user/hadoop/emptydir"}
            ]
        },
        "PostCommands": {
            "FailJobOnCommandFailure" :true,
            "Commands" : [
                {"put" : "localfile hdfs://nn.example.com/user/hadoop/file"}
            ]
        }
    }

_{Back to top}

Job:Hadoop:Sqoop

The following example shows how to use Job:Hadoop:Sqoop to run a Sqoop job.

    "LoadDataSqoop":
    {
      "Type" : "Job:Hadoop:Sqoop",
	  "Host" : "edgenode",
      "ConnectionProfile" : "SQOOP_CONNECTION_PROFILE",

      "SqoopCommand" : "import --table foo --target-dir /dest_dir" 
    }

Optional parameters:

    "LoadDataSqoop1" :
    {
        "Type" : "Job:Hadoop:Sqoop",
        "Host" : "edgenode",
        "ConnectionProfile" : "SQOOP_CONNECTION_PROFILE",

        "SqoopCommand" : "import --table foo",
		"SqoopOptions" : [
			{"--warehouse-dir":"/shared"},
			{"--default-character-set":"latin1"}
		],
 
        "SqoopArchives" : "",
        
        "SqoopFiles": "",
        
        "PreCommands": {
            "FailJobOnCommandFailure" :false,
            "Commands" : [
                {"get" : "hdfs://nn.example.com/user/hadoop/file localfile"},
                {"rm"  : "hdfs://nn.example.com/file /user/hadoop/emptydir"}
            ]
        },
        "PostCommands": {
            "FailJobOnCommandFailure" :true,
            "Commands" : [
                {"put" : "localfile hdfs://nn.example.com/user/hadoop/file"}
            ]
        }
    }

_{Back to top}

Job:Hadoop:Hive

The following example shows how to use Job:Hadoop:Hive to run a Hive beeline job.

    "ProcessHive":
    {
      "Type" : "Job:Hadoop:Hive",
      "Host" : "edgenode",
      "ConnectionProfile" : "HIVE_CONNECTION_PROFILE",

      "HiveScript" : "/home/user1/hive.script"
    }

Optional parameters:

   "ProcessHive1" :
    {
        "Type" : "Job:Hadoop:Hive",
        "Host" : "edgenode",
        "ConnectionProfile" : "HIVE_CONNECTION_PROFILE",


        "HiveScript" : "/home/user1/hive.script", 
        "Parameters" : [
            {"ammount": "1000"},
            {"topic": "food"}
        ],

        "HiveArchives" : "",
        
        "HiveFiles": "",
        
        "HiveOptions" : [
            {"hive.root.logger": "INFO,console"}
        ],

        "PreCommands": {
            "FailJobOnCommandFailure" :false,
            "Commands" : [
                {"get" : "hdfs://nn.example.com/user/hadoop/file localfile"},
                {"rm"  : "hdfs://nn.example.com/file /user/hadoop/emptydir"}
            ]
        },
        "PostCommands": {
            "FailJobOnCommandFailure" :true,
            "Commands" : [
                {"put" : "localfile hdfs://nn.example.com/user/hadoop/file"}
            ]
        }
    }

_{Back to top}

Job:Hadoop:DistCp

The following example shows how to use Job:Hadoop:DistCp to run a DistCp job.  DistCp (distributed copy) is a tool used for large inter/intra-cluster copying.

        "DistCpJob" :
        {
            "Type" : "Job:Hadoop:DistCp",
            "Host" : "edgenode",
            "ConnectionProfile": "DEV_CLUSTER",
         
            "TargetPath" : "hdfs://nns2:8020/foo/bar",
            "SourcePaths" :
            [
                "hdfs://nn1:8020/foo/a"
            ]
        }  

Optional parameters:

    "DistCpJob" :
    {
        "Type" : "Job:Hadoop:DistCp",
        "Host" : "edgenode",
        "ConnectionProfile" : "HADOOP_CONNECTION_PROFILE",
        "TargetPath" : "hdfs://nns2:8020/foo/bar",
        "SourcePaths" :
        [
            "hdfs://nn1:8020/foo/a",
            "hdfs://nn1:8020/foo/b"
        ],
        "DistcpOptions" : [
            {"-m":"3"},
            {"-filelimit ":"100"}
        ]
    }

_{Back to top}

Job:Hadoop:HDFSCommands

The following example shows how to use Job:Hadoop:HDFSCommands to run a job that executes one or more HDFS commands.

        "HdfsJob":
        {
            "Type" : "Job:Hadoop:HDFSCommands",
            "Host" : "edgenode",
            "ConnectionProfile": "DEV_CLUSTER",

            "Commands": [
                {"get": "hdfs://nn.example.com/user/hadoop/file localfile"},
                {"rm": "hdfs://nn.example.com/file /user/hadoop/emptydir"}
            ]
        }

_{Back to top}

Job:Hadoop:HDFSFileWatcher

The following example shows how to use Job:Hadoop:HDFSFileWatcher to run a job that waits for HDFS file arrival.

    "HdfsFileWatcherJob" :
    {
        "Type" : "Job:Hadoop:HDFSFileWatcher",
        "Host" : "edgenode",
        "ConnectionProfile" : "DEV_CLUSTER",

        "HdfsFilePath" : "/inputs/filename",
        "MinDetecedSize" : "1",
        "MaxWaitTime" : "2"
    }

_{Back to top}

Job:Hadoop:Oozie

The following example shows how to use Job:Hadoop:Oozie to run a job that submits an Oozie workflow.

    "OozieJob": {
        "Type" : "Job:Hadoop:Oozie",
        "Host" : "edgenode",
        "ConnectionProfile": "DEV_CLUSTER",


        "JobPropertiesFile" : "/home/user/job.properties",
        "OozieOptions" : [
          {"inputDir":"/usr/tucu/inputdir"},
          {"outputDir":"/usr/tucu/outputdir"}
        ]
    }

Optional parameters:

_{Back to top}

Job:Hadoop:MapReduce

 The following example shows how to use Job:Hadoop:MapReduce to execute a Hadoop MapReduce job.

    "MapReduceJob" :
    {
       "Type" : "Job:Hadoop:MapReduce",
        "Host" : "edgenode",
        "ConnectionProfile": "DEV_CLUSTER",


        "ProgramJar" : "/home/user1/hadoop-jobs/hadoop-mapreduce-examples.jar",
        "MainClass" : "pi",
        "Arguments" :[
            "1",
            "2"]
    }

Optional parameters:

   "MapReduceJob1" :
    {
        "Type" : "Job:Hadoop:MapReduce",
        "Host" : "edgenode",
        "ConnectionProfile": "DEV_CLUSTER",


        "ProgramJar" : "/home/user1/hadoop-jobs/hadoop-mapreduce-examples.jar",
        "MainClass" : "pi",
        "Arguments" :[
            "1",
            "2"
        ],
        "PreCommands": {
            "FailJobOnCommandFailure" :false,
            "Commands" : [
                {"get" : "hdfs://nn.example.com/user/hadoop/file localfile"},
                {"rm"  : "hdfs://nn.example.com/file /user/hadoop/emptydir"}
            ]
        },
        "PostCommands": {
            "FailJobOnCommandFailure" :true,
            "Commands" : [
                {"put" : "localfile hdfs://nn.example.com/user/hadoop/file"}
            ]
        }    
    }

_{Back to top}

Job:Hadoop:MapredStreaming

The following example shows how to use Job:Hadoop:MapredStreaming to execute a Hadoop MapReduce Streaming job.

     "MapredStreamingJob1": {
        "Type": "Job:Hadoop:MapredStreaming",
        "Host" : "edgenode",
        "ConnectionProfile": "DEV_CLUSTER",


        "InputPath": "/user/robot/input/*",
        "OutputPath": "/tmp/output",
        "MapperCommand": "mapper.py",
        "ReducerCommand": "reducer.py",
        "GeneralOptions": [
            {"-D": "fs.permissions.umask-mode=000"},
            {"-files": "/home/user/hadoop-streaming/mapper.py,/home/user/hadoop-streaming/reducer.py"}              
        ]
    }

Optional parameters:

_{Back to top}

Job:SAP

SAP-type jobs enable you to manage SAP processes through the Control-M environment. To manage SAP-type jobs, you must have the Control-M for SAP plugin installed in your Control-M environment.

The following JSON objects are available for creating SAP-type jobs:

• Job:SAP:R3:COPY — Creates an SAP R3 job by copying an existing job
  
• Job:SAP:BW:ProcessChain — Defines a job to run and monitor a Process Chain in SAP Business Warehouse (SAP BW)
  

Job:SAP:R3:COPY

This job type enables you to create a new SAP R3 job by duplicating an existing job. The following example shows how to use Job:SAP:R3:COPY:

"JobSapR3Copy" : {
    "Type" : "Job:SAP:R3:COPY",
    "ConnectionProfile":"SAP-CON",
    "SapJobName" : "CHILD_1",
    "Exec": "Server",
    "Target" : "Server-name",
    "JobCount" : "SpecificJob",
    "JobCountSpecificName" : "sap-job-1234",
    "NewJobName" : "My-New-Sap-Job",
    "StartCondition" : "AfterEvent",
    "AfterEvent" : "HOLA",
    "AfterEventParameters" : "parm1 parm2",
    "RerunFromPointOfFailure": true,
    "CopyFromStep" : "4",
    "PostJobAction" : {
        "Spool" : "CopyToFile",
        "SpoolFile": "spoolfile.log",
        "SpoolSaveToPDF" : true,
        "JobLog" : "CopyToFile",
        "JobLogFile": "Log.txt",
        "JobCompletionStatusWillDependOnApplicationStatus" : true
    },
    "DetectSpawnedJob" : {
        "DetectAndCreate": "SpecificJobDefinition",
        "JobName" : "Specific-Job-123",
        "StartSpawnedJob" : true,
        "JobEndInControlMOnlyAftreChildJobsCompleteOnSap" : true,
        "JobCompletionStatusDependsOnChildJobsStatus" : true
    }
}

This SAP job object uses the following parameters:

Job:SAP:BW:ProcessChain

This job type runs and monitors a Process Chain in SAP Business Warehouse (SAP BW).

NOTE: For the job that you define through Control-M Automation API to work properly, ensure that the Process Chain defined in the SAP BW system has Start Using Meta Chain or API as the start condition for the trigger process (Start Process) of the Process Chain. To configure this parameter, from the SAP transaction RSPC, right-click the trigger process and select Maintain Variant.

The following example shows how to use Job:SAP:BW:ProcessChain:

"JobSapBW": {
    "Type": "Job:SAP:BW:ProcessChain",
    "ConnectionProfile": "PI4-BW",
    "ProcessChainDescription": "SAP BW Process Chain",
    "Id": "123456",
    "RerunOption": "RestartFromFailiurePoint",
    "EnablePeridoicJob": true,
    "ConsiderOnlyOverallChainStatus": true,
    "RetrieveLog": false,
    "DetectSpawnedJob": {
        "DetectAndCreate": "SpecificJobDefinition",
        "JobName": "ChildJob",
        "StartSpawnedJob": false,
        "JobEndInControlMOnlyAftreChildJobsCompleteOnSap": false,
        "JobCompletionStatusDependsOnChildJobsStatus": false
        }
    }

This SAP job object uses the following parameters:

_{Back to top}

Job:ApplicationIntegrator

Use Job:ApplicationIntegrator:<JobType> to define a job of a custom type using the Control-M Application Integrator designer. For information about designing job types, see the Control-M Application Integrator Help.

The following example shows the JSON code used to define a job type named Monitor Remote Job:

"JobFromAI" : {
    "Type": "Job:ApplicationIntegrator:Monitor Remote Job",
    "ConnectionProfile": "AI_CONNECTION_PROFILE",
    "AI-Host": "Host1",
    "AI-Port": "5180",
    "AI-User Name": "admin",
    "AI-Password": "*******",
    "AI-Remote Job to Monitor": "remoteJob5",
    "RunAs": "controlm"
}

In this example, the ConnectionProfile and RunAs properties are standard job properties used in Control-M Automation API jobs. The other job properties will be created in the Control-M Application Integrator, and they must be prefixed with "AI-" in the .json code.

The following images show the corresponding settings in the Control-M Application Integrator, for reference purposes.

_{Back to top}

Job:Informatica

Informatica-type jobs enable you to automate Informatica workflows through the Control-M environment. To manage Informatica-type jobs, you must have the Control-M for Informatica plugin installed in your Control-M environment.

The following example shows the JSON code used to define an Informatica job.

"InformaticaApiJob": {
    "Type": "Job:Informatica",
    "ConnectionProfile": "INFORMATICA_CONNECTION",
    "RepositoryFolder": "POC",
    "Workflow": "WF_Test",
    "InstanceName": "MyInstamce",
    "OsProfile": "MyOSProfile",
    "WorkflowExecutionMode": "RunSingleTask",
    "RunSingleTask": "s_MapTest_Success",
    "WorkflowRestartMode": "ForceRestartFromSpecificTask",
    "RestartFromTask": "s_MapTest_Success",
    "WorkflowParametersFile": "/opt/wf1.prop",
}

This Informatica job object uses the following parameters:

_{Back to top}

Job:Dummy

The following example shows how to use Job:Dummy to define a job that always ends successfully without running any commands. 

"DummyJob" : {
   "Type" : "Job:Dummy"
}

_{Back to top}