ETL in the Atrium Integrator adapter

The Atrium Integrator adapter data flow occurs in an Extraction, Transformation, and Loading (ETL) process, or a transformation. The following image shows the components of this process in the Spoon client:

• Transformations—A transformation is a combination of steps and hops that make the integration between source and target data stores possible. It reads data from the source data store, transforms it according to the rules that you specify, and stores it in the target data store. Transformation may be simple or complex. Transformations are saved with the .ktr filename extension.

• Jobs—A job performs high-level data flow control. A job is composed of one or more transformations and it executes the transformations. Jobs are used for coordinating ETL activities such as defining the flow and dependencies for the order in which the transformations should run, or preparing for execution by checking conditions such as, Is my source file available? or Does a table exists in my database? Jobs are saved with the .kjb filename extension.

• Steps—A step is the basic unit of a transformation that performs an action such as data loading. Groups of steps define input and output stores. Steps are organized into categories such as Input steps or Output steps. Steps also exist in jobs.

• Hops—A hop shows data flowing from one step to another in a pictorial representation of a transformation in the Spoon client. The data in a hop originates at an Input step and is destined for an Output step.

AR System repository

Transformations, jobs, and their related components are stored in the AR System form repository, which can be accessed using the Spoon client.

Form repository

When you save a transformation, the form repository starts and commits the client-managed transaction. If the transformation already exists, the data related to the transformation (such as, transformation attributes) is cleared from all related transformation forms. In addition, it saves:

• All database connections and database connection attribute information
• All transformation hops
• All transformation notes
• All step and step attribute information
• Error step information
• Transformation:
  • setting information
  • parameter information
  • slave server information
  • cluster schema information
  • dependency information

When you save a job, the form repository starts and commits the client-managed transaction. If the job already exists, the data related to the job (such as, job attributes) is cleared from all related job forms. In addition, it saves:

• All database connections
• All job entries and job entry attributes
• All job hops
• All job notes
• Job settings information
• Slave server information

When you load a transformation into the Spoon client, the form repository reads all:

• Step and step attributes
• Transformation attributes
• Transformation hops
• Transformation notes
• Transformation settings
• Database connections, slave servers, cluster schemas, and partition schemas

When you load a form into the Spoon client, the form repository reads all:

• Job attributes
• Job entries and job entry attributes
• Job hops
• Job notes
• Job parameters
• Job settings
• Database connections, slave servers, cluster schemas, and partition schemas