Page tree
    Skip to end of metadata
    Go to start of metadata

    This topic describes how to configure and use object relationship CSV parser, that allows to describe custom relationships among entities in TrueSight Capacity Optimization.

    Note

    Running this ETL connector creates a hierarchy rule; for more information, see Adding and viewing Hierarchy rules.

    This topic contains the following sections:

    Integration steps

    To integrate TrueSight Capacity Optimization with the Object Relationship CSV parser, perform the following task:

    1. Navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
    2. In the ETL tasks page, click Add > Add ETL under the Last run tab.
    3. In the Add ETL page, set values for the following properties under each expandable tab. 

      Note

      Basic properties are displayed by default in the Add ETL page. These are the most common properties that you can set for an ETL, and it is acceptable to leave the default selections for each as is.

      Basic properties

      Property Description
      Run configuration
      ETL module Select Generic - Object Relationship CSV parser.
      ETL task name Default name is already filled out for you.
      Run configuration name Default name is already filled out for you.
      Deploy status Select Production.
      Description (Optional) Enter a brief description.
      Log level Select how detailed you want the log to be:
      • 1 - Light: Add bare minimum activity logs to the log file.
      • 5 - Medium: Add medium-detailed activity logs to the log file.
      • 10 - Verbose: Add detailed activity logs to the log file.
      Execute in simulation mode Select Yes.
      When set to Yes, the ETL will not store actual data into the data warehouse. This option is useful while testing a new ETL task.
      Module selection Select the required module:
      • Based on datasource: Select this option to manually configure the options for integrating with Object Relationship CSV parser.
      • Based on Open ETL template: Select this option to integrate Object Relationship CSV parser using an Open ETL template. For more details, see Generic ETL based on a template.
      Module description A link that points you to technical documentation for this ETL.
      Datasets
      1. Click Edit.
      2. Select one (click) or more (shift+click) datasets that you want to include from Available datasets and click >> to move them to Selected datasets.
      3. Click Apply.
      Entity catalog
      Sharing status Select any one:
      • PRIVATE: Select this option if this is the only ETL that extracts data from the given set of resources and the lookup table is not shared with the specified ETL task.
      • SHARED: Select this option if more than one ETL extracts data from the given set of resources and the lookup table is shared with the specified ETL task.
      Object relationships
      Associate new entities to

      Specify the domain where you want to add the entities created by the ETL. You can select an existing domain or create a new one.

      Select any one of the following options:

      • New domain: Create a new domain. Specify the following properties under New domain:
        • Parent: Select a parent domain for your new domain from the domain selector control.
        • Name: Specify a name for your new domain.
      • Existing domain: Select an existing domain. Make a selection for the following property:
        • Domain: Select an existing domain from the domain selector control.
      Objrel CSV parser
      Header columns You can either specify the header column structure according to the dataset or can use Find in data option.
      CSV Separator You can either specify the CSV separator structure or can use Find in data option.
      File location
      File location Select any one of the following methods to retrieve the CSV file:
      • Local directory: Specify a path on your local machine where the CSV file resides.
      • Windows share: Specify the Windows share path where the CSV file resides.
      • FTP: Specify the FTP path where the CSV file resides.
      • SCP: Specify the SCP path where the CSV file resides.
      • SFTP: Specify the SFTP path where the CSV file resides.
      Directory Path of the directory that contains the CSV file.
      Directory UNC Full Path (Windows share) The full UNC (Universal Naming Convention) address. For example: //hostname/sharedfolder
      Files to copy (with wildcards) Before parsing, the SFTP and SCP commands need to make a local temporary copy of the files; this setting specifies which files in the remote directory should be imported.
      File list pattern A regular expression that defines which data files should be read. The default value is (?$<$!done)\$, which tells the ETL to read every file whose name does not end with the string "done". For example, my_file_source.done.
      Recurse into subdirs? Select Yes or No. When set to Yes, TrueSight Capacity Optimization also inspects the subdirectories of the target directories.
      After parse operation Choose what to do after the CSV file has been imported. The available options are:
      • Do nothing: Do nothing after import.
      • Append suffix to parsed file: Append a suffix you add here to the imported CSV file. For example, _done or _impoted, and so on.
      • Archive parsed file in directory: Archive the parsed file in the specified directory.
        • Archive directory (local): Default archive directory path is filled out for you. For example, %BASE/../repository/imprepository
        • Compress archived files: Select Yes or No.
      • Archive bad files in directory: Archive erroneous files in the specified directory.
        • Archive directory (local): Default archive directory path is filled out for you. For example, %BASE/../repository/imprepository
        • Compress archived files: Select Yes or No.
      Parsed files suffix The suffix that will be appended parsed files; default is .done.
      Remote host (Applies to FTP, SFTP, SCP) Enter the name or address of the remote host to connect to.
      Username (Applies to Windows share, FTP, SFTP, SCP) Enter the username to connect to the file location server.
      Password required (Applies to Windows share, FTP, SFTP, SCP) Select Yes or No.
      Password (Applies to Windows share, FTP, SFTP, SCP) Enter a password to connect to the file location server. Applicable if you selected Yes for Password required.
      ETL task properties
      Task group Select a task group to classify this ETL into.
      Running on scheduler Select the scheduler you want to run the ETL on.
      Maximum execution time before warning The number of hours, minutes or days to to execute the ETL for before generating warnings, if any.
      Frequency Select the frequency of ETL execution. Available options are:
      • Predefined: Select a Predefined frequency from Each Day, Each Week or Each Month.
      • Custom: Enter a Custom frequency (time interval) as the number of minutes, hours, days or weeks to run the ETL in.
      Start timestamp: hour\minute (Applies to Predefined frequency) The HH:MM start timestamp to add to the ETL execution running on a Predefined frequency.
      Custom start timestamp Select a YYYY-MM-DD HH:MM timestamp to add to the ETL execution running on a Custom frequency.

      Note

      To view or configure Advance properties, click Advanced. You do not need to set or modify these properties unless you want to change the way the ETL works. These properties are for advanced users and scenarios only.

      Advanced properties

      Property Description
      Format customization
      Timestamp format Specify a format that should be used by the ETL if the user tables or CSV files use an unsupported format. <YYYY-MM-DD HH:MM:SS> is the supported format. Depending on the type of ETL used, you might need to specify a custom format.
      File location
      Subdirectories to exclude (separated by ';' ) (Local directory) Names of subdirectories to exclude from parsing.
      Input file external validator (Local directory, Windows share, FTP) Select any one of the following options:
      • No external validation: Do not use external validation of the CSV file structure.
      • Use external validation script: Use the following script to validate the CSV file:
        • Script to execute: Specify the validation script to use to validate the input file.
      Additional properties
      List of properties
      1. Click Add.
      2. Add an additional property in the etl.additional.prop.n box.
      3. Click Apply.
        Repeat this task to add more properties.
      Loader configuration
      Empty dataset behavior Choose one of the following actions if the loader encounters an empty dataset:
      • Abort: Abort the loader.
      • Ignore: Ignore the empty dataset and continue parsing.
      ETL log file name Name of the file that contains the ETL execution log; the default value is: %BASE/log/%AYEAR%AMONTH%ADAY%AHOUR%MINUTE%TASKID
      Maximum number of rows for CSV output A number which limits the size of the output files.
      CSV loader output file name Name of the file generated by the CSV loader; the default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID.
      BCO loader output file name Name of the file generated by the TrueSight Capacity Optimization loader; the default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID.
      Detail mode Select the level of detail:
      • Standard: Data will be stored on the database in different tables at the following time granularities: Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
      • Raw also: Data will be stored on the database in different tables at the following time granularities: Raw (as available from the original data source), Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
      • Raw only: Data will be stored on the database in a table only at Raw granularity (as available from the original data source).

      For more information on granularities, see Accessing data using public views and Sizing and scalability considerations.

      Reduce priority
      • Normal:
      • High:
      Remove domain suffix from datasource name (Only for systems) If set to True, the domain name is removed from the data source name. For example, server.domain.com will be saved as server.
      Leave domain suffix to system name (Only for systems) If set to True, the domain name is maintained in the system name. For example: server.domain.com will be saved as such.
      Update grouping object definition If set to True, the ETL will be allowed to update the grouping object definition for a metric loaded by an ETL.
      Skip entity creation (Only for ETL tasks sharing lookup with other tasks) If set to True, this ETL does not create an entity, and discards data from its data source for entities not found in TrueSight Capacity Optimization. It uses one of the other ETLs that share lookup to create the new entity.
      Scheduling options
      Hour mask Specify a value to execute the task only during particular hours within the day. For example, 0 – 23 or 1,3,5 – 12.
      Day of week mask Select the days so that the task can be executed only during the selected days of the week. To avoid setting this filter, do not select any option for this field.
      Day of month mask Specify a value to execute the task only during particular days within a month. For example, 5, 9, 18, 27 – 31.
      Apply mask validation By default this property is set to True. Set it to False if you want to disable the preceding Scheduling options that you specified. Setting it to False is useful if you want to temporarily turn off the mask validation without removing any values.
      Execute after time Specify a value in the hours:minutes format (for example, 05:00 or 16:00) to wait before the task must be executed. This means that once the task is scheduled, the task execution starts only after the specified time passes.
      Enqueueable Select one of the following options:
      • False (Default): While a particular task is already running, if the next execution command arises – it is ignored.
      • True: While a particular task is already running, if the next execution command arises – it is placed in a queue and is executed as soon as the current execution ends.
    4. Click Save.
      You return to the Last run tab under the ETL tasks page.
    5. Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click  to run the ETL.
      After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:
      • OK: The ETL executed without any error in simulation mode.
      • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
      • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
    6. Switch the ETL to production mode: To do this, perform the following task:
      1. In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
      2. In the Run configurations table in the ETL details page, click  to edit the active run configuration.
      3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
      4. Click Save.
    7. Locate the ETL in the ETL tasks table and click  to Run it, or schedule an ETL run.
      After you run the ETL, or schedule the ETL for a run, it will extract the data form the source and transfer it to the TrueSight Capacity Optimization database.

    Creating a valid CSV file containing relations

    Input file has the following format:

    TS;CHANGETYPE;ENTCATNM;ENTNM;DS_ENTNM;ENTCATNMPARENT;DS_ENTNMPARENT;ENTTYPENM
    2016-01-01 00:00:00;ASSERT;APP;Development;Development;APP;_ROOTAPP_
    2016-01-01 00:00:00;ASSERT;APP;Support;Support;APP;_ROOTAPP_
    2016-01-01 00:00:00;ASSERT;SYS;appsrv101;appsrv101.domain.com;;;as
    2016-01-01 00:00:00;ASSERT;SYS;dbsrv102;dbsrv102.domain.com;;;db
    2016-01-01 00:00:00;ASSERT;SYS;websrv103;websrv103.domain.com;;;wss
    2016-01-01 00:00:00;ASSERT;SYS;appsrv104;appsrv104.domain.com;;;as
    2016-01-01 00:00:00;ASSERT;SYS;dbsrv105;dbsrv105.domain.com;;;db
    2016-01-01 00:00:00;ASSERT;SYS;websrv106;websrv106.domain.com;;;wss
    2016-01-01 00:00:00;ASSERT;APP;Development;Development;;;svc
    2016-01-01 00:00:00;ASSERT;SYS;appsrv101;appsrv101.domain.com;APP;Development;as
    2016-01-01 00:00:00;ASSERT;SYS;dbsrv102;dbsrv102.domain.com;APP;Development;db
    2016-01-01 00:00:00;ASSERT;SYS;websrv103;websrv103.domain.com;APP;Development;wss
    2016-01-01 00:00:00;ASSERT;APP;Support;Support;;;svc
    2016-01-01 00:00:00;ASSERT;SYS;appsrv104;appsrv104.domain.com;APP;Support;as
    2016-01-01 00:00:00;ASSERT;SYS;dbsrv105;dbsrv105.domain.com;APP;Support;db
    2016-01-01 00:00:00;ASSERT;SYS;websrv106;websrv106.domain.com;APP;Support;wss
    

    Structure is strictly aligned with OBJREL dataset format.

     

     Click here to view the details of the above displayed input file...

     2016-01-01 00:00:00;ASSERT;APP;Development;Development;APP;_ROOTAPP_
    Indicates that an APP object named Development exists in the _ROOTAPP_. In this case the value of _ROOTAPP_  is ACME Services.

     

    2016-01-01 00:00:00;ASSERT;APP;Support;Support;APP;_ROOTAPP_
    Indicates that an APP object named Support exists in the _ROOTAPP_.


    2016-01-01 00:00:00;ASSERT;SYS;appsrv101;appsrv101.domain.com;;;as
    Indicates that a SYS object named appsrv101 of type as exists.

     

    2016-01-01 00:00:00;ASSERT;SYS;dbsrv102;dbsrv102.domain.com;;;db
    Indicates that a SYS object named dbsrv102 of type db exists.

     

    2016-01-01 00:00:00;ASSERT;SYS;websrv103;websrv103.domain.com;;;wss
    Indicates that a SYS object named websrv103 of type wss exists.

     

    2016-01-01 00:00:00;ASSERT;SYS;appsrv104;appsrv104.domain.com;;;as
    Indicates that a SYS object named appsrv104 of type as exists.

     

    2016-01-01 00:00:00;ASSERT;SYS;dbsrv105;dbsrv105.domain.com;;;db
    Indicates that a SYS object named dbsrv105 of type db exists.

     

    2016-01-01 00:00:00;ASSERT;SYS;websrv106;websrv106.domain.com;;;wss
    Indicates that a SYS object named websrv106 of type wss exists.


    2016-01-01 00:00:00;ASSERT;APP;Development;Development;;;svc
    Indicates that an APP object named Development of type svc exists.


    2016-01-01 00:00:00;ASSERT;SYS;appsrv101;appsrv101.domain.com;APP;Development;as
    Indicates that a SYS object named appsrv101 is the child of an APP object named Development.

     

    2016-01-01 00:00:00;ASSERT;SYS;dbsrv102;dbsrv102.domain.com;APP;Development;db
    Indicates that a SYS object named dbsrv102 is the child of an APP object named Development.

     

    2016-01-01 00:00:00;ASSERT;SYS;websrv103;websrv103.domain.com;APP;Development;wss
    Indicates that a SYS object named websrv103 is the child of an APP object named Development.

     

    2016-01-01 00:00:00;ASSERT;APP;Support;Support;;;svc
    Indicates that an APP object named Support of type svc exists.


    2016-01-01 00:00:00;ASSERT;SYS;appsrv104;appsrv104.domain.com;APP;Support;as
    Indicates that a SYS object named appsrv104 is the child of an APP object named Support.

     

    2016-01-01 00:00:00;ASSERT;SYS;dbsrv105;dbsrv105.domain.com;APP;Support;db
    Indicates that a SYS object named dbsrv105 is the child of an APP object named Support.

     

    2016-01-01 00:00:00;ASSERT;SYS;websrv106;websrv106.domain.com;APP;Support;wss
    Indicates that a SYS object named websrv106 is the child of an APP object named Support.

    Note

    When you are defining:

    • An object, the ENTCATNMPARENT and DS_ENTNMPARENT columns do not contain any values.
    • A relationship between objects, the ENTCATNMPARENT and DS_ENTNMPARENT columns contain the required values.

    As an example, a CSV file that contains the lines in the example above produces the following tree structure:

    The CSV file should provide all entities assertion and parent/child relationships among all entities. Remember to assert all entities before providing relations. For entity assertion, following fields are required:

    Field name

    Note

    Mandatory

    TS

    Timestamp of the relation

    Yes

    ENTNM

    Resource name

    Yes

    DS_ENTNM

    Resource name in the data source

    Yes

    ENTCATNM

    Entity type (APP, WKLD, SYS)

    Yes

    ENTTYPENM

    Type definition

    No

    DESCRIPTION

    Description

    No

    LOCATIONNM

    Location Name

    No

    For asserting relations, following fields are required:

    Field name

    Note

    Mandatory

    TS

    Timestamp of the relation.

    Yes

    ENTNM

    Child resource name.

    Yes

    DS_ENTNM

    Child resource name in the data source.

    Yes

    ENTCATNM

    Child entity type (APP, WKLD, SYS).

    Yes

    ENTCATNMPARENT

    Father entity type (APP, WKLD, SYS).

    Yes

    DS_ENTNMPARENT

    Father resource name in the data source.

    Yes

    Related topics

    Using ETL datasets

    Generic - CSV file parser

    Developing custom ETLs

    Dataset reference for ETL tasks

    Horizontal and Vertical datasets

    Viewing datasets and metrics by dataset and ETL module