Test Data Privacy Composite Rules


BMC AMI Test Drive 
MC Compuware Test Data Privacy – Composite Rules

This tutorial will introduce you to Composite Rule processing to handle multiple data elements within the same column or field.

Revised: 2024/09/20 00:00 





Getting Started

Instructions:

  • This guide contains many screenshots to provide a visual reference
  • Please note each place that you must enter your own specific ID or number
  • You must complete each step before proceeding to the next to successfully follow the test drive script

If, at any point during your experience, your host connection times out, you may need to log back in to the TestDrive host connection.

image-2023-4-10_9-26-41.png

If at any time during the execution of this script the Common Enterprise Services Login popup is shown, enter your test drive ID and password under User ID and Password, check the Save credentials box and then depress the ENTER key or click OK.


BMC AMI DevX Data Studio Data

Data privacy rules are created in the DevX Data Studio plug-in and stored in a repository. These rules are then available to disguise data from each of the following products:

  • File-AID/EX
  • File-AID/Data Solutions
  • File-AID for DB2
  • File-AID/RDX
  • File-AID for IMS

The data to be disguised may reside in z/OS files, IMS databases, or relational database tables in DB2, DB2 UDB, Microsoft SQL Server, Sybase, or Oracle.

In this exercise you will:

  • Create a Composite Rule to handle First and Last Names combined into one field or column
    • Translation rules for first and last names were created in the previous exercise
    • Uses the provided translate table
  • Execute a single table Extract from SQL Server to view how Composite rule will be applied
  • Execute a Load to load the newly disguised data to a new table
  • View the disguised data in a File-AID Data Editor session


Open Your Own TDP Project


Do This
  • From the menu, select BMC, DevX Data Studio, and select the Rules Repository tab.
  • Right-click on the TDPRepos repository and Open Repository.

image-2023-4-3_15-31-33.png

If you have closed your Test Drive session you may no longer have the Project created in the previous exercise. In that case, you may wish to recreate that project using the Disguise Rules Script, or you can use the TEST DRIVE DISGUISE PROJECT to see the completed results for this exercise. You may select this Project for execution in the Extract process.


Do This
  • Double-click on your Data Privacy Project.

image-2023-4-3_15-32-43.png


Do This
  • Click on the Data Elements tab at the bottom.

image2021-12-21_10-1-27.png

Notice you have already created DE's for FIRST LAST NAMES, FIRST NAME, and LAST NAME. First and Last Names have translation rules created for each to give a replacement value. The Composite will allow the replacement values to be used correctly within the First Last Names field or column.


Composite Creation


Do This
  • In your Project, Click on the Composites tab at the bottom of the screen.

image2022-1-11_13-53-49.png


Do This
  • Click on bottom right Global button under Categories.
  • Import Name and Separators categories from Global by clicking Import.

image-2023-4-3_16-12-43.png

Categories 

Do This
  • In the Data Items section in the top right, click on the Global button under Data Items.
  • Select Name-First Name, Name-Last Name and Name-Middle Name.


image-2023-4-3_16-14-25.png

 

Do This
  • Scroll down and also select White Space Separator (reducible).
  • Click Import.

image-2023-4-3_16-15-13.png


Do This
  • Under Composite on the left-hand side click the Add button.

image2021-12-21_10-12-38.png


Do This
  • Give your Composite a name as shown.
  • Under Data Item on the left grab Global: Name-First Name and drag to the black box under Structure.
  • Line it up on the left of that box until you see the arrows and drop it.

image-2023-4-3_16-18-19.png

Make sure that all the components are directly touching one another in the Structure box. If you get a pop-up window asking for positions you have dropped it in the wrong place. In that case click Cancel and retry.


Do This
  • Drag the White Space Separator to the right of the first name in the Structure box and drop it directly beside the First Name.

image-2023-4-3_16-19-32.png


Do This
  • Continue lining up Middle Name, then a Space, then Last Name.

image-2023-4-3_16-20-28.png

 

Do This

The data in our column or field may or may not contain a Middle Name.
To handle this: 

  • Click on the Middle Name in the Structure box.
  • Click the Optional box below under the General tab.

image-2023-4-3_16-21-21.png

Logically, if the Middle Name may or may not exist, then the second Space may or may not exist.

Do This
  • Click on the Second Space in the Structure box and select Optional below.
  • Click on Test. at the bottom of the screen

image-2023-4-3_16-22-3.png


Do This
  • Type a name into the field to test your newly created Composite definition.
  •  Click OK to view your results.

image-2023-4-3_16-22-59.png

You will now see how it breaks the combined field into individual elements.

Since we have already created a rule for First Name, that rule can be applied to that portion of the field or column. And the rule for Last Name can be applied to that portion of the field.
 We do not have a rule for Middle Name. In this scenario we will leave the existing data.

Do This
  • Click the X to close this box. Click OK to save your Composite Definition.

image2022-1-11_14-5-41.png

Composite Rule

We will now use the Composite definition just created within a Rule.


Do This
  • Click on the Rules tab at the bottom of the screen.
  • Click Add to add a new Rule.

image2022-1-11_14-6-51.png


Do This
  • Type a new Rule name as shown and click Composite for the Create Rule Action.
  • Click Next.

image-2023-4-3_16-26-22.png


Do This
  • Type in an Action name as shown in the first field.
  • Use the drop-down menu for Project Resource and select the Data Element already created for FIRST LAST NAMES.
  • Click Next.

image-2023-4-3_16-27-37.png


Do This
  • Select your Composite field definition from the drop-down menu.

The different data items will appear below.

  • Click on a Data Item and use the drop-down under the Data Element column to match to the correct Data Element.

Since we are not disguising Middle Name or Spaces we will leave those blank.

image-2023-4-3_16-28-47.png


Do This
  • Click Finish.

image-2023-4-3_16-29-38.png


Do This
  • Click the X in the tab to Close the updated Data Privacy project.

image2022-1-11_14-13-56.png

You have now successfully created disguise rules to replace sensitive data in fields or columns containing Names, Phone Numbers, Social Security Numbers, Credit Card Numbers, Dates, and Email addresses.

These rules exist in your project in the repository and can be used against multiple data types. In the next step these rules will be called to disguise sensitive data as we move the data (example taking production data to another environment).


Single Table Extract with Disguise


Do This
  • Select the Data Explorer tab next to the Rules tab.  These are both available within the DevX Data Studio perspective..

image-2023-4-3_15-43-38.png

 

Do This
  • Right click on the FAEXRepos Repository in the left-hand tree view. 
  • Select Open.


image-2023-4-3_15-45-42.png

 

Do This
  • Right click on Related Extract and select New.

image-2023-4-3_16-31-55.png

 

Do This
  • Input an Extract name as shown and then click Browse for the Driving Object. 

image-2023-4-3_16-32-59.png


Do This
  • Select Database (JDBC) for the Type.
  • Use the Host dropdown to select the SQL Server Sample database.
  • Type dbo (in lowercase) for the Schema Name.
  • You may be prompted for credentials to connect to the database. Enter "testdrive" for the userid and password and click OK.
  • Click List beside schema name.

worddav45c32159a3140af3266182865b57f18b.png


Do This
  • Select the CONTACT_TABLE from the list and click OK.

image2021-12-21_10-53-44.png


Do This
  • Click Finish.

image-2023-4-3_16-34-7.png


Do This
  • Click Browse for the Extract file location. This is where the extract file will be written.

image2021-12-21_10-55-50.png


Do This
  • Navigate to Temporary Files and then click on Select Folder.

image-2023-4-3_16-35-27.png

The extract will be saved in this directory under a folder name the same as your job name, such as CONTACT_TABLE_EXTRACT.

Do This
  • From the Extract Overview screen, select the Data Privacy tab along the bottom of this section.

image2021-12-21_10-57-43.png


Do This

Here you can select the Disguise Rules already created and saved within your Data Privacy Project. This will determine the data eligible for disguise and the rules to be applied to the data during this extract process.

  • Click on the Browse button for Project Name.

image-2023-4-3_16-36-38.png


Do This
  • Use the Repository drop-down to select the TDPRepos repository.
  • Select your project, or you may use the TEST DRIVE DISGUISE PROJECT for this exercise.
  • Click OK.

image-2023-4-4_10-7-55.png


Do This
  • Execute by clicking on the Blue Arrow execution button in the top right corner.

image2021-12-21_11-2-17.png


Do This
  • Look in the Console tab for execution information. 
  • Double click the Console tab to maximize this view.
  • Here you will see number of rows extracted.
  • Scroll down to see information on how the data privacy rules were applied to the different columns.

image2021-12-21_11-5-20.png


Do This
  • Double click again on the Console tab to minimize.
  • Close the Extract job by clicking on the X in the tab.

image2021-12-21_11-6-20.png


Do This
  • Save the Extract job when prompted.

image-2023-4-3_16-38-7.png


Load Disguised Data to a New Table


Next you will load the disguised data to a table.


Do This
  •  Navigate to the Data Explorer view and right click on Related Loader and select New.

image-2023-4-3_16-39-11.png


Do This
  • Name this Load job CONTACT_TABLE_LOAD.
  • Click on the Browse button to navigate to the target table to load the data.

image-2023-4-3_16-40-4.png


Do This
  • Take the default Type Database and Host of SQL Server Sample and click OK.

image-2023-4-3_16-40-49.png


Do This
  • Click Finish to open the Related Loader.

image-2023-4-3_16-41-39.png


Do This
  • Under Extracted Data navigate to the recently run extract. 
  • Click Browse.

image2021-12-21_11-13-53.png


Do This
  • Select the CONTACT_TABLE_EXTRACT job from the list.
  • Click OK.

image-2023-4-3_15-52-28.png


Do This
  • From the Overview click on the Objects tab across the bottom of this section.

Here is listed the Source and Target table names. The Source is derived from information in the Extract file, and the target defaults to this same name.


  • Right click on Target Name and select Change Target Object Name.

image2021-12-21_11-15-52.png


Do This
  • Change the target table name to CONTACT_TABLE_DIS. 

This table does not exist and will be created during the Load job. 

  • Click OK.

image-2023-4-3_15-53-56.png


Do This
  • Click on the Field Mappings tab along the bottom of this section.

If the source and target tables are different column mapping can be done here.

image2021-12-21_11-17-34.png


Do This
  • Click on the DDL tab along the bottom of this section.

image2021-12-21_11-18-28.png

Here you will see the DDL needed to create the target table. The DDL is generated from the source table information.

Changes can be made on these screens. For example, if the Contact table has a Constraint with the CONT_IDX index. To avoid a conflict during the load the index name will need to be changed.

image2021-12-21_11-25-14.png


Do This
  • Click on the Options tab along the bottom of this section.

Listed are options including for Commits and Discards. 

  • Click the Blue execute button in the top right.

image2021-12-21_11-26-9.png

 

Do This
  • Look in the Console tab for execution information.
  • Double click the Console tab to maximize this view.



Here you will see number of rows loaded.

worddavb5b16becea31ad814f00e41d18226920.png


Do This
  • Double click the Console tab to minimize this view. 
  • Close the Load job by clicking the X in the tab. 
  • Click Save to save this job in the repository.

image-2023-4-3_16-43-31.png

BMC AMI DevX Data Studio Data Editor

In this exercise you have extracted data and applied disguise to the sensitive elements and loaded that data to a new table. Now open both the original data and the newly created disguised table and visually compare the two.

Do This
  • Select the Host Explorer tab.

image-2023-4-3_15-58-18.png

Do This
  • Right click on Hosts
  • Select Configure
  • Click Host Types and check SQLServer Databases.
  • Click Apply and Close.

image-2023-4-3_15-59-39.png

Do This
  • Expand Hosts
  • Expand SQL Server Sample by clicking on the arrow.
  • Expand Microsoft SQL Server. 

(If it cannot be expanded, right click on Microsoft SQL Server and select Add Filter. Type "dbo" in lowercase for the Schema and * in the table field.)


image-2023-4-3_16-2-17.png

Do This
  • Double click on the CONTACT_TABLE.

image-2023-4-3_16-5-30.png


Do This
  • Click Run to open the table in an Edit session

image2021-12-21_11-34-43.png


Do This
  • Double click on CONTACT_TABLE_DIS.

image-2023-4-3_16-7-15.png


Do This
  • Click Run to open this table in another tab within the Edit session.

worddav40b44e88a516b3fb9432b7408600e1a9.png


Do This
  • Place your cursor on the second tab and drag to the right and drop to see the two tables side by side.


Notice the gray line that pops up during the drag.

worddav5af7ab5c70768ca2ab32669842eedf8e.png


Do This

You can now see the Contact Name column in both tables and see how the Composite rule replaced the original First and Last Names with new values. Note Row 9 had a Middle Name in the original data and that is retained in the new table.

image2022-9-20_11-33-32.png


Do This
  • Scroll to the right for each table to view the Telephone Number columns.

image2022-9-20_11-35-36.png


Remember for the Phone Rule the first 3 bytes of data are masked and no disguise is applied. In this example, only the last four digits are changed.

Do This
  • Close both tables by clicking on the X within the tabs.
  • Close all open tabs.

image2021-9-28_14-23-3.png

This completes this exercise. 

You have now successfully created and applied disguise rules to data from a single SQL Server table and loaded newly disguised data to a new table. Continue on to Part 3 Related Extract with Disguise to learn more about extracting data from multiple tables while keeping the database integrity.


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*