Test Data Privacy Disguise Rules

BMC AMI Test Drive 
MC Compuware Test Data Privacy – Disguise Rules

This tutorial will introduce you to the BMC AMI DevX Data Studio Test Data Privacy Manager to create Disguise Rules to be used to disguise different data elements.

Revised: 2024/03/13 00:00 






Getting Started

Instructions:

  • This guide contains many screenshots to provide a visual reference
  • Please note each place that you must enter your own specific ID or number
  • You must complete each step before proceeding to the next to successfully follow the test drive script

If, at any point during your experience, your host connection times out, you may need to log back in to the TestDrive host connection.

image-2023-4-10_9-25-37.png

If at any time during the execution of this script the Compuware Enterprise Services Login popup is shown, enter your test drive ID and password under User ID and Password, check the Save credentials box and then depress the ENTER key or click OK.


BMC AMI DevX Data Studio Data Privacy Manager

Data privacy rules are created in the DevX Data Studio plug-in and stored in a repository. These rules are then available to disguise data from each of the following products:

  • File-AID/EX
  • File-AID/Data Solutions
  • File-AID for DB2
  • File-AID/RDX
  • File-AID for IMS

The data to be disguised may reside in z/OS files, IMS databases, or relational database tables in DB2, DB2 UDB, Microsoft SQL Server, Sybase, or Oracle. Other file types such delimited, flat files, CSV, or XML can be disguised as well.
 In this exercise you will:

  • Create at least two Rule Action Types
    • Format Preserving Encryption for Phones, SSNs, and Credit Cards
    • Translation for names
  • Use the provided translate table
  • Use Expression Builder
    • Literal Values
    • Date Aging
  • Coverage
    • View the rules against the metadata


Data Privacy Rules Repository

Data Privacy Rules are created and stored in a Rules Repository. A windows service called File-AID Services enables connectivity to defined repositories. These rules can then be called later to disguise data at the time of execution of a copy, convert, or extract job.

BMC AMI DevX Workbench for Eclipse connects to the File-AID Services through a setting in Workbench Preferences.

Do This
  • In Workbench go to Window, then Preferences.

image2022-1-11_10-6-15.png


Do This
  • Expand BMC by clicking on the arrow 
  • Select File-AID Services

image-2023-4-3_15-0-26.png

Here you will see the link to the machine and port where File-AID Services is running.

Do This
  • Click Test Connection, then OK.
  • Click Apply and Close.

image-2023-4-3_15-1-52.png



Create Your Own TDP Project


Do This
  • Click on BMC in the top menu.
  • Select the DevX Data Studio perspective.

image-2023-10-2_16-19-5.png

Do This
  • Click the Rules Explorer tab in the top left.
  • Double-click on TDPRepos repository to open.
  • Right-Click on TDPRepos.
  • Select Create New Project.

image-2023-4-3_15-5-32.png


Do This
  • Name your project with your Test Drive ID (CWEZ###).
  • Hit the Tab key to populate the short name.
  • Click OK.

image2022-1-11_10-16-20.png

image2022-1-11_10-17-24.png

Here you will see the overview for the data privacy project. The three sections that we will work with on the right are Data Elements, Rules, and Coverage. These correspond to the tabs across the bottom.


Add Data Elements

Data Elements are containers that allow you to normalize multiple fields into a common item based on data identification instructions.

Do This
  • Click on the Data Elements tab at the bottom of this pane. 
  • Use the Add button in the left-hand Data Elements pane to create the list of elements to be disguised.

image2022-1-11_10-19-24.png


Do This
  • Enter SSN in the first field and click Finish.

image2022-1-11_10-20-30.png


Do This
  • Repeat this process for the following Data Elements as shown.
  • Create Data Elements for: CREDIT CARD, EMAIL, FIRST LAST NAME, FIRST NAME, LAST NAME, PHONE, and STREET ADDRESS.

image2022-1-11_10-23-12.png


Do This
  • Add another Data Element: DATE.
  • Enter DATE in the first field and click Next.

image2022-1-11_10-24-10.png


Do This
  • Select DATE from the Processing type dropdown.
  • Select the correct date format as shown from the dropdown and enter the initialize date values.
  • Click Next.

image2022-1-11_10-25-38.png

The other data handling characteristics available when creating Data Elements are accessible through the left-hand pane.  The screens are shown below in order:

Do This
  • Click Next through each of these screens.


 Value Alignment: 
image2022-1-11_10-26-55.png


Invalid and Long Data Values:
image2022-1-11_10-27-57.png 


Null Values:
image2022-1-11_10-28-45.png 


References:
image2022-1-11_10-29-29.png

Do This
  • Click Finish.


Add Source Data Identifiers

Source Data Identifiers locate the actual data associated with a data element. At disguise execution time, the data identification process is invoked to match the content of the Source Data Identifier against the metadata of the object being disguised. For z/OS files, the metadata is the COBOL or PL/I layout. For DBMS objects, the metadata is the column definition from the DBMS catalog.

Do This
  • Click on the Credit Card Data Element.
  • Click Add in the Source Data Identifier right-hand pane.

image2022-1-11_10-30-29.png


Do This
  • Type CREDIT*CARD*NUM in the first field and again in the last field.


The data identifier name field will show in your list and the Part 1 field is the actual name that will be searched for to identify the field for disguise. It is not Case sensitive unless box is checked.



  • Click Finish.

image2022-1-11_10-31-51.png



Do This
  • Add Source Data Identifiers (SDIs) for each of the Data Elements. Data Elements can have multiple SDIs defined. 
  • Create separate SDI(s) for the listed Data Elements below:
    • DATE:  *DATE
    • EMAIL:  EMAIL
    • FIRST LAST NAME: *CONTACT*NAME
    • FIRST NAME:  *FIRST*NAME 
      FNAME
    • LAST NAME: *LAST*NAME
    • PHONE:  *PHONE*
    • SSN: REP_ID
      SOC*SEC*NUM
      SSN
    • STREET ADDRESS: CONTACT*ADDR
      ADDRESS

image2022-1-11_10-44-50.png


The * are used as wildcards. This will also mean that field names using a dash "-" or column names using an underscore "_" will be identified.
For example, any column or field name containing the string "phone" will be identified as a phone number and have the phone rule disguise technique applied to that data.


Above is an example of the different Source Data Identifiers (SDI) for the SSN Data Element. Any columns or fields matching these identifiers will be disguised by the rule for SSN.

Disguise Rules

Encryption Rule


Do This
  • Click on the Rules tab at the bottom of your Project. 
  • Click Add.

image2022-1-11_10-45-53.png


Do This
  • Create a new Rule called SSN Rule.
  • Select Format Preserving Encryption for the Rule Action.
  • Click Next.

image2022-1-11_10-47-9.png


Do This
  • Create an Action name "SSN Encryption" as shown.
  • Encryption requires a key. Use the dropdown menu to select Global Encryption.
  • Select the SSN Data Element in the list under Project Resource.
  • Click Finish.

image2022-1-11_10-48-23.png


Do This
  • Repeat adding an Encryption Rule for the Credit Card Data Element following the above example.
  • Type "CREDIT CARD Rule" for the rule name.
  • Click Format Preserving Encryption.
  • Click Next.
  • Type "CC Encryption Rule" for the Action name.
  • Select the Global Encryption Key.
  • Select CREDIT CARD for the Project Resource.
  • Click Finish.

image2022-1-11_10-52-46.png


Field Masking

Use Field Masking to determine which bytes of data to apply disguise. In this example we will disguise only a portion of the phone number.

The first 3 bytes will be excluded from encryption and will retain the original values.

Do This
  • Create an Encryption Rule for the Phone Data Element called "Phone Rule". Select Format Preserving Encryption. Click Next.
  • Enter an Action name of "PHONE Encryption".
  • Check Enter Key Value and type in a value.
  • Check the box for the PHONE Project Resource.
  • Enter "NNN" under the Field Mask column as shown below by either using your Tab key or clicking in the Field Mask column.
  • Click Finish.

image2022-1-11_10-55-7.png

note.png
Golden principle: Rules are created for Data Elements. The Source Data Identifiers defined to the Data Element determine which columns and fields will be disguised by that rule.


Translation Rules


Translation Rules are commonly used for data that needs to be meaningful and readable such as names and addresses.


Do This
  • Add a new Rule for First Names.
  • Select Translation for the rule action.
  • Click Next.

image2022-1-11_10-56-41.png


Do This
  • Type "FIRST NAME Translation" for an action name.
  • Select the Translate table and Access path previously defined from the dropdown menus. 
  • Click Next.

image2022-1-11_10-57-54.png


Note that the translate tables already exist with fictitious data and have been identified within the Manage Translate Tables utility under Resource Administration.



Do This
  • Select First Name under Project Resource to use in calculating the hash value.
  • Click Next.

image2022-1-11_10-58-48.png


For example, if the translate table has 1000 rows, this will hash the first name to a value between 1 and 1000 and point to that row. 

Do This
  • Select the Project Resource to be replaced with new values by checking the box next to FIRST NAME.
  • Click in the Translate Table Data Column to see the dropdown menu.
  • Select FIRST by using the drop-down menu in that column to determine what data to bring back from the table.
  • Click Finish.

image2022-1-11_11-14-21.png



Do This

Repeat for Last Names using last name as the hash field and the replacement field.

image2022-1-11_11-16-20.png



Rule Logic

Literal Values


Do This
  • Add a new Rule for Email Data Element.
  • Leave Create Rule Action as None.
  • Click Finish.

image2022-1-11_11-17-26.png



Do This
  • On the right-hand side select the Rule Logic tab.
  • Click the Build button to open the Expression Editor.

image2022-1-11_11-19-11.png


Do This
  • Click on the Resources tab in the bottom of the left pane (you may need to stretch this box to see it).
  •  Expand the Email Data Element in the list by clicking on the arrow.

image2022-1-11_11-21-11.png


Do This
  • Click on the setValue under EMAIL and drag to the Expression area on the Right.
  • Double click on the red word string and replace with a literal value such as "John.Doe@company.com". Double quotes are required.

image2022-1-11_11-23-42.png


Do This
  • Click the Validate Expression button to verify syntax and logic.
  • Click OK and OK.

image2022-1-11_11-25-17.png


This is an example of replacing sensitive data with a literal value for every row or record.

Date Aging Logic


Do This
  • Create a new rule for the DATE Data Element with a Rule name of DATE Rule.
  • This rule will use Rule Logic.
  • Leave None checked for Rule Action.
  • Click Finish.

image2022-1-11_11-26-54.png


Do This
  • On the right-hand side select the Rule Logic tab.
  • Click the Build button to open the Expression Editor.

image2022-1-11_11-27-50.png



Do This
  • In Expression Builder click the Resources tab and expand the DATE Data Element by clicking on the arrow.
  • Click on setValue and drag to the right Expression area.

image2022-1-11_13-33-0.png


Do This
  • Click the Functions tab at the bottom.
  • Expand Date Functions in the top list.
  • To drag and drop the function into our expression:

Click the ADD Days function and drag to the right Expression area and hover over the red word String
until it is highlighted.

Drop the function to replace that word.

image2022-1-11_13-35-10.png


Do This
  • Click Resources, expand DATE and drag getOriginalValue right to replace the word Date


    (may need to manually remove the word Date if it is not replaced – see following screenshot).

  • Replace the word Integer with a value of 3.

image2022-1-11_13-36-58.png


This will age the original values by 3 days. Calendar intelligence is built in and will roll dates to the next month or year accordingly and never return an invalid date such as February 30th.


Do This
  • Validate Expression and click OK for Successful Validation and click OK to close the Expression Editor.

image2022-1-11_13-38-44.png

You have now successfully created disguise rules to replace sensitive data in fields or columns containing Names, Phone Numbers, Social Security Numbers, Credit Card Numbers, Dates, and Email addresses.

These rules exist in your project in the repository and can be used against multiple data types including but not limited to SQL Server, Oracle, Sybase, Db2 and DB2 z/OS, VSAM, sequential, IMS, Excel, Access, Flat Files, Delimited Files, and XML.


Coverage

When a project is created, a Coverage tab is available for project metadata. However, there is no requirement to define metadata to the project. If metadata is defined to the project, however, coverage can be displayed for any single metadata, and coverage reports can be requested against the list of project metadata.

Let's view how the rules you created in this project would be applied to a table that we will be disguising.

Check Coverage against a data source.


Do This
  • Click Coverage tab along the bottom of your Project.
  • Click Add button to the right.

image2022-1-11_13-40-21.png

 

Do This
  • Select the host SQL Server Sample from the dropdown menu.
  • Type "dbo" (in lowercase) for the Schema name.
  • Click List.




    If prompted, use testdrive for both the userid and password.

image2022-1-11_13-42-14.png


Do This
  • Select the Customer, Order, and Sales Rep tables with a CTRL click.
  • Click OK.

image2022-1-11_13-43-20.png


Do This
  • Highlight the Sales Rep table.
  • Click View Coverage on the right.

image2022-1-11_13-44-11.png


This will open a window at the bottom with the metadata from that table showing what columns are being identified as Data Elements and what rules will be applied when we move the data.


Do This
  • Click on the tab and drag the coverage view up and to the right.
  • Click on the Data Elements tab at the bottom of the Project.

image2022-1-11_13-46-19.png

In this view you can see how the rules created will be applied. The DE: denotes the Data Element and the worddavdc35904c1d4e89399511da1b64e01669.pngsymbol denotes the Source Data Identifier for each column eligible to be disguised. Under the Rule column is listed the Disguise Rule that will be applied.

Address is identified with a Data Element and an SDI, but there is no Rule listed. This is because we have not created a Street Address rule in this Project.
The Phone SDI identified two columns in this table containing phone numbers.


You can also use this Coverage View to add Source Data Identifiers.

Do This
  • Click on the SSN Data Element.
  • Drag the PERSONAL_ID_NUM column from the coverage to the SDI fields to add as another column name that may contain SSN.

image-2023-10-2_16-49-43.png

 

Do This
  • Click the Refresh blue arrows icon on the top right of coverage to see changes applied to that column.

You now see the SSN Rule will be applied to that column.

image-2023-10-2_16-51-43.png


Do This
  • Close the Coverage View by clicking on the X on the tab.
  • Close the Disguise Project by clicking on the X on the tab.



You will use these rules saved in the project in the next script.

This completes the Disguise Rules section. Please proceed to the Composite Rules Script before logging out of Test Drive to build upon your Data Privacy Project.

To use the Data Disguise Project that you created in this exercise do not leave Test Drive. Otherwise, you can view an existing project that has been created for you but the work has already been completed.



 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*