Adding an extraction rule to a custom field

An extraction rule determines which session, page, object, content, or header data that the custom field extracts.

Adding an extraction rule to a custom field involves the following procedures:

Users with Operator or Administrator roles can configure custom fields with content extractions, but a user with the Security role must review and approve the configuration before either role can activate the extractions. After you have configured the content extraction, the system displays a message stating that the new custom field is waiting for security approval.

Note

You can use different character encoding (including right-to-left languages) with extraction rules. For instance, you can perform transformations and create regular expressions with Chinese, Japanese, Arabic, Hebrew, or Cyrillic characters:
extraction.png

To set traffic selection for a custom field

  1. In a Real User Analyzer, point to the Administration > Data flow settings, and then click Custom fields.
  2. Expand the custom field to which you want to add an extraction rule.
  3. On the Action menu for this custom field, click Add new rule.
  4. In the Format list, select one of the following traffic selection options:
    • Domain only — Scope the traffic down to a specific domain.
    • Filter expression only — Scope the traffic down to a specific filter.
    • Domain and filter expression — Use domain and filter to scope the traffic down.
    • Default (catch-all) — Do not use a domain or filter and apply your extraction rule to all traffic.

      If you select domain or filter, you can use a preconfigured domain or filter, or create a new one.
  5. In the Extraction name box, enter a name for your extraction.
     If you selected Default in the previous step, the name will be set to Default automatically.
  6. Click Next.

    Proceed with setting the data source for a custom field.

To set the data source for a custom field

  1. Ensure that you have set the traffic selection.
  2. If necessary, point to the Administration > Data flow settings, and then click Custom fields. Then expand the custom field which extraction rule you want to edit, and click Edit rule.
  3. In the Source field list, select a source type.
  4. Certain source types require more configuration than others. For more information about configuring these source types, see the Data source configuration optionstable.
  5. If you want to set transformations for the custom field, click Next; otherwise, click Save.

    If you clicked Next, the system adds the data source for the custom field. Proceed to setting transformations for the custom field.

Data source configuration options

Field type

Configurable options

Fixed String

In the String box, type the string that you want to extract.

Geolocation Information

In the Geolocation list, select the geolocation field that you want to extract information about.

Parameter

In the Key box, type the key that you want to extract, or select the value from the lookup.

Path parameter

In the Element list, select the position index, and in the Parameter box, type the key name within the path element that you want to extract.

Stem element

In the Element section, use lists to select the indexing position.

To strip path parameters, select Strip Path Parameters.

HTTP Request or Response Headers

  1. In the Header name box, type the name of the header that you want to filter on, or select the value from the lookup.
  2. Do one of the following actions:
    • If you want to capture the entire contents of the header, click entire header contents.
    • If you want to capture a specific segment of the header contents, click comma separated list and use the Index box and the from list to specify the segment.
  3. In the Extraction limit box, type the number of characters to which you want to limit the extraction.

Request or response content

  1. In the Configurationsection, use option buttons to select the part of the content that you want to extract:
    • To extract a segment of the content, click When first occurrence of string and specify a limit for the segment.
    • To customize an extraction, click Use regex and type a regular expression in the box.
  2. If you want to make the search case sensitive, select Case-sensitive.
  3. In the Extraction limit box, type the number of characters to which you want to limit the extraction to.

    Extraction rules for Request Content or Response Content defined in a primitive custom field on a Real User Collector or defined in any custom field on a Real User Monitor (session, page, or object). For more information, see Using-request-or-response-content-as-data-sources.

To set transformations for a custom field

After the system has extracted the data, you can modify this data before storing it in the session record. The system supports a variety of built-in transformations, such as substring extraction, regular expressions, and lookup mappings.

  1. Ensure that you have set the traffic selection and the data source.
  2. If necessary, point to the Administration > Data flow settings, and then click Custom fields. Then expand the custom field which extraction rule you want to edit, and click Edit rule.
  3. Click Add transformation.
     If you have already configured transformations, select where in the order of execution to add the new transformation.
  4. In the Transformation list, click a transformation type. For more information about transformation types you can use the following table to configure the transformation.
    Transformation types for custom fields

Transformation

Description

Case conversion

Applies one of the following case conversions:

  • Capitalize all words — Convert the first character of each word in a string to uppercase. If some characters after the first ones in each word are already capitalized, they are not changed.
  • Capitalize first word — Convert only the first character of the first word only to uppercase.
  • Convert to lowercase — Convert all characters to lowercase.
  • Convert to uppercase — Convert all characters to uppercase.

Find/Replace

Replaces all occurrences of a specific substring with another static string.

Geolocation substitution

Converts to a geolocation identity, such as City, ISP, or Region.

Indexed substring extraction

Extracts a portion of a string based on a specified index. For example, extract the first 12 characters.

IP mapping

Converts an IP address to its corresponding value, according to a defined mapping.

You can provide a list of IP addresses and the corresponding values attached to each individual address. For example, if you know the IP addresses of your customers, you can map IP addresses to customer names.

You can enter up to 200 mapping key-value pairs and upload up to 10,000 values through the mapping upload interface. Select Upload mapping file and choose the file from the dialog box.

For the #Mode parameter, you can specify user, original, or none, as in the following examples:

  • user, which takes the default value specified using the #Default parameter
     #Mode, "user"
    #Default,"Unmapped"
    #Key, Value
    "10.160.0.0/16","intranet mtl"
     "172.19.*.*","intranet houston"
     "::FFFF:0:0/120","ipv6 mapped ipv4"
  • original, which keeps the initial value as the new mapped value
    #Mode, "original"
    #Key, Value
    "10.160.0.0/16","intranet mtl"
     "172.19.*.*","intranet houston"
     "::FFFF:0:0/120","ipv6 mapped ipv4"
  • none, which does not assign a default value
    #Mode, "none"
    #Key, Value
     "10.160.0.0/16","intranet mtl"
     "172.19.*.*","intranet houston"
     "::FFFF:0:0/120","ipv6 mapped ipv4"

Prepend/Append

Connects a static string to the extracted value at the beginning or the end.

Regular expression

Applies a regular expression to your data source.

String mapping

Converts a string from the source data to its corresponding value in a defined mapping.

  • You can provide a list of strings and the corresponding values attached to each individual string.
  • You must use UTF-8 encoding.
  • You cannot use wildcards for string mapping.
  • You can enter up to 200 mapping key-value pairs and upload up to 10,000 values through the mapping upload interface. Select Upload mapping file and select the file from the dialog box.

For the #Mode parameter, you can specify user, original, or none, as in the following examples:

  • user, which takes the default value specified using the #Default parameter
     #Mode, "user"
     #Default,"Unmapped Page"
     #Key, Value
     "/marketing/intro.html", "Marketing Home"
     "/marketing/catalog.html", "Marketing Catalog"
     "/partners/portal.html", "Partner Portal"
     "/support/login.html", "Support Home"
  • original, which keeps the initial value as the new mapped value
     #Mode, "original"
     #Key, Value
     "/marketing/intro.html", "Marketing Home"
     "/marketing/catalog.html", "Marketing Catalog"
     "/partners/portal.html", "Partner Portal"
     "/support/login.html", "Support Home"
  • none, which does not assign a default value
    #Mode, "none"
     #Key, Value
     "/marketing/intro.html", "Marketing Home"
     "/marketing/catalog.html", "Marketing Catalog"
     "/partners/portal.html", "Partner Portal"
     "/support/login.html", "Support Home"

Substring extraction

Extracts or deletes a portion of a string.

  • Between two different substrings
  • Between two occurrences of the same substring
  • From the start of the string to the occurrence of the substring
  • From the occurrence of the substring to the end of the string

Whitespace trimming

Deletes all leading and trailing spaces around a string.

Value counter

Counts the number of substrings within a string separated by a token, such as semicolon or comma.
For example, if you have a list of five users separated by semicolons (john; joe; james; jack; jorge), this transformation returns a count of the items in the list (5).

  1. (Optional) To test the transformation, click Test transformation and follow the on-screen instructions.
  2. Click Save.

    The system adds the extraction rule for the custom field.

Related topics

Using-request-or-response-content-as-data-sources
Priorities-of-extraction-rules
Defining-a-custom-field-extraction-domain
Verifying-content-extraction-for-a-custom-field

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*