dedup search command

This search command removes data records that contain fields with duplicate values.

Records are removed based on the field names specified. Depending on the field name, the first instance(s) of the record(s) is kept, while the rest of the records are removed. You can specify a number (N) as the count of records (with duplicate field values) to be kept. For example, you can keep the first three records with the same value and remove the remaining duplicate records.

You can also see the number of duplicate records removed. This can be done by specifying the showDetails option. By doing this, the DuplicatesRemoved field is added to each record kept. The value of this field is a number that indicates the count of the records removed. Specifying the showDetails option is required for running the sortby and the multiple parameters.

If you want to sort the search results, then you can specify the sortby parameter based on a field. By default the sorting happens in a lexicographical order. If you want to sort the results based on a field with numeric values, then you also sort the results numerically.

If you want to further analyze your data to see values of another field in the removed records, then you can use the multiple parameter. This can help you see multiple values of the field for the records returned by the dedup command.

dedup [N] [showDetails] <field>+ [sortby <sort-by-option>(<field>)] [multiple <field>]

In the preceding syntax, the following definitions apply:

• N indicates a number that represents the count of records with duplicate field values to be kept. By default, this number is 1.
• + indicates one or more similar expressions separated by a space
• [Expression] indicates it is optional.
• showDetails indicates whether you want to know the number of duplicate records removed for each unique field value. If you specify this option, the DuplicatesRemoved field is added to each record.
• <field> refers to field name on which you want to run this command or the field name on which you want to run the sortby or the multiple parameter.

• sortby indicates an optional parameter that you can run on a field name. You can add this parameter to sort the search results based on the value of the field specified. You can use one of the following options for sorting the search results:

  Option	Description
  num	Sorts the search results in a numerical order.
  str	Sorts the search results in a lexicographical order.

  Notes

  • Even if you do not specify the str option and only specify the field name, by default the search results are sorted in a lexicographical order.
  • This parameter can be used only when your command syntax uses the showDetails option.

• multiple <field> indicates an optional parameter that you run on a field name to see the unique values of that field occurring in the records returned by running this command.

  Note

  This parameter can be used only when your command syntax uses the showDetails option.

Short examples

Example 1: Remove duplicate search results with the same ClientIp field value.

Example 2: Remove duplicate search results containing the same values, for both the ClientIp and ResponseCode fields.

Example 3: Remove duplicate search results with the same RequestType field value. Additionally, see the number of duplicate records removed and see the unique values of the ResponseSize field in the duplicate records removed.

Example 4: Remove duplicate search results with the same RequestType field value. Additionally, see the number of duplicate records removed and sort the search results by the ResponseSize field in an ascending order.

Long examples

The following sample data and sample indexed data (displayed on the Search tab) will help you understand the examples of using the dedup command.

• Sample data
•  
• Sample indexed data
• dedup (single field)
•  
• dedup (multiple fields)
•  
• dedup with showDetails
• showDetails and multiple
•  
• showDetails and sortby (num)
•  
• showDetails, multiple, and sortby (num)
•  

Sample data

Back to examples ↑

Sample indexed data

Back to examples ↑

dedup (single field)

In this example, you use the command to remove duplicate search results with the same ClientIp field value.

Command

Output

Back to examples ↑

dedup (multiple fields)

In this example, you use the command to remove duplicate search results containing the same values, for both the ClientIp and ResponseCode fields.

Command

Output

Back to examples ↑

dedup with showDetails

In this example, you use the command to perform the following actions:

• Remove duplicate search results with the same RequestType field value.
  
• See the number of duplicate records removed (indicated by the DuplicatesRemoved field).

Command

Output

Back to examples ↑

showDetails and multiple

In this example, you use the command to perform the following actions:

• Remove duplicate search results with the same RequestType field value.
  
• See the number of duplicate records removed (indicated by the DuplicatesRemoved field).
• See the unique values of the ResponseSize field in the duplicate records removed (indicated by the ResponseSize:UniqueValues field).
  

Command

Output

Back to examples ↑

showDetails and sortby (num)

In this example, you use the command to perform the following actions:

• Remove duplicate search results with the same RequestType field value.
  
• See the number of duplicate records removed (indicated by the DuplicatesRemoved field).
• Sort the search results by the ResponseSize field in an ascending order. 

Command

Output

Back to examples ↑

showDetails, multiple, and sortby (num)

In this example, you use the command to perform the following actions:

• Remove duplicate search results with the same RequestType field value.
  
• See the number of duplicate records removed (indicated by the DuplicatesRemoved field).
• See the unique values of the ResponseSize field in the duplicate records removed (indicated by the ResponseSize:UniqueValues field).
   
• Sort the search results by the ResponseSize field in an ascending order.

Command

Output

Back to examples ↑

Notes

By specifying the showDetails option, the search results are displayed only after all the events are processed. Therefore, if you run this option on a large volume of data, the search execution might take longer to complete.