Page tree
Skip to end of metadata
Go to start of metadata
This search command groups events depending on the group options (conditions) specified to create a transaction of events. This command can help you correlate event data based on specific fields and provide a single transaction entry containing related event data from multiple systems, tiers, and so on.

When you run this command, the following three fields are automatically added in the grouped transaction:

  • duration: Provides the time lapse (in seconds) between the timestamp assigned to the group and the timestamp of the first event used to start the group formation.

    The timestamp assigned to the group can be one of the following:

    • If the group was completed: The end time of the last event used for completing the group.
    • If the group was not completed: The time at which the group was evicted to the UI. This can also be the start time of the search duration used for running the group command.
  • numentries: Provides the number of events grouped together in the transaction.
  • _status: Provides the status of the grouped transaction depending on the group options specified. For more information, see About the _status field.

You can control the way in which this command displays the output in the form of search results. For more information, see Customizing settings.

This topic contains the following information:

For a list of all search commands, see Search commands.

Syntax

group [<field-list>] <group-options>*

In the preceding syntax, the following definitions apply:

  • * indicates zero or more similar expressions.
  • [expression] indicates it is optional.
  • <field-list> refers to field names used for grouping the events into a single transaction. The field names can be separated by comma (,) or space.
  • <group-options> refers to the following group options to specify for creating the grouped transaction. These group options are used as conditions for distributing transaction entries.

    ItemSyntaxDescription
    maxspan

    maxspan=<int>(<timescale>)?

    Maximum time span (in seconds, minutes, hours) for grouping the transaction event entries.

    Default: 600 seconds (10 minutes) with a maximum of 7200 seconds (2 hours)

    maxpause

    maxpause=<int>(<timescale>)?

    Maximum interval (in seconds, minutes, hours) between the grouped transaction entries.

    Default: No limit, but cannot be greater than value of the maxspan option

    maxevents

    maxevents=<int>

    Maximum number of entries in a transaction.

    Default: 100 with a maximum limit of 10,000 events

    startswith

    startswith=<filter-string>

    Any string in the raw event data that must mark the start of the grouped transaction.

    endswith

    endswith=<filter-string>

    Any string in the raw event data that must mark the end of the grouped transaction.

    where,
    ? indicates zero or more similar expressions

    <int> refers to an integer value

    <timescale> refers to a time scale such as seconds, minutes, or hours. You can use one of the following time scale options in the command syntax:

    • Seconds: s | sec | secs | second | seconds
    • Minutes: m | min | mins | minute | minutes
    • Hours: h | hr | hrs | hour | hours

Short examples

Example 1: Group events occurring in the span of a maximum of 2 minutes.

... | group maxspan=2m

Example 2: Group events occurring with a time gap of 1 minute but limited to a maximum of two event entries.

... | group maxpause=1m maxevents=2

Example 3: Group data entries with the same HOST field value and with no gaps of longer than 5 seconds between two entries.

...| group HOST maxpause=5s

Example 4: Group data entries with a unique combination of userid and sessionid fields. The first and last entries are within the 25-second time gap. All sessions tied with some user ID are also associated in related transactions and the user ID is not required to be present in such log entries.

...| group userid sessionid maxspan=25s 

Example 5: Group data entries with the HOST field where the first entry contains signon and the last entry contains purchase. The startswith and endswith options mark the transaction boundary for log events.

...| group HOST startswith=signon endswith=purchase

Example 6: Create multiple groups based on a transaction that starts with the login entry and ends with the logout entry.
... | group startswith=login endswith=logout

Long examples

The following sample data and sample indexed data (displayed on the Search tab) will help you understand the examples of using the group command.

Sample data

10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 404 100
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 201 150
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
HTTP/1.1" 201 200
10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
HTTP/1.1" 200 100

Back to examples ↑

Sample indexed data

10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
HTTP/1.1" 200 100
HOST=local.bmc.com |ResponseSize=100|COLLECTOR_NAME=u4 |ClientIp=10.1.1.141 |ResponseCode=200 |RequestType=POST|RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
HTTP/1.1" 201 200
HOST=local.bmc.com |ResponseSize=200|COLLECTOR_NAME=u4 |ClientIp=10.1.1.141 |ResponseCode=201 |RequestType=PUT|RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 201 150
HOST=local.bmc.com |ResponseSize=150|COLLECTOR_NAME=u4 |ClientIp=10.1.1.140 |ResponseCode=201 |RequestType=GET|RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 404 100
HOST=local.bmc.com |ResponseSize=100|COLLECTOR_NAME=u4 |ClientIp=10.1.1.140 |ResponseCode=404 |RequestType=GET|RequestURL=/themes/ComBeta/images/bullet.png

Back to examples ↑

group based on a field

In this example, you use the command to create transactions per client for analyzing a pattern of requests from the various clients.

This command groups all HTTP requests (a maximum of 500 events per transaction) for each client based on its IP address.

The value of the duration field depends on the time range for which you executed the search.

Command

... | group ClientIp

Output

10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
HTTP/1.1" 201 200
10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png
HTTP/1.1" 200 100
COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=27292 | numentries=2 |  _status=timeExpiry | ResponseSize=200,100 |RequestType=PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 404 100
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png
HTTP/1.1" 201 150
COLLECTOR_NAME=u4 | ResponseCode=404,201 | HOST=local.bmc.com | duration=27172 | numentries=2 | _status=timeExpiry | ResponseSize=100,150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

Back to examples ↑

group based on multiple fields

In this example, you use the command to create transactions for each client and each request for analyzing a pattern of requests and request types from the various clients.

This command groups all HTTP requests (a maximum of 500 events per transaction) from a client with the same request type.

The value of the duration field depends on the time range for which you executed the search.

Command

... | group ClientIp RequestType

Output

10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
HTTP/1.1" 200 100
COLLECTOR_NAME=u4 | ResponseCode=200 | HOST=local.bmc.com | duration=27292 | numentries=1 | _status=timeExpiry | ClientIp=10.1.1.141 | ResponseSize=100 | RequestType=POST | RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
HTTP/1.1" 201 200
COLLECTOR_NAME=u4 | ResponseCode=201 | HOST=local.bmc.com | duration=27232 | numentries=1 | _status=timeExpiry | ClientIp=10.1.1.141 | ResponseSize=200 | RequestType=PUT | RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 404 100
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png
HTTP/1.1" 201 150
COLLECTOR_NAME=u4 | ResponseCode=404,201 | HOST=local.bmc.com | duration=27172 | numentries=2 | _status=timeExpiry | ClientIp=10.1.1.140 | ResponseSize=100,150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

Back to examples ↑

maxspan

In this example, you use the command to group events occurring in a span of two minutes.

Command

... | group maxspan=2m

Output

10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 201 150
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png
HTTP/1.1" 201 200
10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png
HTTP/1.1" 200 100
COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=180 | numentries=3 | _status=maxSpan | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=150,200,100 | RequestType=GET,PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 404 100
COLLECTOR_NAME=u4 | ResponseCode=404 | HOST=local.bmc.com | duration=27112| numentries=1| _status=timeExpiry| ClientIp=10.1.1.140 | ResponseSize=100 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

Back to examples ↑

maxpause (with a gap of one minute)

In this example, you use the command to group event entries occurring with a time gap of one minute.

The value of the duration field depends on the time range for which you executed the search.

Command

... | group maxpause=1m

Output

10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 404 100
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png
HTTP/1.1" 201 150
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png
HTTP/1.1" 201 200
10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png
HTTP/1.1" 200 100
COLLECTOR_NAME=u4 | ResponseCode=404,201,200 | HOST=local.bmc.com | duration=27292 | numentries=4 | _status=timeExpiry | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=100,150,200 | RequestType=GET,PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png

Back to examples ↑

maxpause (with a gap of one second)

In this example, you use the command to group event entries occurring with a time gap of one second.

Command

... | group maxpause=1s

Output

10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
HTTP/1.1" 200 100
COLLECTOR_NAME=u4 | ResponseCode=200 | HOST=local.bmc.com | duration=60 | numentries=1 | _status=maxPause | ClientIp=10.1.1.141 | ResponseSize=100 | RequestType=POST | RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
HTTP/1.1" 201 200
COLLECTOR_NAME=u4 | ResponseCode=201 | HOST=local.bmc.com | duration=60 | numentries=1 | _status=maxPause | ClientIp=10.1.1.141 | ResponseSize=200 | RequestType=PUT | RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 201 150
COLLECTOR_NAME=u4 | ResponseCode=201 | HOST=local.bmc.com | duration=60 | numentries=1 | _status=maxPause | ClientIp=10.1.1.140 | ResponseSize=150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 404 100
COLLECTOR_NAME=u4 | ResponseCode=404 | HOST=local.bmc.com | duration=27112 | numentries=1 | _status=timeExpiry | ClientIp=10.1.1.140 | ResponseSize=100 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

Back to examples ↑

maxpause and maxevents

In this example, you use the command to group events occurring with a time gap of one minute but limited to the number of event entries specified.

Command

... | group maxpause=1m maxevents=2

Output

10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
HTTP/1.1" 201 200
10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png
HTTP/1.1" 200 100
COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=60 | numentries=2| _status=maxEvents | ClientIp=10.1.1.141 | ResponseSize=200,100 | RequestType=PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 404 100
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png
HTTP/1.1" 201 150
COLLECTOR_NAME=u4 | ResponseCode=404,201 | HOST=local.bmc.com | duration=60| numentries=2| _status=maxEvents | ClientIp=10.1.1.140 | ResponseSize=100,150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

Back to examples ↑

maxevents and maxspan

In this example, you use the command to group events occurring within two minutes but limited to a maximum of two event entries.

Command

... | group maxevents=2 maxspan=2m

Output

10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
HTTP/1.1" 201 200
10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png
HTTP/1.1" 200 100
COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=60| numentries=2| _status=maxEvents | ClientIp=10.1.1.141 | ResponseSize=200,100 | RequestType=PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 404 100
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png
HTTP/1.1" 201 150
COLLECTOR_NAME=u4 | ResponseCode=404,201 | HOST=local.bmc.com | duration=60 | numentries=2 | _status=maxEvents | ClientIp=10.1.1.140 | ResponseSize=100,150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

Back to examples ↑

startswith

In this example, you use the command to create multiple groups of events based on whether the string specified exists in the event data.

Command

... | group startswith=GET

Output

10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 201 150
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png
HTTP/1.1" 201 200
10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png
HTTP/1.1" 200 100
COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=120 | numentries=3 | _status=complete | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=150,200,100 | RequestType=GET,PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 404 100
COLLECTOR_NAME=u4 | ResponseCode=404 | HOST=local.bmc.com | duration=0| numentries=1| _status=complete | ClientIp=10.1.1.140 | ResponseSize=100 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

Back to examples ↑

startswith and endswith

In this example, you use the command to create multiple groups by marking the start and end of the group, depending on whether the string specified exists in the event data.

Command

... | group startswith=GET endswith=PUT

Output

10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 201 150
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png
HTTP/1.1" 201 200
COLLECTOR_NAME=u4 | ResponseCode=201 | HOST=local.bmc.com | duration=60 | numentries=2 | _status=complete | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=150,200 | RequestType=GET,PUT | RequestURL=/themes/ComBeta/images/bullet.png
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
HTTP/1.1" 404 100
COLLECTOR_NAME=u4 | ResponseCode=404 | HOST=local.bmc.com | duration=0 | numentries=1| _status=startsWith | ClientIp=10.1.1.140 | ResponseSize=100 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

Back to examples ↑

About the _status field

This field indicates whether or not the grouped transaction is completed. Completion of the transaction is determined by whether or not the group options specified in the search query are met. When you specify particular group options, the group begins to form on the basis of the options specified. The groups are kept in memory before being sent to the UI, until the conditions for the group completion (based on the specified options) are met. Open groups occur for all the events where the group formation started but did not complete.

The grouped transaction is marked with a particular status based on the following conditions:

  • Number of group options that you specify
  • Volume of data on which you are running this command

The following table describes the status available for groups:

StatusWhen is this status reported?
complete

When the condition for both startswith and endswith options together is met.

startsWithWhen the condition for the startswith option is met.
maxSpan

When the condition for the maxspan option is met.

maxPause

When the condition for the maxpause option is met.

maxEvents

When the condition for the maxevents option is met.

maxOpenGrps

When the maximum number of open groups surpass the default limit of 5000 in memory before they are sent to the UI.

The events with open groups are evicted based on the FIFO (First In, First Out) policy.

maxInMemoryEvents

When the total number of events across all open groups surpass the default limit of 100000 in memory before they are sent to the UI.

The total number of events evicted when the maximum limit is reached is based on the FIFO (First In, First Out) policy.

timeExpiryWhen none of the conditions are met.

Customizing settings

The following table provides a list of properties that you can change to change the way in which the command output is displayed.

These properties can be changed in the searchserviceCustomConfig.properties file. For more information, see Modifying the configuration files.

After changing the properties, ensure that you restart the Search component service. For more information, see Starting or stopping product services.

PropertyDescription
search.group.maxspan.sec

Maximum time span used for grouping the transaction event entries.

Default: 600 (in seconds)

search.group.maxspan.sec.limit

Limit for the maximum time span used for grouping the transaction event entries.

Default: 7200 (in seconds)

search.group.maxevents

Maximum number of entries in a grouped transaction.

Default: 100

search.group.maxevents.limit

Limit for the maximum number of entries in a grouped transaction.

Default: 10000

search.group.max.open.transactions

Maximum number of open groups that are still in memory before they are sent to the UI.

Default: 5000

search.group.max.inmemory.events

Total number of events across all open groups that are still in memory before they are sent to the UI.

Default: 100000

Notes

  • If in a search query, both startswith and endswith options are specified, then the following conventions apply:

    • If the startswith criteria is not met, the search query start time is used.

    • If the endswith criteria is not met, the search query end time is used.

  • The group maxspan defaults to 10 minutes. Therefore, you might find that transactions are automatically completed after every 10 minutes.

  • Using a very high value for maxspan and maxevents on data containing high cardinality fields (fields with a large number of values) can result in slow performance.

  • This command only accepts descending results as input. Therefore, you cannot precede this command with the head search command command or dedup search command command.

  • The following scenarios apply for events with incomplete transactions. Incomplete transactions refer to events with status as maxOpenGrps, maxInMemoryEvents, or timeExpiry.

    • Drilling down on events with incomplete transactions on the timeline chart might not produce accurate records.
    • The value of the duration field shows as the difference between the end time of the transaction and the time at which the first event occurred.

5 Comments

  1. Hello

    In many examples from this docs page, the _status value is equal to timeExpiry and the duration value doesn't match whith the events timestamp.

    I am running an ITDA PoC and I am having the same "wierd" results. Is there a bug with the group command timings agregation?


    10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 404 100
    10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png
    HTTP/1.1" 201 150
    10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png
    HTTP/1.1" 201 200
    10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png
    HTTP/1.1" 200 100
    COLLECTOR_NAME=u4 | ResponseCode=404,201,200 | HOST=local.bmc.com | duration=27292 | numentries=4 | _status=timeExpiry | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=100,150,200 | RequestType=GET,PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png
    1. Thank you for your comment.

      We have referred this to the team for discussion. We will get back to you shortly.

      Regards,

    2. Hello Alvaro Hojas,

      The team would like to know more about this. Could you please share the query that you used to get these results? 

      Thank you and regards,



      1.  

        1.