group search command
When you run this command, the following three fields are automatically added in the grouped transaction:
- duration: Provides the time lapse (in seconds) between the timestamp assigned to the group and the timestamp of the first event used to start the group formation.
The timestamp assigned to the group can be one of the following:
- If the group was completed: The end time of the last event used for completing the group.
- If the group was not completed: The time at which the group was evicted to the UI. This can also be the start time of the search duration used for running the
group
command.
- numentries: Provides the number of events grouped together in the transaction.
_status: Provides the status of the grouped transaction depending on the group options specified. For more information, see About the _status field.
You can control the way in which this command displays the output in the form of search results. For more information, see Customizing settings.
This topic contains the following information:
For a list of all search commands, see Search commands.
Syntax
group [<field-list>] <group-options>*
In the preceding syntax, the following definitions apply:
*
indicates zero or more similar expressions.- [expression] indicates it is optional.
<field-list>
refers to field names used for grouping the events into a single transaction. The field names can be separated by comma (,) or space.<group-options>
refers to the following group options to specify for creating the grouped transaction. These group options are used as conditions for distributing transaction entries.Item Syntax Description maxspan maxspan=<int>(<timescale>)?
Maximum time span (in seconds, minutes, hours) for grouping the transaction event entries.
Default: 600 seconds (10 minutes) with a maximum of 7200 seconds (2 hours)
maxpause
maxpause=<int>(<timescale>)?
Maximum interval (in seconds, minutes, hours) between the grouped transaction entries.
Default: No limit, but cannot be greater than value of the maxspan option
maxevents
maxevents=<int>
Maximum number of entries in a transaction.
Default: 100 with a maximum limit of 10,000 events
startswith
startswith=<filter-string>
Any string in the raw event data that must mark the start of the grouped transaction. endswith
endswith=<filter-string>
Any string in the raw event data that must mark the end of the grouped transaction. where,
?
indicates zero or more similar expressions<int>
refers to an integer value<timescale>
refers to a time scale such as seconds, minutes, or hours. You can use one of the following time scale options in the command syntax:- Seconds:
s
|sec
|secs
|second
|seconds
- Minutes:
m
|min
|mins
|minute
|minutes
- Hours:
h
|hr
|hrs
|hour
|hours
- Seconds:
Short examples
Example 1: Group events occurring in the span of a maximum of 2 minutes.
... | group maxspan=2m
Example 2: Group events occurring with a time gap of 1 minute but limited to a maximum of two event entries.
... | group maxpause=1m maxevents=2
Example 3: Group data entries with the same HOST field value and with no gaps of longer than 5 seconds between two entries.
...| group HOST maxpause=5s
Example 4: Group data entries with a unique combination of userid and sessionid fields. The first and last entries are within the 25-second time gap. All sessions tied with some user ID are also associated in related transactions and the user ID is not required to be present in such log entries.
...| group userid sessionid maxspan=25s
Example 5: Group data entries with the HOST field where the first entry contains signon and the last entry contains purchase. The startswith and endswith options mark the transaction boundary for log events.
...| group HOST startswith=signon endswith=purchase
login
entry and ends with the logout
entry.... | group startswith=login endswith=logout
Long examples
The following sample data and sample indexed data (displayed on the Search tab) will help you understand the examples of using the group
command.
Sample data
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png |
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png |
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png |
10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png |
Sample indexed data
10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png HOST=local.bmc.com |ResponseSize=100|COLLECTOR_NAME=u4 |ClientIp=10.1.1.141 |ResponseCode=200 |RequestType=POST|RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png HOST=local.bmc.com |ResponseSize=200|COLLECTOR_NAME=u4 |ClientIp=10.1.1.141 |ResponseCode=201 |RequestType=PUT|RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png HOST=local.bmc.com |ResponseSize=150|COLLECTOR_NAME=u4 |ClientIp=10.1.1.140 |ResponseCode=201 |RequestType=GET|RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png HOST=local.bmc.com |ResponseSize=100|COLLECTOR_NAME=u4 |ClientIp=10.1.1.140 |ResponseCode=404 |RequestType=GET|RequestURL=/themes/ComBeta/images/bullet.png |
group based on a field
In this example, you use the command to create transactions per client for analyzing a pattern of requests from the various clients.
This command groups all HTTP requests (a maximum of 500 events per transaction) for each client based on its IP address.
The value of the duration field depends on the time range for which you executed the search.
Command
... | group ClientIp
Output
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=27292 | numentries=2 | _status=timeExpiry | ResponseSize=200,100 |RequestType=PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=404,201 | HOST=local.bmc.com | duration=27172 | numentries=2 | _status=timeExpiry | ResponseSize=100,150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png |
group based on multiple fields
In this example, you use the command to create transactions for each client and each request for analyzing a pattern of requests and request types from the various clients.
This command groups all HTTP requests (a maximum of 500 events per transaction) from a client with the same request type.
The value of the duration field depends on the time range for which you executed the search.
Command
... | group ClientIp RequestType
Output
10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=200 | HOST=local.bmc.com | duration=27292 | numentries=1 | _status=timeExpiry | ClientIp=10.1.1.141 | ResponseSize=100 | RequestType=POST | RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=201 | HOST=local.bmc.com | duration=27232 | numentries=1 | _status=timeExpiry | ClientIp=10.1.1.141 | ResponseSize=200 | RequestType=PUT | RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=404,201 | HOST=local.bmc.com | duration=27172 | numentries=2 | _status=timeExpiry | ClientIp=10.1.1.140 | ResponseSize=100,150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png |
maxspan
In this example, you use the command to group events occurring in a span of two minutes.
Command
... | group maxspan=2m
Output
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=180 | numentries=3 | _status=maxSpan | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=150,200,100 | RequestType=GET,PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=404 | HOST=local.bmc.com | duration=27112| numentries=1| _status=timeExpiry| ClientIp=10.1.1.140 | ResponseSize=100 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png |
maxpause (with a gap of one minute)
In this example, you use the command to group event entries occurring with a time gap of one minute.
The value of the duration field depends on the time range for which you executed the search.
Command
... | group maxpause=1m
Output
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=404,201,200 | HOST=local.bmc.com | duration=27292 | numentries=4 | _status=timeExpiry | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=100,150,200 | RequestType=GET,PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png |
maxpause (with a gap of one second)
In this example, you use the command to group event entries occurring with a time gap of one second.
Command
... | group maxpause=1s
Output
10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=200 | HOST=local.bmc.com | duration=60 | numentries=1 | _status=maxPause | ClientIp=10.1.1.141 | ResponseSize=100 | RequestType=POST | RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=201 | HOST=local.bmc.com | duration=60 | numentries=1 | _status=maxPause | ClientIp=10.1.1.141 | ResponseSize=200 | RequestType=PUT | RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=201 | HOST=local.bmc.com | duration=60 | numentries=1 | _status=maxPause | ClientIp=10.1.1.140 | ResponseSize=150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=404 | HOST=local.bmc.com | duration=27112 | numentries=1 | _status=timeExpiry | ClientIp=10.1.1.140 | ResponseSize=100 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png |
maxpause and maxevents
In this example, you use the command to group events occurring with a time gap of one minute but limited to the number of event entries specified.
Command
... | group maxpause=1m maxevents=2
Output
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=60 | numentries=2| _status=maxEvents | ClientIp=10.1.1.141 | ResponseSize=200,100 | RequestType=PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=404,201 | HOST=local.bmc.com | duration=60| numentries=2| _status=maxEvents | ClientIp=10.1.1.140 | ResponseSize=100,150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png |
maxevents and maxspan
In this example, you use the command to group events occurring within two minutes but limited to a maximum of two event entries.
Command
... | group maxevents=2 maxspan=2m
Output
10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=60| numentries=2| _status=maxEvents | ClientIp=10.1.1.141 | ResponseSize=200,100 | RequestType=PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=404,201 | HOST=local.bmc.com | duration=60 | numentries=2 | _status=maxEvents | ClientIp=10.1.1.140 | ResponseSize=100,150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png |
startswith
In this example, you use the command to create multiple groups of events based on whether the string specified exists in the event data.
Command
... | group startswith=GET
Output
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=120 | numentries=3 | _status=complete | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=150,200,100 | RequestType=GET,PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=404 | HOST=local.bmc.com | duration=0| numentries=1| _status=complete | ClientIp=10.1.1.140 | ResponseSize=100 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png |
startswith and endswith
In this example, you use the command to create multiple groups by marking the start and end of the group, depending on whether the string specified exists in the event data.
Command
... | group startswith=GET endswith=PUT
Output
10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=201 | HOST=local.bmc.com | duration=60 | numentries=2 | _status=complete | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=150,200 | RequestType=GET,PUT | RequestURL=/themes/ComBeta/images/bullet.png |
10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png COLLECTOR_NAME=u4 | ResponseCode=404 | HOST=local.bmc.com | duration=0 | numentries=1| _status=startsWith | ClientIp=10.1.1.140 | ResponseSize=100 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png |
About the _status field
This field indicates whether or not the grouped transaction is completed. Completion of the transaction is determined by whether or not the group options specified in the search query are met. When you specify particular group options, the group begins to form on the basis of the options specified. The groups are kept in memory before being sent to the UI, until the conditions for the group completion (based on the specified options) are met. Open groups occur for all the events where the group formation started but did not complete.
The grouped transaction is marked with a particular status based on the following conditions:
- Number of group options that you specify
- Volume of data on which you are running this command
The following table describes the status available for groups:
Status | When is this status reported? |
---|---|
complete | When the condition for both |
startsWith | When the condition for the startswith option is met. |
maxSpan | When the condition for the |
maxPause | When the condition for the |
maxEvents | When the condition for the |
maxOpenGrps | When the maximum number of open groups surpass the default limit of 5000 in memory before they are sent to the UI. The events with open groups are evicted based on the FIFO (First In, First Out) policy. |
maxInMemoryEvents | When the total number of events across all open groups surpass the default limit of 100000 in memory before they are sent to the UI. The total number of events evicted when the maximum limit is reached is based on the FIFO (First In, First Out) policy. |
timeExpiry | When none of the conditions are met. |
Customizing settings
The following table provides a list of properties that you can change to change the way in which the command output is displayed.
These properties can be changed in the searchserviceCustomConfig.properties file. For more information, see Modifying the configuration files.
After changing the properties, ensure that you restart the Search component service. For more information, see Starting or stopping product services.
Property | Description |
---|---|
search.group.maxspan.sec | Maximum time span used for grouping the transaction event entries. Default: 600 (in seconds) |
search.group.maxspan.sec.limit | Limit for the maximum time span used for grouping the transaction event entries. Default: 7200 (in seconds) |
search.group.maxevents | Maximum number of entries in a grouped transaction. Default: 100 |
search.group.maxevents.limit | Limit for the maximum number of entries in a grouped transaction. Default: 10000 |
search.group.max.open.transactions | Maximum number of open groups that are still in memory before they are sent to the UI. Default: 5000 |
search.group.max.inmemory.events | Total number of events across all open groups that are still in memory before they are sent to the UI. Default: 100000 |
Notes
If in a search query, both startswith and endswith options are specified, then the following conventions apply:
If the startswith criteria is not met, the search query start time is used.
If the endswith criteria is not met, the search query end time is used.
The group maxspan defaults to 10 minutes. Therefore, you might find that transactions are automatically completed after every 10 minutes.
Using a very high value for maxspan and maxevents on data containing high cardinality fields (fields with a large number of values) can result in slow performance.
This command only accepts descending results as input. Therefore, you cannot precede this command with the head search command command or dedup search command command.
The following scenarios apply for events with incomplete transactions. Incomplete transactions refer to events with status as
maxOpenGrps
,maxInMemoryEvents
, ortimeExpiry
.- Drilling down on events with incomplete transactions on the timeline chart might not produce accurate records.
- The value of the duration field shows as the difference between the end time of the transaction and the time at which the first event occurred.
Comments
Log in or register to comment.