• Spaces
  • Collections
  • Help
    • ×
     
     
    •  
    BMC TrueSight IT Data Analytics 2.7

    Page tree

    This search command groups events depending on the group options (conditions) specified to create a transaction of events. This command can help you correlate event data based on specific fields and provide a single transaction entry containing related event data from multiple systems, tiers, and so on.

    When you run this command, the following three fields are automatically added in the grouped transaction:

    • duration: Provides the time lapse (in seconds) between the timestamp assigned to the group and the timestamp of the first event used to start the group formation.

      The timestamp assigned to the group can be one of the following:

      • If the group was completed: The end time of the last event used for completing the group.
      • If the group was not completed: The time at which the group was evicted to the UI. This can also be the start time of the search duration used for running the group command.
    • numentries: Provides the number of events grouped together in the transaction.

    • _status: Provides the status of the grouped transaction depending on the group options specified. For more information, see About the _status field.

    You can control the way in which this command displays the output in the form of search results. For more information, see Customizing settings.

    This topic contains the following information:

    For a list of all search commands, see Search commands.

    Related topics

    Getting started with search

    Syntax

    group [<field-list>] <group-options>*

    In the preceding syntax, the following definitions apply:

    • * indicates zero or more similar expressions.
    • [expression] indicates it is optional.
    • <field-list> refers to field names used for grouping the events into a single transaction. The field names can be separated by comma (,) or space.

    • <group-options> refers to the following group options to specify for creating the grouped transaction. These group options are used as conditions for distributing transaction entries.

      ItemSyntaxDescription
      maxspan

      maxspan=<int>(<timescale>)?

      Maximum time span (in seconds, minutes, hours) for grouping the transaction event entries.

      Default: 600 seconds (10 minutes) with a maximum of 7200 seconds (2 hours)

      maxpause

      maxpause=<int>(<timescale>)?

      Maximum interval (in seconds, minutes, hours) between the grouped transaction entries.

      Default: No limit, but cannot be greater than value of the maxspan option

      maxevents

      maxevents=<int>

      Maximum number of entries in a transaction.

      Default: 100 with a maximum limit of 10,000 events

      startswith

      startswith=<filter-string>

      		Any string in the raw event data that must mark the start of the grouped transaction.

      endswith

      endswith=<filter-string>

      		Any string in the raw event data that must mark the end of the grouped transaction.

      where,
      ? indicates zero or more similar expressions

      <int> refers to an integer value

      <timescale> refers to a time scale such as seconds, minutes, or hours. You can use one of the following time scale options in the command syntax:

      • Seconds: s | sec | secs | second | seconds
      • Minutes: m | min | mins | minute | minutes
      • Hours: h | hr | hrs | hour | hours

    Short examples

    Example 1: Group events occurring in the span of a maximum of 2 minutes.

    ... | group maxspan=2m

    Example 2: Group events occurring with a time gap of 1 minute but limited to a maximum of two event entries.

    ... | group maxpause=1m maxevents=2

    Example 3: Group data entries with the same HOST field value and with no gaps of longer than 5 seconds between two entries.

    ...| group HOST maxpause=5s

    Example 4: Group data entries with a unique combination of userid and sessionid fields. The first and last entries are within the 25-second time gap. All sessions tied with some user ID are also associated in related transactions and the user ID is not required to be present in such log entries.

    ...| group userid sessionid maxspan=25s 

    Example 5: Group data entries with the HOST field where the first entry contains signon and the last entry contains purchase. The startswith and endswith options mark the transaction boundary for log events.

    ...| group HOST startswith=signon endswith=purchase

    Example 6: Create multiple groups based on a transaction that starts with the login entry and ends with the logout entry.
    ... | group startswith=login endswith=logout

    Long examples

    The following sample data and sample indexed data (displayed on the Search tab) will help you understand the examples of using the group command.

    Sample data

    10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 404 100
    10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 150
    10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 200
    10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 200 100

    Back to examples ↑

    Sample indexed data

    10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 200 100
    HOST=local.bmc.com |ResponseSize=100|COLLECTOR_NAME=u4 |ClientIp=10.1.1.141 |ResponseCode=200 |RequestType=POST|RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 200
    HOST=local.bmc.com |ResponseSize=200|COLLECTOR_NAME=u4 |ClientIp=10.1.1.141 |ResponseCode=201 |RequestType=PUT|RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 150
    HOST=local.bmc.com |ResponseSize=150|COLLECTOR_NAME=u4 |ClientIp=10.1.1.140 |ResponseCode=201 |RequestType=GET|RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 404 100
    HOST=local.bmc.com |ResponseSize=100|COLLECTOR_NAME=u4 |ClientIp=10.1.1.140 |ResponseCode=404 |RequestType=GET|RequestURL=/themes/ComBeta/images/bullet.png

    Back to examples ↑

    group based on a field

    In this example, you use the command to create transactions per client for analyzing a pattern of requests from the various clients.

    This command groups all HTTP requests (a maximum of 500 events per transaction) for each client based on its IP address.

    The value of the duration field depends on the time range for which you executed the search.

    Command

    ... | group ClientIp

    Output

    10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 200 
    10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 200 100
    COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=27292 | numentries=2 |  _status=timeExpiry | ResponseSize=200,100 |RequestType=PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 404 100
    10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 150
    COLLECTOR_NAME=u4 | ResponseCode=404,201 | HOST=local.bmc.com | duration=27172 | numentries=2 | _status=timeExpiry | ResponseSize=100,150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

    Back to examples ↑

    group based on multiple fields

    In this example, you use the command to create transactions for each client and each request for analyzing a pattern of requests and request types from the various clients.

    This command groups all HTTP requests (a maximum of 500 events per transaction) from a client with the same request type.

    The value of the duration field depends on the time range for which you executed the search.

    Command

    ... | group ClientIp RequestType

    Output

    10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 200 100
    COLLECTOR_NAME=u4 | ResponseCode=200 | HOST=local.bmc.com | duration=27292 | numentries=1 | _status=timeExpiry | ClientIp=10.1.1.141 | ResponseSize=100 | RequestType=POST | RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 200
    COLLECTOR_NAME=u4 | ResponseCode=201 | HOST=local.bmc.com | duration=27232 | numentries=1 | _status=timeExpiry | ClientIp=10.1.1.141 | ResponseSize=200 | RequestType=PUT | RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 404 100
    10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 150
    COLLECTOR_NAME=u4 | ResponseCode=404,201 | HOST=local.bmc.com | duration=27172 | numentries=2 | _status=timeExpiry | ClientIp=10.1.1.140 | ResponseSize=100,150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

    Back to examples ↑

    maxspan

    In this example, you use the command to group events occurring in a span of two minutes.

    Command

    ... | group maxspan=2m

    Output

    10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 150
    10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 200
    10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 200 100
    COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=180 | numentries=3 | _status=maxSpan | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=150,200,100 | RequestType=GET,PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 404 100
    COLLECTOR_NAME=u4 | ResponseCode=404 | HOST=local.bmc.com | duration=27112| numentries=1| _status=timeExpiry| ClientIp=10.1.1.140 | ResponseSize=100 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

    Back to examples ↑

    maxpause (with a gap of one minute)

    In this example, you use the command to group event entries occurring with a time gap of one minute.

    The value of the duration field depends on the time range for which you executed the search.

    Command

    ... | group maxpause=1m

    Output

    10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 404 100
    10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 150
    10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 200
    10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 200 100
    COLLECTOR_NAME=u4 | ResponseCode=404,201,200 | HOST=local.bmc.com | duration=27292 | numentries=4 | _status=timeExpiry | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=100,150,200 | RequestType=GET,PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png

    Back to examples ↑

    maxpause (with a gap of one second)

    In this example, you use the command to group event entries occurring with a time gap of one second.

    Command

    ... | group maxpause=1s

    Output

    10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 200 100
    COLLECTOR_NAME=u4 | ResponseCode=200 | HOST=local.bmc.com | duration=60 | numentries=1 | _status=maxPause | ClientIp=10.1.1.141 | ResponseSize=100 | RequestType=POST | RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 200
    COLLECTOR_NAME=u4 | ResponseCode=201 | HOST=local.bmc.com | duration=60 | numentries=1 | _status=maxPause | ClientIp=10.1.1.141 | ResponseSize=200 | RequestType=PUT | RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 150
    COLLECTOR_NAME=u4 | ResponseCode=201 | HOST=local.bmc.com | duration=60 | numentries=1 | _status=maxPause | ClientIp=10.1.1.140 | ResponseSize=150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 404 100
    COLLECTOR_NAME=u4 | ResponseCode=404 | HOST=local.bmc.com | duration=27112 | numentries=1 | _status=timeExpiry | ClientIp=10.1.1.140 | ResponseSize=100 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

    Back to examples ↑

    maxpause and maxevents

    In this example, you use the command to group events occurring with a time gap of one minute but limited to the number of event entries specified.

    Command

    ... | group maxpause=1m maxevents=2

    Output

    10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 200
    10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 200 100
    COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=60 | numentries=2| _status=maxEvents | ClientIp=10.1.1.141 | ResponseSize=200,100 | RequestType=PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 404 100
    10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 150
    COLLECTOR_NAME=u4 | ResponseCode=404,201 | HOST=local.bmc.com | duration=60| numentries=2| _status=maxEvents | ClientIp=10.1.1.140 | ResponseSize=100,150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

    Back to examples ↑

    maxevents and maxspan

    In this example, you use the command to group events occurring within two minutes but limited to a maximum of two event entries.

    Command

    ... | group maxevents=2 maxspan=2m

    Output

    10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 200
    10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 200 100
    COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=60| numentries=2| _status=maxEvents | ClientIp=10.1.1.141 | ResponseSize=200,100 | RequestType=PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 404 100
    10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 150
    COLLECTOR_NAME=u4 | ResponseCode=404,201 | HOST=local.bmc.com | duration=60 | numentries=2 | _status=maxEvents | ClientIp=10.1.1.140 | ResponseSize=100,150 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

    Back to examples ↑

    startswith

    In this example, you use the command to create multiple groups of events based on whether the string specified exists in the event data.

    Command

    ... | group startswith=GET

    Output

    10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 150
    10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 200
    10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 200 100
    COLLECTOR_NAME=u4 | ResponseCode=201,200 | HOST=local.bmc.com | duration=120 | numentries=3 | _status=complete | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=150,200,100 | RequestType=GET,PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 404 100
    COLLECTOR_NAME=u4 | ResponseCode=404 | HOST=local.bmc.com | duration=0| numentries=1| _status=complete | ClientIp=10.1.1.140 | ResponseSize=100 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

    Back to examples ↑

    startswith and endswith

    In this example, you use the command to create multiple groups by marking the start and end of the group, depending on whether the string specified exists in the event data.

    Command

    ... | group startswith=GET endswith=PUT

    Output

    10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 150
    10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 201 200
    COLLECTOR_NAME=u4 | ResponseCode=201 | HOST=local.bmc.com | duration=60 | numentries=2 | _status=complete | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=150,200 | RequestType=GET,PUT | RequestURL=/themes/ComBeta/images/bullet.png
    10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
    HTTP/1.1" 404 100
    COLLECTOR_NAME=u4 | ResponseCode=404 | HOST=local.bmc.com | duration=0 | numentries=1| _status=startsWith | ClientIp=10.1.1.140 | ResponseSize=100 | RequestType=GET | RequestURL=/themes/ComBeta/images/bullet.png

    Back to examples ↑

    About the _status field

    This field indicates whether or not the grouped transaction is completed. Completion of the transaction is determined by whether or not the group options specified in the search query are met. When you specify particular group options, the group begins to form on the basis of the options specified. The groups are kept in memory before being sent to the UI, until the conditions for the group completion (based on the specified options) are met. Open groups occur for all the events where the group formation started but did not complete.

    The grouped transaction is marked with a particular status based on the following conditions:

    • Number of group options that you specify
    • Volume of data on which you are running this command

    The following table describes the status available for groups:

    StatusWhen is this status reported?
    complete

    When the condition for both startswith and endswith options together is met.

    startsWithWhen the condition for the startswith option is met.
    maxSpan

    When the condition for the maxspan option is met.

    maxPause

    When the condition for the maxpause option is met.

    maxEvents

    When the condition for the maxevents option is met.

    maxOpenGrps

    When the maximum number of open groups surpass the default limit of 5000 in memory before they are sent to the UI.

    The events with open groups are evicted based on the FIFO (First In, First Out) policy.

    maxInMemoryEvents

    When the total number of events across all open groups surpass the default limit of 100000 in memory before they are sent to the UI.

    The total number of events evicted when the maximum limit is reached is based on the FIFO (First In, First Out) policy.

    timeExpiryWhen none of the conditions are met.

    Customizing settings

    The following table provides a list of properties that you can change to change the way in which the command output is displayed.

    These properties can be changed in the searchserviceCustomConfig.properties file. For more information, see Modifying the configuration files.

    After changing the properties, ensure that you restart the Search component service. For more information, see Starting or stopping product services.

    PropertyDescription
    search.group.maxspan.sec

    Maximum time span used for grouping the transaction event entries.

    Default: 600 (in seconds)

    search.group.maxspan.sec.limit

    Limit for the maximum time span used for grouping the transaction event entries.

    Default: 7200 (in seconds)

    search.group.maxevents

    Maximum number of entries in a grouped transaction.

    Default: 100

    search.group.maxevents.limit

    Limit for the maximum number of entries in a grouped transaction.

    Default: 10000

    search.group.max.open.transactions

    Maximum number of open groups that are still in memory before they are sent to the UI.

    Default: 5000

    search.group.max.inmemory.events

    Total number of events across all open groups that are still in memory before they are sent to the UI.

    Default: 100000

    Notes

    • If in a search query, both startswith and endswith options are specified, then the following conventions apply:

      • If the startswith criteria is not met, the search query start time is used.

      • If the endswith criteria is not met, the search query end time is used.

    • The group maxspan defaults to 10 minutes. Therefore, you might find that transactions are automatically completed after every 10 minutes.

    • Using a very high value for maxspan and maxevents on data containing high cardinality fields (fields with a large number of values) can result in slow performance.

    • This command only accepts descending results as input. Therefore, you cannot precede this command with the head search command command or dedup search command command.

    • The following scenarios apply for events with incomplete transactions. Incomplete transactions refer to events with status as maxOpenGrps, maxInMemoryEvents, or timeExpiry.

      • Drilling down on events with incomplete transactions on the timeline chart might not produce accurate records.
      • The value of the duration field shows as the difference between the end time of the transaction and the time at which the first event occurred.

    5 Comments

    1. Alvaro Hojas

      Hello

      In many examples from this docs page, the _status value is equal to timeExpiry and the duration value doesn't match whith the events timestamp.

      I am running an ITDA PoC and I am having the same "wierd" results. Is there a bug with the group command timings agregation?


      10.1.1.140 - - [11/Jul/2013:15:01:52 -0700] "GET /themes/ComBeta/images/bullet.png 
      HTTP/1.1" 404 100
      10.1.1.140 - - [11/Jul/2013:15:02:52 -0700] "GET /themes/ComBeta/images/bullet.png 
      HTTP/1.1" 201 150
      10.1.1.141 - - [11/Jul/2013:15:03:52 -0700] "PUT /themes/ComBeta/images/bullet.png 
      HTTP/1.1" 201 200
      10.1.1.141 - - [11/Jul/2013:15:04:52 -0700] "POST /themes/ComBeta/images/bullet.png 
      HTTP/1.1" 200 100
      COLLECTOR_NAME=u4 | ResponseCode=404,201,200 | HOST=local.bmc.com | duration=27292 | numentries=4 | _status=timeExpiry | ClientIp=10.1.1.140,10.1.1.141 | ResponseSize=100,150,200 | RequestType=GET,PUT,POST | RequestURL=/themes/ComBeta/images/bullet.png

      1. Sonali Deshmukh

        Thank you for your comment.

        We have referred this to the team for discussion. We will get back to you shortly.

        Regards,

      2. Sonali Deshmukh

        Hello Alvaro Hojas,

        The team would like to know more about this. Could you please share the query that you used to get these results? 

        Thank you and regards,



        1. Harihara Subramanian

           

          1. Harihara Subramanian

             

    © Copyright 2005-2024 BMC Software, Inc. Use of this site signifies your acceptance of BMC’s Terms of Use . BMC, the BMC logo, and other BMC marks are assets of BMC Software, Inc. These trademarks are registered and may be registered in the U.S. and in other countries.