Receiving data over a TCP or UDP connection

To receive data over a TCP/UDP connection, you need to create the Receive over TCP/UDP data collector.

Note

The HOST field is not displayed on the Search tab for data collected over a TCP/UDP connection (by default).

However, if the data pattern used for collecting data includes the HOST field, then it is displayed and available for search.

The following information describes the use case and the process of creating this data collector:

Standalone Agent and Standalone Collection Agent

All references to the Standalone Agent or Standalone Collection Agent in this document is applicable only if you are using IT Data Analytics version 11.3.01. The latest version released for a Standalone Agent is 11.3.01. Starting from version 11.3.02, no more versions will be released for the Standalone Agent. However, you can make a note of the following information:

  • You can continue to use Standalone Agent version 11.3.01 with IT Data Analytics version 11.3.02.
  • If you have created Data Collectors using a Standalone Agent in version 11.3.01, the data collection will continue to work with IT Data Analytics version 11.3.02.
  • You can also edit the Data Collectors to use PATROL Agent instead of a Standalone Agent in IT Data Analytics version 11.3.02.

To receive data over a TCP or UDP connection

  1. Navigate to Administration > Data Collectors > Add Data Collector .
  2. In the Name box, provide a unique name to identify this data collector.
  3. From the Type list, select Receive over TCP/UDP.
  4. Provide the following information, as appropriate:

    FieldDescription
    Target/Collection Host
    Collection Host (Agent)

    Type or select the collection host depending on whether you want to use the Collection Station or the Collection Agent to perform data collection.

    The collection host is the computer on which the Collection Station or the Collection Agent is located.

    By default, the Collection Station is already selected. You can either retain the default selection or select the Collection Agent.

    Note: For this type of data collector, the target host and collection host are expected to have different values.

    Collector Inputs (Note that data becomes available for searching only after a client sends data to the specified host (on the corresponding port)).
    Protocol

    Select UDP or TCP as appropriate.

    By default, UDP is selected.

    Bind address
    Provide the IP address to which you want to bind for creating a connection.
    Port

    Provide the port to connect to the UDP/TCP protocol.

    By default, this value is set to 514.

    Note: This data collector does not work with the default port on a Linux computer. This is because only root users can access ports with values less than 1024. Users other than root users must ensure that the port value is greater than 1024. To allow the Standlone Agent or Collection Station to run with root privileges when you want to capture TCP data on the default port of 514 (which is the default for syslog), run the following commands:

    cd $BMC_ITDA_HOME/services/bin/

    chown root:root collection_station

    sh collection_station restart

    Data Pattern
    Pattern

    Assign a matching data pattern (and optionally date format) for indexing the data.

    By default the data pattern is set to Free Text without Timestamp. Unlike other data collectors, this type of data collector does not support automatic detection of relevant data patterns. And because this data collector receives events asynchronously, you cannot see a preview of the results right after selecting the data pattern.

    After creating the data collector, if you are not satisfied with the search results, then you can assign a new data pattern by manually creating it or by customizing an existing data pattern that closely matches the data that you are collecting (by using the clone feature). You can obtain the sample data for creating (or cloning) the data pattern from the search results on the Search page (data that just got indexed).The data pattern and date format together decide the way in which your data will be indexed. When you select a data pattern, the matching date format is automatically selected. However, you can override the date format by manually selecting another date format or by selecting the option to create a new date format. By doing this, the date format is used to index the date and time string, while rest of the data is indexed as per the data pattern selected. If you select only a date format, then the date format is used for indexing the timestamp, while the rest of the data is displayed in a raw format in your search results.

    For more information, see Assigning the data pattern and date format to a data collector.


    Note: Before specifying the data pattern, under the Advanced Options section, ensure that the correct file encoding is set.

    Date Format
    Date Locale

    (Optional) You can use this setting to enable reading the date and time string based on the language selected. Note that this setting only applies to those portions of the date and time string that consist letters (digits are not considered).

    By default, this value is set to English.

    You can manually select a language to override the default locale. For a list of languages supported, see Language information for IT Data Analytics

    File Encoding

    If your data file uses a character set encoding other than UTF-8 (default), then do one of the following:

    • Filter the relevant character set encodings that match the file.
      To do this, click Filter relevant charset encoding next to this field.
    • Manually scan through the list available and select an appropriate option.
    • Allow TrueSight IT Data Analytics to use a relevant character set encoding for your file by manually select the AUTO option.

    Note: This data collector receives events asynchronously; therefore, filtering the relevant file encodings and using the AUTO option are not supported for this data collector. Also, because most files are expected to use the UTF-8 file encoding, UTF-8 is set as the default option. If the data that you want to collect and index uses a format other than UTF-8, you can manually select the correct file encoding option.

    Event Delimiter

    (Optional) All the records processed using the Free Text without Timestamp data pattern are assumed to be a single line of data with a line terminator at the end of the event.

    Records are distinguished on the basis of the new line separator.

    If you want to distinguish records in a custom way, then you can specify a custom string or regular expression in the Event Delimiter box that decides where the new line starts in the data. This string or regular expression must correspond to some text in your data which appears at the beginning of a line.

     See examples

    The following regular expression distinguishes records when the line starts with "INFO" or "ERROR" or "WARN".

    ^(INFO|WARN|ERROR)

    The following regular expression distinguishes records when the line starts with “com.bmc.ola”.

    ^(com\.bmc\.ola)

    Start/Stop Collection(Optional) Select this check box if you want to start the data collection immediately.
     Advanced Options

    Ignore Data Matching Input

    (Optional) If you do not want to index certain lines in your data file, then you can ignore them by providing one of the following inputs:

    • Provide a line that consistently occurs in the event data that you want to ignore. This line will be used as the criterion to ignore data during indexing.
    • Provide a Java regular expression that will be used as the criterion for ignoring data matching the regular expression.

    Example: While using the following sample data, you can provide the following input to ignore particular lines.

    • To ignore the line containing the string, "WARN", you can specify WARN in this field.
    • To ignore lines containing the words both "WARN" and "INFO", you can specify a regular expression .*(WARN|INFO).* in this field.
    Sample data
    Sep 25, 2014 10:26:47 AM net.sf.ehcache.config.
    ConfigurationFactory parseConfiguration():134
    WARN: No configuration found. Configuring ehcache from 
    ehcache-failsafe.xml  found in the classpath:
    
    Sep 25, 2014 10:26:53 AM com.bmc.ola.metadataserver.
    MetadataServerHibernateImpl bootstrap():550
    INFO: Executing Query to check init property: select * 
    from CONFIGURATIONS where userName = 'admin' and 
    propertyName ='init'
    
    Sep 30, 2014 07:03:06 PM org.hibernate.engine.jdbc.spi.
    SqlExceptionHelper logExceptions():144
    ERROR: An SQLException was provoked by the following 
    failure: java.lang.InterruptedException
    
    Sep 30, 2014 04:39:27 PM com.bmc.ola.engine.query.
    ElasticSearchClient indexCleanupOperations():206
    INFO: IndexOptimizeTask: index: bw-2014-09-23-18-006 
    optimized of type: data
    Data Block

    Indicates the index block with which you want to associate the data collector. You can associate a data collector to one of the various index blocks, each having a configurable retention period.

    By default, this value is set to Small.

    The maximum number of index blocks allowed are 5. Besides the three defined index blocks, Small, Medium and Large, you can create two more custom index blocks.

    When you select an index block, the properties of that index block are displayed below it. The properties that are displayed are:

    • Archive: This indicates whether the data that you index using the selected index block will be archived.
    • Retention Days: This indicates the retention days associated with the index block.

    Following are the retention days associated with the typical index blocks. The retention days displayed can be as configured by your Administrator.

    Index BlockRetention
    Small7
    Medium14
    Large30
    Metrics7
    Select the index block as per your needs of retention days and the Archive status. If the Archive status is Off and you need to archive your data, contact your administrator to set the Archive status for the index block to On. For more information on how to set the archive status of the index block, see Changing System Settings.

    Note

    If you select the ITDA Metrics data pattern while creating a data collector, the Index Block field is unavailable since the Metrics Index Block is automatically associated with the data collector.

    Best Effort Collection

    (Optional) If you clear this check box, only those lines that match the data pattern are indexed; all other data is ignored. To index the non-matching lines in your data file, keep this check box selected.

    Note: Non-matching lines in the data file are indexed on the basis of the Free Text with Timestamp data pattern.

    Example: The following lines provide sample data that you can index by using the Hadoop data pattern. In this scenario, if you select this check box, all lines are indexed. But if you clear the check box, only the first two lines are indexed.

    Sample data
    2014-08-08 15:15:43,777 INFO org.apache.hadoop.hdfs.server.
    datanode.DataNode.clienttrace: src: /10.20.35.35:35983, dest: 
    /10.20.35.30:50010, bytes: 991612, op: HDFS_WRITE, cliID:
    
    2014-08-08 15:15:44,053 INFO org.apache.hadoop.hdfs.server.
    datanode.DataNode: Receiving block blk_-6260132620401037548_
    683435 src: /10.20.35.35:35983 dest: /10.20.35.30:50010
    
    2014-08-08 15:15:49,992 IDFSClient_-19587029, offset: 0, 
    srvID: DS-731595843-10.20.35.30-50010-1344428145675, 
    blockid: blk_-8867275036873170670_683436, duration: 5972783
    
    2014-08-08 15:15:50,992 IDFSClient_-19587029, offset: 0, 
    srvID: DS-731595843-10.20.35.30-50010-1344428145675, 
    blockid: blk_-8867275036873170670_683436, duration: 5972783

     Tags

    Inherit Host Level Tags From Target Host(Optional) Select this check box to inherit your tag selections associated with the target host that you selected earlier. This option is not applicable if you did not select a target host. Note: After selecting this check box, you can further manually select additional user groups. When you manually select additional user groups, both the inherited permissions as well as the manually assigned permissions are applied. To remove the inherited permissions, clear this check box.
    Select Tag name and corresponding value

    (Optional) Select a tag name and specify the corresponding value by which you want to categorize the data collected. Later while searching data, you can use these tags to narrow down your search results.

    Example: If your are collecting data from hosts located at Houston, you can select a tag name for "Location" and in the value specify "Houston". While searching the data, you can use the tag, Location="Houston" to filter data and see results associated with the Houston location.

    To be able to see tag names, you need to first add them by navigating to Administration > System Settings.

    To specify tag names and corresponding values, in the left box select a tag name and then type the corresponding tag value in the right box. While you type the value, you might see type-ahead suggestions based on values specified in the past. If you want to use one of the suggestions, click the suggestion. Click Add to add the tag name and corresponding value to the list of added tags that follow. Click Remove Tag to remove a tag.

    The tags saved while creating the data collector are displayed on the Search tab, under the Filters panel, and in the Tags section.

    Note: At a time, you can specify only one value for a tag name. To specify multiple values for the same tag name, each time you need to select the tag name, specify the corresponding value, and click Add.

    For more information about tags, see Understanding tags.

     Group Access

    Inherit Host Level Access Groups From Target Host(Optional) Select this check box to inherit your group access configurations associated with the target host that you selected earlier. This option is not applicable if you did not select a target host.

    Note: After selecting this check box, you can further manually select additional user groups. When you manually select additional user groups, both the inherited permissions as well as the manually assigned permissions are applied. To remove the inherited permissions, clear this check box.
    Select All Groups

    (Optional) Select this option if you want to select all user groups. You can also manually select multiple user groups.

    Notes: You can access data retrieved by this data collector based on the following conditions.

    • If user groups are not selected and data access control is enabled: Only the creator of the data collector can access data retrieved by this data collector.
    • If user groups are not selected and if data access control is not enabled: All users can access data retrieved by this data collector. You can restrict access permissions by selecting the relevant user groups that must be given access permissions. To enable data access control, navigate to Administration > System Settings.

    For more information, see Managing user groups in IT Data Analytics.

  5. Click Create to save your changes.

Use case for sending log4j output directly into TrueSight IT Data Analytics

If you are monitoring an application that uses log4j for logging data, you can directly collect and search this data by using TrueSight IT Data Analytics. You can collect this data in a number of ways. However, one of the easiest ways of collecting this data is to configure log4j to send data directly into TrueSight IT Data Analytics.

You can do this by adding the SyslogAppender in the log4j .properties file. After doing this, you can create the Receive over TCP/UDP data collector to receive this data on the TCP or UDP port. Doing this can help you easily search, analyze, and visualize your log4j data and get meaningful insights.

Best practice

Ensure that the collection host (where the Collection Station or Collection agent resides) is operating in the same timezone as the Log4j files you are trying to collect.

If there is no timestamp in the files, then ensure that the collection host is operating in the same timezone as the server hosting the files.

To send log4j data into TrueSight IT Data Analytics

  1. Identify the .properties file that you want to send to TrueSight IT Data Analytics.
  2. Edit the file, add details regarding the SyslogAppender, and configure appenders for the logger as follows (if it is not already present).

    The following code block provides a few example lines from one of the log4j .properties file.

    # Configured appenders for the logger
    log4j.rootLogger=SYSLOGFILE
    
    # The syslog appender to be configured for the syslog 
    configuration to affect 
    log4j.appender.SYSLOGFILE=org.apache.log4j.net.SyslogAppender
    
    # The  hostname to log the syslogger information
    log4j.appender.SYSLOGFILE.SyslogHost = CollectionHost.bmc.com
    
    # The facility name in the logger where the log file shall be logged 
    log4j.appender.SYSLOGFILE.facility=local7
    
    # If true, the SyslogAppender will generate the header (timestamp 
    and host information) in the message
    log4j.appender.SYSLOGFILE.Header=false
    
    # The log filename layout of the syslogger appender 
    log4j.appender.SYSLOGFILE.layout=org.apache.log4j.PatternLayout
    
    # The syslogger configuration pattern 
    log4j.appender.SYSLOGFILE.layout.ConversionPattern=
    %-5p %d{MMM/dd HH:mm:ss} %-20c{20} [%t] %m%n
  3. Replace the values of the following properties to be able to connect with TrueSight IT Data Analytics and save the file.

    PropertyDescription
    If you are using log4j versions prior to 1.x
    log4j.appender.SYSLOGFILE.SyslogHost
    Value must be set to the collection host name that you want to use for receiving the log4j data.
    This value can be set to the host name where the Collection Station or Collection Agent is installed.

    Note:
    Log4j versions prior to 1.x can only send data over the UDP protocol.
    If you are using log4j versions after 1.x
    host
    Value must be set to the collection host name that you want to use for receiving the log4j data.
    This value can be set to the host name where the Collection Station or Collection Agent is installed.
    port

    Value must be set to the port that you want to use for receiving the log4j data.

    Default port: 514

    protocol
    Depending on the protocol that you want to use for receiving the log4j data, value must be set to TCP or UDP.
  4. (Optional) If you want to extract the timestamp and fields from your log4j data, then it is recommended that you create a data pattern. This data pattern needs to be used while creating the data collector. For more information, see Creating data patterns.
    You can optionally extract only the timestamp and extract rest of the data as free text. To capture the timestamp, you need to create a new date format at the time of creating the data collector.
Was this page helpful? Yes No Submitting... Thank you

Comments