You can create a data collector for monitoring data by using an SSH connection to a Microsoft Windows or Linux computer and retrieving event data.
This topic contains the following information:
Navigate to Administration > Data Collectors > Add Data Collector.
In the Name box, provide a unique name to identify this data collector.
From the Type list, select Monitor File over SSH.
Provide the following information, as appropriate:
(Optional) Select from a list of hosts that you have already configured under Administration > Hosts.
The target host is the computer from which you want to retrieve the data. You can choose to select the target host and inherit the host-level tags and group access permissions already added to the host, or manually enter the host name in the Server Name field.
|Collection Host (Agent)|
Type or select the collection host depending on whether you want to use the Collection Station or the Collection Agent to perform data collection.
The collection host is the computer on which the Collection Station or the Collection Agent is located.
By default, the Collection Station is already selected. You can either retain the default selection or select the Collection Agent.
Note: For this type of data collector, the target host and collection host are expected to have different values.
Enter the host name of the server from which you want to retrieve the data.
Note: If you selected a target host earlier, this field is automatically populated. The value of this field is necessary for generating the "HOST" field that enables effective data search.
(Optional) Select one of the following options:
Provide the user name for connecting with the server from which you want to retrieve the data.
Note: This field is disabled if you applied a security profile earlier.
The product supports only password-based authentication for connecting with the SSH server.
Provide the password for connecting with the server from which you want to retrieve the data.
Click Add Credential Administration > Credentials., provide a credential name, and click OK to create a new credential (profile) from the credentials that you provided in the user name and password fields. Once this credential is created, it is displayed under
Note: This field is disabled if you applied a security credential earlier.
Provide the absolute path of the data file.To retrieve data files from subdirectories, provide the path up to the parent directory.
|Include sub-directories||(Optional) Select this check box if you want to retrieve log files from subdirectories of the file path specified.|
Specify the file name only, or specify the file name with a rollover pattern to identify subsequent logs.
You can use the following wild card characters:
Specifying a rollover pattern can be useful to monitor rolling log files where the log files are saved with the same name but differentiated with some variable like the time stamp or a number.
Note: Ensure that you specify a rollover pattern for identifying log files that follow the same data format (which means they will be indexed with the same data pattern).
Suppose you want to collect log files saved with succeeding numbers once they reach a certain size; for example:
Rollover pattern: In this scenario, you can specify the rollover pattern as IAS?.log.
Suppose you want to collect log files that roll over every hour and are saved with the same date but a different time stamp in the YYYY-MM-DD-HH format; for example:
Rollover pattern: In this scenario, you can specify the rollover pattern as 2013-10-01-*.log.
By default, the Use file time zone option is selected. This means the data is indexed as per the time zone available in the data file. If the data file does not contain a timezone, then the by default the time zone of the Collection Host (Collection Station or Collection Agent) server is used.
You can also manually select a timezone from the list available. This timezone must match the timezone of the server from which you want to collect data. If your data file contains a timezone and you manually specify the timezone, then the manually specified timezone overrides the file timezone.
Select the data pattern to use for indexing the data file.
Use one of the following methods to specify the data pattern:
Note: If you select both – a pattern and a date format, the product uses the date format to index the timestamp and the pattern to index rest of the event data.
When you select a data pattern, the matching date format is automatically updated. However, you can specifically find date formats matching the timestamp in your data file.
Use one of the following methods to specify a date format:
You can use this setting to enable reading the date and time string based on the language selected. Note that this setting only applies to those portions of the date and time string that consist letters (digits are not considered).
By default, this value is set to English.
You can manually select a language to override the default locale. For a list of languages supported, see Language information.
|Poll Interval (mins)|
Enter a number to specify the poll interval (in minutes) for the log collection.
By default, this value is set to 1.
|Start/Stop Collection||(Optional) Select this check box if you want to start the data collection immediately.|
If your data file uses a character set encoding other than UTF-8 (default), then do one of the following:
|Ignore Data Matching Input|
(Optional) If you do not want to index certain lines in your data file, then you can ignore them by providing one of the following inputs:
Example: While using the following sample data, you can provide the following input to ignore particular lines.
|Best Effort Collection|
(Optional) If you clear this check box, only those lines that match the data pattern are indexed; all other data is ignored. To index the non-matching lines in your data file, keep this check box selected.
Note: Non-matching lines in the data file are indexed on the basis of the Free Text with Timestamp data pattern.
Example: The following lines provide sample data that you can index by using the Hadoop data pattern. In this scenario, if you select this check box, all lines are indexed. But if you clear the check box, only the first two lines are indexed.
|Host Key Fingerprint|
(Optional) Provide the fingerprint of the RSA host key to connect with the server from which you want to retrieve the data.
This is the host key that is configured to be used by the SSH server with which you want to connect.
For more information, see About the SSH host key fingerprint (BMC contributor page).
|Log File Contains Header|
(Optional) Providing this value is mandatory only if you are trying collect a file that contains a constant header which must not be indexed.
The value must be the actual header appearing in the data.
|Log File Contains Footer|
(Optional) Providing this value is mandatory only if you are trying collect a file that contains a constant footer which must not be indexed.
The value must be the actual footer appearing in the data.
|Inherit Host Level Tags From Target Host||(Optional) Select this check box to inherit your tag selections associated with the target host that you selected earlier. This option is not applicable if you did not select a target host. Note: After selecting this check box, you can further manually select additional user groups. When you manually select additional user groups, both the inherited permissions as well as the manually assigned permissions are applied. To remove the inherited permissions, clear this check box.|
|Select Tag name and corresponding value|
(Optional) Select a tag name and specify the corresponding value by which you want to categorize the data collected. Later while searching data, you can use these tags to narrow down your search results.
Example: If your are collecting data from hosts located at Houston, you can select a tag name for "Location" and in the value specify "Houston". While searching the data, you can use the tag, Location="Houston" to filter data and see results associated with the Houston location.
To be able to see tag names, you need to first add them by navigating to Administration > System Settings.To specify tag names and corresponding values, in the left box select a tag name and then type the corresponding tag value in the right box. While you type the value, you might see type-ahead suggestions based on values specified in the past. If you want to use one of the suggestions, click the suggestion. Click Add to add the tag name and corresponding value to the list of added tags that follow. Click Remove Tag to remove a tag.
The tags saved while creating the data collector are displayed on the Search tab, under the Filters panel, and in the Tags section.
Note: At a time, you can specify only one value for a tag name. To specify multiple values for the same tag name, each time you need to select the tag name, specify the corresponding value, and click Add.
For more information about tags, see Understanding fields.
|Inherit Host Level Access Groups From Target Host||(Optional) Select this check box to inherit your group access configurations associated with the target host that you selected earlier. This option is not applicable if you did not select a target host. |
Note: After selecting this check box, you can further manually select additional user groups. When you manually select additional user groups, both the inherited permissions as well as the manually assigned permissions are applied. To remove the inherited permissions, clear this check box.
|Select All Groups|
(Optional) Select this option if you want to select all user groups. You can also manually select multiple user groups.
Notes: You can access data retrieved by this data collector based on the following conditions.
For more information, see Managing user groups in IT Data Analytics.
To understand the troubleshooting scenarios related to this data collector, see Troubleshooting common issues with the Category filter set to Data collection.