The data collector collects data and sends it to the Indexer for indexing. When you perform a search, the indexed data is made available as a series of individual records (or search results). You can also view these search results in the form of charts. For more information, see search results.
This topic contains information about the following aspects regarding the data collection process:
Data collectors are responsible for actually collecting your data, indexing it, and making it available for search.
Data collectors contain inputs necessary for connecting with the target host (where data resides) and other inputs such as the data pattern, the rollover pattern if you are collecting rolling logs, and so on. You can collect data by creating data collectors on the Administration > Data Collectors tab. Data collectors can be of various types; depending on the type of data you want to collect, you need to create data collectors of that type. For example, if you want to collect Windows events, you must create a data collector of the type, Monitor Windows events.
You can collect the following kinds of data:
You cannot collect data that contains non-English characters appearing as the time stamp.
You can collect data for one-time or continuous monitoring.
When you create a data collector, at a minimum, you need to specify information about:
This information is used by the Indexer to index data and make it available in the form of events that can be searched immediately. If the manner in which the data was indexed is not as per your requirement, you can modify the data pattern and see if the results match your criteria.
The data that you want to collect can be on the same computer on which theis installed (local data), or it can be on a different computer (remote data). You can collect data remotely by creating an SSH connection or connecting to a shared network drive on a Windows computer.
For more information, see Agent types.
You can collect data by using one of the following collection mechanisms:
At the time of data collection, the product automatically extracts particular knowledge from the data such as the timestamp present in the data and fields. But bulk of the extraction happens depending on the data pattern used for collecting the data. Data patterns are objects that define the pattern in which to extract, organize, and categorize data. Based on the data pattern, the data collector collects data and makes it available as a series of individual records, on which you can search. For more information, see Managing data patterns.
Fields are name=value pairings such as HOST=clm.bmc.com that add meaning to the data and help you search more effectively. They help you classify particular portions in your data that might otherwise go unnoticed. Fields are extracted automatically by the product (such as timestamp and name=value pairs already present in the data) and additionally defined by the data pattern. Fields act as the building blocks for running search commands and creating dashboards. You can additionally define tags to group or categorize fields with similar values. For example, you can add a tag called Location with the values Houston, San Jose, and California. These tags can be added to your search query to help improve your search. Tags can be added at the time of creating a data collector. For more information about fields and tags, see Understanding fields.
You can collect data for one-time or continuous monitoring. This depends on the polling interval defined in the data collector and the type of data collector. For more information, see Data collector types.
The following table lists the supported data collector types categorized on the basis of the data sources and whether you want to perform local or remote collection:
|Data sources||Description||Data collectors||Local / remote?|
|Files and directories|
Collect data that comes from various files and directories.
Note: The Upload File data collector can be used to upload a file for one-time collection of data.
|Monitor file over SSH||Remote|
|Monitor file over Windows share||Remote|
|Script outputs||Collect data that is generated as a result of running a script.||Local|
|Monitor script output over SSH||Remote|
|Windows events||Collect and index Windows events remotely.||Monitor remote Windows events||Remote|
|Collect and index Windows events locally.||Monitor local Windows events||Local|
|Events from external systems|
Collect events directly from supported external systems such as BMC ProactiveNet, or BMC TrueSight Infrastructure Management.
|Data receivers||Collect Syslog events over a TCP or UDP connection.||Remote|
|Collect data over an HTTP or HTTPS connection.||Receive over HTTP/HTTPS||Remote|
In the process of creating data collectors, various functions are supported. For example, you can add tags that will eventually help you search data effectively. You can assign group permissions by which the data collected will be accessible only to particular user groups, and so on. These functions can be performed at an individual data collector level. A list of functions is available at Functions available while creating data collectors. These functions might vary depending on the data collector type that you select.
There are other functions related to managing multiple data collectors already created. For example, changing the maximum data retention period, starting or stopping data collection, and refreshing the data collector list. These functions can be performed by navigating to the Administration > Data Collectors page. For more information about these functions, see Viewing and searching configured data collectors.
Based on the following information available on the Administration > Data Collectors page, you can conclude the data collection status.
|Data collector polling status|
Polling status can be of the following types:
|Data collector state|
Information about whether the data collector was started or stopped, and whether the data collection is in-progress or has completed.
This information is available under the State column.
For more information, see Data collection status.
After creating a data collector, data collection starts when the first poll happens.
Data retention period has implications on the Read from Past (# days) setting. The Read from Past (#days) setting in the data collector determines the number of days in the past for which data must be read and indexed. The value of this setting must be lesser than or equal to the data retention period. This setting is available for the following data collectors only:
The data retention period acts as a moving window (depicted in green in the following figure).
Consider that on the following scale of time, you created a data collector at time T1, now data collection starts from T1 when the first poll happens. Data collected at T1 remains in the system until T1+7. As time passes, the data older than the seven days period starts getting deleted and is no longer available for searching.
After the data collector is created, it might take some time (approximately 1 minute) for the first poll to happen. The first poll is used to make the data collector ready for data collection. The data is fetched only from the second poll.
Expected time delay (to see the first set of data for a search) = (Time for first poll) + (Poll interval set for the data collector).
The following table provides scenarios in which data might not be collected and therefore is not searchable.
|When a data collector is stopped and then started.||The time for which the data collector remains down.|
|When a new Collection Station is added to the pool.||This involves restart of all the Collection Agents. Data is not be collected for the time taken by the Collection Agents to restart.|
|When the configuremasters is run.||This involves the restart of the Collection Station. Data is not be collected for the time taken by the Collection Station to restart.|
|When the movecomponents is run.||This involves the restart of all the Collection Agents. Data is not be collected for the time taken by the Collection Agents to restart.|
|When the data collector is started, past data is not collected.|
|If the Payload channel is full.||This can occur due to a number of reasons. For example, if the Collection Station is not reachable or there is a sudden burst in incoming data.|
The Data Collectors tab allows you to manage data collectors. To access this tab, navigate to Administration > Data Collectors.
This tab displays a default data collector for collecting the data in the Collection_metrics.log file. The Search tab displays a graph summarizing the data collected by this data collector. For more information, see Collecting product metrics.
You can perform the following actions on the Data Collectors tab.
|Add Data Collector|
Add a new data collector.
For more information, see Creating data collectors.
|Edit Data Collector |
Edit the selected data collector.
You can modify the same details that you provided while adding a data collector.
Note: You cannot modify a data collector if the data collector is of the Upload File type.
|View Data Collector ||View details of the selected data collector|
|Delete Data Collector |
Delete the selected data collectors. Optionally, select the Delete data for this Data Collector check box if you want to delete all the data collected by that data collector so that it is no longer available for searching.Click OK to confirm your action.
Note: There might be some residual data remaining in the system that was still being collected when you decided to delete the data collected by the data collector. Such data is deleted from the system when the data retention period is over.
|Clone Data Collector|
Make a copy of the selected data collector.
|Collection Status History|
View the individual status of the last ten polls for a data collector.
For more information, see Data collection status.
|Start Data Collector(s)|
Start the selected data collectors.
This actioAftern is not relevant for the Upload file data collector as it performs one-time data collection after the data collector is created.
|Stop Data Collector(s)|
Stop the selected data collectors that are already started.
This action is not relevant for the Upload file data collector as it performs one-time data collection after the data collector is created.
If you want to avoid data collection during a particular time period (for example, during your maintenance window), you can stop the data collection. No data is collected during this time. The next time you start data collection, it begins from that point onward.
|Refresh Data Collector List||Manually refresh the list of data collectors to see the latest poll status and any other updates made to the data collectors.|
|Change Maximum Data Retention Period|
Change the maximum data retention period (in days) for the selected data collectors.
Use the following options to set the data retention period:
After changing the property value, you need to restart the Search component to apply the change.
|Search by Data Collector Name/Path/Host/Created By/Tags|
In the search bar, at the top-right side of your screen, you can filter data collectors in the following ways:
To search for tags, you need to specify the tag name with the corresponding value in the format TagName=TagValue. You can also specify a comma-separated list of tag name=value pairs.
Note: If the data collector contains an equals sign (=) in the name, you can search that data collector in one of the following ways:
The Data Collectors tab provides the following information:
|Show Data Collected|
Click Show Data Collectednext to one of the data collectors in the last column on the right, to search the data collected by that data collector.
When you click Show Data Collected, by default the search is run for the search string, COLLECTOR_NAME="DataCollectorName".
In the preceding search string, DataCollectorName refers to the name of the data collector.
Name of the data collector configured.
|Path||File path of the data file used while creating the data collector.|
|Host||Host name of the server on which the data exists.|
List of tag names with their corresponding values added to the data collector.
Overall polling status for the data collector, as follows:
For more information, see Data collection status.
Displays the data collector state, to indicate whether the data collector was started or stopped, as follows:
Note: The start and stop actions are not relevant for the Upload file data collector as it performs one-time collection. Therefore, for this data collector, the State column displays a dash (-).
Date and time when the data collector was modified.
If you did not modify the data collector, then the date and time when the data collector was created is displayed.
|Last Event Timestamp||Date and time of the last event that got indexed by the data collector.|
|Type||Type of the data collector.|
|Data Pattern||Data pattern used for creating the data collector.|
You can collect and analyze metrics (or logs) generated by the BMC TrueSight IT Data Analytics product for the Collection Station and Search components. After installing the product, the data collector for collecting the Collection Station is automatically created. You can also view a line chart summarizing the total data indexed on the Default dashboard page. But you need to create the data collector for collecting the Search component logs. For more information, see Monitoring the product metric files.