Data collectors are responsible for actually collecting your data, indexing it, and making it available for search. The Data Collectors tab allows you to configure data collectors for collecting data from particular data sources depending on the data pattern used.
This topic contains the following information:
You can collect the following kinds of data:
Notes
You can collect data for one-time or continuous monitoring.
When you create a data collector, at a minimum, you need to specify information about:
This information is used by the Indexer to index data and make it available in the form of events that can be searched immediately. If the manner in which the data was indexed is not as per your requirement, you can modify the data pattern and see if the results match your criteria.
The data that you want to collect can be on the same computer on which the Collection Station (or Collection Agent) is installed (local data), or it can be on a different computer (remote data). You can collect data remotely by creating an SSH connection or connecting to a shared network drive on a Windows computer.
For more information, see Local and remote data collection.
You can collect data by using one of the following collection mechanisms:
The Collection Agent is useful in the following scenarios:
To understand how to choose a data collection mechanisms for your environment, see Data collection mechanisms.
For information about setting up the Collection Agent, see Setting up Collection Agents.
After creating a data collector, data collection starts when the first poll happens. Data starts getting collected from the time when the first poll happens. Supposing you want to monitor a file in which data is being continuously added. After creating the data collector, data starts getting collected from the point when the first poll happened and the previous data available in the file is ignored. By default, the product defines the data retention period as seven days. This period defines the maximum duration of time for which data must be retained in the system. You can change the default setting by navigating to Administration > System Settings.
The data retention period acts as a moving window (depicted in green in the following figure).
Consider that on the following scale of time, you created a data collector at time T1, now data collection starts from T1 when the first poll happens. Data collected at T1 remains in the system until T1+7. As time passes, the data older than the seven days period starts getting deleted and is no longer available for searching.
Data retention period has implications on the Read from Past (# days) function which defines the maximum limit (of time) for collecting data older than the current time. This setting is available for the following data collectors:
Note
After the data collector is created, it might take some time (approximately 1 minute) for the first poll to happen. The first poll is used to make the data collector ready for data collection. The data is fetched only from the second poll.
Expected time delay (to see the first set of data for a search) = (Time for first poll) + (Poll interval set for the data collector).
Depending on the data sources and whether you want to perform local or remote collection, data collectors can be categorized as follows:
Data sources | Description | Data collectors | Local / remote? |
---|---|---|---|
Files and directories | Collect data that comes from various files and directories. Note: The Upload File data collector can be used to upload a file for one-time collection of data. | Local | |
Monitor file over SSH | Remote | ||
Monitor file over Windows share | Remote | ||
Upload file | Remote | ||
Script outputs | Collect data that is generated as a result of running a script. | Local | |
Monitor script output over SSH | Remote | ||
Windows events | Index and search Windows events. | Monitor Windows events | Remote |
Events from external systems | Collect events directly from supported external systems such as BMC ProactiveNet Performance Management. | Remote | |
Syslog events | Collect Syslog events over a TCP or UDP connection. | Remote |
If none of the filtered data patterns suit your needs, you can add a new data pattern.
You can collect and analyze metrics (or logs) generated by the BMC TrueSight IT Data Analytics product for the Collection Station and Search components. You can also view events indexed in the last week by adding a graph in one of the four quadrants on the Search tab.
For more information, see Monitoring the product metrics files.
The Data Collectors tab allows you to manage data collectors. To access this tab, navigate to Administration > Data Collectors. From here, you can perform the following actions:
Action | Icon | Description |
---|---|---|
Add Data Collector | Add a new data collector. You can add one of the following types of data collectors: | |
Edit Data Collector | Edit the selected data collector. You can modify the same details that you provided while adding a data collector. Note: You cannot modify a data collector if:
| |
View Data Collector | View details of the selected data collector | |
Delete Data Collector | Delete the selected data collector. Optionally, select the Delete data for this Data Collector check box if you want to delete all the data collected by that data collector so that it is no longer available for searching.Click OK to confirm your action. Note: There might be some residual data remaining in the system that was still being collected when you decided to delete the data collected by the data collector. Such data is deleted from the system when the data retention period is over. | |
Clone Data Collector | Make a copy of the selected data collector. | |
Last 10 Polls Status of Data Collector | View the overall status of the selected data collector. The overall status consists of information about the last 10 polls for that data collector. Note: The overall status is not available in the following scenarios:
| |
Search | Search for a data collector by entering appropriate text in the search bar at the top-right side of your screen. Searching is supported only for the Name, Path, and Host columns. |
The Data Collectors tab provides the following information:
Field | Description |
---|---|
Name | Name of the data collector configured. |
Path | File path of the data file used while creating the data collector. |
Host | Host name of the server on which the data exists. |
Type | Type of the data collector. |
Data Pattern | Data pattern used for creating the data collector. |
Status | Individual polling status for the data collector, as follows:
If the individual status of the data collector has changed to red but the next data collection happens successfully, the status changes to green. You can view details of the overall status (for the last 10 polls) by selecting the data collector and clicking Last 10 Polls Status of Data Collector . |
Action |
When the data collection is successful, you can see Data Collection Completed .For the Upload File data collector, the Start Data Collection and Stop Data Collection icons are not relevant. If you want to avoid data collection during a particular time period (for example, during your maintenance window), you can stop the data collection. No data is collected during this time. The next time you start data collection, it begins from that point onward. |
Using the command line to export a data collector
Using the command line to import a data collector
Using the command line to list data collectors
Using the command line to start collection of a data collector
Using the command line to stop collection of a data collector