Collecting data from an individual file
You can collect individual log files (up to 100 MB in size) from the computer on which you are accessing the product Console. To collect data from an individual file, you need to create the Upload file data collector.
Note
BMC recommends that you not use this data collector for files that are updated dynamically.
Only that data which is present in the file at the time of uploading is collected. Any additional data added to the file, after the file is already uploaded into the system, does not get collected. Uploading images, compressed files and application files are not supported when you create the Upload file data collector.
The following information describes the data collection process:
To collect data from an individual file
- Navigate to Administration > Data Collectors > Add Data Collector .
In the Name box, provide a unique name to identify this data collector.
From the Type list, select Upload file.
Provide the following information, as appropriate:
Field Description Target/Collection Host Target Host (Optional) Select from a list of hosts that you have already configured under Administration > Hosts.
The target host is the computer from which you want to retrieve the data. You can choose to select the target host and inherit the host-level tags and group access permissions already added to the host, or manually enter the host name in the Server Name field.
Collection Host (Agent) Type or select the collection host depending on whether you want to use the Collection Station or the Collection Agent to perform data collection.
The collection host is the computer on which the Collection Station or the Collection Agent is located.
By default, the Collection Station is already selected. You can either retain the default selection or select the Collection Agent.
Note: For this type of data collector, the target host and collection host value is the same.
Collector Inputs Server Name Enter the host name of the server from which you want to retrieve the data.
Note: If you selected a target host earlier, this field is automatically populated. The value of this field is necessary for generating the "HOST" field that enables effective data search.
This field is mandatory to enable you to search data that you are uploading by host name.
File Path Provide the path of the log file. Time Zone (Optional) Accept the default Use file time zone option or select a time zone from the list.
With the default option, data is indexed as per the time zone available in the data file. If the data file does not contain a timezone, then the time zone of the Collection Host (Collection Station or Collection Agent server) is used.
Keep in mind that the selected timezone must match the timezone of the server from which you want to collect data. If you manually specify the timezone despite the file containing a timezone, then the manually specified timezone overrides the file timezone.
The field Time Zone takes into account the changes due to Daylight Savings Time (DST) where ever applicable.
Data Pattern Pattern Assign the data pattern (and optionally date format) for indexing the data file.
The data pattern and date format together decide the way in which the data will be indexed. When you select a data pattern, the matching date format is automatically selected. However, you can override the date format by manually selecting another date format or by selecting the option to create a new date format. By doing this, the date format is used to index the date and time string, while rest of the data is indexed as per the data pattern selected.
Instead of manually browsing through the list of available data patterns, you can click Auto-Detect to automatically find a list of matching data patterns. If no matching data patterns are found, then a list of matching date formats is displayed. By selecting the date format, the date and time string (in the data) is indexed with the selected date format, while rest of the data is indexed as free text.
If you cannot find both matching data patterns and date formats, then you can choose to index the data as free text. Depending on whether the data contains a date and time string, you can choose to assign the data pattern as Free Text with Timestamp or Free Text without Timestamp. All the records processed by using the Free Text without Timestamp option are assumed to be a single line of data with a line terminator at the end of the event. To distinguish records in a custom way, you can specify a custom string or regular expression in the Event Delimiter box, which decides where the new line starts in the data.
If you are collecting JSON data, then depending on whether the data contains a date and time string, you can assign the data pattern as JSON with Timestamp or JSON without Timestamp.
After assigning the data pattern (and optionally date format), you can preview the sample records.
For more information, see Assigning the data pattern and date format to a data collector.
Notes:
- Before filtering the relevant data patterns by clicking Auto-Detect, ensure that the correct file encoding is set.
- If you select both – a pattern and a date format, the product uses the date format to index the timestamp and the pattern to index rest of the event data.
Date Format Date Locale (Optional) You can use this setting to enable reading the date and time string based on the language selected. Note that this setting only applies to those portions of the date and time string that consist letters (digits are not considered).
By default, this value is set to English.
You can manually select a language to override the default locale. For a list of languages supported, see Language information.
File Encoding If your data file uses a character set encoding other than UTF-8 (default), then do one of the following:
- Filter the relevant character set encodings that match the file.
To do this, click Filter relevant charset encoding next to this field. - Manually scan through the list available and select an appropriate option.
- Allow TrueSight IT Data Analytics to use a relevant character set encoding for your file by manually select the AUTO option.
Read from Past (#days) Indicates the number of days for which past data can be collected and indexed. The maximum amount of past data that can be collected into the system is defined by the data retention period. You can change this value by navigating to Advanced Options and changing the Data Retention Period (in days) field value.
By default, this value is set to 7.
Recommendation: BMC recommends you to not use a very high value in this field (for example, 365). This is necessary to avoid a very large amount of data collected into the system in a short time.
- Click Create to save your changes.
Changing the default limit for uploading data
By default, the limit for uploading a file is set to 100 MB. You can change this limit by adding the collection.upload.maximumAllowedFileSizeInMB
property in the olaengineCustomConfig.properties file located at %BMC_ITDA_HOME%\custom\conf\server. The value of this property is the size of data (in MB) that you want to collect.
After adding the property, save the file, and then restart the service for the Console Server. For more information, see Starting or stopping product services.
Recommendation
BMC recommends that you upload files less than 100 MB in size. A file size larger than 100 MB may lead to the data collector timing out.
Comments
Log in or register to comment.