Collecting data by using a Windows Share connection

The following video (6:42) illustrates the process of creating this data collector for collecting data from a custom file. To see the detailed instructions for creating this data collector, see Procedure of collecting data by using a Windows Share connection.

 https://youtu.be/RZatSlpGjGY 

To collect data by using a Windows Share connection

1. Navigate to Administration > Data Collectors > Add Data Collector .
2. In the Name box, provide a unique name to identify this data collector.
3. From the Type list, select Monitor over Windows Share.

4. Provide the following information, as appropriate:

    

   Field	Description
   Target/Collection Host
   Target Host	Excerpt named targetHost was not found in document xwiki:IT-Operations-Management.Operations-Management.BMC-TrueSight-IT-Data-Analytics.ITDA113.Administering.Setting-up-data-collection.Collecting-data-into-the-system.Creating-data-collectors.Collecting-data-from-an-individual-file.WebHome.
   Collection Host (Agent)	Type or select the collection host depending on whether you want to use the Collection Station or the Collection Agent to perform data collection. The collection host is the computer on which the Collection Station or the Collection Agent is located. By default, the Collection Station is already selected. You can either retain the default selection or select the Collection Agent. Note: For this type of data collector, the target host and collection host are expected to have different values.
   Collector Inputs
   Server Name	Excerpt named server name was not found in document xwiki:IT-Operations-Management.Operations-Management.BMC-TrueSight-IT-Data-Analytics.ITDA113.Administering.Setting-up-data-collection.Collecting-data-into-the-system.Creating-data-collectors.Collecting-data-from-an-individual-file.WebHome.
   Credentials	(Optional) Select one of the following options: Apply security credential to automatically populate the user name and password fields. Then select the appropriate credential (profile) from the Available Credential list that you already configured under Administration > Credentials. Provide Credential to manually add user name and password credentials. Then enter the credentials in the User Name, Password, and Domain fields. You can also create a credential that uses the manually entered details by clicking Add Credential next to the Domain field.
   User Name	Provide the user name for connecting with the server from which you want to retrieve the data. Note: This field is disabled if you applied a security profile earlier.
   Password	Provide the password for connecting with the server from which you want to retrieve the data.
   Domain	Provide the domain of the Windows user with which you want to connect to retrieve the data files. Click Add Credential, provide a credential name, and click OK to create a new credential profile from the credentials that you provided in the user name, password, and domain fields. Once this profile is created, it is displayed under Administration > Credentials.
   Directory Path	Provide the Windows share path in the format \\serverName\shareName\directoryLocationPath. To retrieve data files from subdirectories, specify two asterisks (**) as the wildcard at the end of the directory path. For example, you can specify \\server\itdashare\local\**\ to collect the following logs: \\server\itdashare\local\stats_log \\server\itdashare\local\logs/login_log/ \\server\itdashare\local\mailman/log Notes: You cannot use wildcards in place of a file share name in the directory path. When creating a data collector template as a part of the collection profile, you need to provide the path in the following format: \shareName\directoryLocationPath The host name is added as a prefix to the preceding path when the collection profile is applied to a host.
   Filename/Rollover Pattern	Specify the file name only, or specify the file name with a rollover pattern to identify subsequent logs. You can use the following wildcard characters: Asterisk (*)—Can be used to substitute zero or more characters in the file name. Question mark (?)—Can be used to substitute exactly one character in the file name. Specifying a rollover pattern can be useful to monitor rolling log files where the log files are saved with the same name but differentiated with some variable like the time stamp or a number. Specifying a wildcard can also be useful when you remember the file name only partially. Note: Ensure that you specify a rollover pattern for identifying log files that follow the same data format (which means they will be indexed with the same data pattern). See examples Scenario 1 Suppose you want to collect log files saved with succeeding numbers once they reach a certain size; for example: IAS0.log IAS1.log IAS2.log Rollover pattern: In this scenario, you can specify the rollover pattern as IAS?.log. Scenario 2 Suppose you want to collect log files that roll over every hour and are saved with the same date but a different time stamp in the YYYY-MM-DD-HH format; for example: 2013-10-01-11 .log 2013-10-01-12.log 2013-10-01-13.log Rollover pattern: In this scenario, you can specify the rollover pattern as 2013-10-01-*.log or 2013-10-01-??.log. In this scenario, if you are sure that exactly two digits at the end of timestamp are likely to change, then you can specify the ?? wildcard sequence to capture exactly two changing digits. Otherwise, specifying a single asterisk is recommended.
   Time Zone	Excerpt named time zone was not found in document xwiki:IT-Operations-Management.Operations-Management.BMC-TrueSight-IT-Data-Analytics.ITDA113.Administering.Setting-up-data-collection.Collecting-data-into-the-system.Creating-data-collectors.Collecting-data-from-an-individual-file.WebHome.
   Data Pattern
   Pattern	Excerpt named pattern was not found in document xwiki:IT-Operations-Management.Operations-Management.BMC-TrueSight-IT-Data-Analytics.ITDA113.Administering.Setting-up-data-collection.Collecting-data-into-the-system.Creating-data-collectors.Collecting-data-from-an-individual-file.WebHome.
   Date Format
   Date Locale	(Optional) You can use this setting to enable reading the date and time string based on the language selected. Note that this setting only applies to those portions of the date and time string that consist letters (digits are not considered). By default, this value is set to English. You can manually select a language to override the default locale. For a list of languages supported, see Language information for IT Data Analytics
   File Encoding	If your data file uses a character set encoding other than UTF-8 (default), then do one of the following: Filter the relevant character set encodings that match the file. To do this, click Filter relevant charset encoding next to this field. Manually scan through the list available and select an appropriate option. Allow TrueSight IT Data Analytics to use a relevant character set encoding for your file by manually select the AUTO option.
   Poll Interval (mins)	Enter a number to specify the poll interval (in minutes) for the log collection. By default, this value is set to 1.
   Start/Stop Collection	(Optional) Select this check box if you want to start the data collection immediately.

   Advanced Options

   Ignore Data Matching Input	(Optional) If you do not want to index certain lines in your data file, then you can ignore them by providing one of the following inputs: Provide a line that consistently occurs in the event data that you want to ignore. This line will be used as the criterion to ignore data during indexing. Provide a Java regular expression that will be used as the criterion for ignoring data matching the regular expression. Example: While using the following sample data, you can provide the following input to ignore particular lines. To ignore the line containing the string, "WARN", you can specify WARN in this field. To ignore lines containing the words both "WARN" and "INFO", you can specify a regular expression .*(WARN|INFO).* in this field. Sample data Sep 25, 2014 10:26:47 AM net.sf.ehcache.config. ConfigurationFactory parseConfiguration():134 WARN: No configuration found. Configuring ehcache from ehcache-failsafe.xml  found in the classpath: Sep 25, 2014 10:26:53 AM com.bmc.ola.metadataserver. MetadataServerHibernateImpl bootstrap():550 INFO: Executing Query to check init property: select * from CONFIGURATIONS where userName = 'admin' and propertyName ='init' Sep 30, 2014 07:03:06 PM org.hibernate.engine.jdbc.spi. SqlExceptionHelper logExceptions():144 ERROR: An SQLException was provoked by the following failure: java.lang.InterruptedException Sep 30, 2014 04:39:27 PM com.bmc.ola.engine.query. ElasticSearchClient indexCleanupOperations():206 INFO: IndexOptimizeTask: index: bw-2014-09-23-18-006 optimized of type: data
   Data Block	Indicates the index block with which you want to associate the data collector. You can associate a data collector to one of the various index blocks, each having a configurable retention period. By default, this value is set to Small. The maximum number of index blocks allowed are 5. Besides the three defined index blocks, Small, Medium and Large, you can create two more custom index blocks. When you select an index block, the properties of that index block are displayed below it. The properties that are displayed are: Archive: This indicates whether the data that you index using the selected index block will be archived. Retention Days: This indicates the retention days associated with the index block. Following are the retention days associated with the typical index blocks. The retention days displayed can be as configured by your Administrator. Select the index block as per your needs of retention days and the Archive status. If the Archive status is Off and you need to archive your data, contact your administrator to set the Archive status for the index block to On. For more information on how to set the archive status of the index block, see Changing System Settings. Note If you select the ITDA Metrics data pattern while creating a data collector, the Index Block field is unavailable since the Metrics Index Block is automatically associated with the data collector.
   Best Effort Collection	(Optional) If you clear this check box, only those lines that match the data pattern are indexed; all other data is ignored. To index the non-matching lines in your data file, keep this check box selected. Note: Non-matching lines in the data file are indexed on the basis of the Free Text with Timestamp data pattern. Example: The following lines provide sample data that you can index by using the Hadoop data pattern. In this scenario, if you select this check box, all lines are indexed. But if you clear the check box, only the first two lines are indexed. Sample data 2014-08-08 15:15:43,777 INFO org.apache.hadoop.hdfs.server. datanode.DataNode.clienttrace: src: /10.20.35.35:35983, dest: /10.20.35.30:50010, bytes: 991612, op: HDFS_WRITE, cliID: 2014-08-08 15:15:44,053 INFO org.apache.hadoop.hdfs.server. datanode.DataNode: Receiving block blk_-6260132620401037548_ 683435 src: /10.20.35.35:35983 dest: /10.20.35.30:50010 2014-08-08 15:15:49,992 IDFSClient_-19587029, offset: 0, srvID: DS-731595843-10.20.35.30-50010-1344428145675, blockid: blk_-8867275036873170670_683436, duration: 5972783 2014-08-08 15:15:50,992 IDFSClient_-19587029, offset: 0, srvID: DS-731595843-10.20.35.30-50010-1344428145675, blockid: blk_-8867275036873170670_683436, duration: 5972783
   Log File Contains Header	(Optional) Providing this value is mandatory only if you are trying collect a file that contains a constant header which must not be indexed. The value must be the actual header appearing in the data.
   Log File Contains Footer	(Optional) Providing this value is mandatory only if you are trying collect a file that contains a constant footer which must not be indexed. The value must be the actual footer appearing in the data.

   Tags

   Inherit Host Level Tags From Target Host

   (Optional) Select this check box to inherit your tag selections associated with the target host that you selected earlier. This option is not applicable if you did not select a target host. Note: After selecting this check box, you can further manually select additional user groups. When you manually select additional user groups, both the inherited permissions as well as the manually assigned permissions are applied. To remove the inherited permissions, clear this check box.

   Select Tag name and corresponding value

   (Optional) Select a tag name and specify the corresponding value by which you want to categorize the data collected. Later while searching data, you can use these tags to narrow down your search results. Example: If your are collecting data from hosts located at Houston, you can select a tag name for "Location" and in the value specify "Houston". While searching the data, you can use the tag, Location="Houston" to filter data and see results associated with the Houston location. To be able to see tag names, you need to first add them by navigating to Administration > System Settings. To specify tag names and corresponding values, in the left box select a tag name and then type the corresponding tag value in the right box. While you type the value, you might see type-ahead suggestions based on values specified in the past. If you want to use one of the suggestions, click the suggestion. Click Add to add the tag name and corresponding value to the list of added tags that follow. Click Remove Tag to remove a tag. The tags saved while creating the data collector are displayed on the Search tab, under the Filters panel, and in the Tags section. Note: At a time, you can specify only one value for a tag name. To specify multiple values for the same tag name, each time you need to select the tag name, specify the corresponding value, and click Add. For more information about tags, see Understanding-tags.

   Group Access

   Inherit Host Level Access Groups From Target Host

   (Optional) Select this check box to inherit your group access configurations associated with the target host that you selected earlier. This option is not applicable if you did not select a target host. Note: After selecting this check box, you can further manually select additional user groups. When you manually select additional user groups, both the inherited permissions as well as the manually assigned permissions are applied. To remove the inherited permissions, clear this check box.

   Select All Groups

   (Optional) Select this option if you want to select all user groups. You can also manually select multiple user groups. Notes: You can access data retrieved by this data collector based on the following conditions. If user groups are not selected and data access control is enabled: Only the creator of the data collector can access data retrieved by this data collector. If user groups are not selected and if data access control is not enabled: All users can access data retrieved by this data collector. You can restrict access permissions by selecting the relevant user groups that must be given access permissions. To enable data access control, navigate to Administration > System Settings. For more information, see Managing-user-groups-in-IT-Data-Analytics.

5. Click Create to save your changes.