Installing and configuring the cognitive search data crawler for BMC Helix ITSM: Knowledge Management articles

After crawling, the knowledge articles in BMC Helix ITSM: Knowledge Management are uploaded to the IBM Watson Discovery collection. For more details about the data flow, see Leveraging-cognitive-search-in-your-application.

Before you begin

Make sure that you have completed the following tasks:

Process for installing and configuring the BMC crawler utility

The following image illustrates the end-to-end process to install and configure the BMC crawler utility:

Task 1: To set up your system and download the BMC crawler utility

1. Download the BMC crawler utility.

   1. As an administrator, download the BMC crawler utility file. For information about the utility, see BMC crawler utility.
   2. Save the crawler utility in your local machine. 

   3. Extract the .zip file of the utility. 

      The extracted folder includes the following files:

   • 
     
     • bmccrawler.properties
     • bmcCrawler-versionNumber.jar
2. Set the Java environment variables.

   You must set the Java environment variables in your system to run the BMC crawler utility.

   1. Open the command prompt.
   2. Perform one of the following steps:

   • 
     

     • For Linux, run the following command:

       Commands to set environment variables

       export JAVA_HOME=/opt/jdk
       export PATH=/opt/jdk/jre/bin:$PATH
       export PATH=$JAVA_HOME/bin:$PATH

     • For Windows, Set the Java_Home variable.

3. Configure the bmcCrawler.properties file.

   1. Navigate to the folder where you downloaded the BMC crawler utility. 
   2. Open the extracted folder and then open the bmccrawler.properties file. 

   3. Specify the following parameter values:

      Mandatory parameters
      Parameter	Description	Example value
      discoveryApiKey	Specify the API key of the IBM Watson Discovery instance.	zABCd4EFgHijKLmn1O1pqrStuu0vWXYzaBCDEfGHIjk_
      discoveryEndPoint	Specify the endpoint URL of the IBM Watson Discovery instance.	https://gateway-/location.watsonplatform.net/discovery/api
      remedyHostName	Specify the hostname of the AR System server.	abc-tenantname-1234
      remedyPort	Specify the port number of the AR System server.	46262
      remedyUser	Specify the AR System administrator username.	Administrator
      remedyPassword	(If you are running the BMC Crawler utility to encrypt the password)  Specify the password in plain text. (If you are running the BMC Crawler utility after encrypting the password)  Specify the encrypted password.	Plain text password—password123 Encrypted password—1234567891106a28b29c843d98e5fg1hi59j0k12345678953592258d3a981312b1
      formName	Specify the BMC Helix ITSM: Knowledge Management form name. Important: You can specify only one form at a time. Comma-separated form names are not valid.	RKM:HowToTemplate_Manageable_Join
      fieldNamesForParagraphs	Specify the field names whose values are appended to the Text field of the IBM Watson Discoverycollection. During a cognitive search, matching paragraphs are derived from the Text field. Important: For file system articles, specify only one attachment field. If you specify multiple attachment fields, only the first one is considered.  For database tables, do not specify attachment fields.	Article_Keywords,ArticleTitle
      fieldNamesForDetails	Specify the field names that should be added as metadata fields in the IBM Watson Discovery collection. During a cognitive search, the metadata fields are used to display the knowledge article details. Important: For file system articles, specify only one attachment field. If you specify multiple attachment fields, only the first one is considered.  For database tables, do not specify attachment fields.	DocID, ArticleTitle
      discoveryConfigurationName	If you do not want to specify the value for discoveryConfigurationFilePath, specify the name of the IBM Watson Discovery configuration. If you specify the value for discoveryConfigurationFilePath, the configuration name is taken from the JSON file.	HowTo-Config
      discoveryCollectionName	Specify the name of the IBM WatsonDiscovery collection.	HowTo-Collection
      customerId	Specify the customer ID that you want to use for labeling data for General Data Protection Regulation (GDPR).	BMC
      Optional parameters with default values
      chunkSize	Specify the batch size for querying entries from the BMC Helix ITSM: Knowledge Management form.	Default value—1000
      threadPoolSize	Specify the number of knowledge articles that will be crawled simultaneously.	Default value—10 Maximum valid value—50
      deleteCollectionBeforeCrawl	Specify whether you want to delete the collection before you start crawling.	Default value—no Valid values—yes/ no
      qualification	Specify a valid AR System qualification to include BMC Helix ITSM: Knowledge Management articles.	Default value—'1=1'
      language	Specify the language of the IBM Watson Discoverycollection. Valid values: en, es, de, fr, it, ja, ko, pt, nl, zh-CN	Default value—en
      discoveryEnvironmentSize	Specify the IBM Watson Discovery plan. This is supported only for IBM Watson Discovery V1.	Default value—LT (for the Lite plan) Valid values—LT/ S (for all other plans)
      fieldNameForGettingModifiedRecord	Specify the field name that represents the modified date of a form in BMC Helix ITSM: Knowledge Management.	Default value—Modified Date
      isApplyModifiedDateForDelete	Specify the field value that will be considered to delete the retired ITSM Knowledge Articles from the IBM Watson collection.	Valid values—Yes/No Default value—Yes
      Optional parameters without default values
      discoveryConfigurationFilePath	(After you download and modify the out-of-the-box files) Specify the path of the JSON file that you modified.	C:\\Path\\Demo_Discovery_Collection_Configuration_HowTo_.json
      titleFieldName	Specify the field name that will be considered as the title for the knowledge article in the search results.	Title
      documentUniqueID	Specify the document ID to be used in IBM Watson Discovery. Important: If the parameter value is blank, the BMC Helix ITSM: Knowledge Managementarticle request ID is set as the unique ID in IBM Watson Discovery. If there are multiple versions of the knowledge article, all the versions of the article might be displayed to the end user. If you want to use the BMC Helix ITSM: Knowledge Management knowledge base ID as the unique ID in IBM Watson Discovery, you must set the value to DocID. If there are multiple version of the knowledge article, only the latest version of the article are displayed to the end user.	docID

4. Restrict the properties file to administrators.

   After downloading the BMC crawler utility, you must restrict the bmcCrawler.properties file so that only authorized administrators can access and modify this file.

   Best Practice

   We recommend that you provide access to the file to only that user who owns the bmcCrawler.properties file. Do not provide access to any other users.

   • To restrict the file on Windows, perform one of the following steps:
     • Restrict file permissions by using Windows Explorer.

     • Open the command prompt and Apply Discretionary Access Control Lists (DACLs) to the file.

   • To restrict the file on Linux, open the command prompt and run the chmod 700 command as specified in The chmod command description.
     For example, use one of the following commands:

     • 
       
       • chmod 700 bmcCrawler.properties
       • chmod u+rwx,go-rwx bmcCrawler.properties
         
         

Task 2: To download the out-of-the-box files

BMC provides the following files out-of-the-box:

• Stop words file—Text file with a list of words that you can filter out from the data collection.
• Discovery Collection configuration file—JSON file format of knowledge articles that are crawled and uploaded to the IBM Watson Discovery collection in this format.

After downloading, you can modify the stop words list or the enrichments of the knowledge articles. 

1. Download the sample stop words file. 
2. Download the Discovery Collection configuration file.
3. (Optional) If required, modify the stop words list. 
4. Modify the enrichments section in the Discovery Collection configuration file according to the BMC Helix ITSM: Knowledge Managementform that you want to crawl. 
5. Save the modified file with a different name. 

Task 3: To encrypt the password and forcefully pause the crawler for stop words

1. Run the crawler for the first time to encrypt the Remedy password.

   You run the BMC crawler for the first time to encrypt the Remedy password. The password is encrypted by using the AES with GCM cipher and 256-bit key.

   1. Open the command prompt. 

   2. Run the following command:

      Example of command to encrypt the password

      java -jar -DbmcCrawlerPropertyFile=.\bmcCrawler.properties bmcCrawler-20.2.0.jar -encpassword

      The encrypted password is displayed in the command prompt. 

   3. Note the encrypted password for future reference. 
2. (For Microsoft Windows only) Restrict the Keys file to administrators

   After the password is encrypted, the key.txt file is generated and is located in the same directory from where you run the BMC crawler utility. If you have Microsoft Windows, you must restrict this file to administrators only. If you have Linux, this file is automatically restricted and you do not have to restrict it manually.

   1. Navigate to the location of the key.txt file.
   2. Perform one of the following steps:
      • Restrict file permissions by using Windows Explorer.

      • Open command prompt and Apply Discretionary Access Control Lists (DACLs) to the file.

Task 4: To run the crawler for a second time and forcefully pause the crawler for stop words

Perform this task if you want to upload and activate the stop words.

1. Open a command prompt. 

2. Run the following command:

   Example of command to encrypt the password

   java -jar -DbmcCrawlerPropertyFile=.\bmcCrawler.properties bmcCrawler-20.2.0.jar -stopToCreateStopwordList

   The BMC Crawler displays a message that the crawler has stopped. The user must go to the IBM Watson Discovery UI to upload and activate the stop words file.

Task 5: To upload the stop words file

1. Log in to BMC Helix Innovation Studio.
2. Open an existing project.
3. Open the Improvize and customize pane.
   
   
   
4. Go to Improve Relevance and select Stopwords.
   
   
   
5. Click Upload stopwords, to upload the stop word file.
   File format: <filename>.json
   The IBM Watson Discovery search will ignore the word mentioned in the stop word file.

For more information about uploading stop words, see Identifying words to ignore in IBM documentation. 

Task 6: To run the crawler

You run the BMC crawler for a third time to crawl BMC Helix ITSM: Knowledge Management articles so that they are uploaded to the IBM Watson Discovery collection.

1. Open a command prompt. 

2. Run the following command:
   
   

   Example command to run BMC crawler

   java -jar -DbmcCrawlerPropertyFile=.\bmcCrawler.properties bmcCrawler-20.2.0.jar

   A log file is generated every time you run the BMC crawler and is saved in the same directory from where you ran the crawler. 

To run the crawler to delete retired knowledge articles

Run the BMC crawler on BMC Helix ITSM: Knowledge Management articles to delete retired knowledge articles from the IBM Watson Discovery Collection. The deletion process ensures that when the user sends their queries to the chatbot, the chatbot does not display retired knowledge articles to users when they query the chatbot. 

1. Open the command prompt. 

2. Run the following command:

   java -jar -DbmcCrawlerPropertyFile=./<filename>.properties com.bmc.dsm.bmcCrawler-99.0.00-SNAPSHOT.jar

Where to go from here

To define search data sets, see Defining-search-data-sets.