Evaluating the cognitive service test results

As an administrator, you can test and evaluate the BMC Helix Cognitive Automation to ensure that the cognitive service correctly auto-categorizes service requests raised by the end users. The test results provide the exact problem area when the data sets do not predict the appropriate categories, so that administrators can rectify the data sets.

For more information about the test metrics, see Leveraging-machine-learning-metrics-to-improve-cognitive-service-data-sets.

Before you begin

After uploading the CSV data set or selecting the application data that you want to test, ensure that you have trained and tested the data set. 

To view the test metrics

1. Log in to BMC Helix Innovation Studio and navigate to the Administration tab. 
2. Click the configuration that you created for training the cognitive service. 
   For example, select My application > Cognitive Training.

3. Based on the value defined in the Classification-Service-Provider setting, one of the following tabs is displayed:

   Value	Tab
   WATSON	Auto-classification Training and Evaluation - IBM Watson Assistant
   NATIVE	Auto-classification Training and Evaluation - BMC Native (Google)

4. From the tab, select Test Results. 
   The test metrics—accuracy, precision, recall, and F-score are displayed. 
   
   The following image shows the test metrics for each test data and the CSV file of test results for BMC Native (Google) classification:
   
   
   
5. In the Results column, select the CSV file corresponding to the data set for which you want to view the test results. 
   
   The following image is an example of the CSV test results file:
   
   
   

To evaluate the cognitive service from the BMC Helix Innovation Studio UI

1. Log in to BMC Helix Innovation Studio and navigate to the Administration tab. 
2. Click the configuration that you created for training the cognitive service.
   For example, select My application > Cognitive Training.

3. Based on the value defined in the Classification-Service-Provider setting, one of the following tabs is displayed:

   Value	Tab
   WATSON	Auto-classification Training and Evaluation - IBM Watson Assistant
   NATIVE	Auto-classification Training and Evaluation - BMC Native (Google)

4. From the tab, select Test Results > Interactive Evaluation. 
5. On the Auto-classification Interactive Evaluation page, from the Trained Data Set list, select the data set that you want to evaluate. 
6. In Text, enter the input text from the data set that you want to categorize.
7. Click Classify.
   The input text is categorized and the results are displayed on the screen.

To download and rectify the data sets

The CSV data set that you uploaded is split into training data and test data. The test data shows the input texts that were not  categorized as desired.

1. To download the test results, perform the following steps:
   1. Log in to BMC Helix Innovation Studio and navigate to the Administration tab.
   2. Click the configuration that you created for training the cognitive service.
      For example, select My application > Cognitive Training.

   3. Based on the value defined in the Classification-Service-Provider setting, one of the following tabs is displayed:

      Value	Tab
      WATSON	Auto-classification Training and Evaluation - IBM Watson Assistant
      NATIVE	Auto-classification Training and Evaluation - BMC Native (Google)

   4. From the tab, select Test Results.
   5. In the Test Data column, select and download the test data CSV file corresponding to the data set that you want to rectify. 
      
   6. In the CSV test data file, in the Match column, view the rows that have the value N. 
2. To rectify the data set, add more input texts for the corresponding categories or remove email addresses, email signatures, salutations, or any other information that is not relevant for categorization. 

Related topics

Leveraging-machine-learning-metrics-to-improve-cognitive-service-data-sets

Types-of-data-sets-used-to-train-and-test-the-cognitive-serviceTesting the chatbot training data to improve predictability