This documentation supports the 23.3 version of BMC Helix Innovation Suite (AR System and BMC Helix Innovation Studio).

To view an earlier version, select the version from the Product version menu.

Evaluating the cognitive service test results

As an administrator, you can test and evaluate the cognitive service test results to make sure that the cognitive service correctly auto-categorizes service requests raised by the end users. The test results provide the exact problem area when the data sets do not predict the appropriate categories so that administrators can rectify the data sets.

For more information about the test metrics, see Leveraging machine learning metrics to improve cognitive service data sets.

Related topics

Leveraging machine learning metrics to improve cognitive service data sets

Types of data sets used to train and test the cognitive service

Testing the chatbot training data to improve predictability Open link

Before you begin

After uploading the CSV data set or selecting the application data that you want to test, make sure that you have trained and tested the data set. For more information, see Training and testing the cognitive service for a custom application.

To view the test metrics 

  1. Log in to BMC Helix Innovation Studio and navigate to the Administration tab. 
  2. Click the configuration that you created for training the cognitive service. 
    For example, select My application > Cognitive Training.
  3. Based on the value defined in the Classification-Service-Provider setting, one of the following tabs is displayed:

    ValueTab
    WATSON

    Auto-classification training and evaluation - IBM Watson Assistant

    NATIVE

    Auto-classification training and evaluation - BMC Native (Google)

    For more information about the Classification-Service-Provider setting, see Configuration settings C-D.

  4. From the tab, select Test Results
    The test metrics—accuracy, precision, recall, and F-score are displayed. 
    The following image shows the test metrics for each test data and the CSV file of test results for BMC Native (Google) classification:


  5. In the Results column, select the CSV file corresponding to the data set for which you want to view the test results. 
    The following image is an example of the CSV test results file:


Tip

To delete the test results that are no longer required, you can create a process by using the Delete record element. For more information, see Creating or modifying record instances using Record Service Tasks.

To evaluate the cognitive service from the BMC Helix Innovation Studio UI

  1. Log in to BMC Helix Innovation Studio and navigate to the Administration tab. 
  2. Click the configuration that you created for training the cognitive service.
    For example, select My application > Cognitive Training.
  3. Based on the value defined in the Classification-Service-Provider setting, one of the following tabs is displayed:

    ValueTab
    WATSON

    Auto-classification training and evaluation - IBM Watson Assistant

    NATIVEAuto-classification training and evaluation - BMC Native (Google)

    For more information about the Classification-Service-Provider setting, see Configuration settings C-D.

  4. From the tab, select Test Results > Interactive Evaluation
  5. On the Auto-classification Interactive Evaluation page, from the Trained Data Set list, select the data set that you want to evaluate. 
  6. In Text, enter the input text from the data set that you want to categorize.
  7. Click Classify.
    The input text is categorized and the results are displayed on the screen.

To download and rectify the data sets

The CSV data set that you uploaded is split into training data and test data. The test data shows the input texts that were not categorized as desired.

  1. To download the test results, perform the following steps:
    1. Log in to BMC Helix Innovation Studio and navigate to the Administration tab.
    2. Click the configuration that you created for training the cognitive service.
      For example, select My application > Cognitive Training.
    3. Based on the value defined in the Classification-Service-Provider setting, one of the following tabs is displayed:

      ValueTab
      WATSON

      Auto-classification training and evaluation - IBM Watson Assistant

      NATIVEAuto-classification training and evaluation - BMC Native (Google)

      For more information about the Classification-Service-Provider setting, see Configuration settings C-D.

    4. From the tab, select Test Results.

    5. In the Test Data column, select and download the test data CSV file corresponding to the data set that you want to rectify. 
    6. In the CSV test data file, in the Match column, view the rows that have the value N. 
  2. To rectify the data set, add more input texts for the corresponding categories or remove email addresses, email signatures, salutations, or any other information that is not relevant for categorization. 
Was this page helpful? Yes No Submitting... Thank you

Comments