Testing the chatbot training data to improve predictability

As an administrator, you can test whether the chatbot application is able to understand the conversation context and provides appropriate responses when interacting with users in natural language, and perform tasks on behalf of end users. You must create a CSV data set to test the chatbot application. The test results provide the exact problem area when the chatbot does not respond appropriately so that administrators can rectify the chatbot intents, entities, and dialogs (that serve as training data). The tests are particularly important when you are implementing a new data set, localized data set, or if you have made major changes to the data set.

For more information, see Leveraging machine learning metrics to improve chatbot predictability.

CSV data set structure for a chatbot application

You must create a CSV file to test a chatbot application. The CSV file should have the following structure:  

• A row represents a record.

• The first column represents the natural language text to which the chatbot responds. 

  Example: Vacation time
  

• The second column represents the intent that the chatbot understands and responds to. 
  Example: For vacation time as the natural language text, the chatbot should understand the intent as PTO. 

• Each row contains a natural language text and the intent that applies to the text.

Example of a test CSV file

The following image is an example of a CSV test data file for chatbot:

Before you begin

• Ensure that you have created intents, entities, and dialogs. 
• Ensure that you have created the CSV test data file. 
• If you are using localized Chatbot Skill, ensure that your test data locale matches with the workspace locale.
  
  

To test and rectify chatbot data

1. Log in to BMC Helix Innovation Studio and navigate to the Administration tab. 
2. Click the configuration that you created for training the cognitive service.
   Example: My Application > Cognitive Training
3. Expand the Chatbot Evaluation tab.

4. Upload and test the CSV test data file. 

    To upload and test the CSV data file

   1. On the Data Sets tab, click New. 
   2. On the New CSV Data Set page, provide the Data Set Name and Description. 
   3. In CSV File, select the test data file that you created. 
   4. In Chatbot Locale, select the locale of your chatbot.
   5. In Data Set Locale, select the locale of the CSV data set.
   6. Click Save. 
   7. Select the test data file and click Test . 

5. View and download the test results.

    To view and download the test results

   1. On the Test Results tab, you can view the score of all the test metrics—accuracy, precision, recall, and F-score.
   2. To download the test results CSV file, in the Results column, click the test results file.

   

6. Rectify the chatbot training data. 

    To improve the chatbot training data

   1. From the test results CSV file, view the input texts for which the chatbot could not identify the intent correctly.

   1. Add more entities for these intents and test the chatbot again to view the results.

   Tip

   To delete the test results that are no longer required, you can create a process by using the Delete record element. For more information, see Creating or modifying record instances using Record Service Tasks .

Related topics

Types of data sets used to train and test the cognitive service

Training and testing the cognitive service for a custom application