Training a topic model to improve email routing

3 Tasks

20 mins

Pega Customer Decision Hub 8.7

Pega Customer Decision Hub '24.1 Pega Customer Decision Hub '23 Pega Customer Decision Hub 8.8 Pega Customer Decision Hub 8.7 Pega Customer Decision Hub 8.6

Visible to: All users

Beginner Pega Customer Decision Hub 8.7 English

Scenario

U+ Bank uses Pega Customer Service™ to route incoming emails to the appropriate department based on the topic of the email. For several use cases (for example an address change), emails are routed based on keywords detected in the message. To improve the email routing, train the text prediction with a data set that contains classified messages.

Use the following credentials to log in to the exercise system:

Role	User name	Password
Data scientist	DataScientist	rules
Customer service representative	cacsr	rules

Your assignment consists of the following tasks:

Task 1: Test the text prediction

As a data scientist, test the U+ Bank customer support text prediction with the two messages below and inspect the sentiment and topic classifications. Answer the following questions:

• How would you estimate the sentiment of these two messages?
• Why is complaint detected as a topic in a message concerning an address change?

Complaint

I am a very unhappy customer of yours. For the third time, I have noticed an invalid transaction. The amount is small, but could you please check? It is unfortunate because, in general, I like your services.

Address change

I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected]

Task 2: Train the text prediction

Train the U+ Bank customer support text prediction with the CSTrainingData data set and answer the following question:

• On how many topics will this CSV file train?

Task 3: Test the trained prediction

Test the U+ Bank customer support text prediction after it is trained with the message concerning an address change from task 1.

Confirm your work

As a customer, use the enablement email client to send an email to U+ Bank customer support.

To: [email protected]

Subject: Incorrect address

Messsage:

Hi U+ Bank,

I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected].

Cheers, Sara Connors

As a U+ Bank customer service representative, confirm that the incoming email is correctly routed.

 

You must initiate your own Pega instance to complete this Challenge.

Initialization may take up to 5 minutes so please be patient.

• Log In

Challenge Walkthrough

Detailed Tasks

1 Test the text prediction

1. On the exercise system landing page, click Pega CRM suite to log in to Prediction Studio.
2. Log in as a data scientist with user name DataScientist and password rules.
3. On the Predictions landing page, click the U+ Bank customer support tile.
   
4. In the upper-right corner, click Test to open the Test prediction dialog box.
5. Test prediction dialog box, in the Text field, enter the following message concerning a complaint:
   
   I am a very unhappy customer of yours. For the third time, I have noticed an invalid transaction. The amount is small, but could you please check? It is unfortunate because, in general, I like your services.
6. Click Test.
7. Observe that a complaint is detected as the topic of the message with a confidence score of 1.
   
8. Click the Sentiment tab.
9. In the Sentiment section, notice that the complaint has a negative overall sentiment score.
   Tip: Check if your estimation of the sentiment of the message matches the sentiment score.

10. Click Highlight in input text to identify the sentences with positive and negative sentiments.
    
    Note: Although the email has a mixed positive and negative sentences, the overall sentiment is negative.

11. In the Text field, enter the following message that concerns an address change.
    
    I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected]
12. Click Test.
13. Click the Topic tab.
14. In the Topic section, observe that both a complaint and an address change are detected as topics with a confidence score of one.
    
    Tip: Check if your estimation of the sentiment of the message matches the sentiment score.

15. Close the test panel to return to the prediction.
16. Question – Why is complaint detected as a topic in a message concerning an address change?
    1. To the right of the complaint topic, click the Gear icon to view the keywords for this topic.
       
    2. On the Keywords tab, notice that wrong is one of Should words.
       
       Tip: Because the word 'wrong' is present in the message, the complaint topic is detected.

17. Click Cancel.

2 Train the text prediction

1. Download CSTrainingData and extract the CSV file.
2. Question - How many topics does this CSV file contain?
   1. Open the CSV file in an editor, and then inspect the result column and check the possible values.
      Tip: This file has training data for two topics: Account Address Change and Complaint.

3. Switch back to the U+ Bank customer support text prediction.
4. On the right, click Import.
   
5. Click Choose file, and then select the CSTrainingData.csv file.
6. Click Upload.
   
   Tip: Note that the two topics have over a hundred records pending training each.

7. In the upper-right corner, click Build.
8. Select the U+ Bank customer support topic model.
9. Click Build to begin training the text prediction.
   
10. After the build is triggered, note that there is a training job in a pending state.
    
11. In the upper-left corner, click Actions > Refresh.
    Tip: It may take up to a minute to build the model. Click Refresh until the build is complete.

12. Once the prediction build is complete, click View report to see the result of the model building process.
13. Close the Model training report.
    Note: The training data consists of 70 percent of the records. The remaining 30 percent has been used for validation of the model.

3 Test the trained prediction

1. In the upper-right corner, click Test.
2. In the Text field, enter the following message concerning an address change that you have tested in the first task.
   
   I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected]
3. Click Test.
4. Observe that after training the model, only an address change is detected as a topic with a high confidence score.
   
   Note: The observed confidence score values may vary, as 30 percent of the training data is randomly used for validation.

5. In the Entities section, confirm that the email address, city, state, and ZIP code mentioned in the message are detected by the entity extraction models.
   
6. In the lower-left corner, click the user icon, and then select Log off to log out of Prediction Studio.

Confirm your work

1. On the exercise system landing page, click Email.
   
2. In the navigation pane on the left, click Mail.
3. In the top menu, click Write mail.
4. Compose an email concerning an address change:
   
   To: [email protected]
   
   Subject: Incorrect address
   
   Messsage:
   
   Hi U+ Bank,
   
   I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected].
   Cheers, Sara Connors
5. Click Send message.
6. On the exercise system landing page, click Pega CRM suite to log in to Prediction Studio.
7. Log in as a customer service representative with user name cacsr and password rules.
8. On the My workbaskets tab, select the Account Maintenance workbasket.
9. If there is no incoming email, click the Refresh button until the email is processed.
   
   Note: Depending on the environment, it might take up to two minutes until the email is sent and processed in Customer Service.

10. Click the ID number of the incoming email.
    
11. Notice the neutral sentiment and the entities that are detected.
    

---

---