Training a topic model to improve email routing

5 Tasks

20 mins

Pega Customer Decision Hub '24.2

Pega Customer Decision Hub '24.1 Pega Customer Decision Hub '23 Pega Customer Decision Hub 8.8 Pega Customer Decision Hub 8.7 Pega Customer Decision Hub 8.7 Pega Customer Decision Hub 8.6

Visible to: All users

Beginner

Pega Customer Decision Hub '24.2

English

Scenario

U+ Bank plans to use text predictions to route incoming emails to the appropriate department based on the topic of the email. For several use cases (for example an address change), emails can be routed based on keywords detected in the message. To implement email routing, create an email channel interface, and train the text prediction with a data set that contains classified messages.

Use the following credentials to log in to the exercise system:

Role	User name	Password
Data scientist	NLPDataScientist	rules
Application Developer	NLPApplicationDeveloper	rules

Your assignment consists of the following tasks:

Task 1: Create a channel interface

As an application developer, create and configure U+ Bank customer support email channel interface to respond to requests from customers. The system creates the U+ Bank customer support prediction automatically after you create the email channel.

Task 2: Test the text prediction

As a data scientist, test the U+ Bank customer support text prediction with the two messages below and inspect the sentiment and topic classifications. Answer the following questions:

• How would you estimate the sentiment of these two messages?
• Why is complaint detected as a topic in a message concerning an address change?

Complaint

I am a very unhappy customer of yours. For the third time, I have noticed an invalid transaction. The amount is small, but could you please check? It is unfortunate because, in general, I like your services.

Address change

I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected]

Task 3: Train the text prediction

Train the U+ Bank customer support text prediction with the CSTrainingData data set and answer the following question:

• On how many topics will this XLSX file train?

Task 4: Test the trained prediction

Test the U+ Bank customer support text prediction after it is trained with the message concerning an address change from task 1.

Task 5: Test the routing conditions

Test the U+ Bank customer support email channel routing conditions. Confirm that the address change request is recognized and routed to a correct work queue.

 

You must initiate your own Pega instance to complete this Challenge.

Initialization may take up to 5 minutes so please be patient.

• Log In

Challenge Walkthrough

Detailed Tasks

1 Create a channel interface

1. On the exercise system landing page, click Launch Pega Infinity™ to log in to App Studio.
2. Log in to App Studio as an application developer:
   1. In the User name field, enter NLPApplicationDeveloper.
   2. In the Password field, enter rules.
3. In the navigation pane of App Studio landing page, click Channels.
4. In the Create new channel interface section, click Email to create a new channel interface.
   
5. In the Details section, in the Email channel name text box, enter U+ Bank customer support.
6. In the Email handling section, click Add email account.
7. In the drop-down list, select ubanksupport.
8. Click Verify to verify the address and check if the email server is responsive.
   
9. In the Additional settings section, in the Reference operator list, select DataScientist.
   
10. Click the Behavior tab.
11. In the Text Analytics section, click Edit topics:
    1. In the Edit topics window, click Add item to add a row.
    2. In the new row, in the Topic column, enter Complaint.
    3. In the Approximate match column, enter wrong,invalid,service,transaction,unhappy,protest,customer,services.
    4. click Add item to add another row.
    5. In the new row, in the Topic column, enter Address Change.
    6. In the Approximate match column, enter new,change,moved,acct,acount,chang,cange,newe,addr,annount.
    7. Click Submit to save the topic list.
       
12. In the Intelligent routing section, click Add condition.
    1. In the Action list, select Route to work queue.
    2. In the Value field, enter or select AccountMaintenance.
    3. In the When list, select NLP > Topic.
    4. In the comparator list, select is equal.
    5. In the value field, enter or select Address Change.
       
13. In the upper right, click Save.
14. In the lower-left corner, click the operator icon, and then select Log off to end the application developer session.

2 Test the text prediction

1. Log in to Prediction Studio as a data scientist:
   1. In the User name field, enter NLPDataScientist.
   2. In the Password field, enter rules.
2. On the Predictions landing page, click the U+ Bank customer support tile.
   
3. In the upper-right corner, click Test to open the Test prediction dialog box.
4. In the Test prediction dialog box, in the Text field, enter the following message concerning a complaint:
   
   I am a very unhappy customer of yours. For the third time, I have noticed an invalid transaction. The amount is small, but could you please check? It is unfortunate because, in general, I like your services.
5. Click Test.
6. Observe that a complaint is detected as the topic of the message with a confidence score of 1.
   
7. Click the Sentiment tab.
8. In the Sentiment section, notice that the complaint has a negative overall sentiment score.
   Tip: Check if your estimation of the sentiment of the message matches the sentiment score.

9. Click Highlight in input text to identify the sentences with positive and negative sentiments.
   
   Note: Although the email has a mixed positive and negative sentences, the overall sentiment is negative.

10. In the Text field, enter the following message that concerns an address change.
    
    I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected]
11. Click Test.
12. Click the Topic tab.
13. In the Topic section, observe that both a complaint and an address change are detected as topics with a confidence score of one.
    
    Tip: Check if your estimation of the sentiment of the message matches the sentiment score.

14. Close the test panel to return to the prediction.
15. Question – Why is complaint detected as a topic in a message concerning an address change?
    1. To the right of the complaint topic, click the Gear icon to view the keywords for this topic.
       
    2. On the Keywords tab, notice that wrong is one of Should words.
       
       Tip: Because the word 'wrong' is present in the message, the complaint topic is detected.

16. Click Cancel.

3 Train the text prediction

1. Download CSTrainingData and extract the XLSX file.
2. Question - How many topics does this XLSX file contain?
   1. Open the XLSX file in an editor, and then inspect the result column and check the possible values.
      Tip: This file has training data for two topics: Address Change and Complaint.

3. Switch back to the U+ Bank customer support text prediction.
4. On the right, click Import.
   
5. Click Choose File, and then select the CSTrainingData.xlsx file.
6. Click Upload.
7. In the Action > Address Change or Action > Complaint row, in the Total training data column, click 0 to view feedback information.
   
   Tip: Note that the two topics have over a hundred records pending training each.

8. In the upper-right corner, click Build.
9. Select the U+ Bank customer support topic model.
   
10. Click Build to begin training the text prediction.
11. After the build is triggered, note that there is a build job in progress.
    
12. In the upper-right corner, click Actions > Refresh.
    Tip: It may take up to a minute to build the model. Click Refresh until the build is complete.

13. Once the prediction build is complete, click View build report to see the result of the model building process.
    
14. Close the Model training report.
15. Notice the new values in Total training data column.
    
    Note: The training data consists of 70 percent of the records. The remaining 30 percent has been used for validation of the model.

4 Test the trained prediction

1. In the upper-right corner, click Test.
2. In the Text field, enter the following message concerning an address change that you have tested in the first task.
   
   I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected]
3. Click Test.
4. Observe that after training the model, only an address change is detected as a topic with a high confidence score.
   
   Note: The observed confidence score values may vary, as 30 percent of the training data is randomly used for validation.

5. In the Entity section, confirm that the address and email mentioned in the message are detected by the entity extraction model.
   
6. Close the Test prediction window.
7. In the upper-right corner, click Save.
8. In the lower-left corner, click the user icon, and then select Log off to log out of Prediction Studio.

5 Test the routing conditions

1. Log in to App Studio as an Application Developer:
   1. In the User name field, enter NLPApplicationDeveloper.
   2. In the Password field, enter rules.
2. In the navigation pane of App Studio, click Channels to view the list of current channels.
3. In the Current channel interfaces section, click the icon that represents your existing U+ Bank customer support email channel.
4. In the upper-right corner of the Email channel, click Test to test the routing conditions:
   1. In the Test routing conditions window, in the From (email) field, enter [email protected]
   2. In the To (email) field, enter [email protected]
   3. In the Subject field, enter Wrong address.
   4. In the Message field, enter:
      
      I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected]
   5. Click Test.
   6. Confirm, the the topic of the email is correctly recognized, and that the email is routed to the AccountMaitenance work queue.
      
   7. Close the Test routing conditions window.
5. In the lower-left corner, click the operator icon, and then select Log off to end the application developer session.

---

This Challenge is to practice what you learned in the following Module:

• Text analytics for email routing v6

---

Available in the following mission:

• Pega NLP Essentials v5