Training a topic model to improve email routing

4 Tasks

20 mins

Pega Customer Decision Hub 8.8

Pega Customer Decision Hub '24.2 Pega Customer Decision Hub '24.1 Pega Customer Decision Hub '23 Pega Customer Decision Hub 8.7 Pega Customer Decision Hub 8.7 Pega Customer Decision Hub 8.6

Visible to: All users

Beginner

Pega Customer Decision Hub 8.8

English

Scenario

U+ Bank plans to use Pega Customer Service™ to route incoming emails to the appropriate department based on the topic of the email. For several use cases (for example an address change), emails can be routed based on keywords detected in the message. To implement email routing, create an email channel interface, and train the text prediction with a data set that contains classified messages.

Use the following credentials to log in to the exercise system:

Role	User name	Password
Data scientist	DataScientist	rules
Application Developer	ApplicationDeveloper	rules
Customer service representative	cacsr	password123!
Customer1	customer1	citadel

Your assignment consists of the following tasks:

Task 1: Create a channel interface

As an application developer, create and configure U+ Bank customer support email channel interface to respond to requests from customers.

Task 2: Test the text prediction

As a data scientist, test the U+ Bank customer support text prediction with the two messages below and inspect the sentiment and topic classifications. Answer the following questions:

• How would you estimate the sentiment of these two messages?
• Why is complaint detected as a topic in a message concerning an address change?

Complaint

I am a very unhappy customer of yours. For the third time, I have noticed an invalid transaction. The amount is small, but could you please check? It is unfortunate because, in general, I like your services.

Address change

I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected]

Task 3: Train the text prediction

Train the U+ Bank customer support text prediction with the CSTrainingData data set and answer the following question:

• On how many topics will this CSV file train?

Task 4: Test the trained prediction

Test the U+ Bank customer support text prediction after it is trained with the message concerning an address change from task 1.

Confirm your work

As a customer1, use the enablement email client to send an email to U+ Bank customer support.

To: [email protected]

Subject: Incorrect address

Message:

Hi U+ Bank,

I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected].

Cheers, Sara Connors

As a U+ Bank customer service representative, confirm that the incoming email is correctly routed.

 

You must initiate your own Pega instance to complete this Challenge.

Initialization may take up to 5 minutes so please be patient.

• Log In

Challenge Walkthrough

Detailed Tasks

1 Create a channel interface

1. On the exercise system landing page, click Pega CRM suite to log in to App Studio.
2. Log in to App Studio as an application developer:
   1. In the User name field, enter ApplicationDeveloper.
   2. In the Password field, enter rules.
3. In the navigation pane of App Studio landing page, click Channels.
4. In the Create new channel interface section, click Email to create a new channel interface.
   
5. In the Details section, in the Email channel name text box, enter U+ Bank customer support.
6. In the Email handling section, click Add email account.
7. In the drop-down list, select ubanksupport.
8. Click Verify to verify the address and check if the email server is responsive.
   
9. In the Additional settings section, in the Reference operator list, select DataScientist.
   
10. Click the Behavior tab.
11. In the Text Analytics section, click Edit topics:
    1. Delete unnecessary topics, as shown in the following figure:
       
    2. In the Edit topics window, click Add item to add a row.
    3. In the new row, in the Topic column, enter Complaint.
    4. In the Approximate match column, enter wrong,invalid,service,transaction,unhappy,protest,customer,services.
    5. Click Submit to save the topic list.
       
12. In the Intelligent routing section, edit the existing condition:
    1. In the Action list, select Route to work queue.
    2. In the Value field, enter or select TransactionDisputes.
    3. In the When row, in the value field, enter or select Complaint, as shown in the following figure:
       
    4. Click Add condition to add a second condition.
    5. In the Action list, select Route to work queue.
    6. In the Value field, enter or select AccountMaintenance.
    7. In the When list, select NLP > Topic.
    8. In the comparator list, select is equal.
    9. In the value field, enter or select Account Address Change.
       
13. In the upper right, click Save.
14. In the lower-left corner, click the operator icon, and then select Log off to end the application developer session.

2 Test the text prediction

1. On the exercise system landing page, click Pega CRM suite to log in to Prediction Studio.
2. Log in to Prediction Studio as a data scientist:
   1. In the User name field, enter DataScientist.
   2. In the Password field, enter rules.
3. On the Predictions landing page, click the U+ Bank customer support tile.
   
4. In the upper-right corner, click Test to open the Test prediction dialog box.
5. In the Test prediction dialog box, in the Text field, enter the following message concerning a complaint:
   
   I am a very unhappy customer of yours. For the third time, I have noticed an invalid transaction. The amount is small, but could you please check? It is unfortunate because, in general, I like your services.
6. Click Test.
7. Observe that a complaint is detected as the topic of the message with a confidence score of 1.
   
8. Click the Sentiment tab.
9. In the Sentiment section, notice that the complaint has a negative overall sentiment score.
   Tip: Check if your estimation of the sentiment of the message matches the sentiment score.

10. Click Highlight in input text to identify the sentences with positive and negative sentiments.
    
    Note: Although the email has a mixed positive and negative sentences, the overall sentiment is negative.

11. In the Text field, enter the following message that concerns an address change.
    
    I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected]
12. Click Test.
13. Click the Topic tab.
14. In the Topic section, observe that both a complaint and an address change are detected as topics with a confidence score of one.
    
    Tip: Check if your estimation of the sentiment of the message matches the sentiment score.

15. Close the test panel to return to the prediction.
16. Question – Why is complaint detected as a topic in a message concerning an address change?
    1. To the right of the complaint topic, click the Gear icon to view the keywords for this topic.
       
    2. On the Keywords tab, notice that wrong is one of Should words.
       
       Tip: Because the word 'wrong' is present in the message, the complaint topic is detected.

17. Click Cancel.

3 Train the text prediction

1. Download CSTrainingData and extract the CSV file.
2. Question - How many topics does this CSV file contain?
   1. Open the CSV file in an editor, and then inspect the result column and check the possible values.
      Tip: This file has training data for two topics: Account Address Change and Complaint.

3. Switch back to the U+ Bank customer support text prediction.
4. On the right, click Import.
   
5. Click Choose file, and then select the CSTrainingData.csv file.
6. Click Upload.
   
   Tip: Note that the two topics have over a hundred records pending training each.

7. In the upper-right corner, click Build.
8. Select the U+ Bank customer support topic model.
9. Click Build to begin training the text prediction.
   
10. After the build is triggered, note that there is a training job in a pending state.
    
11. In the upper-right corner, click Actions > Refresh.
    Tip: It may take up to a minute to build the model. Click Refresh until the build is complete.

12. Once the prediction build is complete, click View report to see the result of the model building process.
13. Close the Model training report.
    Note: The training data consists of 70 percent of the records. The remaining 30 percent has been used for validation of the model.

4 Test the trained prediction

1. In the upper-right corner, click Test.
2. In the Text field, enter the following message concerning an address change that you have tested in the first task.
   
   I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected]
3. Click Test.
4. Observe that after training the model, only an address change is detected as a topic with a high confidence score.
   
   Note: The observed confidence score values may vary, as 30 percent of the training data is randomly used for validation.

5. In the Entity section, confirm that the ZIP code, state, city, address, country, and email address mentioned in the message are detected by the entity extraction model.
   
6. Close the Test prediction window.
7. In the upper-right corner, click Save.
8. In the lower-left corner, click the user icon, and then select Log off to log out of Prediction Studio.

Confirm your work

1. On the exercise system landing page, click Email.
   
2. Log in to Citadel Server as a customer1:
   1. In the User name field, enter customer1.
   2. In the Password field, enter citadel.
3. In the navigation pane on the left, click Mail.
4. In the top menu, click Write mail.
5. Compose an email concerning an address change:
   
   To: [email protected]
   
   Subject: Incorrect address
   
   Message: Hi U+ Bank,
   I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected].
   Cheers, Sara Connors
6. Click Send message.
7. On the exercise system landing page, click Pega CRM suite to log in Customer Service Portal.
8. Log in to Customer Service Portal as a customer service representative:
   1. In the User name field, enter cacsr.
   2. In the Password field, enter password123!.
9. On the My workbaskets tab, select the Account Maintenance workbasket.
10. If there is no incoming email, click the Refresh button until the email is processed.
    
    Note: Depending on the environment, it might take up to two minutes until the email is sent and processed in Customer Service.

11. Click the ID number of the incoming email.
    
12. Notice the neutral sentiment and the entities that are detected.
    

---

This Challenge is to practice what you learned in the following Module:

• Text analytics for email routing v3

---

Available in the following mission:

• Pega NLP Essentials v2