Skip to main content

Using model-based topic detection to route emails

Archived

4 Tasks

20 mins

Visible to: All users
Beginner Pega Customer Service 8.5 English
This content is now archived and is no longer updated. Progress is not calculated. Pega Cloud instances are disabled, and badges are no longer awarded.

Scenario

U+ Bank currently has a rule-based email routing system in place — all emails with the topic detected as address change are routed to the Account Maintenance department, and emails with the topic detected as dispute transaction are routed to the Transaction Disputes department.

The current topic detection is not extremely intelligent, as in longer emails, with many keywords, multiple topics are detected and the system will simply select the first one. As the next implementation step, U+ Bank wants the email routing system to implement artificial intelligence to select the most prominent topic in an email that contains multiple topics.

As a data scientist, you are tasked with the creation and training of a new text categorization model that outputs a confidence score for each detected topic.

Next, you configure the Text Analyzer to use the new model to select the most prominent topic. Also, to facilitate efficient handling of the emails by the customer service representatives, you enable entity extraction. This will identify useful information, such as account number and location details, in the emails.

Use the following credentials to log in to the exercise system:

Role

Username

Password

Data Scientist

CASysAdmin

rules

 

Caution: To reuse the exercise system from a previous challenge, first complete the Using rule-based topic detection to route emails challenge. Otherwise, click Initialize Pega or Reset Instance in this challenge.

Your assignment consists of the following tasks:

Task 1: Test the Text Analyzer with a message containing multiple topics

Test the Text Analyzer with a message that contains an address change and disputes a transaction.

Task 2: Create a text categorization model

In Prediction Studio, create a new text categorization model, UBankSupport_Trained, and train the model with the TrainingData training set.

Task 3: Configure the text analyzer

Configure the text analyzer to use model-based topic detection, using the newly created UBankSupport_Trained text categorization model. Also, enable entity detection.

Task 4: Confirm your work

Verify your work by sending emails with multiple topics and sentiments to U+ Customer Service. As a customer service representative, confirm that the routing of the emails is based on the topic with the highest confidence score generated by the text categorization model. Also confirm that emails with a negative overall sentiment are routed to CSEscalations.

Challenge Walkthrough

Detailed Tasks

1 Test the Text Analyzer with a message containing multiple topics

  1. Download and extract the TrainingData training set.
  2. Log in as Data Scientist with user name CASysAdmin and password rules.
  3. In the header, click Application: Customer Service > Channels and interfaces.
    Channels
  4. Scroll down and click UBankSupport to open the email channel configuration.
  5. Click the Behavior tab.
  6. In the Text Analyzer section, click Open text analyzer rule.
  7. In the top right, click Actions > Run.
  8. In the Sample text field, write a message concerning an address change and a transaction dispute.
Dear U+,
I write to inform you that I found a new house. The address: 222 West Las Colinas Blvd., Irving, TX 75039, USA.
Also, I have a new email: [email protected]
At the moment, I am a very unhappy customer of yours. I always had good experiences. But now, for the third time, I have noticed an invalid transaction. The amount is small, but could you please check? Again, in general, I like your services.
Bye, Sarah
  1. In the top right, click Run.
  2. In the Topic detection section, notice that both the Account Address Change topic and the Dispute Transaction topic are detected with a confidence score of 1.
    Rule-based confidence
    Note: The confidence score of a topic is either 0 or 1 in rule-based methodology, depending on the presence or absence of keywords in a message.

Close the Run Text Analyzer: UBankSupport window.

2 Create a text categorization model

  1. In the upper left corner, switch from DEV Studio to Prediction Studio.
  2. In the navigation pane on the left, click Models.
  3. Click New > Text categorization.
  4. In the New text categorization model dialog box, in the Name field, enter UBankSupport_Trained.
  5. In the Save to IVA channel list, select UBankSupport.
  6. In the Language list, select English.
    New model
  7. Confirm that the Methodology is set to Use machine learning.
  8. Click Continue.
  9. In the Source selection section, click Choose File and select the TrainingData.csv file.
  10. Click Next.
  11. In the Sample construction section, select User defined sampling to construct training and test sets.
  12. Click Next.
  13. In the Taxonomy review section, expand the Action list and examine the number of records and words counts.
    Taxonomy
  14. Click Next.
  15. In the Model creation section, ensure that the Maximum Entropy model is selected.
  16. Click Next.
    F-score
    Note: The F-score is used as the metric for the expected performance of categorical models.
  1. Click Next.
  2. In the model selection section, ensure that the Maximum Entropy model is selected.
  3. Click Create.
  4. In the upper left, click Test. Enter a message about an address change and click Test:
Dear U+,
My customer number is: 123456789.
I would like to inform you that I have a new address: 222 West Las Colinas Blvd., Irving, TX 75039, USA.
Also, I have a new email: [email protected]
Cheers, Sara
  1. Scroll down and notice the topic with the highest Confidence score.
    Results address
  2. Enter a message about disputing a transaction and click Test:
Dear U+,
I am a very unhappy customer of yours. I always had good experiences. But last month, I have noticed several invalid transactions on my invoice. How can this happen!
Cheers, Sara
  1. Scroll down and notice the topic with the highest Confidence score.
    Results dispute
  2. Close the Test panel.
  3. Click Save.

3 Configure the text analyzer

  1. In the upper left corner, switch from Prediction Studio to DEV Studio.
  2. On the Text Analyzer: UBankSupport tab, in the upper right, click Check out.
  3. In the Topic model field, enter UbankSupport_Trained.
  4. In the Topic preference section, select Use model-based topics if available.
    Topic preference
    Note: This option ensures that the topic detection looks at available models first and then applies the rule-based topic detection only if there are no models.
  1. In the Text extraction section, select Enable text extraction.
  2. Inspect the entities that are detected by the installed extraction models and the Ruta script.
  3. On the Advanced tab, in the Topic settings section, select Sentence level granularity.
    Sentence level
  4. In the top right, click Check in and enter appropriate check-in comments.

4 Confirm your work

  1. In the top right, click Actions > Run.
  2. In the Sample text field, write a message concerning an address change and a transaction dispute.
Dear U+,
I write to inform you that I found a new house. The address: 222 West Las Colinas Blvd., Irving, TX 75039, USA.
Also, I have a new email: [email protected]
At the moment, I am a very unhappy customer of yours. I always had good experiences. But now, for the third time, I have noticed an invalid transaction. The amount is small, but could you please check? Again, in general, I like your services.
Bye, Sarah
  1. In the top right, click Run.
  2. Scroll down and, in the Topic detection section, confirm that a confidence score is calculated by the model for each topic and that an address change and a transaction dispute are recognized as the prominent topics.
    Model-based confidence
  3. In the Text extraction section, notice the entities that are identified in the message.
  4. Optionally, alter the text and observe how changes in the text influence the confidence scores for topic detection and sentiment.


Available in the following mission:

We'd prefer it if you saw us at our best.

Pega Academy has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice