KPIs reporting and value measurement case study

Interviewer: Bahar Gand

Interviewee: Thomas Francis

Bahar Gand: In this module, we have been discussing the importance of defining correct KPIs and value measurement tools in a custom decision app implementation. Now we would like to bring the subject to life with a use case from the field. For that, we will have a discussion with Thomas Francis from our Pega Consulting team. Hello Thomas, thank you for joining us on the call today. Could you please introduce yourself?

Thomas Francis: Hi, Bahar, thanks for having me. I'm really excited to be part of this discussion. Hi, guys. I'm Thomas Francis. I've been in the martech industry before using Pega and I've been using Pega for the past 10 years leading deliveries for multiple clients across comms as well as the banking industry. I have started off initially with Quadient and then kind of moved through the ranks of Pega 6, Pega 7, Pega 8 and now the latest versions of 23 and 24 as well, right? So that's a brief about my background and my exposure to Pega.

Bahar Gand: That's great. So as an introduction to our use case, can you explain to us about your client and give an overview of your client?

Thomas Francis: Yeah, so for the past couple of years I've been involved with a comms client. They are one of the largest telcos in the US. They have over 100 million subscribers. So, I've been helping them to drive value from Pega over the last couple of years.

Bahar Gand: Great. And how did the project start? What were your initial use cases and the channels that were defined in the first phases and how did the project evolve over time?

Thomas Francis: Sure. So, my current clients have been using Pega for almost a decade right now and they initially got started with using next best action in the call center channel and they started evolving into multiple channels, right. To start off with, they have around 150 actions that are there in their catalog right now. From a channel perspective, they have grown from initially using it in the call center to the web and mobile channels as well and they have almost 15 to 20 million impressions per day from all channels, right. So, they got started with the call center, they moved into web and mobile and they also have a presence in their IVR. Right now, they have next best actions that are doing call reason prediction and then kind of doing call deflections as well. Very soon they're going to get started with their outbound use case. Happy to announce also that a couple of weeks back we got rolled out in the retail channel as well. So, we're seeing some good numbers from there. So yeah, that's the journey for my client. They started off and then this is where we are right now and looking to unfold more value from the platform.

Bahar Gand: That sounds really great. So, what were the initial KPIs when you started the project? How did you show the value of CDH in the first phases so that the client continued investing in other channels and use cases?

Thomas Francis: Yep. Thank you for that question. I think that's a really good question because everyone initially starts off very small and then they realize that over a period of time, they need to get a little bit more structured, right. So, in the initial use cases, they were tracking retention mainly to understand what's the retention rate and eventually when we started expanding across multiple channels, especially web and mobile, we started measuring other KPIs as well, right. The second KPI we started measuring was the CLV customer lifetime value over a period of time and then instead of just being in a click or an accept driven KPI, we also started tracking conversion rates to see what is the actual product conversions that's happening as a result of the interactions. Another KPI we started building on top of the existing ones is the calls to care as well. And that became very important because of the call deflection and the IVR use case where we needed to make sure that the CLV and retention was in addition to the CLV and retention. They also wanted to make sure that the calls are being deflected and the cost for care is coming down right. So these were the four KPIs that we evolved to and then I think as we go through the journey we might end up measuring more KPIs to see what impact the platform's having.

Bahar Gand: Actually, it already sounds like a lot of KPIs that you have, that's great. So, it's impressive in my opinion.

Thomas Francis: Yeah, thank you.

Bahar Gand: So how about the use of test and control groups? Do you use test and control groups to measure lift, and can you talk about the lifts measured and if you are having any holdout groups, etc.?

Thomas Francis: Sure. So yeah, we've always done a target versus control or treatment versus control methodology. We have always a 5% holdout group and the 95% of the interactions get treated with the next best action and the 5% of the holdout group do not get any interactions from the platform. Over a period of time, we kind of measure lift and revenues from these numbers. So, all the KPIs are mainly driven off these two groups, right, which is the 95% group which gets actions from NBA and the 5% which are not treated with anything from the next best action platform.

Bahar Gand: Yeah, that makes sense. And what kind of reports do you use, like what type of data do you export from CDH and what is the main reporting tool that you are using for that?

Thomas Francis: Sure. I will grow in how they've evolved over reporting which is with like how the project was earlier used, right. So we extract the interaction history and all the dimension tables from our databases and this information goes into the data warehouse and all the target versus control comparisons in the different KPIs are all measured on this data that is exported and it's represented in a Power BI dashboard which is self-serve, right. So, all their users, the business users as well as their product managers go into this Power BI platform or dashboard to see what impact their actions are having right on the entire base. Now in addition to that, two additional data sets that have been exported are related to the adaptive model which their data science teams use pretty extensively. The first one is the predictor data itself on all the inactive and active predictors and what is the predictor performance of each predictor. And these predictors keep getting recycled, right? So, they look at the predictor data exported, and they also make use of the GitHub scripts that our Pega data sciences team have put together which helps them take a standard view of how the predictors are performing and then kind of do the assessment that way. And the last thing that they do also is the actual raw data as well is also being exported as part of the latest feature where there is a data export feature that is there in Pega 8 now. So, these are all the data that gets exported, and these are the dashboards that the clients create and use in the back end to keep track of all the KPIs, all the reports, as well as improve the performance of the platform.

Bahar Gand: That's great. And how about the simulation capabilities? Do you use any simulation in CDH to monitor your KPIs and look for opportunities for more improvement?

Thomas Francis: Yep. So off late as I said the earlier reports that I called out are reports that they had already in place as a result of the older versions. Now with the new versions they're really excited with all the capabilities and discovery tools that have been introduced as part of the strategy optimizer bundle. So, they use the simulation capabilities pretty actively. The way they kind of use it is that before every deployment that happens each week, they run the scenario planner to get an idea of how the new actions are performing and what kind of redistribution of the actions is happening as a result of the new actions being introduced. So, they do that before every BAU and post the deployment of the BAU irrespective of the deployment, they kind of use the action performance tracker on a daily basis. So, all the enterprise and leadership level reports and KPIs are driven from the back end, the data warehouse, whereas the operations team kind of use the action performance tracker on a daily basis to keep track of it, right. So that tying it back to how they manage their BAU deployments, they run a scenario planner report, get an idea of how many proposals they are expecting post deployment and after deployment. They come in the next day and check if the action performance tracker numbers that they're seeing in production are aligned with what they saw before the deployment from their scenario planner simulation, right. So, they're pretty excited to use it and they are able to in fact accurately predict what's going to happen before their deployment through their simulations.

Bahar Gand: That's great. So, they simulate and then they see the real results with action performance and then they use the external reports as well at the higher level. So, it makes sense like they are doing all.

Thomas Francis: Yep, it sounds so.

Bahar Gand: Yeah, that's a great loop. So as a final question, I want to hear about your next steps. So, what do you have in the pipeline for your client and what new features do you want to adopt and what are their plans about CDH?

Thomas Francis: Sure. So, in addition to going live with retail in the last couple of weeks, we're happy to inform that we also started using one-to-one OPS manager. The current throughput from the OPS manager is we configure at least 30 to 60 actions per week, and we do deployment once per week, right. So, from a one-to-one OPS management tracker perspective, we want to start doing deployments on demand or deployments every day because they have a pretty high number of actions that's being requested by the product team. So, we want to move the velocity and frequency of deployment, right. The main advantage that we have seen is that the entire deployment has become automated with the use of one-to-one OPS manager, right. So, a little bit on one-to-one OPS manager there. So apart from the product features itself, the next use case that they're trying to enable is their outbound messaging. We're going to get started with a trigger-based messaging system in the next 60 days. So that's going to be the short-term goal of the new use case that they're going to set out to achieve and post that. There is going to be another instance of Pega that's going to be spun up to cater to their enterprise segments, enterprise and SMB segments, right. Currently, they're serving the consumer segment using their NBA instance. So, there is a plan to expand out to implementing Pega for NBA for their enterprise segments as well.

Bahar Gand: So, is it the B2B customers?

Thomas Francis: Correct. Yes, it's on the B2B segment. So, yeah, so that's what the two next milestones that we're looking forward to and hopefully we're able to drive more value for that segment than what we're driving for the consumer right now.

Bahar Gand: Yep, it sounds great. Thank you, Thomas. Thank you so much for sharing all these insights with us. I think it has been really helpful. And thank you everyone for listening today.

Thomas Francis: Thank you, Bahar. Thanks for having me. Bye.

Bahar Gand: Thanks.