Latest / Receivables Podcast | Discussions About Debt Collection / How Collection Agencies Can Use AI Without Losing Control | Pete Klipa | Ep. 294

Transcript

How Collection Agencies Can Use AI Without Losing Control | Pete Klipa | Ep. 294 · Receivables Podcast | Discussions About Debt Collection

• 0:07
  SPEAKER_00: Hello everybody, Adam Parks here with another episode of Receivables Podcast.
• 0:13
  SPEAKER_00: Today, very lucky to have with me Pete Klippa, the Chief Client Experience Officer with Harvest Strategy Group, to talk to me about, well, kind of the current wave of what's happening across the deck collection industry in terms of the use of artificial intelligence and other technology tools and how that is ultimately impacting a change in our dynamic.
• 0:38
  SPEAKER_00: It used to be we have a problem and we would could throw bodies at it, get more people on the phone, but that's not really the way of the future, and it's sure not the way things are going right now.
• 0:49
  SPEAKER_00: So, Pete, thank you so much for coming on with me today.
• 0:52
  SPEAKER_00: I really appreciate you sharing your insights.
• 0:55
  SPEAKER_01: My pleasure.
• 0:56
  SPEAKER_01: Looking forward to chatting about this topic.
• 0:58
  SPEAKER_01: Uh, it has uh many angles to consider.
• 1:01
  SPEAKER_00: So, Pete, for anyone who has not been as lucky as me to get to know you through the years, could you tell everyone a little about yourself and how you got to the seat that you're in today?
• 1:09
  SPEAKER_01: Absolutely.
• 1:10
  SPEAKER_01: Uh, you know, I've been in the collections and recovery space for the better part of 25 years, several different uh verticals during that time, including you know, public utility, credit card, had a stint in credit counseling where I've got an opportunity to work with a lot of the banks in that uh nonprofit uh sector.
• 1:28
  SPEAKER_01: And uh currently I'm with uh Harvest Strategy Group.
• 1:32
  SPEAKER_01: I'm the chief client experience officer, and I'm in my fourth year at Harvest.
• 1:36
  SPEAKER_01: Harvest um is an outfit that uh does work for regional creditor clients as well as um you know debt buying clients, and uh, we have uh a master servicing role.
• 1:46
  SPEAKER_01: And you know, we're like a lot of folks in the industry, really seeing some outstanding growth uh in the current environment.
• 1:52
  SPEAKER_00: Which is part of the reason I think you've got such unique a perspective, not just your history in the space, but kind of your current role in helping organizations to manage those vendor networks and and to be part of that entire process.
• 2:06
  SPEAKER_00: So I feel like it gives you really good visibility upstream and downstream as to how these things are are actually happening.
• 2:14
  SPEAKER_00: And we've seen this constant increase for the last few years in our survey results that it's harder to hire people and it's harder to retain the people that we're able to hire.
• 2:24
  SPEAKER_00: And with those challenges in place or those challenges still be being very real, it's difficult for us to throw bodies at the problem.
• 2:33
  SPEAKER_00: And as you've seen the an increase in the volume of accounts that you're managing, what have you seen, Pete, organizations doing to offset that need to hire?
• 2:44
  SPEAKER_01: Yeah, absolutely.
• 2:45
  SPEAKER_01: It's a dynamic where you know I think there's a an expectation, you know, from clients that we find efficiencies.
• 2:53
  SPEAKER_01: And um we have to keep pace, you know, with those efficiencies to try to continue to keep the lid somewhat on, you know, um the extensive hiring because the the hiring process and training process, it's expensive.
• 3:08
  SPEAKER_01: You know, there's still a dynamic, you know, in a post-COVID world of um you know the whole hybrid environment of working from home versus you know working from an office.
• 3:17
  SPEAKER_01: So I think as all of us in the industry navigate that, you know, being able to find efficiencies, uh to not have to do as much hiring in uh the environment of growth that we're in is you know um a target.
• 3:31
  SPEAKER_01: You know, just like a lot of things that you target, you know, there needs to be some parameters and some some rules around it.
• 3:39
  SPEAKER_01: And particularly with something like AI, it has you know outstanding potential application and even some really market uh actual application.
• 3:51
  SPEAKER_01: But at the same time, there's uh some rules to consider around it.
• 3:55
  SPEAKER_01: And it can't be unlimited, uh, it needs to have human involvement as well as uh client and and regulatory sign-off and consideration.
• 4:04
  SPEAKER_00: We hear a lot of conversations around the ideas of deploying artificial intelligence in the space.
• 4:10
  SPEAKER_00: And as I've gone down the research rabbit hole, looking at a variety of, let's call it, academic reports and things, it sounds like some use cases are having deeper impacts than others when it comes to the deployment of this technology, whether it be for an agency or for a servicer.
• 4:28
  SPEAKER_00: Are there any specific use cases that you've seen that have been more impactful than others in these early stages of AI deployments?
• 4:37
  SPEAKER_01: Yeah, absolutely.
• 4:37
  SPEAKER_01: You know, I think a general comment about it is that um to the degree that it can replace um the use of manual tasks, that um you know really does have a sweet spot kind of element to it because manual repetitive tasks are things that um can be displaced with uh something that's just a lot more efficient.
• 5:01
  SPEAKER_01: It could be in you know the summary of of data packages so that you know you get a uh a quicker feel you know for what a large body of data is telling you, or even a large body of data that's already summarized, what are some of the AI findings around that?
• 5:18
  SPEAKER_01: So that can can be very useful.
• 5:20
  SPEAKER_01: I have a couple of examples of of use case that I could mention that are you know things that um happen with us at Harvest, you know, and one of them is is kind of a personal thing with me.
• 5:30
  SPEAKER_01: You know, I don't know if you guys remember the um the old um legal black books.
• 5:36
  SPEAKER_01: Um, you know, for for non-attorney lay people, there used to be in the industry a little black book that was like a reference book around legal terminology, you know, mostly things that attorneys, you know, would have pretty close at hand.
• 5:50
  SPEAKER_01: But um, you know, when they start spotting off some things, you know, around Latinness and and you know, courtroom detail that that sometimes is beyond uh the untrained guy.
• 6:02
  SPEAKER_01: I used to use that anti-black book, you know, for reference, especially when I was new and early in the industry.
• 6:07
  SPEAKER_01: But recently, you know, kind of got turned on to a to a program called Perplexity that um the ability to reach into you know website data that uh has detail, you know, at uh various court levels within states and jurisdictions.
• 6:22
  SPEAKER_01: It's uh it's a very handy reference tool, you know, um to understand local happenings in legal and legals like politics.
• 6:31
  SPEAKER_01: It kind of plays out at the local level, and uh it's important to have a base of understanding, you know, based on what's public information out there that a program like Perplexity can uh pull up on.
• 6:43
  SPEAKER_01: So that was one application, had uh some data that was already summarized from account level data on our side at uh Harvest that needed to be uh, I guess for lack of a better uh phrase, had to be packaged up in such a way that we were pulling information from uh desperate um uh spreadsheets into one spreadsheet that had to be consolidated, you know, for a particular client.
• 7:11
  SPEAKER_01: Some people at Harvest were able to write, you know, very quickly uh a program to pull a large amount of spreadsheet data together into a package.
• 7:20
  SPEAKER_01: And you know, probably the the human time to to pull that together would probably have been two or three hours, and it was reduced to two or three minutes, you know.
• 7:30
  SPEAKER_01: So that type of efficiency, I guess, is an example of AI and action, you know, to um deliver a result that resulted in, you know, a lot less human time.
• 7:43
  SPEAKER_01: Those are the types of things I think we as an industry are trying to find those nuggets, those opportunities, because the more you can do those, the more it can be duplicated.
• 7:52
  SPEAKER_01: And you know, kind of through that experimentation, there's an opportunity for expansion.
• 7:57
  SPEAKER_00: What I'm hearing is that it's not just about the debt collection specific use cases, it's about those general business use cases that transcend industries and really, you know, kind of individual businesses, because what I heard there was about moving data, right?
• 8:14
  SPEAKER_00: Not necessarily just from a collection agency standpoint, making litigation decisions, for example, but going and looking at our businesses as businesses and and not just debt collection businesses and identifying some of those opportunities where we can really deploy some tools to improve.
• 8:29
  SPEAKER_00: I mean, perplexity, I'm in it every day.
• 8:32
  SPEAKER_00: I love perplexity from a deep research standpoint.
• 8:35
  SPEAKER_00: That sonar model is one of my favorites that and the tools of availability to use additional models like JatGPT or some of the Claude models, I think makes it a really versatile tool.
• 8:47
  SPEAKER_00: The level of detail and description that I'm able to retrieve from the web or from research documentation, et cetera, has been really impactful, at least for me from a research and an understanding standpoint.
• 8:59
  SPEAKER_00: So I definitely get where you're going there.
• 9:02
  SPEAKER_00: Has you seen organizations start to deploy AI, let's say vendors, whether it be law firms or agencies?
• 9:09
  SPEAKER_00: Have you start are are you mostly looking at the use cases from a debt collection standpoint, or are you looking at their use of artificial intelligence across the board, kind of the way that you're viewing it in your it's a little bit of both.
• 9:24
  SPEAKER_01: Um there's a very high importance for client review and sign-off on AI use cases for sure at the vendor level, you know, that we've manage.
• 9:36
  SPEAKER_01: And we as ourselves, uh a vendor and a provider that sort of um serves as that intermediary, you know, with vendors to clients.
• 9:44
  SPEAKER_01: There's a lot of client interest in it, but there's also this need, you know, to make sure that um there's protections, you know, particularly around account level data.
• 9:54
  SPEAKER_01: So I think in a lot of ways, it's leading toward you know more summaries of existing data that's already summarized, you know.
• 10:03
  SPEAKER_01: So in other words, it's not involving PII, you know, but it's maybe AI use of summarized data, I guess, which you know puts it in a little bit of a safer place with the deliverable that comes from that.
• 10:17
  SPEAKER_01: There's client interactions, you know, in the way of like presentation material that is lending itself to uh a review, the AI with analysis that ultimately can get packaged for a client review that is faster than a human putting that together initially, and maybe 80 to 90 percent of it is already in kind of a near final form, and then you move forward with it, you know, with some tinkering and get it into the final form with a with the last bit of human touch.
• 10:50
  SPEAKER_01: You know, um there's applications in contract review.
• 10:53
  SPEAKER_01: You know, when you think about you know managing contact detail and you know the the type of review uh that that involves with a lot of reading, um, it can uh assist with um interpretation, you know, around that of the subtleties and differences uh, you know, in a contract.
• 11:10
  SPEAKER_01: You know, even uh policy review or policy rewriting or creation, you know, it it has the ability to synthesize some things from that and get it into a form that can then be reviewed, you know, without a ton of heavy lifting and then more of a fine-tuning uh in the end state.
• 11:30
  SPEAKER_01: You know, there's there's lots of things around, you know, the the language, written and verbal.
• 11:36
  SPEAKER_01: You can cast a wider net around that, you know, to understand what is being said in review year and where do I need to focus my attention from an exception standpoint.
• 11:47
  SPEAKER_01: So it's a streetlining while casting a wider net at the same time so that you can focus on the things that are most important in the the review of that that compliance and quality activity.
• 11:59
  SPEAKER_01: So, you know, the um the applications are are endless and you know, with that becomes a lot of potential, but also you know, the need to to to make sure that we go about it the right way with you know foundational controls and um and then building from there to the next level uh as you kind of get to the place where there's a comfort level with what you complete, and then then it has it lends itself to expansion.
• 12:25
  SPEAKER_00: Are you seeing that the organizations that are best performing with these tool sets have kind of revert reworked their workflows versus bolting on tools to their existing processes?
• 12:38
  SPEAKER_00: And where you go so deep in your due diligence and your auditing processes, has that been your experience for high performers?
• 12:47
  SPEAKER_01: Yeah, you know, the integration with your enterprise system is really important.
• 12:52
  SPEAKER_01: You know, it's a lot like digital that way, in the sense that, you know, I think digital has shown that it's at its best when it's integrated with my system, you know.
• 13:02
  SPEAKER_01: And I I I think that's that's true of AI as well, because you know, it has to be established in such a way that it's it's not on the outside of the system, it's it's in a way kind of tied into what the system is telling you, you know, and that could, you know, help in in reports and analysis of those reports of what those outcomes are so that you can you know kind of focus on review and decision making, you know, from what AI is kind of setting the table for.
• 13:34
  SPEAKER_01: And um there there's there there's some efficiency to be gained with that for sure as it as it gets more and more sophisticated.
• 13:41
  SPEAKER_00: Do you think that we're focusing too much on the artificial and not enough on the intelligence portion of it?
• 13:48
  SPEAKER_00: I hear so much conversation around, oh, we're gonna be able to call cheaper, we're gonna be able to do this less expensively, but it feels like we're kind of missing the boat there when the intelligence portion, the decisioning engines, the machine learning at scale, those kinds of things seem to be where we can find the largest impact in the short term.
• 14:11
  SPEAKER_00: What would his experience been?
• 14:13
  SPEAKER_01: Yeah, uh, you know, I I'm I may be a little bit old school on this, and um I am Ohio-based, so you know, there's a little bit of uh a high state butt eye in me as it relates to this, but you know, a famous uh high state coach from way back, Woody Hayes, you know, wrote a book, you know, you win with people.
• 14:32
  SPEAKER_01: And yeah, I do think collections, you know, for all its system advantages that continue to um to come into the uh industry and structure changes, you know, that we implement, I do think at the end of the day it is a business and an industry that you win with people on.
• 14:49
  SPEAKER_01: And that'll be true, I believe, even in an AI environment, you know, that human intelligence, sort of side by side or in parallel with artificial intelligence, I think that's where our industry examples will be at their best and are at their best, you know, um, where you have that regulating human function that you know comes in and says, hey, you know, is this really what we want to do?
• 15:15
  SPEAKER_01: You know, does this pass the test of you know my my history, my knowledge of the industry?
• 15:20
  SPEAKER_01: Because sometimes it will take you down a rabbit hole and and you have to be careful about that.
• 15:24
  SPEAKER_01: You know, it it can be a time snapper if you know not involving some human elements to it.
• 15:31
  SPEAKER_01: I did hear of an example where you know, from a programmatic standpoint, collections partner kind of turned it over to um AI, you know, um with a database of some sort, and you know, the database was somehow erased, and then it went so far as erasing the backup database as well.
• 15:52
  SPEAKER_01: And you know, that's kind of a double whammy and um certainly uh would be you know resulting in a loss of productivity with um with that uh outcome.
• 16:02
  SPEAKER_01: So those are the types of things you want to avoid.
• 16:04
  SPEAKER_01: And I think with experimentation comes documentation and uh and kind of um understanding and and sort of sharing that understanding from a training standpoint so that others can benefit you know from from something that's so sort of new for many of us.
• 16:19
  SPEAKER_00: I always encourage people to to write an eighth grade experiment document along with their test.
• 16:25
  SPEAKER_00: Look, just defining your hypothesis and where you're trying to go with this thing and what you're going to experiment for.
• 16:30
  SPEAKER_00: And I also find that defining success before we begin the experiment helps us to kind of set our bearings as to whether or not this met our expectations, because so often our expects our expectations change on the fly as we're going through this learning process.
• 16:48
  SPEAKER_00: And it sounds like you you kind of gave me one nightmare scenario there with an organization deleting the database and the backups.
• 16:55
  SPEAKER_00: But as we think about agencies and law firms and other collection vendors on a on a higher level, where do you think they're failing or missing the opportunity from an artificial intelligence deployment standpoint?
• 17:09
  SPEAKER_00: Is there something that that you feel like they're maybe avoiding that could have serious impact?
• 17:16
  SPEAKER_01: Well, it's almost like there's um there's a tipping point um of this where you know if you're really gonna set up an environment that um you know it is safe in terms of uh PII and you know being able to protect you know key client information, that sort of thing, you almost have to to make that investment to get things set up on your side of the firewall, if you will, so that you don't have to go outside of it, you know.
• 17:45
  SPEAKER_01: You still can benefit, you know, from from the tools and the technology that some are benefiting in the industry.
• 17:52
  SPEAKER_01: And that's setup, it's not cheap, you know, to get that kind of expertise.
• 17:57
  SPEAKER_01: But I do think that um it makes sense to make that investment at the businesses at a certain scale.
• 18:04
  SPEAKER_01: Because the environment we're dealing with right now, it reminds me a lot of the environment that we had, you know, really pre-CFTB, you know, when we had so many accounts that were coming, you know, and just not enough time and not enough hiring ability to grab them all and give them the necessary attention that they need.
• 18:25
  SPEAKER_01: You know, so AI can help a lot with that, particularly, you know, with things like scoring, you know, it can lend itself to some input around the scoring so that we're working as smart as we can in light of um all the uh the accounts that are flowing through the system right now.
• 18:41
  SPEAKER_01: So it um I don't know what that scale point is, you know, because we have small vendors in our space for sure, and we have large ones, but um, I don't know that anybody can be exempt from it, you know, because the the industry's changing, and you know, you sort of have to keep pace with that, you know, with a commitment as a company to to gather some knowledge of it so that you can apply it in your world.
• 19:06
  SPEAKER_00: Orchestration seems to be one of those key pieces that we're finding value in.
• 19:11
  SPEAKER_00: And I know that there's more than a few use cases where we can find value, but it felt before like people talked about omnichannel, but without that coordination engine, without being able to collaborate across those channels, it felt more like disparate channels with you know similar context.
• 19:31
  SPEAKER_00: What have you seen in terms of organizations' results as they've coordinated the efforts across their various communication channels?
• 19:42
  SPEAKER_01: You know, I think with digital, you know, we've we've seen a whole lot of difference that it made, you know, in the out-of-the-gate, you know, as the digital adoption expanded, you know, it was clear the folks that were using digital and the ones that had not yet adopted it, you know, so it became a differentiating factor.
• 20:01
  SPEAKER_01: With AI, I don't know that it's as far along to be able to evaluate how much of a difference it can make in results.
• 20:10
  SPEAKER_01: You know, um, we're having more and more conversations, you know, with our vendors about it because you know, we expect our vendors, just like we expect of ourselves, to have a uh a process and a program of innovation.
• 20:26
  SPEAKER_01: And AI is part of that, you know.
• 20:28
  SPEAKER_01: And if if we ask them, you know, what are the things that you're pursuing in the way of improvement initiatives that that we as a company can benefit from and our clients can benefit from, and they kind of just have a blank well, if a blank stare, you know, that's that's not the answer that we want.
• 20:44
  SPEAKER_01: So, you know, there there has to be this thought of what am I gonna commit to to innovation?
• 20:50
  SPEAKER_01: How am I going to get my business to the future so that when we look at it five years from now, it's gonna look different, you know.
• 20:58
  SPEAKER_01: So those types of things I think are part of the dialogue, but you know, it's a type of thing where I think the AI is still pretty new to say, hey, this is paying dividends here, uh, because you also have this balancing effect with the clients and their expectation for for sign-off and review, uh, that um you know you can't bypass that.
• 21:18
  SPEAKER_01: You know, that that part is is in a way a type of regulating mechanism.
• 21:22
  SPEAKER_00: Interesting.
• 21:23
  SPEAKER_00: And and as you're looking at organizations that are providing services and and doing it with any type of AI in their workflows, and and now we see that 93% of debt collection organizations are deploying some level, some type, or exploring artificial intelligence.
• 21:40
  SPEAKER_00: How do you distinguish between an AI enabled partner and an AI dependent partner?
• 21:47
  SPEAKER_00: It feels like we're gonna start and and we've all read the articles about, you know, Chat GPT and brain usage and all of that.
• 21:53
  SPEAKER_00: But how do you how do you watch for that going into the future?
• 21:57
  SPEAKER_01: Yeah, that's a that's a good whole map.
• 22:00
  SPEAKER_01: I'm not as worried about the dependent piece.
• 22:02
  SPEAKER_01: I'm I'm more worried about um the the the flip side of that, which is you know, a lack of dependence or an independence from dependency.
• 22:15
  SPEAKER_01: Right.
• 22:16
  SPEAKER_01: And and as those practical uses get more documented um and and increasingly safer, you know, with uh them on their way to perfection.
• 22:26
  SPEAKER_01: You know, I th I think it's a conversation, you know, around what are you doing with this?
• 22:30
  SPEAKER_01: And and we've seen you know differentiation, you know, amongst um vendors that we've talked to in terms of what they what they're doing, what they're claiming to doing, and some you know that are a little bit slow out of the starting blocks.
• 22:45
  SPEAKER_01: And you know, I think what we try to do is understand better, you know, from a best practice standpoint.
• 22:50
  SPEAKER_01: Some of the vendors are out about doing, you know, asking questions like, is it safe, is it approved, you know, um, how do you know that, you know, that type of thing, versus some of these others that have said, hey, you know, I'm I'm just not willing to commit to it because I see it as too risky.
• 23:06
  SPEAKER_01: You know, um that that that that may not be the right answer either, but I think the best place is somewhat in between.
• 23:13
  SPEAKER_00: Have you started looking at different KPIs or measuring vendors or your own performance in a different way as you've started deploying this technology and watching it be deployed throughout your network?
• 23:25
  SPEAKER_01: Yeah, I mean, one of the um best KPIs to look at kind of at the highest level is you know the financial viability of the vendors.
• 23:34
  SPEAKER_01: You know, um if they are you know working through this the right way and gaining some efficiencies, you know, there should be evidence of cost savings.
• 23:43
  SPEAKER_01: You know, there should be evidence of holding a line on hiring growth with expenses while you know achieving some top-line increase in revenue.
• 23:53
  SPEAKER_01: You know, that's probably the ultimate KPI measure that that that um profitability as well as solvency that you see in a financial statement.
• 24:03
  SPEAKER_01: Beyond that, we have our performance and compliance metrics that you know really are how we differentiate one vendor to the next in terms of their outcomes, you know, beyond the financial.
• 24:16
  SPEAKER_01: And um with that, you know, I think if they're performing at their best, it's going to translate, you know, um to those outcomes that we need them to see which will result in you know increased market share.
• 24:29
  SPEAKER_01: So yeah, I think we in the industry are seeing expanding pie.
• 24:32
  SPEAKER_01: You know, I don't think that uh that's at all unusual for most of the folks in our space.
• 24:37
  SPEAKER_01: But how do you carve that pie out, you know, in terms of market share, I think comes down to how these folks are performing in compliance and performance and you know it should translate.
• 24:48
  SPEAKER_01: So in the best of all situations, you know, we're having conversations about their financials, we're having conversations about their strategy, which would be inclusive of you know things like digital and AI.
• 25:00
  SPEAKER_01: And we're seeing those outcomes in the day-to-day metrics that we measure, so or then feeding back to the clients and saying, you know, uh these vendors are performing great, and we're rewarding them with more business.
• 25:12
  SPEAKER_01: These vendors are not performing so great, and they're the ones that are uh having business taken away.
• 25:20
  SPEAKER_00: Well, you're watching a lot of the same things, you're just expecting a change in those measurements based on whether it be that reduction in cost or that increase in productivity or both, depending on the use cases that vendor or organization is is actively deploying.
• 25:38
  SPEAKER_01: Aaron Powell I mean it's a it's a it's a it's a results review, but it's also coupled with a dialogue.
• 25:44
  SPEAKER_01: The dialogue is around that that strategic push.
• 25:48
  SPEAKER_01: You know, we we tend to want to challenge our vendors a little bit about what the future looks like for them.
• 25:54
  SPEAKER_01: One of the things that we know is the case in the law firm space, you know, because there's a lot of folks who are more gray beard oriented, like myself.
• 26:04
  SPEAKER_01: I put myself in that category.
• 26:06
  SPEAKER_01: And, you know, there's succession planning and things like that that are discussed, you know, as a matter of routine.
• 26:12
  SPEAKER_01: And as we have those conversations, you know, it's a conversation about the future, but it's a good um opportunity to have a dialogue about what the future initiatives of the firm or the agency look like, you know, and that goes beyond the people side.
• 26:28
  SPEAKER_01: That's the commitment to technology and some of these tools and enhancements.
• 26:32
  SPEAKER_01: You know, it it might be a system migration.
• 26:35
  SPEAKER_01: You know, that's something that uh many folks consider.
• 26:38
  SPEAKER_01: And um, you know, in the course of that system migration, especially with um AI consideration around programmatic advantages and being able to program quicker and customize in mass fashion quicker, you know, those things are things that can be differentiating, you know, as it relates to changes, you know, at vendor providers.
• 27:00
  SPEAKER_01: So we want to understand what is their thinking on that today, as well as kind of in the short to intermediate term, you know, how's that going to change for us?
• 27:10
  SPEAKER_01: And ultimately, what do they expect to happen from it?
• 27:13
  SPEAKER_01: You know, because you know, all this AI stuff, it's important, but if it doesn't translate to results that will be beneficial, you know, then I think it somewhat misses the mark.
• 27:23
  SPEAKER_00: I think organ I think some organizations are struggling with how to truly measure the return on investment for the money that they're putting into these AI tools.
• 27:34
  SPEAKER_00: And it'll take us some time, I think, before we truly understand it and can isolate it.
• 27:38
  SPEAKER_00: And I don't know that everything can be isolated, but you bring up some really good points and and how and some perspective for our audience that are those vendors that are being looked at by their clients and some of the things that you may find to be more important than others because I don't think it's always clear-cut out, and although a lot of clients may be asking for the same things, they may be interpreting them somewhat differently.
• 28:04
  SPEAKER_00: So, Pete, I really do appreciate you coming on, sharing all of your experience and insights.
• 28:09
  SPEAKER_00: This has been a great conversation.
• 28:11
  SPEAKER_01: Well, my pleasure, as always, you guys uh do so much to further the industry and um it's your knowledge amongst a very large uh group across our space.
• 28:22
  SPEAKER_01: And you always look forward to these updates that you guys have.
• 28:26
  SPEAKER_00: Well, I really do appreciate you.
• 28:27
  SPEAKER_00: For those of you that are watching, if you have additional questions you'd like to ask Pete or myself, you can leave those in the comments on LinkedIn and YouTube, and we'll be responding to those.
• 28:34
  SPEAKER_00: Or if you have additional topics you'd like to see us discuss, you can leave those in the comments below as well.
• 28:38
  SPEAKER_00: And hopefully I can get Pete back here at least one more time to help me continue to create great content for a great industry.
• 28:45
  SPEAKER_00: But until next time, Pete, thank you so much.
• 28:47
  SPEAKER_00: I appreciate all your insights.
• 28:48
  SPEAKER_00: Thank you.
• 28:49
  SPEAKER_00: Appreciate it, and as always.
• 28:51
  SPEAKER_00: And thank you everybody for watching.
• 28:52
  SPEAKER_00: We appreciate your time and attention.
• 28:54
  SPEAKER_00: We'll see you all again soon.
• 28:55
  SPEAKER_00: Bye, everyone.