Latest / The Experimentation Edge / Truist on shipping faster and more safely with AI and human-in-the-loop banking
Transcript
- Ashley Stirrup (00:33) Thank you for joining us on the Experimentation Edge. We're excited to have Charles Williams, Senior Vice President and Software Engineering Director at Truist. Charles, welcome to the show. Charles Williams (00:44) Glad to be here, Ash, thank you. Ashley Stirrup (00:47) Could you tell us a little bit about Truist and your role there? Charles Williams (00:51) So, Truist is a financial institute, a very large super regional bank that was a merger of BB &T and SunTrust Bank, happened around, I'm gonna say three or four years ago. My role there right now is focused on developer experience, which has a lot to do with whether it's onboarding policies and processes or automation needs. So I focus on helping facilitate the improvement of that developer experience. Ashley Stirrup (01:20) Got it. And are there some kind of key strategic initiatives going on at Truss these days? Charles Williams (01:27) Yeah, I mean, like a lot of folks in the industry and across multiple industries, ⁓ artificial intelligence is obviously key. Move faster and more efficiently and help folks get stuff through the pipelines and get things out to the consumer. So yeah, I think artificial intelligence is the key thing. Ashley Stirrup (01:44) Yeah, yeah, such a common trend. ⁓ One of the interesting things with that is just as you see developer productivity increasing, it obviously puts pressure on every other step in the whole development process, particularly if you're building on top of AI, then maybe doing evals or doing A-B testing ⁓ once it's live. Are you working on things that kind of focus on that kind of the quality of the code aspect of it? Charles Williams (01:47) Yeah. A little bit. when it comes to the quality, what I'm trying to do, my team in particular, we're wanting to put the tools needed to, I guess, know that quality is seen first, far left of the pipeline, if you will. So put it in the developer's hands. So give them the ability to not have to rely on a human touch for quality, but more like automation. So we're working with a bunch of different platforms. And you're right, as you said before. Ashley Stirrup (02:30) Mm-hmm. Yeah. Charles Williams (02:41) You only as fast as your slowest point of the of pipe of the flow rate. Ashley Stirrup (02:45) Right, yeah. And so are there certain metrics that you track to see if the teams you're enabling are performing or they're delivering a good customer experience? How do you measure success? Charles Williams (02:58) Yeah, success is measured in a couple of ways, right? So you've got the folks out there would know what the door metrics are, right? So that's your developer DevOps metrics that are standard as far as throughput and turnover time onboarding and so forth. But we also want to measure satisfaction from a just a happiness perspective. And then it's going to sound weird, but developers aren't just necessarily satisfied because they're moving faster. They have the tools that enable them, right? They're happy with the work they do and they're not doing mundane work that takes away from their core development needs, right? Or core development likes is, I want to code as a developer. So we measure that as well. And that's usually done through surveys and sentiment and things like that. But we do measure the classic stuff as far as co-quality coverage and throughput as you onboard into a pipeline, for example. Ashley Stirrup (03:46) Yeah, and how does Truist think about technology as far as like a differentiator in the marketplace and trying to win more customers? Charles Williams (03:54) Yeah, I can probably find like our CIO speaking on LinkedIn and things like that where technology is definitely a partner to the business operations, right? So, I there is no doubt. And I think I don't think any FinTech organization on planet would say anything differently, but we're right in line. Ashley Stirrup (03:59) Mm-hmm. Yes. Yeah, right, right. Yeah, I would imagine that it's ⁓ such a key area of differentiation for the company. Like, the better your app and your web experience, the happier your customers are. Charles Williams (04:24) Right, and the key thing is, and this kind of goes against what we're generally talking about, but it partners with it, is the technology is there, but understanding the impacts of that technology to your business operations and being able to have the KPIs backwards to say, if I build faster and I deliver products and services sooner, then the business can make changes to the business structures and models in order to satisfy the needs of these consumers. Ashley Stirrup (04:30) Mm-hmm. Charles Williams (04:51) So all that is one thread that we have to maintain, not just as us as truism, but any bank or any facility, I think, in the industry. Ashley Stirrup (04:58) Yeah. Yeah, well, it's pretty interesting as far as AI goes, because there's a lot of industries where a few mistakes from the LLM, no big deal. But I would imagine in your business, failure is not an option. Charles Williams (05:07) Right. No, and that's why we haven't removed the human in the loop component to AI, right? So if I'm an AI, well, I'm saying, if I'm a developer using AI as a companion, I can tell it based on a business requirement, a feature, write this set of code, right? And before you commit it, I'm gonna overview it. So we're not autonomously just like, go out there and do a thing. Ashley Stirrup (05:30) Yeah. Right, right, right, Yeah, I would also imagine that you've probably got a human in the loop in the customer experience a lot too. So you're making sure that somebody's ensuring that yes, that is the right answer before it actually gets given to the customer. Charles Williams (05:41) Fuck worth it. Yeah, yeah, for sure. Yeah. And that's I like what I work on is it's a little bit of that backend processes. But yeah, as far as a human consumer facing AI and all the services we have there, absolutely. Ashley Stirrup (05:59) Yeah, yeah. Are there any interesting trends you're seeing in terms of new technology? Like, is customer service more of a focus with AI for banking or kind of how is the business thinking about where the biggest opportunities are to apply it? Charles Williams (06:14) Yeah, I think from what what's probably been talked about, right? Obviously, I can't share anything that's not obviously public, of course. The customer experience, custom experience is definitely a focus for the AI initiatives because the quicker we can get answers to problems or known issues, the better obviously the experience would be. Same thing with internal processes. The quicker we can get past two resolution, the faster folks can operate. Ashley Stirrup (06:20) Sure, of course. Yeah, yeah. And so is that kind of one of the dimensions you think about is like, how are you enabling your branches to be able to deliver better service with fewer people and things like that? Charles Williams (06:52) Yeah, yeah, I think our model and what I heard our leadership say is that it's not necessarily taking folks out of branches, it's making branches more efficient, right? More effective, right? So we're not replacing bodies, we're enhancing experiences is I think how you would want to work that. We would want to work that rather. ⁓ Ashley Stirrup (07:03) right yeah ⁓ Yeah, for sure. It opens up all sorts of new opportunities to deliver just a better experience. So, hopefully that leads to more revenue per branch or maybe more assets under management. I guess it's probably a metric you look at more. Yeah. Charles Williams (07:24) Yeah. Yeah, think as a business, that's obviously as any banking business would, financial industry would look at that. So we are no different. Ashley Stirrup (07:32) Yeah, yeah. And as you're evaluating new technology to enable your developers, how do you kind of test those and, you know, just get the right feedback to make sure, yes, this is something I want to put into the developer's supply chain. Charles Williams (07:39) Mm-hmm. Yeah, it starts with the understanding the developers journey and understanding the personas in which you're looking to improve. Right. So it's a matter of where the pain points, what are the redundant tasks that they repetitive tasks that they have to do and what are they telling us the pain is. We look, we know there are some pain somewhere. Those are known to, but then what is the developers telling us? So we start there and experiment with the low hanging fruit as the nomenclature goes is for example, Ashley Stirrup (07:56) Mm-hmm. Charles Williams (08:17) ⁓ unit testing or documentation. That's an easy one. That's a very easy one. How well can they do that? So let's start with something very simple. How well can you tell me about my project? How well do you know, help me with this error that I'm seeing in my security logs? Things like that we start with. Ashley Stirrup (08:33) Yeah, that makes a lot of sense. And are there certain vendors you're really partnering with on the AI side? Like, are you working with one of the LLMs in particular or Microsoft and kind of a broader stack? Charles Williams (08:47) Yeah, I would say, ⁓ so we definitely have a partnership with Microsoft. From the developer perspective, we're not necessarily using their tooling for that, although we have Microsoft Co-pilot in-house. We are partnering with, publicly partnering with GitLab as our developer side. So we're rolling out and testing and demoing their dual, their developer agenda platform. Ashley Stirrup (08:51) Mm-hmm. Mm-hmm. Yeah. okay. Yeah. Got it. Yeah, because I would imagine in your line of work, the level of trust with your technology partners needs to be really high just because you're dealing with people's money. Yeah. Charles Williams (09:25) Yes. Yeah. Yeah. And the good thing, a good thing from my perspective is what I'm that, and the AI influences don't touch the money. It's more of the applications and things like that. So, ⁓ but we're definitely, it's a, it's a, it's still important. And, and the way is used, like it's not one single model, right? So the way GitLab functions is there are almost like an AI gateway. So there's multiple models based on your needs that can be invoked if need be. Ashley Stirrup (09:36) Great, got it. Yeah. Mm-hmm. Yeah, that makes sense. And are you kind of measuring like utilization of AI across your developer teams? Charles Williams (10:02) Yeah, so we're starting to that. Actually, we're starting to figure out not only just consumption of who's actually using it, what are they using it for in the four main categories around the key things that everyone's measuring, right? Co suggestions, ⁓ co quality reviews, security ⁓ flags and use case documentation. like units, assessing things like that. Ashley Stirrup (10:21) Yeah, yeah, one thing I'm hearing is really transformational. I guess they call them Ralph loops where you're not only having the AI build the code, but actually do the testing as well. And then, you know, identifying issues and fixing those. So that by the time it gets back to the developer, you've got more, much more production ready code than you would have, you know, just a few months ago. Is that. Charles Williams (10:44) Yeah, mean, we have examples where I've heard developers say, you know, I've written 25,000 lines of code, 10,000 lines of that was unit testing. was, developers are learning is that not only you don't need to get rid of your development skillsets, but you add on prompt engineering. So that's the other thing, right? It's learning how to prompt and letting AI do its work and then coming back and be the oversight. Ashley Stirrup (10:55) Yeah. Right. Yeah. Yeah, it's almost like every developer is now a manager too. Yeah. That's right. Yeah, so it must be an exciting time to see how those teams are becoming so much more productive and capable of doing so much more. Charles Williams (11:15) Yeah, you're managing your AI agents, right? Your AI agents are your employees. Yeah, correct. Yeah, they are super excited, almost like the floodgates. We started this whole process, it was mid last year, really started getting heavy into it and we were intentionally being purposeful about the beta test group and so forth and moving it through. And when WING got out, it just became an explosion of interest. So excited is, I don't even think it completely captures what I'm seeing in the business. Ashley Stirrup (11:54) Yeah. Yeah, well that's really good to hear because you you always worry about kind of more traditional businesses, maybe not a software company, being you know maybe a little more like resistant to change but it sounds like your teams are really embracing it. Charles Williams (12:16) Yeah, I think that'd be the differentiator between us and any other in our industry is like, if you do not embrace this, you will get passed. That's a simple way looking at it for us. Ashley Stirrup (12:25) Yeah. Yeah, and with so much change going on in the industry, how are you staying up to date and deciding what to try next and what's safe to include and what like, maybe we'll give that one a little more time to bake. Charles Williams (12:44) Right. Well, I don't sleep a lot. No, ⁓ no, but you have to build in your guard rails and you build in your governance models and things like that, you know, to help. we do have a verbal verbose team of steering committee around technologies. have our ⁓ team involved. it's not a Charles just doing it. It is a my working group, my steering committee, the enterprise architect. So this is a entire business initiative. It's not just one initiative. So Ashley Stirrup (12:47) What? Charles Williams (13:12) I may think something is really cool or some elements to whatever pick one feature, but that may be a hidden, you know, stroging horse into something that he found a while ago. So it's always what my team needs to do and has been doing is making sure we go around the horn to all our stakeholders to be sure that what we're bringing in makes sense and everyone agrees on it. Ashley Stirrup (13:19) Mm-hmm. Yeah. Yeah. And so are you tasking some people with like, this is now your job to go evaluate AI or is it something that everybody's taking on and then as they find things, they bring it back to the larger team. Charles Williams (13:48) So what Churus has done is kind of a little bit of a hybrid of both of those. So as team creates use cases for AI, what they're doing is saying, let's hear your AI story, let's hear your use case, and then we'll evaluate it against our current AI approved and what's already down the road, man. Well, we don't want to have duplicative AI instances, right? It's okay to have AI ecosystem, but its purpose needs to be... Exactly, like co-pilot would be productivity and use cases, right? But then GitLab could be your developer. So, and then there's another AI just built and driven for consumer facing, right? So we've got to figure out that. So when folks have ideas, they come to the steering committee, present their case, and then it's processed through appropriately. Ashley Stirrup (14:34) Yeah, yeah, there must be ⁓ no shortage of ideas. And so then it becomes like, how do you pick? Right? Yeah. Charles Williams (14:37) Yeah. Yeah. And that's yeah. we're in the learning stage in this, right? It is how you put it. No one's I have not seen anyone who's got it down unless you're just completely involved in one ecosystem. Like if you're just doing a Microsoft ecosystem, then you know how to you know what's the big right. But we're still learning. We're still in our learning process. Ashley Stirrup (14:45) Yeah. Yeah. Yeah. Yeah. And speaking of that learning, know, AI always, the demo always looks so great, right? And then, you you use it and like the seventh time, suddenly it's lost track of what you talked about the first time or what have you. So you have, have you had some of those learning moments when you thought, this is going to really work. And then as you started to do the pilot, you're like, hmm, this one's not quite ready for prime time. Charles Williams (15:05) yes. Yes, yes. I have actually, I've experienced it directly myself, where the context limitations, I hit the context limitations on my prompt or my session rather, and thinking, ooh, I really needed to continue doing this. And I was trying to build an application to pull metrics for something or another, and it just died. It just stopped. So, wait a minute, this is supposed to be one of my, these custom, well, these stock agents that they had, and it would not. Ashley Stirrup (15:37) Mm-hmm. wow. Yeah. Charles Williams (15:55) He just couldn't do it. So I'm like, OK, well, maybe this is not quite this age is not quite ready for. Ashley Stirrup (15:57) Yeah. Yeah, yeah, yeah, and the amazing thing is, you you wait two months and suddenly that agent that wasn't ready is now ready. And so it's not like you can just say, AI is ready for this and not ready for that, you know, because it's improving so quickly. yeah. Charles Williams (16:07) Yeah. Right. Yeah, it is a matter of just wait because we spoke to our vendors and I think every release that happens, think every two or three weeks, there's a new edition, there's a new prompt change, there's a new something. yeah, it's fast paced. It's probably the most exponential growth I've seen in technology in my entire 35 years of being in IT. Yeah. Ashley Stirrup (16:36) Yeah. Yeah. And how are you investing in your people to help them stay on top of all this? I'm sure they are all worried about falling behind and eager to adopt it, but also not always sure what to do first. So how do you help them in terms of that development? Charles Williams (16:57) I started with, I had a mentor tell me once, ⁓ two, a of years ago that, that really reinforced it with me is, is AI is not taking your job. The person who knows how to use AI, that person is the one taking your job. So I, I re instill that in the team is embrace it, learn it. I'm not asking folks to become AI engineers necessarily and understand how to build an LLM themselves, but understand your function and day to day and how it can age you in that. So if you can do things faster, that means you can do more things. Ashley Stirrup (17:09) Right. Charles Williams (17:25) with the same amount of hours. That's what leadership is looking for. They're not looking for you to be AI expert. They're looking for you to leverage the tool in order to accomplish your role. That's I still have my team. Ashley Stirrup (17:29) Yeah. Yeah, yeah. And I know the challenge for me and my team in marketing is there's so many great stories. And so you always feel like you're behind. And so you always want to be doing more, but you've got a day job too. And so it's like, am I really going to carve out the time to do this thing that might 10x this one part of my job, or it might be a huge time suck? And so I'm sure your teams are struggling with those kinds of challenges as well. Charles Williams (17:44) Right? Right. Yeah. Yeah, because my team's understanding and learning just prompt engineering and figuring out like, and I had to have someone on my team say, this is going to replace, you I don't need to learn to code. said, well, if you have a passion for coding, then stay on your passion. This will help you understand it faster and understand some of the nuances in your logic that it'll help you get past that when you're struggling and maybe you're using way too many lines and it's doing it more efficiently. That's you want to learn that, right? And that would make you more valuable quicker to the business. Ashley Stirrup (18:29) Yeah. Yeah, yeah. Well, also, at least today, humans can kind of see further down the road and probably have that bigger context on where you're trying to go. And so you need to be guiding the coding agents to your longer term vision so that you're not doing something that works short term but isn't going to enable your long term goals. So, yeah. Charles Williams (18:40) Yes. Yes. Yeah. And like I said, spoke about the context is, I was on a meeting today and, and one of my power users said, yeah, I asked AI, it wasn't even the one we're using. was talking about a home thing. I asked it five different times, the same question. They gave me five different answers. So he said, have to guide it. No, didn't know. make sure. Cause we had to keep the context in the bigger picture. So he understands and it can help us versus taking over for us. Ashley Stirrup (19:06) Yeah. Yeah, yeah, it's funny because it's so easy to get really confident in AI because it's doing something so well. And then you can ask it a very simple question like, how many points did Steph Curry score last night? And give you some answer. You're like, where did you get that? So it can be a little bit of a scary reminder that, OK, I can trust AI, but trust but verify. So. Charles Williams (19:29) Right. Right. I'll say this. So I had it today. I asked AI to help me write an HTML application to do something right. And it did it. It wrote beautiful lines of code and it did everything. I got the HTML, but it didn't work. it told me, ⁓ know, it literally said, I know the problem is you're not running this. You're running it locally. You need to run it from your public folder. Great. I will move it there for you, Charles. Great. Hit accept. Go do the merge. It did everything and said, OK, you're fine now. And I went to look public. It wasn't there. I was like, did you move it? Yes, I did. No, you didn't. It was literally arguing with me. So it's not always like you said, trust, but verify. Absolutely. Ashley Stirrup (20:15) Yeah. Yes, yeah, and that realizes, oh, actually I moved it to this other folder. yeah, and so as you're looking down the road in financial services, where do you think the biggest innovations will come around AI? Is it on the customer service side? Are there? Charles Williams (20:25) Right. Like, no you didn't. Ashley Stirrup (20:43) certain product offerings that you think, know, like I would guess maybe ⁓ processing loan applications, things like that, that you might be able to get a lot quicker at. Charles Williams (20:53) Yeah. And I would speak from that. I'm to speak from a consumer side voice and saying that yes, that would be the biggest leaps as far as helping me process and help me understand the choices that I have available to me for wealth management or insurance or whatever. Like, so now you're not just talking to me about one product. You can talk to me and understand context of my family and give me a wealth of products in order to identify the best route for me, for my financial future. Now the caveat there, which I think everyone knows is Ashley Stirrup (20:58) Mm-hmm. Yeah. Charles Williams (21:23) How much do we give AI permission to do? Right? So how much of my data, which is the major restriction and frankly, risk at everyone's stand across the board is, do I want AI getting that deep or do I want to have it be a consult against that data? You know I mean? So that's the plus or minus. Ashley Stirrup (21:26) Mm-hmm. Yeah. Yeah, yeah, the interesting thing there is that I think the younger generation has much higher expectations around, expect you to use my data, I expect a more personalized experience. The older generation's more like, hey, what are you doing with my data? Yeah, and. Charles Williams (21:54) Yes. Right, right. And I think, I think going back to your statement about brick and mortar traditional companies, don't banks or, you know, brick and mortar banks were probably erring aside of call it the older generation. When you got the FinTech startups, well, yeah, they'll probably err on the other side. So I think we haven't bridged that gap yet. And frankly, I'm not brushing that because I'm on the old generation. So I'm not brushing that at all. Ashley Stirrup (22:11) Yeah. huh. Right, right, right. It does open up some pretty interesting avenues though around kind of the disruption that comes from the fintech providers. But with AI, maybe that's now more possible for the kind of more traditional financial services companies to essentially be fintech companies too. Charles Williams (22:43) Right. I I would think that's what they want to expire and grow up into to be more like Fintech because Fintech can be more mobile. So you can have a Fintech agility, but with the grounding and background and stability of a brick and mortar. That would be the ideal unicorn, right? Ashley Stirrup (22:57) Yeah. Yeah. And so as you think about how AI has helped truest, would you say that developer productivity is the biggest benefit so far? Or is it more on ⁓ code quality? Or how do you think about where it's had the biggest impact? Charles Williams (23:21) I think our initial impacts are going to be around their productivity. So their gains are just onboarding into pipelines and not having to do the manual tow gating and processes. think just getting a developer up and running, a new developer, actually comes into the team brand new. How can I create him? How can I make him be more productive in 10 minutes so he can start? I think that's going to be our key. Ashley Stirrup (23:35) Yeah. Yeah. Yeah. Yeah, yeah, that makes a lot of sense, especially with a large pre-existing code base. You know, it's hard for a human to wrap their head around all that. Much easier for the AI to help and then kind of educate you as you go. Yeah. Charles Williams (23:52) Exactly. Exactly. Exactly. think that's going to be the key. And then part of that will be the quality and all that stuff. That'll just come as part of your journey down that map, down that road. Ashley Stirrup (24:10) Yeah, yeah. And so we're starting to run out of time here. ⁓ Maybe one last question. If you were to say, go back in time and talk to yourself 10 years ago, what advice would you be giving to yourself ⁓ as you kind of thinking about your career and your own personal development? What things would you tell your younger self to work on so that you can be as successful as possible in your career? Charles Williams (24:16) for sure. Yeah. Yeah, I would tell myself, and I've actually thought about this recently, I would definitely tell myself that invest your time and education learning and machine learning at the time, because it was the original database management relationship and understanding how things are interconnected. And that'll feed the natural logic into what AI is today. So if you can learn that and the principles today, it would be that I would be way ahead, honestly, that I had been. Ashley Stirrup (25:05) Yeah. Yeah. Yeah. That one's a tough one, right? Because that's almost seeing around corners. ⁓ How about ⁓ more on like the personal development side or management side as you're managing teams? What skills have you built over time? Yeah. Charles Williams (25:09) Yeah, yeah, yeah, for sure. So, ⁓ yeah, so I definitely built up, ⁓ I call it, well, it's kind of two major things is understanding the empathy and the neurology, what did I call it? Neurodiversity, that's what I called it. Neurodiversity of your team, right? And that has nothing to do with anything other than how people think and perceive what's in front of them. And what I've learned over my years is everyone approaches the, Ashley Stirrup (25:40) Yeah. Yeah. Charles Williams (25:50) problem differently. If I ask five different people with DevLive experiences, I'll get five different answers. None of them is wrong, but they're different perspective. The other thing is, which I practice, try to practice all my life, is understanding change management and how change happens and evolves in an organization. This is not change management from a technology. This is things are changing and how do you flow? How do you pull people with you and understanding the motivations around it? So I've used that to help me over the last few years. Ashley Stirrup (25:55) Yeah. Yeah. I think those are two really powerful concepts. go really well together as well because it's interesting. I was thinking about that the other day as a leader on my team. And we're trying to charge that hill. But somebody wants to go left and somebody wants to go right. And sometimes it's like, it's OK to go a little to the left if that helps with kind of. Charles Williams (26:23) Yes. Mm-hmm. The ultimate, yeah. Ashley Stirrup (26:44) Yeah, it helps the team feel like a little more empowered and... you know, that they have a little more say in the direction of where things are going. And so I think that neurodiversity is a really important thing to keep in mind, because there's not always one right answer to things and figuring out a way that helps the overall team be the most productive, ⁓ you know, sometimes involves you some flexibility that, you know, as a leader, sometimes you know, I just want to go straight to the hill. Charles Williams (27:02) Exactly. Exactly. Ashley Stirrup (27:13) And it's like, Charles Williams (27:13) Yeah. Ashley Stirrup (27:14) no, okay, we need to stop over here on the way. yeah. Well, thank you so much. I've really enjoyed this conversation. ⁓ Charles, it's such an exciting time and it's clear you're doing a lot of exciting things to help apply AI in the financial services space. So really enjoyed your time. Thank you so much. Charles Williams (27:16) Yeah, yeah, for sure. Yeah. Yeah, thank you. Thanks for having me. I really appreciate the discussion as well. I enjoyed myself.