Latest / Architecting the AI Enterprise / Episode 4: The Unseen Role of Judgment and Accountability in AI-Driven Decisions
Transcript
- Marketing: Hi, welcome. I'm Gary Hoberman, CEO and founder Unquark. Hi, Dave Ferrucci. AI Officer and CTO at Unquark. And welcome to Architecting the AI Enterprise podcast. In the podcast, we get to bring amazing and credible leaders across technology and enterprise, specifically enterprise. They live breathe this to share their experiences, their background, but more importantly, where do we think we're all going to? in this crazy new world of AI we're living. And with that, I am honored to have Jenny Larson join us here. Jenny, I first got to know Jenny as I did as a former global CIO. And Jenny is a world-class technology leader, AI advocate. And just Jenny, thank you for joining us here today. Yeah, thank you for having me, Gary. And nice to meet you too, Dave. Awesome. So Jenny, we always start off with like the hero's journey of how you got to where you are. Did you always want to be a CIO? us the Jenny story, the background, the technology journey, and how that came about. Well, I started out as a developer at one point, which was great. I coded embedded systems, C, C++, Java. That was kind of my backdrop. And then I moved into banking and financial and insurance and just kind of went through the ranks. And all of a sudden, I found myself kind of as this In this global company, I was with GE for more than 10 years, amazing journey, just working with amazing people across the world. Then I moved on from GE when we saw, I was first part of just selling, you know, $150 billion of assets when all of GE capital was sold. And then I found myself coming out of that and thinking, okay, what do I do now? And then I went for a CIO job and I realized, you know what? I could actually do that and I enjoyed it. was great. was, I moved into, so I was in GE, kind of large global organization, operations across Asia, Europe, and then found myself into this fast moving company that was growing a hundred percent year over year in the financial services space. And it was just a completely different experience. But it was just amazing. Right. And I'm, really happy about just the global insurance companies that I've been with. but then also the global banking companies and then just a fast scaling environment. Cause you learn completely different things in terms of how you operate just with organizational inertia, or if you have a huge momentum that you have when you're in kind of the growth mode and you're just trying to figure out how do you enable your organization for the next month so that you can grow with the pace that you're at, right? And so tell us jumping right in. So five years from now, jumping a time machine. come out of the time machine? What is the most important role within a company? What do you see the future organization being structured? So for me, it's not just the IT org. I think about the overall organization. So two layers to this question. IT organization itself, but then every other organization. And you guys have touched on this in previous episodes, I believe, because even now, I am seeing leaders and people are taking on AI agents alongside their teams, right? So in, in the next few years, everybody's going to be having teams of AI agents, right? Either working alongside them or even managing them. And then you also say that architects, AI architects or enterprise architects is the most important role. I would, I would push a bit further on that because the most important capability isn't necessarily a role. I think about judgment and I think about critical thinking ⁓ and knowing when to step in and knowing when to trust. When you think about what the IT organization is going to evolve like, there is a more radical shift that's most likely on the horizon where IT becomes the backbone, Infrastructure, security, governance, architecture, coupling it all together, all of the things that nobody really wants to care about. But then the big IT development pieces that we've had in the past, they're creating the building that gets democratized across the entire organization because you have citizen development, you have the tools that are available for people now. And then you're going to be blending domain experts and software builders as part of that. So just building up, typical build versus run. Build versus run in a bank or insurance, like sometimes Organizations split their IT group into build and run and hey, the run is the boring stuff. That's keeping the lights on and you just- And you typically need different people, right? Because there's the people that love the building, loves the change. And then there's the people that run and they love the run as well, right? It's almost different personality types. So in your view though, the build group gets merged back into the business. Yeah. They become business. And that's what it was. That was the way it was when I started coding is like there was no concept of CIO function and IT and, you know, like the, you sat on the trading floor, you built systems with the traders, you sat with the business, your bonuses were based on how well the business did. a concept. You know, if I remember when I started building, was the same thing. I started building. was building with the business. I developed products and C C sharp and I shipped products, right? And I was the UA tester, I was developer, I was the one gathering requirements. It was all, you know, in one world. ⁓ So, Dave, what do you think in that world where there's the, I'm going to call it the janitorial staff in some ways, because it's a cleanup. It's the, it's the keep it running cleanup. And then the build goes to the business. I'm curious. What do you, where does architecture play in that? That's an interesting question. Is architecture a build or is it a run? First of all, ⁓ Yeah. So I mean, look, I'm an engineer and a scientist. I'm not a business person, but I'm very applied. So I think that the relationship between engineering, design, development is always in service of the business. You have to understand why you're doing things or problem you're trying to solve. You don't really want to throw things over the wall. So. I think the, the integrated view where everyone's focused on the business outcome and the business function, you know, makes a ton of sense. One of the things I wanted to pick up on on what Jenny was saying was, you know, about how things are evolving for all the players with regard to the use of artificial intelligence. And, you talked about judgment and, and, I, and I think the role humans play in all of that as AI. ⁓ and agents get better is around accountability, judgment, and trust. And I often think about the job of an engineer or an architect making judgments all the time. And they're not just making judgments. They're, when you combine that with accountability, they're making bets in some sense, right? They have their projecting forward in time about how they expect a lot of things to change, the surrounding technical environment, the surrounding market, the surrounding technology, how the system itself, how different components will change independently of each other. And they make assumptions, they make bets, and they engineer and design with respect to those assumptions, and then they're accountable for them. And I don't think that complexity ultimately goes away. I think the AI accelerates, implements many of those things for us, moves things faster, but that judgment, ultimately that accountability, ultimately that insight that makes us, makes us humans have to say, I'm going to bet on this being the right direction and then being accountable for it. I don't think that goes away. I think you're going to have to have accountability for sure as part of it. I was recently in this agentic course at Harvard because I felt, you know, I'm going to go out and see what the research says, the top universities in the world. And what stood out to me wasn't necessarily the tech. It was the ethical design frameworks and how you need to think about things going forward and how you need to put that into your design framework upfront. So the best example of this is your self-driving car, right? you have to design upfront, what are you going to do? Are you going to hurt your driver or are you going to hurt pedestrians? And that's the ethical design framework. And that's not necessarily a software engineering question in the future. That is going to have to be values, leadership, and people are going to have to step in and be part of that process. And that's a big gap that I'm seeing in most organizations. As you're rolling out agents in the organization, are you thinking about that enough? interesting is like so there's an MIT driving game where I play it where you could go online and say okay the car is driving I can either go left and hit the elderly couple or I go right and hit the three kids and the mom and what should the car do based on and what's interesting is the car itself potentially might have a say to say I don't want to get hurt either down the road the car could say which which of these is going to be least damage for my myself as a car ⁓ but it's interesting because the the fear that I have is Like Jenny, when you would deploy a change in the global insurance company, or GE, like every change you deployed on a weekend or a Friday night and then tested the weekend and rolled back probably, every change was impossible to get right. Just adding a simple field in a database required altering tables, offloading data, loading data back in, changing the, and it wasn't done well. And I just keep picturing when the model changes or variable changes in the model, the answers change and we're not aware. Like suddenly the behavior changes and the car is going to, you know, do something you didn't expect. Like, and that's, that to me is the interesting part is like what, what, what's your acceptable rate of error in that case? Right. That's, that's the scary part. Scary part for sure. And the way, the way I think about this, first of all, I think about it from a standpoint of, for me, it depends where you are in the process and what the consequence of that error is going to be. So in regulated industries, and I know you guys have touched on that in previous episodes, right? Healthcare banking, it's a regulated industry. You cannot go wrong, right? And you have to upfront, you have to plan for that. You have to put that as part of the design process, governance and human oversight. But then you have... So many other areas of the organization where, you know, if an agent gets you 70 % of the way, you know, when it used to take three days for somebody to do it and you finish off the last 20 minutes and 30, you know, the last 30 % in 20 minutes, that's, that's a huge leap forward for an organization. I was going to say, I think there's just a, um, another kind of more fundamental issue going on with this probabilistic nature of large language models. And their usage, because when we use them for things that are actually not designed for that creates more fragility in a system, I think we're, just getting carried away. I mean, this is a bad design. So it's very easy to start asking agents to do things. And then agents are making decisions, possibly hallucinating, possibly making errors that are really. They shouldn't be making it all because there are very clear deterministic processes for doing that. But it's so easy to throw an agent as a problem rather than step back and say, this aspect of the system should be absolutely deterministic, error-free. There's no reason to have any probabilistic process involved. How do I re-architect that? Now you might have the AI help you write the deterministic program. That's fine. But throwing certain problems at probabilistic agents is just a bad design point that I think we're getting carried away with. And we're really not stepping back and thinking about architect these things more effectively. think to your point, there's always like this different types of risk. So if, if when you have to do that risk assessment, In some processes, especially for finance or regulatory issues or healthcare, your tolerance is zero. It's just zero. ⁓ And throwing an LLM at that when there is a deterministic process is just a bad judgment, speaking about judgment. On the other hand, to your point, if you can accelerate things with LLMs ⁓ and you understand the nature of the errors that might happen, and they're worth the risk, that's fine. But that's another judgment to come to that determination. You know, it's interesting. So we didn't talk like so the car example. So you could do vision, you could do LIDAR depending on which one. I guess there's is Rivian coming out later this year with both is what I heard they're going to have. But the whole vision aspect of knowing what's around the environment is a great AI use case and understanding what's there. that's but I guess once you know it's a child versus the elderly, That Dave that could just be a deterministic process at that point. It could just be a rule, right? It could be a value. It could be a value judgment that's made and predetermined. that's right. That can be a rule. I mean, I'm just, I'm just the reason why I don't like that analogy is because it's, it's good on one hand, but the reason I don't like it when it applies to software and engineering and because everything's always not that stark. mean, there are micro judgments that are made all the time. that when you accumulate, they amplify throughout the system. There's no one stark, you know, go to the right or go to the left. know, do they, it's, know. And then when you have teams that are responsible for different parts of it, right? They, you know, they don't need to how your system error just rises to the ranks, right? Exactly. So Jenny go in this period, we've advanced five years. So build is now inside the business, run is here. I'm curious, you yourself, where are you in this role? What do you see your, if you were gonna be able to define your role right now for what you'd wanna be in five years in that picture, are you in the business? Are you run? Are you, where do you see that? I'm curious. I would see myself on the business development side, for sure. Because if you think about my background, it's all been transformational work. very tightly coupled with how are you driving your organization forward. So I would definitely see that being more on the business development side. then, however that's going to look, who knows, right? Because we're just evolving. So in the CIO world, remember before I got the MetLife CIO world, the recruiter who called me said, there's a matrix. And there's this matrix. And the matrix, the quadrant on the right is innovative CIOs that understand cost maintenance. then there's, so it's cost control versus innovation as the two matrices. And where do you sit in here? Are you the innovative, but you couldn't care less about the cost? Are you the cost CIO who just wants to focus on expense reduction, but can't think of anything new? In this new world, I guess that that's the bifurcation of like, hey, if you're innovative, you're going to the business and you're to help them directly drive and you're not going to be constrained if you're cost-cutting, you're going to be the run guy or girl and basically get that done. And it could be a good way to think about it is that that's the dimensions we've always talked about in the past. The car analogy, bringing it back to insurance, we are helping insurance companies today to take in unstructured data through emails and documents and PDF and Word and Excel and making meaning of that using foundational models and LLMs and... extracting out the data where it used to take days to your point. It used to take three days and now it could take 20 minutes. then applying the deterministic underwriting rules. then applying the... Because underwriting is not a... You don't need a foundational model for that. It's pretty cut and dry. There's a run book that says, here's what the regulator will allow you to do or not allow you to do. And you can't consider anything else except for what's in this run book. When I think about people that are saying, okay, so like, what's the error rate and what's allowed? The others... The other piece that I feel we're not talking enough about is we're expecting it to be a hundred percent. But then when you take any team and you put an assignment in front of five people, are you going to get the same exact five answer back from them? Most likely not in a lot of cases. So even in operations or whatever function that you're running, you will have human judgment or subjectivity, right? That is, you know, But you're always expecting agents all of a sudden to run at a hundred percent. But do you even know what your own organization is running at? Yeah. I mean, I, go, so I agree with that. I think you always have to have a baseline and having the human baseline, ⁓ is really, really important. ⁓ it's not just a human baseline though, right? There's also solving the problem differently with a computer and a traditional conventional algorithm. Is yet another baseline. So the times a computer incorrectly looks up your social security number, I'm sure approaches zero. ⁓ we're not doing that probabilistically. So, ⁓ I think when you look at a problem, you have to sit there and say, what is the right baseline to apply? What is the right expectation? You're absolutely right. Errors are, are humans are error prone. The driving cars is great. Like if all the cars were. you know, AI self-driving cars, I'm sure the overall accident rate would be dramatically lower than, than humans driving. And so that's true in that case. So when you look at a system, you don't, know, and Gary's going to laugh because I love this phrase. You don't want to brush your teeth through your ear. If you know, there's a deterministic process that is 99.9999 % accurate that becomes your new baseline. And you don't want to introduce a probabilistic process when you know how to solve that deterministically with that level of error rate. There are other problems that we just can't easily do deterministically where these LLMs are incredibly powerful. And then our baseline switches over to, who's doing it now? Humans are. What's their error rate? Yeah. And how long is it taking? Yes. And how long is it taking? I mean, if you could do underwriting and explain the thought process. that's so Dave's Dave's model he before he came to Uncorick was deterministic, we call reasoning AI reasoning engine that was not just repeatable, but explainable to come back with. How did you come up with the answer? And you you approved that loan. How did you come up with that answer for the loan and go through the steps? If you're able to keep the log of the steps that the AI went through and record that along with the answer. That's better than a human. I know you can't ask a human. Jenny, you approved this loan. Why did you approve it? You'd be like, I don't know. That's right. That's right. Yeah. I that's true. I mean, look, I absolutely I hate what I'm doing right now is I'm tweaking everything that you guys are saying, but I can't, I can't, I I can't, I can't resist because you can get the, you can get the LLM to to follow a procedure ⁓ and you might as well just have it write a program that follows a very specific procedure. But if you can't, because that is so ambiguous and fuzzy and so language-based and so experience-based, you can certainly get it to do something that would otherwise be very, very hard to program and to mirror the intuitions that humans use or whatever. And then you ask it to explain itself. There's a difference between asking it to explain how it went through a series of rules in a very systematic way and asking it to explain a prediction it made, a probabilistic result. Because what's fascinating about LLMs is they give post facto explanations. Often like humans do, they look at the result and they go, huh, how might I have come up with that in a way that makes me look good? But isn't that why you need to build it in as part of the decision process? Like, you need to find the flow, to find it upfront to log, here's why, here's the data I'd used, here's the decision I made. Move on, right? Right. You need to make sure it follows that process. and again, you can always, and this is why you can always use the LLM, the AI to help you define that process. Now you have to assure it follows that process. And then give you the explanation that reflects that process. And then you have the best of both worlds. Yeah. But the other, the other piece to this, so people aren't talking a lot about the human error. The other piece that I don't feel people are talking enough about either is when you design some of these agentic workflows, or you design, you're, you're taking a lot of them. And I'm very passionate about this coming from GE, like black belt, sigma. was all ingrained in us, right? designing the workflows from an agentics perspective, as opposed to just taking the human process. Because when you do agentic, you can do workflows in parallel, which, you know, in the human, you had to do in a certain sequence, right? So it's the analogy of, yeah, you can put the Ferrari, right? You can put the Ferrari in the traffic jam, but it's not going to go any faster. So if you don't design your workflows from, from an agent standpoint, if you're taking those, you're, just going to be in the same situation. I agree with that. I think it has to be kind of ⁓ AI first. You have to look at it holistically. I'm solving this problem differently now. How do I use the tools most effectively? Yeah. I agree. what, Jenny, if you looked back and said in the corporate world, which department would you think is most able to be replaced by agents today? what do you, what in reality, I'm asking you, which department was the most painful to work with? I guess I'm saying. First of all, I think the opportunities are endless for any department, to be honest. I always think about the mantra, which is finding the biggest pain point that nobody wants to do, but that's critical to the business and making you money. And that's where you should be putting AI first. I would also push back a bit on the word replace because I have not, maybe you guys have seen this, but Have you guys seen organizations accidentally replace complete departments? I've only seen the best ones and they're actually elevating them. They're finding their best people and then they're giving them AI as coworkers. ⁓ That was my kind of cue to it. But if you look at what the market is doing, right? Customer service, a lot of them are going after customer service, marketing, finance. There's a lot of kind of repetitive admin work that people are doing. So I think they are for sure. ⁓ opportunities, even HR recruiting, there's a lot of repetitive work, right? But I haven't seen complete departments. What have you guys seen? Well, I mean, I, I, I could speak to a couple of examples where I agree that there aren't completely eliminating department, but I think there are definitely junior people are not as valuable. think there's definitely this sense that I. I definitely agree with, and I definitely see, which is that more experienced senior people are more valuable because they, their insights, their experience, their judgment, their accountability, their, the trust that people have in, in the wealth of their knowledge is now amplified with AI. and they become like the more, you know, the more valuable. ⁓ people and they don't need as many as assistance or they don't need as many people working with them to achieve the same result. I think that's a very significant trend. It does raise the question or the challenge. How do you create more people like that? If you're, if you don't have entry level positions. So you need more mentorship. need different sorts of programs, to allow those senior people to, you know, to keep the pipeline open for that. Anyway, it's an interesting, it's a, it's an interesting question. I have a more provocative version of Gary's question, is at this point in time in your work life or your personal life, do you find yourself thinking, gee, I rather discuss this with an AI than with a person. Yeah. Of course. Yes. I hope Dave doesn't say it he's like, before I ask Gary, I'm just going to go talk to my little chat phone over there. Haven't you guys built a Gary brain yet? So that we can go I had someone reach out, Jenny, and said they want to basically give me a twin CEO to make decisions for the team. And I'm like, how do you know how my brain works? That's what people are doing, right? Or even, your mentorship, right? You build out your mentor, the brain of your mentor, right? Yeah. Yes. It's an interesting phenomenon, right? I mean, it's where you have a topic you want to discuss and you think I'd just rather discuss it if I had to pick anybody in my life to discuss this with. It's just an interesting place, I mean, for humanity to have reached, quite honestly. I diverge from the topic of software. But I mean, use so I'm using all the different chat bots and all those chat bots know probably more about on cork AI than we've disclosed publicly in any any of them. And I last weekend for fun, I just said each of them, said, build me a pitch deck. You know, assume tell me whatever you knew about this from me and from everyone else. Build me the pitch deck. And it did a really good job. It was it was surprising. OK, Jenny. Let's get onto the most favorite part, the speed round of questions. By the way, you can always ask us any as well. So feel free. And Dave's going to, Dave's going to pick the first question for you. All right. So I like your favorite use of gen AI in your personal life, not your professional life. My, my kids typically like they have a, they have a saying, ⁓ who's mommy talking to? ⁓ she's talking to her best friend AI. I use it for everything. I plan my garden. cook my. I myself in terms of being a better parent. I build up, you know, brains or workspaces with all kinds of things. So yeah, it's... How do you not get so... Like, the one problem is I do the same thing and then I have to stop myself and go, it's making me feel really good about myself, which is not good. Like, I almost want, like, the grandmother AI behind the scenes slapping me back into... You know what my husband did? He actually did this because in your custom instructions, my husband did this. So he put in his custom instructions something about a mentor and then he had a version that was just being really harsh. Like you're going to critique the heck out of this, which it did. And he ended up being really discouraged. So now he's softened that out and he had more of a mentoring voice on that one. Jenny, favorite book? So I have this author ⁓ and I utilize like the tips and tricks from the book all over. his name is Scott Halford. It's activate your brain. And I had the privilege of just meeting him at one of the GE Crotonville kind of leadership courses. And it's how you nurture your brain, how you think, how you handle pressure. You need to scribble on your notepad, right? To really learn what you're hearing ⁓ or just how do you feed your brain in? before an important conversation, for example. So like it's Slope Centrics. I've, yeah. I guess my morning is always playing Sudoku or Ken Ken, Wordle, Connections. go through a series all the way up to chess puzzles. And then at the end of that, I kind of feel like I'm ready for my morning. That's it. So that's my activation. How long do you do that for? I spend, I take a cup of coffee. And by the time the coffee is cold, I should be done with that, which is about 20 minutes, I would say. that's, that's my, before I exercise, I used to, I've got a broken ankle now, so I can't exercise. But so now I spend a lot longer doing puzzles and then exercise, but I'll be back soon. how about you, Dave, how are you using your AI tools in your day to day? ⁓ my goodness. Well, I'm similar to you. I, I use it for a lot of, a lot of things. I mean, I use it for coding and prototyping, like, but Even for personal projects, but it's a huge research assistant for me. I debate with that. I, ⁓ I, I use the deep research. have a lot of, ⁓ hypotheses and I wonder if they're true. A lot of them in healthcare, a lot of them around biohacking and understanding the human body and, ⁓ health related stuff. And I just get into it. I mean, one of the things I did before it was one of the first rag systems. that we did at elemental cognition was really directed toward, toward bio, ⁓ bioscience and bio research. And so I do a lot, I do a lot of that. ⁓ I, I've gotten to a point where I will actually in a restaurant because I'm on a particular diet and protocol right now to try to achieve certain things. I'll just take a picture of the menu and say, consistent with my dietary goals and my health goals, what would you pick off of this menu and tell me why? I just did it this weekend. Same thing. It does an amazing job. I, you know, yeah, I'm a very, very aggressive user now for a very long time. I've had amazing experiences. You know, I did a big project in composite laminate theory, which I knew nothing about when I started. And without AI would have never taken it on. And at one point it sent me in loops where I wasted, I don't know, 10 hours because it was basically lying to me. It was telling me that it was checking the work and writing programs and running those programs. And it actually wasn't, ⁓ it was insisting that it was right and had validated its answers. ⁓ And then eventually when I. I proved it wrong, know, unconstrued, you know, ⁓ undeniably wrong. And I said, please analyze how the hell you did this to me. And I want you to write a confession. and you know, it was interesting because it did, and it admitted it lied and cheated and it gave me reasons. I was just kind of a really bizarre thing and it stands that kind of, it, it. It's word to the wary, right? So if I'm doing anything seriously, I always ask for citations and then I'll use another LLM that doesn't have the same context because it's going to start looking for coherence across that context. So I take it out of context and go to another system and I say, can you validate for this for me? Explain why. So it's fascinating. don't, I, I'm never going back, but I'm also conscious of, you know, some of the traps. love the cartoon that's all over right now where it shows the hand holding a mushroom and asks the AI, is this poisonous? And the next part of the cartoon, it shows an RIP with the robot standing and able to wink. ⁓ yes, I actually do see now that this was poisonous and I'm making a note for next time. Of course it wasn't. I'm very sorry. And this is, Jenny, as I expected, having you on is amazing. And thank you for being a guest here on the show with us and sharing with the audience your beliefs, your feelings, your excitement, your passion for AI. I think the idea you have about the business and tech and coming together around the build and the innovation part is exactly where we'll be. And I hope we'll be there. And just again, appreciate that you're sharing this. And it's great to see you again, of course. Thank you. same. It's great. Great conversation. Until next time everyone, thanks again for watching another episode. Remember to like and subscribe and see you next time. Thank you. Bye bye.