Latest / Architecting the AI Enterprise / Episode 3: Deploying Generative AI to 80,000 Employees

Transcript

Episode 3: Deploying Generative AI to 80,000 Employees · Architecting the AI Enterprise

• Marketing: Hi, welcome. I'm Gary Huberman, CEO, founder Uncork. Welcome to Architecting AI Enterprise. Hi, Dave Ferrucci, Chief AI Officer and CTO at Uncork, AI researcher and engineer practitioner. Glad to be here, Gary. Awesome. And in this series, we're exploring what it takes to really deploy AI at enterprise at scale, security, reliability, all the illities that are critical to an enterprise. Today we welcome Jeff Millen, global AI leader, former head of AI at one of the top investment banks, Morgan Stanley, which is very impressive. like Jeff, your background was way before that, right? And I'd love if you could share your story a little bit with the audience. How'd you get here? Well, first of all, I went to West Point and I was sitting in a foxhole in South Korea for a while and ⁓ decided to make my move to the civilian sector. ⁓ like many veterans, and by the way, let me just do a shout out if you want to, if you find this useful, hire a veteran. I love it. Many veterans like myself struggle. I bounced around quite a bit, wound up actually at Morgan Stanley for the first time, working on a customer account renovation project. This was right before long-term capital hit and many firms struggle to identify counterparty risk because of guess what? Data quality problems. That project went from a six week initiative to about three years and somewhere along the line I became a data quality expert. And in many ways I think my history is the history of data and analytics and machine learning and artificial intelligence. I've pretty much ridden that wave my whole career. In 09 I came back to Morgan Stanley, was actually very much involved in merging the Morgan Stanley Smith Barney entities. But I had always gone back to this concept of how do we use data at scale in financial services, you similar to what was Google and Amazon and Netflix were doing. And when in 2016, I became our chief analytics and data officer and we launched what we call Next Best Action, which was the first algorithmically driven recommendation engine for financial advisors, which was very successful, primarily because smart people with good information, with a good process, produce better outcomes than smart people without it. And then somewhat fortuitously, I was on the West Coast about four years ago, almost to the day, and we met with this small unknown company at the time called OpenAI. And we became their first financial services customer. And we deployed our first solution not far after ChetGPT was launched. And candidly, I really pivoted my... career, because we talk about AI as one thing. mean, traditional machine learning is obviously powerful, and by the way, still is powerful, but it's very different than the non-deterministic large language models. Two years ago, I took over as the head of firm-wide AI, focused on scaling AI, architecture, education training, which by the way, is the biggest challenge for organizations. It's less technology and more understanding. And then more recently, I decided to radically shift my life. and left very terrific relationships with everyone at Morgan Stanley. And I still consider that home. And I started my own business focused on education, consulting, and I have an online training platform. And so far, it's been a lot of fun. Now, what's amazing is it's a very small world. So when Jeff and I first met, what we realized is I started in Smith Barney, Salmon Smith Barney City, where I led mutual fund sales performance, fund accounting. did that was the group that actually went over that you helped integrate on your side. It was like a Passover. It was like, ⁓ I stayed behind at Citi at that point. That's an amazing trajectory. So definitely data quality beginning to, I got to ask next best action. I've always seen these run into problems where they start to, it's easy to predict what they will predict because it's always based on past. Did you overcome that? Yeah. So I would say the following when you're talking about investment decisions. Next best actions are very difficult because of, you know, historical performance is not equal future, right? Yes. That's a little disclaimer on the bottom of every discussion. exactly. But the thing about human behavior is we're actually quite, we're not that dynamic in terms of our actions. So, you know, what we did six months ago doesn't actually change very much in terms of our core preferences. They do shift over years, but part of the power of these tools when you're dealing with clients is that a couple things. Number one, you can demographically put people in, but more importantly, they can understand what you want. And you you might be someone that likes tech investments and fixed income and alternatives and, you know, Dave may be more into lifestyle issues, right? And all of a sudden he's not waking up the next day and saying, I want to be long only, right? So people are actually a little bit more predictable than markets and next best actions tend to work. much more effectively and efficiently with demographics and products and services than they do in investment markets. That makes a lot of sense. That's very cool. So I got to, the questions we're going to go through are questions we're hearing from customers regularly. ⁓ Customers will ask questions like, what is the future of SaaS? Is SaaS dead? It's in the news all the time. What's your, and before we start, ⁓ Jeff had created something you call the Jeff Bot. which you should definitely tell the audience about. It's available. I actually tested it. You can go on Macmillanai.com and if you get lonely, you can talk to me or my digital version of myself. Do you call it a digital twin? You know, I don't really like that term because I don't think they're twins. By the way, these are technologies. They're very clever technologies, but they're not people. And by the way, use JeffBot as sort of a joke because it's not Jeff. It's sort of a, I don't know, in some ways it's smarter than me, but in a lot of ways it's not. So does JeffBot do any good restaurant recommendations or No, JeffBot, by the way, my friends have been asking it really inappropriate questions. It has been specifically trained ⁓ not to answer anything outside the world of AI. Did you see the Chipotle one where someone said save all your cloud credits just ask the Chipotle agent and you could generate the code? I think there's a lot of developers out there that are probably using Chipotle to run their architecture off of these days. Yes, awesome. Dave, do you want to ask the first question? Let's go. SAS dead? SAS is not dead, but it is repriced. I think that that there will always be a need for third party solutions. The only thing that I will say is that that third party solution has to have three elements for it to be useful. One, it either has to have a unique proprietary data set that it's trained on that creates knowledge and understanding that you don't have internally. Two, it needs to have been trained by somebody that knows what good looks like. It needs to have, if you're going to build a legal app, you'd probably need a lawyer involved in building that. If you're to build a financial services app, you'll probably need to build a financial service, a financial advisor. And then finally, and we'll probably talk more about this, I think that there's a lot to be said about these orchestration layers. And I believe over time, there's going to be a greater degree of normalization on those orchestration layers. And what's going to happen is that firms are going to want to work with third party providers that are going to seamlessly integrate into their stack. And I think this question of integration is going to be an important one. It's still very early in the process, but I think if you're going to build SAS, it needs to sort of meet those criteria. And unfortunately, the reason I say repriced is, I mean, there are six, by the way, this is a real number, 62,000 AI startups in the United States right now. 80 plus percent of them are built on top of Claude or Chachi BT, right? That is not a defensible business model. But, but, and I have seen some of them that have really spent the time understanding workflows at a very deep level or have some unique data set or content. And in those cases, they're going to be differentiated in the marketplace that I think people will pay for them. Okay. So there's a future which scares me a little bit, which I remember the days when Lotus notes came out or SharePoint came out. And as technologists, we viewed it as a platform to enable us to move faster. And then you suddenly discovered the business themselves, without us knowing, built a million lines application that's powering their critical path. In technology, we call that end user technologies, end user computing, and ⁓ the regulators do not like those things, including Excel. There's ⁓ a vision I could see happening, a pattern, which would say, similar, engineers right now in CIOs are highlighting productivity gains from AI coding assistants. Look how much faster I can move 20, 30 % ⁓ more accurate. And at the same time, the business was looking and saying, I pay the check, I know the requirements, I write the requirements, I give them to IT, IT paste them in a window and it generates the app. Why can't I do that myself? Why can't the business directly take control? So like Citizen Developer 2.0, completely rewired. And in that vision, which I could easily see happening, IT would relinquish that back to the business, but suddenly become the janitor's cleaning up technical debt. So what I would first say is I don't think it's one or the other. I think you have to, and this is actually a super important point, that what people raise this issue, but they're not putting the right guardrails and the process in place. Tech is not sure what they, where their lines start and stop. The business is insure. There's a lot of end user tools out there. You even in copilot, they have their concept of agent, which isn't really an agent in the way I guess we would describe it, but there's a lot of power in those tools. So the first thing is I would say, is you have to define, I think three types of use cases. The first use case is when a business owner can design a solution. And I'm going to say a solution like a GPT or a project or an agent in Microsoft. And they can do that. completely on their own. It has all of the controls in place around PII and MMPI. Now you want to make sure those people are fully trained on how to do that. And you might want to have some type of lightweight control process, which by the way, you can use AI for, you can have AI to check how people are using AI. So I would sort of put that category. And by the way, there's a lot of things that the business wants to do today that are simply very, want to do a little audit reporting tool or a mutual fund, right? I mean, probably 40 or 50 % of the things on your list right now, you could solve through what I would describe as a enterprise-wide certified tool with the right tools. So let's call that category one and by the super value. But again, you want to make sure people are trained in how to prompt and all those issues. The second category is what I'm going to call the hybrid, where you may want to have, you're building something more complex. There's APIs, there's connectivity. You might have multiple databases to work against. You've got entitlements issues. Your citizen developer is not really capable to do that. And if they build that alone, they're going to create a problem for the organization. But what the citizen developer could do is instead of writing a 20 page business requirements document or doing agile development, they could create a prototype that they could hand to their technology partners. And in those cases, I think it's a hybrid, right? And again, you just have to define what does that look like? What does good look like from a development perspective? but super powerful for me. Instead of me describing what I want, I can give you what I want in a reasonable format and then you can say, okay, and you can ask me a million questions, you can make stores hardened. And then the last example, what I'm gonna talk about is like enterprise wide. You're building an entirely new platform for your business that probably should completely sit with your technology team with maybe some input. So I think you have to define those three routes. Probably over time, more of it will move. To the left. To the left, for sure. But I think you have to do it consciously. And the other thing I'll say, and this is a really important point, is the business is excited to have these tools. There's lots of enthusiasm. And in many cases, they're able to do it, but they don't necessarily, they don't know what a prompt injection is, or they don't really care about the issues around entitlements or, you know, multi-layer and, am I making too many? Is my context window big enough? Right? Like all these issues are completely unfathomable to business people. But the second problem is that these business people are not like they're waiting around to do eight hours of development a day. So yes, we're creating efficiency on the one side, but organizations need to acknowledge that if you want to give your citizen developers more capacity, you have to give them more capacity. And by the way, that's probably your best person already who's selling your most X or solving the whatever. And you need to then say, I'm going to take that person out of that day job at least four hours a day so they can be a citizen developer. And that's problem for organizations because they don't want to let that individual go. Isn't it a bit of not so much slippery slope, the three categories you define, there's not like a hard line between them. You start off doing something relatively simple. And I'm sure as you know, before you know, you're adding requirements to that. Now you need integrations. You need to grow as capability and the sophistication of that solution. And then you're bleeding into the second category and then you're bleeding into the third. And it's interesting to what happens organizationally, right? Because if a citizen developer is doing that first category, simpler one, and then they get to a point where they're ⁓ underwater, if you will, with regard to what they know and what they don't know, now all of sudden are they handing it over to IT? Like, now it becomes a burden that was not anticipated. Yeah, I mean, I think, first of all, We are two years into enterprise development of this technology. If you look at mainframe, distributed data sets, the internet, it took 10 to 12 years to figure out what the hell we were doing. So I think first we should acknowledge that there is no answer. I guess what I will say though, and use a great example where somebody comes in, they should be able to describe what they're doing. They should be able to articulate the degree of risk associated with it. and they should act appropriately. the system, the AI or some person needs to be able to see when that threshold is exceeded and to your point, that may need to get thrown over the transom. Now, again, maybe that wasn't budgeted for, right? Which is why the other thing that I would say is so critical here is that particularly in large organizations, you need a human being who is watching this stuff every single day. is monitoring stuff. By the way, means you have an inventory, right? You just want to people, people can't just do this stuff. You want to inventory every single thing that happens in the system. You want to evaluate it before it happens. You want to evaluate after it happens. And then finally, you need a decision maker who's able to make those balance choices between, this just got a lot bigger. We should not be spending the money on this. And by the way, that needs to happen at a very senior level in the organization and generally speaking, that human being doesn't exist. So, organizationally, you need to create sort of this mechanism that's really trying to find the balance between letting people do whatever they want, which is what they want to do, but also doing it in a way. And by the way, both sides are right here, but it's a very challenging, I used to joke when I was my old job that someone was yelling at me pretty much every day that I was either moving too slow or I was moving too fast. And the reality is, is, know. Yes. And is architecture critical? ⁓ so tell me, I know you have a five stage architecture. Six. Six. I got one. See, I missed one. Everyone's got, why Dave does architecture. That's the. Everyone's well, Dave probably knows this better than I do, but the way I think about, everyone talks about applications, right? The tools. And they talk about models, which by the way, doesn't matter what model you use. They're all good. I mean, we can argue and you, and someone will say, ⁓ no, Jeff, this is better for this. And it's probably true. but give it six months and the other company will catch up. So I make the hypothesis that the model is not the differentiation. What the differentiation is what I call the architecture. And by the way, tech is, I know tech is concerned, like I'm not gonna build apps, like you don't wanna build apps anymore. You wanna build this infrastructure because it's the infrastructure that's built in the proper way that is going to allow you not to do two agents, but 200 or 2000 or 20,000, right? That's what you're trying to build for. So what does that architect looks like? So first of all, you have a data layer and that could be, know, Databricks, Snowblakes or whatever else you've got. Then I have, you know, call it your semantic layer, which has maybe your knowledge graphs on top, does your rag, right? It makes the data, and this is a more sophisticated audience, so that you know we're talking about here, makes your data accessible to AI. Then I have what I like to call your control layer. So that's your entitlements. It's where you explicitly tell the system what you do and don't want it to do. It's the guard rails that you build into the infrastructure. So maybe somebody who's a citizen developer can't do something or even an engineer. Then you have the models. And by the way, most firms are using multi-models right now. And by the way, you can often use the models to check the models and check the other model to check the model. mean, super, super sophisticated stuff there that no human could do. Then you have what I'm going to call your orchestration layer. And I'm not talking about An agent, I'm talking about this platform that is able to both communicate individually into these agents, right, these applications, but it's also able to communicate across your firm. And I think there's an inordinate value by creating that one infrastructure because over time you have no idea where this AI is going to go. And you're going to want to be able to do handoffs and leverage data and get tasks on a single You know, the analogy I use is like in the early 1830s, there were 27 different gauges of track in the United States. And it wasn't until government and railroad tracks. Wasn't until the government and industry got together and said, is how wide tracks are going to be, that really the expansion took place. So you got that agentic layer. And then finally, you have the applications themselves. And everyone is focused on those two. but I will tell you the value, comes from the other pieces and most senior executives don't even think about this. And what I say when I talk to CEOs, I'm like, you should be talking to your CTO about their agenda. mean, whether it's my Jeff McMillan six layer or Dave's got seven or you've got five, it doesn't matter. The point is, if you are not talking about these, these are fundamental strategic decisions that people need to make and you should be really, really aware. By the way, one other, there's actually a horizontal layer too, which I call the evaluation observability layer. And that's where your independent infrastructure that's able to monitor what your agents are doing, are they acting with intolerance, and it's also your evaluation framework. mean, one of the challenges with these tools is not building them, it's feature-proofing them and making sure they're producing quality and you need infrastructure to do that as well. Dave, how does that map to, I'm curious, your thoughts on architecture? No, I think that sounds roughly right. I mean, I think that, or to the point, I mean, whether you count that as four, five, six, seven, whatever. mean, those are, you know, that spells it out. I think that if you look at it in relation to traditional software architecture, if you take AI out of the picture, it's interesting to say, see where it's changed. I mean, on the surface at the very least, it's changed in this, you know, agentic layer. I think a lot of the other pieces, I mean, the foundational AI is driving, is enabling the agentic layer. I agree that the, you know, by and large, the foundational models, mean, ones that had ones not, but I think for the sake of argument, you could say they're generally equal. And now you're dealing with the real differentiator relative to conventional software stacks is that agentic layer. And it's interesting to focus on how different that is, what responsibility we're giving it, what expectations we have. Of it and where the failure modes are because there are failure modes relative to the conventional software. It's interesting to think about clearly there's failure modes for conventional software as well. I think one of the differentiating things is the probabilistic nature of agents where they give you enormous breath, meaning you could specify much less right? The big bottleneck and conventional systems is. I have to specify every little corner case in tremendous detail. have to anticipate it with LLM driven agents. You don't have to do as much of that. They're more autonomous in their ability to work through a problem where you're not doing that enormous amount of specification and then coding. ⁓ At the same time for low tolerance applications, that's a real issue and how you manage and guardrail that like that's where all the differences I think relative to conventional systems. I think that's right. And I think two points, you know, so what do you do about that? Number one, you have to have robust evaluation frameworks, right? And that means that means your engineers are doing testing. It means your business people are doing testing because those corner cases are what kill you. Right. And that's number one. And number two, That observability piece is really important too, right? You need a mechanism that is independently judging the quality of the output. And by the way, that could be, you in some cases, you you might have two LLMs and a human. You might have a human and an LLM, right? There's a whole bunch of different ways you can do it. But I often say that like AI can help you manage the AI. But you have to be very clever and it is an architectural problem too. So you've got context window sizes and I mean, there's a whole bunch of issues that are really infrastructure related. the in fact, we just got out of a company meeting where we were talking about exactly how we're approaching this because it is different. know, AI is probabilistic. You have to look at the statistical distribution of your performance. ⁓ and you can exactly as you said, you can use AI to generate test cases, to generate corner cases. but you do have to regularly run these tests and the LLMs do change their behavior do change. So now you have the cost of regression testing every time the foundational models change. it, there is more cost to that building that infrastructure for achieving similar levels of reliability. And I think that cost is balanced by what I said before, which is you're leveraging that AI for being intelligent about managing all these specifications you would otherwise have to do that you don't have to do. And you can't underestimate that. I mean, we do things now we would not even attempt to do before because of how much time that saves us. 100%. So curious in coming from Morgan Stanley and City Group and Bridgewater. The compensation model in many of these companies is based on how many people work for you in many cases, majority. But the future IT group, it's almost the inverse is most important. It's how do we actually do more with less? Well, first of all, we're going to be, well, probably not us, because we'll be too old. The next generation will be flexing themselves at the local bar on a Friday saying, I've got 16 agents that I'm responsible for. Your Jeff Bot will still be around though. Yeah, it will be. will be. The point is, and I teach at Columbia and we talk a lot about this as well, like, you know, the flex will be how many agents you have, whether you have the premier agent that you're managing. I also think that... Seriously, so I haven't heard this before. So that's good. So we're... Three of us are sitting around a bar, kids are us. Yeah, you'll be like, like I've got, you know, they just promoted me to run the most important HR agent at the firm. And that's going to be a big responsibility because, and by the way, because that agent's going to do stupid stuff some days, it's not going to, it's going to go down. It's not going to communicate. It's going to need to be improved. Someone's going to come along. You know, there's a, by the way, I'm not saying this is going to happen. This is actually not my prediction, but the bull case on employment, is that in a world in which every business has 100 use cases they want to get done this week or this year, and they can do 18 because their budget is constrained, they're not only going to do those 82, they're going to come up with to day's point another 100 things that they're not even doing today. So instead of having 20 use cases today, we may have 2000. And when we have 2000 use cases each, each business, guess what, we're gonna create a complete and utter mess for ourselves. And we're gonna need more people to manage this. And by the way, I'm even just talking the four walls of your organization with the right controls. What happens when they start talking to each other, when Bridgewater starts talking to Citi, starting talking Morgan Stanley? Now we're moving it, by the way, I don't even wanna go here, because it's a level of complexity and sophistication that we're not ready for right now. But that is the argument that the cost of tech is going closer to zero, not zero to be clear. The cost to create code. It's going to zero. So therefore we're going to create more stuff. We're not going always build it the right way. And it's going to create a whole set of messes, which is why, again, I think this architecture is so important. You have to lay out. I just have to butt in there because you're hitting on so many, I think, critical things. So people say, ⁓ my gosh, what are we going to do with AI? What are we going to do with AI? It's going to take all our jobs. I said, you know what you're to do with AI? You're going to end up doing more. And exactly, is a Jayvon's paradox, right? I mean, as you go, as these, the underlying facility gets cheaper and cheaper, you start to realize, wow, there's more and more I could do that demand actually goes up. You're expecting more. It's already happening. I mean, it's all around me all the time from myself. I expect more from myself. I expect more from the people around me. How come that took so long? Why didn't you use AI? I don't understand. What are you doing? And so the expectations of doing more is going through the roof. As we do more exactly to your point the guardrails the architecture because you're getting more moving parts You're grading greater and greater complexity. You're getting more opacity with regards to how these agents ultimately are reaching the conclusions They're reaching you to have to put in all these guardrails and all this other kind of stuff I mean, I couldn't agree more. I don't we have to be careful to to Think we know what's gonna happen here ⁓ I think there's a lot to be learned around that. It's interesting to think too about trust and responsibility. Why is it that you just said, okay, we're going to hire somebody who's going to control the HR agent and we need a human there. Why do we need a human there? And this is just an interesting question. This is almost a philosophical question about how humans interact with humans. in the end, there's a human we need to hold responsible. Well, I there's a couple of reasons for that. First of all, the tech's not good enough to run without human supervision. think that's your first comment, right? mean, most people don't see that though. Most people are, because they haven't coded before and now they can code. Because you're building, you know, yeah, if you're building your little website, mean, go to my website. built it in 45 minutes. It's great. But God forbid, God forbid I had like, maybe if you have a million listeners, they're going to come in and hit me. mean, the thing's going to fall over. And by the way, it could ask it to order a Tripoli for me, my point is, here's the other point is if it goes down, I'm going to sweat. Right. Cause I don't actually know how to fix it. Absent Claude. So number one, these things are not good enough. Number two, until the law changes, I can't sue Claude and the AI bot for not performance. So I think there's a legal issue here too. And you never will. That'll never happen. Right. So one, it can't do it yet. Number two, I can't sue it. ⁓ And the other thing is it only knows what it knows. So in theory, if you can put every single thing in the world, Every single interaction that you and I have had today, over the last six months, and you put into a system, and all the memory that we have, right, the context wasn't that big enough, but let's say you could do that. Like, this is, we're talking, you guys, we're long gone before that happens, right? So I think people need to appreciate how fragile these systems can be, how they do stupid, I mean, I'm sure you do it all the time, you're like, I'll say why, I've told you four times, I want the button. right there and it doesn't give me the button. And then I have to like be like, you are not listening to me, Claude, or OpenAI, why are you not doing that? Like, ⁓ I'm really, and he keeps apologizing and he keeps not doing it. Like this is the world that we live in right now. So I want to be clear. We can go on and on with stories like that. mean, what, you know, one AI, I won't mention its name, essentially lied to me intentionally. Ultimately explained why it lied to me I had it write me a three-page confession because it wasted like 12 hours of my time Lying to me about what it was doing. But yeah, I mean look I mean these things happen and so you do have the issue of of in spite of that the acceleration amplification they give you the the ability to solve problems you would have never even tried before is You know is is remarkable, but in the end To your point, like we end up saying they're not perfect. We can't rely on them 100%. We have to hold a human accountable to know what the hell is going on and how to control them and how to use them effectively. And, but there's another point that you made that I think is a key one, which is at least part of the reason why this sort of thing happens, not for all the reasons, but why you run into these problems with these unexpected things. is because most problem definitions are underspecified. And the power of AI, which is to fill in all the assumptions and all the details so that it always gives you an answer, is to fill in that underspecification. And that's such an important thing to realize as a business user or as an engineer using these systems, is do I realize that that underspecification is on me. Yeah, I mean, you're getting to one of the most important responsibilities, right? And I, you know, I teach a class on prompting. Do you really? Columbia? this? Well, I mean, it's part of the course, but when I do senior executives, the first 30 minutes, we talk about how to write a prompt. Who am I? Who's my audience? What's the task? What's the format and tone? What do you not want it to do? and then upload the supporting documentations. then we actually, because by the way, no disrespect to some of my peers, some of them are using AI to find a good dim someplace in lower Manhattan, right? They are not like, this conversation we're having is not the conversation that a lot of people are having in this world. So to be able to kind of give them the skill, and what I say is that you have to practice that for several months. I mean, when vibe coding happened, right, I was on it right away. Anyway, I'm not a great vibe coder today either, but I am a hundred times better at five coding than I was four months ago. And the reason I'm a hundred times better is I do it a lot. And, you know, I say that part of your skill as a human is to be able to talk to the machine and to get it to do what you want. And that is a skill that you have to develop. You have to learn. You it's like, it's like, know, your kids are all different and they've got different temperaments. So are models, different techniques, you know, which can rely on. And that does not come with a six hour course that comes with practice and experience and the only way to learn it is by doing it. You know, it's interesting. I'm on a board of a nonprofit and the auditor came to see us to tell us in the last meeting, you know, a lot of boards are being asked to govern AI for the company responsible. Things will stop if that's the case. If suddenly there's a legal responsibility as to what prompts are used and how it's done correctly, like that changes the world. Well, this is I make a living. I do a lot of board work. ⁓ And I have like a framework for this. And the point I make is that you're not, your job is to ask the questions. Right? And make sure those questions are smart. ⁓ You don't have to have all the answers. Right. But you should be asking the questions like, what is your tiered architecture look like? What are your data quality controls? What is your governance mechanism? What guardrails do you have in place? What is your training program? That would be a great day when the board start asking all those questions. That would be good. We could do a lightning round and I was just thinking I could also go to Jeff Bott and just ask Jeff Bott these questions to see if it would answer it the same way. ⁓ know, Dave, you and I, let's alternate the questions. I'll jump in first. So, legacy technology 2030, does it exist or is it finally gone? ⁓ it'll be here. It'll be here. ⁓ And part of the reason it'll be here is that in many, even with AI, the cost of retrofitting is more expensive than the cost of leaving it alone. And unless you're digging underneath the covers, sometimes it's best to leave things be, particularly when you've got, you you've got a hundred projects tasting, you know, a million dollars, right? And you can only do four. So yeah, there'll be legacy code for a long time. We'll be long gone. Favorite podcast besides this one? Well, from an AI perspective, like the hard fork guys. And I have to say The Daily. The Daily's probably the best in terms of general. How about favorite book you'd recommend for them? Oh, I, um... Could be fiction, non-fiction. I would say Daniel Goleman, Emotional Intelligence. read it in the late 90s and probably it's probably, you know... By the way, that's more important than all the AI stuff we're talking about right now. you were talking about the predictive analytics, that's best action. I remember reading, is predictable rationality, I think it is, is one of the books. It's one of the common interests I think we all have is ⁓ understanding how predictable and irrational we all are as humans. How crazy Well, the point is, people ask me, like, what should they study? And I have no idea. But I'll tell you what, one, learn to use AI effectively, and two, Learn to be more empathetic, more thoughtful, a better listener, more insightful, right? Because you got to compete with machines not competing. And don't piss off the robots. Understanding your own biases. I think this is something we're going to face more and more is the, the, ⁓ are, you know, we, talk about AI biases, but as humans, have a huge number of ⁓ very serious cognitive biases and understanding them and actually how they relate to ⁓ AI biases to get the best of both worlds if you will ⁓ or to get a ⁓ Positive complementarity is interesting is an interesting question We'll do one more Dave take pick the last one. Let's do it last show you've been watched binge-watched I you know, I rewatched the Queen's Gambit with my 18 year old son who's learning to play play chess and ⁓ Yeah, it's great. Did you, so I love the show. I love the show. The drug use is interesting. The drug, I always wonder, is it a natural ability or is it because of that? I don't know the answer. Well, I think the answer was it was natural because she got off the drugs at the end and won. ⁓ that's true. Okay, I like it. She was able to recreate, we just watched this last night, so it's very close. But remember, she was able to recreate the picture on the roof without being high. My morning routine is I go through a series of puzzles every morning to kind of wake my brain up and so I do go to Wordle to connections and then chess and it's a chess one of the solve puzzles games And it's it's a good exercise. It's a good it's very different Well, my son was telling me because he's really into chess now that ⁓ these these grandmasters are burning You know, their heart rates are going to 140 150 when they're actually playing chess So they're burning a lot of calories as well, which I thought was interesting. By way, no, it's true. That's my 18-year-old. That's not validated information. That was good. Awesome. Well, Jeff, this is amazing to have you on. what you shared is incredible for the audience. Can I just say one other thing? Of course you could. So I did this with my class the other day. We're going to play a game. Yes. Uh-oh. Now I'm nervous. My heartbeat's going to 140. OK, you have to tell me. I'm going to raise some concerns that were raised in the public ecosystem. about a specific technology, and you have to tell me what that technology is. Okay. You guys ready? Yep. So the first concern was that there would be a loss of control of information. Organizations that used to be gatekeepers worried that they would bout to lose power. The second is a rapid spread of harmful or disruptive content. People feared that tech would accelerate the distribution of extreme views, personal attacks, and material considered offensive or destabilizing. Third, information overload. and lower quality thinking. Commentators warned the flood of new content would overwhelm audiences. Four, decline in skills and expertise. What used to require training, time, intentional effort could now be produced cheaply and quickly. Five, job disruption. Entire professions built around the old system fell threatened. And then finally, increased push for regulation. Response institutions called for licensing, approval system restrictions, and penalties. The printing press. You win. You win. The point I try to make is guys like we want to make we want to act like this is that was good, by the way. This is so different. and by the way, it is different. ⁓ want to be clear, large language models are different. ⁓ But is this is not the first time in our world's history that we have gone through a transformation. ⁓ And argue that somehow ⁓ in of giving people the ability to read the world move forward. And I am optimistic that humanity will be just fine. It will look different, but I think we're going to be okay. Jeff, I am sure there are listeners who are saying they want your services. They want you to come help advise them. What's the best way to find them? Sure. You can reach me at ⁓ jeffm at Macmillanai.com. And if you want to just kick the tires ⁓ my offerings, go to Macmillanai.com and you can talk to Jeff Bott. That's awesome. Jeff, thanks again for joining us. Thanks, Dave. Thanks, audience, for tuning in and listening. And for those who are interested, we are doing our annual Unquark Create event, bringing together top leaders across multiple industries to basically discuss AI, future of enterprise tech. For more information, you can find out at unquark.com slash create dash 2026. And some of the most incredible speakers will be joining us as well. Look forward to see you there. Thank you. Thank you to our listeners for tuning in. Make sure to like and subscribe. We hope to see you again in our next episode. Please send ideas for topics or guests that you'd like to hear from. We look forward to seeing you then. Thanks.