Latest / The Adults In The AI Room / Earley AI Podcast Episode 71: The Real Work of Operationalizing AI

Transcript

Earley AI Podcast Episode 71: The Real Work of Operationalizing AI · The Adults In The AI Room

• 0:01
  SPEAKER_00: This is the Early AI Podcast hosted by Steph Early.
• 0:04
  SPEAKER_00: Join us as we delve deep into the passions, expertise, and experiences of thought leaders and practitioners to talk about what's possible with artificial intelligence.
• 0:14
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science, your digital transformation journey with design and deployment of innovative technology solutions, as well as Fector.
• 0:26
  SPEAKER_00: Now enjoy the show.
• 0:28
  SPEAKER_01: Welcome to the Early AI Podcast.
• 0:31
  SPEAKER_01: I'm your host, Seth Early, and today we are going to be talking about some of the intersection of architecture, operations, AI enablement, coding, AI assisting with coding and building applications.
• 0:45
  SPEAKER_01: And you know, in this show, we try to get beyond buzzwords.
• 0:48
  SPEAKER_01: We try to get into what's really practical, what's what can be applied, how it works to uh provide business value, what organizations need to do.
• 0:57
  SPEAKER_01: And so today I'm really excited to have Charlie Betts, who's a principal analyst at Forster Research, and he focuses on digital operating models, enterprise architecture, and the future work.
• 1:09
  SPEAKER_01: Charlie brings a systemic or a systems thinking approach to the way digital initiatives are planned or governed and scaled, and his insights are very relevant to leaders trying to make uh AI real in complex environments.
• 1:23
  SPEAKER_01: Welcome to the show, Charlie.
• 1:25
  SPEAKER_01: Nice to have you.
• 1:26
  SPEAKER_01: Great to be here, Seth.
• 1:28
  SPEAKER_01: We were just you just uh I was just asking you about your weekend, and uh you had said uh you were you were wrestling with with Claude.
• 1:35
  SPEAKER_01: Tell me, tell me what your your wrestling was.
• 1:38
  SPEAKER_01: What were you trying to do and what was uh frustrating?
• 1:40
  SPEAKER_01: Again, we talked last time in our prep call about how this is such a great force multiplier, but sometimes when you get all these additional options, you know, you have to sort through them, you have to figure out what's right, you have to fine-tune, takes a lot of work, and and then you run into other issues and and uh challenges as you're trying to make these things uh operational.
• 2:00
  SPEAKER_01: So, what was your particular challenge this weekend?
• 2:04
  SPEAKER_03: Sure.
• 2:04
  SPEAKER_03: Well, just so that the readers, viewers have full context, Claude is the branded name of the major SaaS uh um service or a major SaaS service from Anthropic.
• 2:19
  SPEAKER_03: They they just as OpenAI names theirs Chat GPT, Anthropic names theirs Claude.
• 2:25
  SPEAKER_03: And Claude has a variant called Claude Code, which you run at the command line.
• 2:32
  SPEAKER_03: And Claude essentially serves as a junior developer, I guess is probably the best term for it.
• 2:41
  SPEAKER_03: And you can instruct Claude to not just auto-complete code that you're writing, you can tell Claude to write whole modules.
• 2:50
  SPEAKER_03: You can have it stand up, you know, minimum viable products and scaffolding.
• 2:55
  SPEAKER_03: You know, it's happy to, you know, stand up a database and uh middle tier and and uh a front end.
• 3:02
  SPEAKER_03: I've had it write React, uh, you know, I've had it write uh, you know, Python-based front ends, uh, I've had it write Jupyter Notebooks, uh it, you know, it it certainly is very adept at managing graphs.
• 3:17
  SPEAKER_03: Uh it will via MCP, Model Context Protocol, it will talk directly to Neo4J, and I have used that a lot.
• 3:27
  SPEAKER_03: And um it will you know do a variety of of very useful things, you know, not not just you know what we started to you know hear about, oh, you know, developers you know can start to write code and then you know Copilot will will suggest what the final you know line of that code might be.
• 3:46
  SPEAKER_03: We're way, way beyond that.
• 3:47
  SPEAKER_03: And so I started to uh work with uh Claude in building an entire system, and it's definitely a big force multiplier.
• 3:58
  SPEAKER_03: Uh, you know, we've I've gotten much further than I ever could have on my own.
• 4:02
  SPEAKER_03: Or Claude has written something like 60,000 lines of production code.
• 4:06
  SPEAKER_03: Uh, there's more like 600,000 lines currently in the working directory, but we as we containerize and you know include stuff intentionally into the containers that actually run, it it's uh at about 60,000 lines, and it's helped me ingest 15,000 nodes and 45,000 edges into a graph that is starting to get pretty far beyond just proof of concept that's becoming useful to me in my work as an analyst.
• 4:35
  SPEAKER_03: And this has all been quite an enlightening journey, uh, but it has its frustrations.
• 4:42
  SPEAKER_03: So I suppose saying to you, I spent the weekend wrestling Claude, or maybe Claude wrangling, whispering, there it is, the the Claude risk, the Claude Whisperer, like that old movie, The Horse Whisperer, right?
• 4:53
  SPEAKER_03: Because I think, you know, boys that they say don't anthropomorphize the large language model.
• 5:00
  SPEAKER_03: No, no, bad, terrible, terrible.
• 5:02
  SPEAKER_03: It's just algorithms.
• 5:05
  SPEAKER_03: Jeez, these things act like junior programmers.
• 5:08
  SPEAKER_03: Unbelievable.
• 5:09
  SPEAKER_03: It is just, I mean all the behaviors, all the behaviors of a contract maintenance programmer or a junior programmer.
• 5:19
  SPEAKER_03: They patch, they post process, they forget which environment they're in.
• 5:25
  SPEAKER_03: Oh my god, you know, they will blow away the database if you aren't careful.
• 5:32
  SPEAKER_03: I mean, there's stories circulating on the internet.
• 5:35
  SPEAKER_03: I've been quite cautious.
• 5:36
  SPEAKER_03: I back up my database.
• 5:38
  SPEAKER_03: I have tested backup and restore scripts, everything gets committed to GitHub, and there are still, you know, I mean, when you take proper precautions, it's still, you know, three steps forward, one step back, and many days.
• 5:51
  SPEAKER_03: Yeah.
• 5:51
  SPEAKER_03: This weekend was more like a a lot of one step back.
• 5:55
  SPEAKER_01: So take take take it up a level, uh, which when you were first uh talking, I know that uh it's possible for people in our industry to speak sentences and paragraphs entirely in accrates.
• 6:09
  SPEAKER_01: I'm sorry.
• 6:11
  SPEAKER_01: No, that's okay.
• 6:12
  SPEAKER_01: No, that's okay.
• 6:12
  SPEAKER_01: You know, it's funny, like you were saying what Claude was, but then you went into a lot of things, you know, that people may not be as familiar with in terms of from the business side.
• 6:20
  SPEAKER_01: But why don't you abstract it a layer and just say, what are the different components and things that you're working with?
• 6:26
  SPEAKER_01: And then what are you trying to accomplish?
• 6:28
  SPEAKER_01: And I know you can't get into too much field because there's proprietary stuff, you're right, but you're also not a professional developer or coder.
• 6:35
  SPEAKER_01: Now, you have uh uh an undergraduate degree in liberal arts, as I recall.
• 6:40
  SPEAKER_01: I do, and now you're doing this kind of work.
• 6:43
  SPEAKER_03: Now, is that because of years and years of diving into self-educating, or you know, how much of this well uh it's a little, I mean, I have a political science degree as an undergrad, but then I have a master's in software engineering.
• 6:57
  SPEAKER_01: Oh, I didn't realize that.
• 6:58
  SPEAKER_03: And I was at the different story.
• 7:01
  SPEAKER_03: Yeah, no, and I was an employee at the University of Minnesota, and I could take courses for free.
• 7:06
  SPEAKER_03: So I took a number of the introduction to computer science courses.
• 7:09
  SPEAKER_03: I've had intro to C, I've had algorithms and data structures and uh C.
• 7:14
  SPEAKER_03: You're not you're not a mere mortal.
• 7:16
  SPEAKER_03: No, so I apologize.
• 7:18
  SPEAKER_03: I'm a little bit of a ringer, but I did take the approach, you know, it's not exactly vibe coding, but I did take an approach of saying I don't want to really particularly look at the code.
• 7:28
  SPEAKER_03: I just want to see how far I get if I don't look at the code.
• 7:32
  SPEAKER_03: Can I read the code?
• 7:33
  SPEAKER_03: Absolutely, I can read the code, but I don't want to any more than if I, as a business owner, were to hire a developer and have them build a system for me.
• 7:44
  SPEAKER_03: I don't want to read the code in that case either.
• 7:47
  SPEAKER_01: Exactly.
• 7:47
  SPEAKER_03: You know, I know there's a lot of purists out there who are like, well, somebody should be reading the code, and you know, don't you care about the craft?
• 7:53
  SPEAKER_03: And no, I really don't care about the craft, I care about the results, right?
• 7:57
  SPEAKER_03: You know, I'm sorry to be so blunt.
• 7:59
  SPEAKER_03: No, not at all.
• 8:00
  SPEAKER_01: And then I I think that's the attitude people have to have.
• 8:03
  SPEAKER_01: You know, too many times we get caught up in the elegance of our code, the elegance of our our technology and our solution, and and you know, the stuff that's most important can be very elegant and not be aligned with the business need.
• 8:16
  SPEAKER_01: So you're saying I don't really care about the elegance behind this.
• 8:19
  SPEAKER_01: The passage yeah, just go ahead.
• 8:22
  SPEAKER_03: As long as it works, and there are a number of things I put in, but you asked me to take it up a level.
• 8:26
  SPEAKER_03: So let me I pride myself on being able to speak at multiple levels, and I know I started off a little technical there, so let me reset.
• 8:32
  SPEAKER_03: They set you up there.
• 8:34
  SPEAKER_03: Um, okay, so if I was explaining this to um a non-technical audience, I would exactly do now.
• 8:43
  SPEAKER_03: As an industry analyst, my job is essentially to provide guidance on where the technology markets are going.
• 8:51
  SPEAKER_03: And on a non-technical level, I compare it when I had to explain my job to my father, I simply said, I do consumer reports for the chief information officer, right?
• 9:04
  SPEAKER_03: That's that's what we do in a nutshell.
• 9:06
  SPEAKER_03: We're not consultants.
• 9:07
  SPEAKER_03: People say, Oh, they'll bump us in together, they'll say, Oh, Forrester and Deloitte.
• 9:12
  SPEAKER_03: One of these things is not like the other.
• 9:14
  SPEAKER_03: Five-year-old Sesame Street, right?
• 9:16
  SPEAKER_03: Just to remember that song.
• 9:17
  SPEAKER_03: I, as an industry analyst, I have certain needs.
• 9:22
  SPEAKER_03: I need to track markets, I need to understand all the companies in a given market.
• 9:28
  SPEAKER_03: And I cover a specific market, and I won't go down the rabbit hole of the market I I cover for right now at least.
• 9:35
  SPEAKER_03: But I cover a market that probably is$250 billion a year of the total five trillion dollars in global IT spend.
• 9:44
  SPEAKER_01: And just a quick thumbnail of just to find what that market is versus the overall spend.
• 9:49
  SPEAKER_03: Sure.
• 9:50
  SPEAKER_03: The global economy is about$50 trillion a year.
• 9:53
  SPEAKER_03: That's all the economic activity in the world.
• 9:56
  SPEAKER_03: So hopefully all the MBAs and finance CFOs are with me.
• 10:02
  SPEAKER_03: 10% of that by a number of decent sources, including Forrester,$55 trillion a year, is spent on Amazon, Oracle, Microsoft, SAP, Deloitte's IT offerings, McKinsey's IT-related offerings, and it just rolls on up to roughly NVIDIA and Intel and AMD and chip manufacturers.
• 10:30
  SPEAKER_03: Now, this is not technology in the broad sense, because that would include biotech and electrical tech and mechanical tech.
• 10:40
  SPEAKER_03: This is info tech.
• 10:42
  SPEAKER_03: It starts with the silicon, goes up from there in terms of all the management and solutions and how do you solve a problem with computers.
• 10:52
  SPEAKER_03: Five trillion dollars a year.
• 10:55
  SPEAKER_03: With me so far?
• 10:56
  SPEAKER_03: Non-techn, I mean financially technical, but yeah.
• 10:59
  SPEAKER_03: And then 250 billion of that, so roughly five percent, is the market I cover.
• 11:08
  SPEAKER_03: And without going into a lot of detail, that segment of the market is characterized by vendors like ServiceNow and Atlassian and Dynatrace and Datadog.
• 11:21
  SPEAKER_03: And now we're getting into domains that probably are unfamiliar to a lot of people, although many people have heard of ServiceNow by now.
• 11:27
  SPEAKER_03: Right.
• 11:28
  SPEAKER_01: And and how are you carried?
• 11:29
  SPEAKER_01: Is that ITSM plus plus.
• 11:33
  SPEAKER_01: Okay.
• 11:34
  SPEAKER_01: Um ITSM uh yeah is IT service management, and then what is the plus plus?
• 11:39
  SPEAKER_01: So I just want to get a quick thumb, but what is it?
• 11:42
  SPEAKER_03: AI AI ops, okay, portfolio management.
• 11:45
  SPEAKER_03: Yep, and basically what we would call the business of IT or the IT control plane.
• 11:51
  SPEAKER_03: Gotcha.
• 11:52
  SPEAKER_03: Okay, is that five percent is extremely, extremely leveraged.
• 11:56
  SPEAKER_03: Right.
• 12:12
  SPEAKER_03: Sure.
• 12:12
  SPEAKER_01: Okay, right.
• 12:13
  SPEAKER_01: So that's that's kind of understanding your your world and managing it and operationalizing it.
• 12:19
  SPEAKER_03: And that is my business.
• 12:20
  SPEAKER_01: And keeping the lights on, be a trusted advisor and a global authority on that problem, and uh and optimizing, and so the whole idea of using these types of accelerators is right in your wheelhouse because this is directly impacting the ability for organizations to operate more efficiently in that space.
• 12:40
  SPEAKER_01: So this is a critical piece.
• 12:42
  SPEAKER_01: So tell me what what does that mean for that industry for that$250 billion uh a year segment?
• 12:51
  SPEAKER_01: What does it mean?
• 12:53
  SPEAKER_03: Let me and let me just also clarify am I started with my personal experience.
• 12:57
  SPEAKER_03: So there's almost like there's two sides of the coin here.
• 13:00
  SPEAKER_03: In order for me to analyze the impact of AI on the 250 billion and then also the broader 5 trillion, and we've got many analysts who are looking at the impact on the broader 5 trillion, but I own the impact on the 250 billion with some with a number of other analysts, but it's a much broader problem.
• 13:20
  SPEAKER_03: That's a big problem.
• 13:21
  SPEAKER_03: And so I'm using AI to help write tools so that I can actually analyze this problem.
• 13:28
  SPEAKER_03: And I can talk about both of these.
• 13:30
  SPEAKER_03: Now, if we talk to answer your question, if we talk specifically about the impact of AI on the IT management infrastructure, I have an architecture or a model that focuses on seven domains, and we see AI influencing the development of software, certainly.
• 13:57
  SPEAKER_03: We see AI influencing the end user experience.
• 14:01
  SPEAKER_03: We see AI influencing our financial understandings of information technology.
• 14:07
  SPEAKER_03: The information technology at a large uh structurally significant bank, for example, is eight to ten billion dollars a year.
• 14:14
  SPEAKER_03: That kind of money would, if you spun it out as its own business, would be in the Fortune 500 itself.
• 14:20
  SPEAKER_03: Right, right, right.
• 14:21
  SPEAKER_03: There is a core of information that always comes up.
• 14:29
  SPEAKER_03: And I call I call it the core, and it includes concepts like business application or service and higher order concepts like capabilities and value streams, but this is essentially at the heart of it, the architecture heart.
• 14:44
  SPEAKER_03: AIOps is a massive domain, a lot of money.
• 14:47
  SPEAKER_03: Vendors like Dynatrace and Datadog, I mean, right there, you know,$2 billion each a year, I think, roughly.
• 14:54
  SPEAKER_03: Then we have the classic information management space, which includes uh vendors like Informatica that just got bought by Salesforce and Data.world that just got bought by ServiceNow.
• 15:08
  SPEAKER_03: But this is your data cataloging and data governance, data quality.
• 15:13
  SPEAKER_03: And then finally, we have risk and security, which is absolutely a control plane function.
• 15:19
  SPEAKER_03: AI is impacting all seven in specific ways.
• 15:22
  SPEAKER_03: I'll pause.
• 15:23
  SPEAKER_01: Right.
• 15:24
  SPEAKER_01: So, what's kind of a summary takeaway for executives when you look at the impact of all those seven different domains within that uh$250 billion segment?
• 15:35
  SPEAKER_01: Uh, what is what is kind of the takeaway?
• 15:37
  SPEAKER_01: You know, we started off talking about the fact that this coding uh can be accelerated and that you're building applications that are very powerful, very sophisticated, there's still nuances there.
• 15:48
  SPEAKER_01: But what's the what's the overarching lesson here?
• 15:51
  SPEAKER_01: Because I still see large organizations struggling with how to align and coordinate and optimize how they're using AI in a way that is producing the most value.
• 16:02
  SPEAKER_01: So this is kind of under the covers a little bit, but uh, what's the takeaway?
• 16:07
  SPEAKER_01: What do you what what's your message for executives to say, here's what you need to think about to embed AI in all of your operations because that's what's happening.
• 16:18
  SPEAKER_01: And if they're not doing it, their vendors are certainly doing it.
• 16:21
  SPEAKER_01: But what do they need to care about versus uh what they're purchasing off the shelf versus building?
• 16:28
  SPEAKER_01: I know it's a broad question, but can you provide some high-level uh guidance?
• 16:34
  SPEAKER_03: Yeah, absolutely.
• 16:35
  SPEAKER_03: There's only one point.
• 16:36
  SPEAKER_03: I would say look for the feedback loop.
• 16:40
  SPEAKER_03: Okay, yeah.
• 16:41
  SPEAKER_01: Um what's the impact?
• 16:42
  SPEAKER_01: How is it being measured?
• 16:44
  SPEAKER_03: And how are and because what's what AI is enabling is essentially a faster feedback loop than we've ever had before in industry.
• 16:53
  SPEAKER_03: And this is where the old becomes new.
• 16:55
  SPEAKER_03: Yeah.
• 16:56
  SPEAKER_03: My view on generative AI is it is fine, is it is enabling these kinds of continuous learning loops at the enterprise level.
• 17:06
  SPEAKER_03: Yep.
• 17:06
  SPEAKER_03: And that is the fundamental economic promise of AI to the enterprise.
• 17:11
  SPEAKER_03: It's not about chatbots displacing service desk operators, it's way more profound than that.
• 17:17
  SPEAKER_01: And it's not about just asking the chatbot the question and having it give you a response based on its worldview, which is a model that has disconnected from your systems and processes.
• 17:26
  SPEAKER_01: It's about using it to operate on your enterprise ecosystems.
• 17:32
  SPEAKER_01: And I like to say we speed up the information metabolism of the entire organization.
• 17:37
  SPEAKER_01: That means people's inputs get to them faster, more contextualized, getting them what they need when they need it, and then their outputs are produced more quickly for someone else's input.
• 17:48
  SPEAKER_01: And that cycles through every part of the organization because you're speeding that value creation, you're speeding the flows of knowledge and data throughout the that ecosystem and those value chains.
• 18:01
  SPEAKER_01: And that's where we're really and then getting that feedback, getting that continuous improvement.
• 18:06
  SPEAKER_01: The fact that it used to take weeks to cycle through, you know, getting feedback or getting iterations from an agency on marketing campaigns is down to, you know, minutes, hours, days.
• 18:19
  SPEAKER_01: You know what I mean?
• 18:19
  SPEAKER_01: It's it's a much, much faster cycle time because you're constantly getting this feedback, you're constantly getting the answers much more quickly, and we're able to process all of that stuff.
• 18:29
  SPEAKER_01: The downside, in my opinion, is that you have so many more opportunities and so many more choices and so many more options that you need to spend time uh trying to figure those out.
• 18:39
  SPEAKER_01: But it is, as you say, it's speeding all of those information flows, and that is really what's critical and what's so interesting.
• 18:46
  SPEAKER_01: Now, when we um we we talked, we prepared, we talked about putting demands on operating models and enterprise infrastructure.
• 18:54
  SPEAKER_01: How are digital operations evolving to accommodate this?
• 18:58
  SPEAKER_01: So again, it's embedded in all of these different things, but what's happening within digital operations and IT service operations and ops and machine learning ops, what's happening to accommodate uh all of these changes?
• 19:16
  SPEAKER_03: It's it's variegated.
• 19:18
  SPEAKER_03: It's it's highly various.
• 19:20
  SPEAKER_03: May I share a slide?
• 19:22
  SPEAKER_03: Sure, yeah, please.
• 19:23
  SPEAKER_03: See our intentions as human beings, as executives and enterprises, will be increasingly informed by the generative AI infrastructure in the middle.
• 19:34
  SPEAKER_03: And I emphasize uh large language models, agents, graph databases, rag, and vector databases is accelerated by the infrastructure, certainly force multipliers like large language model-based coding.
• 19:51
  SPEAKER_03: Realization of value continues on is you know, while we're delivering transactional value, some of that transactional value may itself incorporate large language models.
• 20:02
  SPEAKER_03: But I think the important point here for this feedback loop is that signal emerges from the value realization, from the actual IT operations at the coalface, and the signal is increasingly processed by agents and curated into the knowledge management infrastructure in the middle, which then feeds the intentions and the cycle starts again.
• 20:26
  SPEAKER_03: And this has always been the ideal of the learning organization.
• 20:32
  SPEAKER_01: The trouble is that years ago, just to give you a couple of explain the slide for people who might just be listening to the podcast, you have a cycle of intend, deliver, realize, signal, and that's kind of a loop, right?
• 20:49
  SPEAKER_01: Yes, yes, thank you.
• 20:50
  SPEAKER_01: And that loop represents what organizations want to do or want to accomplish or what their goal is.
• 20:57
  SPEAKER_01: Um, the delivery of that, so the ability to operate on that.
• 21:02
  SPEAKER_01: And then when you say the realization, what does that mean?
• 21:05
  SPEAKER_03: Well, delivery, I'm if I may, intention is strategy, delivery just very simple.
• 21:10
  SPEAKER_03: Let's just keep it super simple.
• 21:12
  SPEAKER_03: That's project management.
• 21:13
  SPEAKER_03: That's that's actually writing the code.
• 21:16
  SPEAKER_03: It's writing the system.
• 21:18
  SPEAKER_03: Delivery is a bit of an ambiguous word.
• 21:21
  SPEAKER_03: Yeah, realization is operations.
• 21:24
  SPEAKER_03: What's what's the result?
• 21:25
  SPEAKER_03: What how is it impacting?
• 21:26
  SPEAKER_03: Yeah, I mean, you have production code running at scale on a you know it's doing something, it's responding to people's inquiries, it's managing a supply chain, it's helping you hire employees, train them all the time.
• 21:40
  SPEAKER_01: And then it's giving the signals to say what's working, what's not working, how is this operating, what are the metrics, and so on.
• 21:49
  SPEAKER_01: And then that goes back to feedback to intent to change the strategy.
• 21:53
  SPEAKER_01: It's really strategy, uh, execution operation operation operationalization, I can say that word.
• 22:00
  SPEAKER_01: Uh feedback and then back into the middle of it are all the things that enable this.
• 22:07
  SPEAKER_01: So you talk about rag, right?
• 22:09
  SPEAKER_01: You know, you're governed ontology.
• 22:10
  SPEAKER_01: I love that you have a box for ontology.
• 22:13
  SPEAKER_01: I love that you have a you know a uh that rag.
• 22:16
  SPEAKER_01: I love that you have the graph database.
• 22:18
  SPEAKER_01: Um, and what is the attention uh Q QK uh QK that's the large language model?
• 22:25
  SPEAKER_03: I've been working with uh just how do you rep how do you graphically represent as an architect the large language model?
• 22:32
  SPEAKER_03: And uh I've settled on a cube.
• 22:34
  SPEAKER_03: Okay, uh core um a very a very brief representation of the core algorithm for machine or uh for large language model transformers, I believe.
• 22:44
  SPEAKER_01: I have some ideas for some graphics to represent uh n dimensions, multi-dimensions.
• 22:50
  SPEAKER_01: I have not realized them yet, but when you can't, it's hard to think beyond four dimensions, but you can start to add more attributes and dimensions.
• 22:58
  SPEAKER_01: So and you know what's so interesting about graph databases is that they do in vector spaces, is they they take all of the different characterizations of a piece of information and they apply attributes, and it can be tens of thousands, hundreds of thousands of attributes.
• 23:14
  SPEAKER_01: And then you're saying we're gonna look for something that has similar attributes in n dimensions in this vector space, and it's very difficult to kind of get your wrap your head around, right?
• 23:25
  SPEAKER_01: But that's kind of how that works.
• 23:27
  SPEAKER_01: But anyway, let's keep going.
• 23:28
  SPEAKER_01: So you're gonna show another one.
• 23:30
  SPEAKER_01: I just want to explain that one.
• 23:31
  SPEAKER_01: Thank you.
• 23:32
  SPEAKER_03: You know, when when we started to think about continuous improvement, it was applied on the shop floor, you know, by the in the 50s or even before.
• 23:42
  SPEAKER_03: I mean, the history of it is sometimes a little obscure, but basically things like Plan Do Study Act, Plan Do Check Act, I mean, they've been around for a long time.
• 23:50
  SPEAKER_03: But the core, the core of of my argument, one of the core premises is that when they we tried to scale them up to the enterprise, they did not scale.
• 24:00
  SPEAKER_03: Right.
• 24:01
  SPEAKER_03: And they did not scale because of the information problem.
• 24:04
  SPEAKER_03: Right.
• 24:05
  SPEAKER_03: And they became bureaucra bureaucratized.
• 24:07
  SPEAKER_03: I mean, you started to have you know departments of continuous improvement, and really, you know, they they they did not fulfill their um their their mission.
• 24:18
  SPEAKER_03: I I saw many failed attempts, and I am not aware of too many successful attempts at the enterprise level, Toyota's culture perhaps being the most notable exception to that.
• 24:30
  SPEAKER_03: Toyota did operationalize continuous learning, but it's it's far from a simple, you know, feedback loop on the shop floor with Toyota.
• 24:38
  SPEAKER_03: It's an entire company built on it.
• 24:40
  SPEAKER_03: So those are the things that then led me back to the feedback loop that I showed you.
• 24:46
  SPEAKER_03: And by the way, the the graphic here that we're showing is just of a large plan to check act gear bogged down and cracking, surrounded by piles and piles of documentation.
• 24:57
  SPEAKER_03: Back to the question you have about well, what's the impact across the IT operational landscape?
• 25:02
  SPEAKER_03: Well, this was the diagram that I was recounting for you.
• 25:05
  SPEAKER_03: And we we talked earlier about uh development and user experience and finance, IT finance and the core, and then AI ops and information management and security and risk.
• 25:17
  SPEAKER_03: Now, and I said you out you asked me, well, what's changing, what's different.
• 25:21
  SPEAKER_03: My response is it's it's complex.
• 25:24
  SPEAKER_03: It's it's very complex.
• 25:25
  SPEAKER_03: You know, you can point to in any one of these domains here, and what we have here for the for the non-visual folks is a there's about uh 50, I think, uh circles of different colors that all represent an information topic or an entity.
• 25:44
  SPEAKER_03: The entities can you know include things like server, asset, alert, service, pipeline, story, defect, epic, risk, exploit, cluster, budget, cost.
• 26:00
  SPEAKER_03: You know, it's it's essentially the entire, it's it's a I think a reasonable ontology of the critical things in IT management.
• 26:09
  SPEAKER_03: And I've been working on this space for a long time now.
• 26:12
  SPEAKER_03: I've been very interested in it, you know, as co-founder of the IT for IT standard and so forth.
• 26:17
  SPEAKER_03: This is still kind of a notional market, as much as it is a rigorous ontology, but it shows, you know, the domain that I'm focusing on.
• 26:26
  SPEAKER_03: And to your answer to your question, there are hundreds, thousands of experiments.
• 26:32
  SPEAKER_03: There are dozens of little startups, there are significant strategic efforts and experiments happening within the big vendors like ServiceNow in Dynatrace.
• 26:43
  SPEAKER_03: You know, you take even something like Dynatrace, and they're building a complete graph infrastructure, and they're focused on the light blue AI ops, you know, the trait tracing from you know, signal up and up into problems.
• 26:57
  SPEAKER_03: Um, and what they're doing may or may not be relevant to finance or security, but they have their own vendors, vendors like Aptio, uh increasingly applying AI to the problem of IT spend, and certainly the cloud cost management vendors and the FinOps vendors doing much the same thing, different algorithms, different data topics, different data sources.
• 27:18
  SPEAKER_03: And so, you know, it's$250 billion worth of market.
• 27:23
  SPEAKER_03: You know, and that's going to attract a lot of experimentation.
• 27:26
  SPEAKER_03: But what I've been telling people, I've been telling, I've been making a number of calls to action.
• 27:34
  SPEAKER_03: Um and, you know, the the uh one of the first calls to action is appointing an architect to manage this.
• 27:43
  SPEAKER_03: If you are operating at scale, just like you might have an architect focused on supply chain.
• 27:49
  SPEAKER_03: If you're a manufacturer, you might take one of your enterprise architects and tell them your focus is supply chain, you're the domain architect for supply chain.
• 27:57
  SPEAKER_03: If you're a bank, you probably have architects focused on payments, and that's all they're gonna do for a good chunk of time.
• 28:05
  SPEAKER_03: And payments architects, when I was in banking, that's all they'd done for quite a while, and they knew everything about big banking payments.
• 28:11
  SPEAKER_03: You need an IT for IT architect or an IT control plane architect, whatever you want to call them.
• 28:18
  SPEAKER_03: You need somebody to plug in and reason about this stuff because it is so highly leveraged.
• 28:25
  SPEAKER_03: The second call to action is govern your data, appoint a data steward, and they might be hand in glove with that architect, but they should come out of your data and information management capability and govern the meaning and do not simply concede to the vendors full control over your graph.
• 28:48
  SPEAKER_03: You own the meaning of what it is you're doing.
• 28:50
  SPEAKER_03: Now it's influenced by industry.
• 28:53
  SPEAKER_03: Certainly, industry, you know, you want to you want to maintain some consistency with industry labels, but at the end of the day, ServiceNow and Atlassian actually might not agree on what certain things should be called, and then you've got a bit of a headache.
• 29:07
  SPEAKER_03: Right.
• 29:08
  SPEAKER_03: And the third call to action is directly to the enterprise architects of the world, always include an improvement loop in your Gen AI architectures.
• 29:18
  SPEAKER_03: Always look for how the loop is closing, because that is the value of generative AI.
• 29:25
  SPEAKER_03: It helps us close feedback loops more quickly and optimize them over time.
• 29:31
  SPEAKER_03: And because Gen AI is stochastic, imprecise, you know, whatever you want to call it, you need that continuous improvement loop because it is not deterministic.
• 29:42
  SPEAKER_03: And how do we build systems in a non-deterministic world out of imperfect components?
• 29:48
  SPEAKER_03: How do we build high-functioning systems?
• 29:51
  SPEAKER_03: We include telemetry, we include control, we include cybernetic loops.
• 29:57
  SPEAKER_03: And this is not a new idea, right?
• 30:00
  SPEAKER_01: Right, right, absolutely.
• 30:02
  SPEAKER_01: No, those are great, those are great calls to action.
• 30:05
  SPEAKER_01: And I think they make they make a lot of sense for organizations when when they're trying to evolve this.
• 30:11
  SPEAKER_01: You know, there is a lot of legacy thinking and a lot of legacy architecture.
• 30:16
  SPEAKER_01: And what's you know, how do organizations overcome that?
• 30:20
  SPEAKER_01: Because I know that uh for one large uh retail uh vendor that we worked with, uh, they had 700 systems that were used to manage that customer and uh vendor experience.
• 30:32
  SPEAKER_01: 700 systems.
• 30:33
  SPEAKER_01: And right how do you rip, you don't rip out 700 systems, right?
• 30:37
  SPEAKER_01: You know, so and yet you know a lot of that stuff is is taking up so much, uh, so many resources to keep the lights on and to you know do patches and just keep it going, but it's there's no room for that real transformation.
• 30:52
  SPEAKER_01: So where where do you see organizations with that those legacy systems and in many cases the legacy mindset uh going?
• 30:59
  SPEAKER_01: Is it just going to be a matter of uh existential reality?
• 31:02
  SPEAKER_01: And you know, they'll either make it or they won't.
• 31:04
  SPEAKER_01: They'll either be uh Kodak or blockbustered or or uh or they're going to um find you know make the harsh uh choices and uh and uh bite the bullet.
• 31:15
  SPEAKER_03: It's like Seth, I'm sorry, it's like you were watching me this weekend.
• 31:21
  SPEAKER_01: Um so the problem when you downloaded that last podcast.
• 31:26
  SPEAKER_03: Right?
• 31:26
  SPEAKER_03: Yeah, yeah.
• 31:27
  SPEAKER_03: No, and this actually ties back to what I kicked off with.
• 31:32
  SPEAKER_03: What was what was one of the things I was wrangling Claude to do this weekend was to build a systems dynamics model for technical debt.
• 31:43
  SPEAKER_03: Wow.
• 31:44
  SPEAKER_03: And this is something I'm very excited about, and this is the first time I've mentioned it in public, other than some casual mentions on LinkedIn, perhaps.
• 31:53
  SPEAKER_03: So the thing is that that IT management is poorly understood.
• 32:00
  SPEAKER_03: Just stating historically, kind of from a clinical quasi-academic perspective, I'm not an academic, but I understand the research process, I understand the peer review process enough to be dangerous.
• 32:14
  SPEAKER_03: And I have a fairly decent grasp of the literature on IT management and technical debt and related topics.
• 32:22
  SPEAKER_03: Most of the industry and even academic concepts out there that purport to provide guidance on IT are static and do not actually look at the feedback loops that are operating.
• 32:36
  SPEAKER_03: Interesting.
• 32:37
  SPEAKER_03: Problem that you characterize, I can't tell from a distance if that's a good thing or a bad thing.
• 32:43
  SPEAKER_03: We know a few things about technical debt, and we know enough about technical debt to actually derive a functioning systems dynamics model in the J.
• 32:52
  SPEAKER_03: Forrester sense.
• 32:53
  SPEAKER_03: We know that statements have been made by Marty Kagan and reinforced by others that companies with a good handle on technical debt work on it every day with a capac with at least 10 to 30 percent of their overall engineering capacity.
• 33:09
  SPEAKER_03: Wow.
• 33:10
  SPEAKER_03: Wow.
• 33:10
  SPEAKER_03: That's actually quantifiable.
• 33:12
  SPEAKER_03: That's a very nice quantifiable assertion.
• 33:15
  SPEAKER_03: And if we, for example, assume for the sake of a model that 20% of engineering app capacity being allocated to technical debt results in a model that operates at equilibrium, that's a powerful constraint.
• 33:29
  SPEAKER_03: A very powerful constraint.
• 33:31
  SPEAKER_03: You know, you don't you wind up with a model that could go anywhere, be anything, do anything, to a model that actually has an access that it has to it must revolve around.
• 33:40
  SPEAKER_01: We also know that when a systems can now when you say working on technical debt, are you saying they're constrained by technical debt and their energy goes into working around and managing, or are you saying that they are working on reducing that technical?
• 33:59
  SPEAKER_03: Both.
• 34:00
  SPEAKER_03: So in the positive sense, in the positive case, and so the the qu I mean, we're gonna, I'm gonna we're starting, you're starting to probe deeply into this complex systems dynamics model, and we have to think very carefully about precisely what we mean in order to quantify it credibly.
• 34:18
  SPEAKER_03: There is operational demand, which is the which I view as a form of bad demand.
• 34:24
  SPEAKER_03: This is the demand for immediate firefighting and fixes and the immediate create, you know, because the systems are degrading and the value must be restored.
• 34:36
  SPEAKER_03: Right.
• 34:36
  SPEAKER_01: Okay.
• 34:37
  SPEAKER_03: And house on fire stuff.
• 34:39
  SPEAKER_03: Yeah, house on fire.
• 34:41
  SPEAKER_03: And the house on fire is not is not the same as what we the house on fire demand is elastic demand.
• 34:47
  SPEAKER_03: Yeah.
• 34:48
  SPEAKER_03: There is also inelastic demand, like the power, the your your monthly, your monthly um, like let's go on-prem, or if you have fixed workloads in Amazon, you know, power licenses, you know, your baseline inelastic floor of what you're absolutely going to spend no matter what every month, right?
• 35:09
  SPEAKER_03: So you have inelastic operational demand, you have elastic operational demand, the elastic operational demand can start to expand, at which point it starts to impact the delivery capacity for a new functionality to respond to market changes.
• 35:25
  SPEAKER_03: Right.
• 35:26
  SPEAKER_03: Now, there are three classes of technical debt that we have identified, and we are stepping intentionally away from Ward, or it's not away, we're expanding very intentionally on Ward Cunningham's initial definition, which was focused on what I would call refactoring debt.
• 35:46
  SPEAKER_03: Refactoring is code that is what may have been fit for purpose but was written hastily or is simply degrading over time, not because code degrades, right?
• 35:57
  SPEAKER_03: Because circumstances change.
• 35:59
  SPEAKER_03: Sure.
• 36:00
  SPEAKER_03: But there are two other classes of technical debt that have that clients come with their hair on fire to us saying we've got horrible technical debt, and you look at the literature and it's like, well, what you are describing isn't recognized in the literature, and I'm tired of telling clients they're wrong, it's not a good look.
• 36:17
  SPEAKER_03: And so the other two classes of technical debt are refreshing, rationalization debt, and refreshing debt.
• 36:25
  SPEAKER_03: Ration refreshing debt is simply is the software up to date or are you stretching it out past end of life?
• 36:33
  SPEAKER_03: Yeah.
• 36:33
  SPEAKER_03: You know, and this is a you know, it's a it's a DK curve.
• 36:37
  SPEAKER_03: You know, software becomes goes through certain, it becomes deprecated, it goes end of life, but you can pay for it, extended support, and then even the extended support dries up and you got to go to some third-party, you know, Joe's garage and you know, software maintenance, you know, to maintain it because you just can't get rid of it.
• 36:56
  SPEAKER_03: And when you hit when you hit um the tail end of of refresh, then you're into the rationalized problem, which is where we see digital transformations failing and really damaging companies and contributing to a lot of problems in the industry.
• 37:11
  SPEAKER_03: This is the problem of you do a digital transformation.
• 37:14
  SPEAKER_03: I talked to a client two weeks ago.
• 37:17
  SPEAKER_03: Said, don't talk to me about digital transformations to reduce technical debt.
• 37:21
  SPEAKER_03: We've done three, and all they've done is add the technical debt because the first one took us into, you know, the first one took us off the mainframe into Power Builder, and we still have Power Builder.
• 37:30
  SPEAKER_03: And then the second one took us off Power Builder into Java and Web Sphere, and we still have Java and Web Sphere.
• 37:37
  SPEAKER_03: And now the third one was all about web native stuff, and so now we've got the whole web native stack and microservices as well, and we still have the mainframe.
• 37:44
  SPEAKER_03: So we have four layers of archaeology.
• 37:46
  SPEAKER_03: We've had three major digital transformations, and our technical debt is, you know, continues is at, you know, we're at technical bankruptcy now.
• 37:56
  SPEAKER_02: Unbelievable.
• 37:57
  SPEAKER_03: And all of these things, these stories I'm telling you, can be captured and modeled quantitatively.
• 38:03
  SPEAKER_03: And ultimately what we see is that there are four or five major vicious cycle technical debt loops, and there's a couple of positive virtuous cycle loops.
• 38:14
  SPEAKER_03: And this is ultimately, I think, you know, how we're going to solve these kinds of problems.
• 38:21
  SPEAKER_03: How we're going to get line of sight there.
• 38:22
  SPEAKER_03: I hope to be a player in this conversation.
• 38:25
  SPEAKER_03: But this is exactly what Claude was so beneficial to me because I've been looking at systems dynamics for years.
• 38:31
  SPEAKER_03: You know, I have Sturman's book, I bought Stella, or I think one of the two years ago.
• 38:36
  SPEAKER_03: I just didn't have the time.
• 38:38
  SPEAKER_03: You know, I mean, life is short.
• 38:40
  SPEAKER_03: I get, you know, day to day.
• 38:42
  SPEAKER_03: Three weeks, two weeks ago, I got up and I said, Hey Claude, could you create a systems dynamics model for me?
• 38:47
  SPEAKER_03: You know, and go off and do that.
• 38:49
  SPEAKER_03: It's like, oh sure.
• 38:50
  SPEAKER_03: And it did.
• 38:53
  SPEAKER_03: You know, and all of a sudden I have power tools at my disposal that I never had before.
• 38:58
  SPEAKER_01: Yeah.
• 38:59
  SPEAKER_01: Wow.
• 39:00
  SPEAKER_01: So when you go back to that company that went through those three digital transformations and each time dug the hole deeper.
• 39:06
  SPEAKER_01: Right.
• 39:07
  SPEAKER_01: What should they have done differently?
• 39:08
  SPEAKER_01: And now they have a mess.
• 39:10
  SPEAKER_01: And now they have the bigger mess, and they have more layers of complexity.
• 39:13
  SPEAKER_01: And as you say, you have to go splunking very deeply into the code in order to fix things at the root cause.
• 39:20
  SPEAKER_01: And then that still impacts their agility.
• 39:23
  SPEAKER_01: They didn't get any, maybe they got some new capabilities, but they overall, I mean, if that maybe that mainframe layer is very, very stable, and those other layers are somewhat stable.
• 39:33
  SPEAKER_01: Where does the where does the evolution and the agility change?
• 39:36
  SPEAKER_01: But what should they have done differently?
• 39:37
  SPEAKER_01: What should they have done differently?
• 39:39
  SPEAKER_01: Where did they mess up?
• 39:39
  SPEAKER_03: Well, I mean, you know, counterfact.
• 39:41
  SPEAKER_03: I mean, none of us knew how to run computers for the first hundred years of computers.
• 39:46
  SPEAKER_03: I think that's kind of the basic, you know, the basic bottom.
• 39:48
  SPEAKER_03: I mean, we have to take a historical perspective.
• 39:50
  SPEAKER_03: We're still in our infancy as an industry.
• 39:53
  SPEAKER_03: If I could wave a wand and if they could travel back in time, that clearly what they need to do is they need to continually pay down and prioritize the refactoring, the refreshing, and the rationalization, which on their face do not have good RO.
• 40:09
  SPEAKER_03: And that's the fundamental problem, is that if you take a green eye shade bean counterpoint of view, you're like, oh, we can sweat this technology a little longer.
• 40:19
  SPEAKER_03: We can defer that.
• 40:20
  SPEAKER_03: Let's focus on real value over and over again.
• 40:24
  SPEAKER_03: And the bottom line is that we see the destructive consequences of those kind of very limited points of view.
• 40:32
  SPEAKER_03: And they are at this point existential crises.
• 40:35
  SPEAKER_03: And at this point, you know, if you're at 60 to 65 tech percent technical debt, you are in technical bankruptcy.
• 40:41
  SPEAKER_03: And I don't have good news for you.
• 40:43
  SPEAKER_03: You know, there's no real easy way out.
• 40:47
  SPEAKER_03: Short, and I I can say what you would need to do.
• 40:50
  SPEAKER_03: First of all, you need visibility, you need to understand the dynamics at play, that would help.
• 40:56
  SPEAKER_03: But companies, companies and products do get to a point where it's just a matter of time before they're displaced.
• 41:03
  SPEAKER_03: You know, and companies wind up in financial bankruptcy all the time.
• 41:07
  SPEAKER_01: Well, I know there's one company that we had as a client that was a financial services company that was 100 years old.
• 41:16
  SPEAKER_01: And uh they had literally uh a thousand different systems that were used because they bought businesses and bought businesses and bought businesses and just never ever you know in never merged them.
• 41:30
  SPEAKER_01: No, there was merged the systems, and so you know, where do you go with that?
• 41:35
  SPEAKER_01: I mean, they had they still had you know, they were trying to manage documentation on these things, and you know, who knows about this stuff?
• 41:42
  SPEAKER_01: Nobody knows.
• 41:43
  SPEAKER_03: Yeah, no, AI will AI will help to a degree.
• 41:47
  SPEAKER_03: It can intervene in legacy code bases, it can document them with far greater precision and patience.
• 41:53
  SPEAKER_03: That's one of the advantages to AI is it's patient, it's relentlessly patient, it will just keep doing work, even if you've got the money, you can't find people to do the work, right?
• 42:03
  SPEAKER_03: You know, they just start phoning it in, they start getting casual, their attention span drops.
• 42:08
  SPEAKER_03: The agents I will just keep hammering away at it until you've got the systems documented.
• 42:14
  SPEAKER_03: So there is some blight at the end of the tunnel there.
• 42:16
  SPEAKER_01: I agree.
• 42:17
  SPEAKER_01: It is still an investment, and it's still hard to get ROI.
• 42:20
  SPEAKER_01: And when people are trading that off with, well, I can make this investment in the sales, you know, or marketing application, or I can and I'm gonna get X return on that, or I can go back and fix stuff that's not very sexy and that's not gonna show ROI.
• 42:35
  SPEAKER_01: You know what's gonna happen every time, and that this that that that is a lack of vision.
• 42:40
  SPEAKER_03: Every every time, except as we it doesn't happen every time, and I'll I'll give a shout out to Marty Kagan, who doesn't know me, but I follow his writing, and he's a great thinker, head of the Silicon Valley product group.
• 42:54
  SPEAKER_03: And it's actually his quote that inspired my dynamics model in some ways.
• 42:59
  SPEAKER_03: He says, you know, over the years it's become clear that the product leading product companies that have a handle on technical debt, and there are some spend 10 to 30 percent.
• 43:12
  SPEAKER_01: Okay, yeah.
• 43:13
  SPEAKER_01: So you so you just have to say this has to be an executive board-level decision.
• 43:18
  SPEAKER_03: It has to be a board-level decision that we are here for the long haul, and we will continually reinvest in our technical platform, in our technical capabilities by paying down the debt, and we understand that it's an egg, you know, that the ongoing temptation because in 10 years you're really not going to be able to do it.
• 43:35
  SPEAKER_03: Well, I mean, if you wanna if you only want to, if you just want to run for five years and then have a success and have an exit, or and this part of it is is that you know, the the incentives for people hey, I come in for two years and I, you know, leave with my bonus, or you know, I make my money and then I take off.
• 43:51
  SPEAKER_01: Go to a new place.
• 43:52
  SPEAKER_03: Um, but this is what the board should be, you know, actually actively managing as a risk.
• 43:56
  SPEAKER_01: That's right.
• 43:57
  SPEAKER_01: They have the longevity.
• 43:58
  SPEAKER_01: You know, we're almost out of Time, but I just wanted to um touch on something very quick briefly, and maybe we can have another podcast about this.
• 44:06
  SPEAKER_01: But sure you talked about swarms of agents working on problems.
• 44:11
  SPEAKER_01: And I just think we should just get into a whole agent discussion on another session.
• 44:17
  SPEAKER_01: But um, what do you have to say about agents?
• 44:19
  SPEAKER_01: Because one of the things you said that was interesting is that there's so many misconceptions about agents, and there's so much advice and bad, you know, uh uh it's just a lot of stuff that's noise and nonsense.
• 44:34
  SPEAKER_03: So the disclaimer is that I I still am learning myself, as we all are, and we have other Forester analysts, very talented ones like Leslie Joseph and and others who are really, really leaning into agentic.
• 44:48
  SPEAKER_03: Yeah.
• 44:49
  SPEAKER_03: But I somebody asks me about agents, and I'm the only person in the room, and Leslie's not there, or whoever.
• 44:56
  SPEAKER_03: There are two things that I do know.
• 44:58
  SPEAKER_03: One is that full agentic autonomy, to me, in the standard enterprise context, is at this juncture absolute non-starter.
• 45:09
  SPEAKER_03: I mean, I've already seen Claude try to wipe out my database, and we know that it has wiped out databases in other situations.
• 45:17
  SPEAKER_03: And when you talk about autonomous, what are we saying?
• 45:20
  SPEAKER_03: That would be bad.
• 45:22
  SPEAKER_03: What are we saying?
• 45:22
  SPEAKER_03: That that the agent would go out to your API catalog and it could introspect into any of those APIs and determine what APIs it wanted to access, and you would give it permissions to do whatever those APIs enabled, and it could just do this on its own without, you know, any kind of I mean, okay.
• 45:42
  SPEAKER_03: Obviously, that sounds ludicrous.
• 45:44
  SPEAKER_03: That's not going to happen anytime soon.
• 45:47
  SPEAKER_03: Right.
• 45:47
  SPEAKER_03: We would need whole infrastructures of control and intent management that are simply not there yet operationally.
• 45:54
  SPEAKER_03: Absolutely.
• 45:55
  SPEAKER_03: Right.
• 45:55
  SPEAKER_03: And the business model and the business impacts, the operating model impacts, which are all concepts that so many of the AI experts out there just find immensely tedious.
• 46:06
  SPEAKER_03: Enterprise operating models.
• 46:08
  SPEAKER_03: Oh, Charlie, gag me now.
• 46:09
  SPEAKER_03: That's so boring.
• 46:10
  SPEAKER_03: You know, agents are cool.
• 46:11
  SPEAKER_03: They're exciting.
• 46:12
  SPEAKER_03: It's like, yeah, right.
• 46:13
  SPEAKER_01: Talk about that.
• 46:14
  SPEAKER_01: Wait a minute.
• 46:15
  SPEAKER_01: Let me check my calendar.
• 46:16
  SPEAKER_01: How about never?
• 46:18
  SPEAKER_03: Right?
• 46:19
  SPEAKER_03: Yeah.
• 46:20
  SPEAKER_03: So I balance that.
• 46:21
  SPEAKER_03: And then and I'm also aware I hear rumblings.
• 46:24
  SPEAKER_03: I don't dig in because time is limited, but I hear rumblings of people who are trying to run these things in well-sandboxed computer science research type labs or enthusiast experimenting.
• 46:36
  SPEAKER_03: And they're like, the agents are completely uncontrollable.
• 46:39
  SPEAKER_03: We don't know what they're doing.
• 46:40
  SPEAKER_03: You know, lie, they lie, they exactly.
• 46:44
  SPEAKER_03: But I want to balance this now with another case that I have experimented with and can work.
• 46:53
  SPEAKER_03: And that's where the swarm, first of all, does not have access to production.
• 46:58
  SPEAKER_03: The swarm is ring fenced off in containers, or you know, has a job to do within that context.
• 47:05
  SPEAKER_03: And within that context, you give it a big job like a system specification.
• 47:10
  SPEAKER_03: And this is what Reuven Cohen has been advocating.
• 47:14
  SPEAKER_03: A guy named Reuven Cohen.
• 47:15
  SPEAKER_03: I knew that name, yeah.
• 47:16
  SPEAKER_03: Yeah.
• 47:16
  SPEAKER_03: Yeah.
• 47:17
  SPEAKER_03: Look him up on LinkedIn.
• 47:18
  SPEAKER_03: And Adrian Cockroft, who is former CTO of Netflix, has been very enthusiastic about Reuven's work, which is why a lot of people in the world are paying attention to Reuven.
• 47:28
  SPEAKER_03: And uh, and for example, it gets back to waterfall development.
• 47:32
  SPEAKER_03: If you can write a 300-page spec, you can give it to one of these agentic swarms, and you can stand up not just one instance of Claude, but you can stand up eight instances of Claude in a swarm-like uh uh choreography with a Queen Bee agent and a testing agent and an architect agent and a coding agent and da-da-da-da-da.
• 47:53
  SPEAKER_03: And they all will do things and they will, in some cases, create real value.
• 47:59
  SPEAKER_03: And people are getting varying results, but there's a lot of people out there saying, yeah, this stuff really works.
• 48:04
  SPEAKER_03: But notice that this swarm has a clear goal, yes, and it is not talking to production systems.
• 48:12
  SPEAKER_01: Yes, it's fenced in.
• 48:14
  SPEAKER_01: It's definitely fenced in.
• 48:15
  SPEAKER_01: I totally agree with you.
• 48:16
  SPEAKER_01: And one of the things we use our agentic approaches, and I think you and I talked about this when I was showing you some pieces of our virtual information architect platform, is that we'll go out when we're doing things like product data remediation, we'll go out to different sources, we'll pull that in, we'll work through things agentically because it is uh going through and accessing other data sources and integrating it and making certain decisions, but it has a very clear set of objectives, has a very clear set of inputs, has a very clear set of outputs, and we're very uh specific about what it can and cannot do, right?
• 48:51
  SPEAKER_01: Right.
• 48:52
  SPEAKER_01: So within the context of that, yes, there's state has autonomy.
• 48:56
  SPEAKER_03: But the guardrails, see, here's what I get troubled with, Seth.
• 48:59
  SPEAKER_03: I'd give Claude guardrails all the time, and Claude loses context and forgets the guardrails.
• 49:06
  SPEAKER_03: I don't trust the agents to remember and keep their guardrails internalized when they're optimizing for an objective, when that's their their function or whatever you would call it, there is a tendency to, you know, tail dropping in the cues or whatever it is.
• 49:23
  SPEAKER_03: There is a tendency for them to drift away from the governance protocol.
• 49:28
  SPEAKER_03: And and at least that's my personal experience.
• 49:30
  SPEAKER_03: Now, could this be resolved over time?
• 49:32
  SPEAKER_03: Yes, but we would need a lot of confidence, a lot of empirical evidence.
• 49:37
  SPEAKER_03: I wouldn't need to see a lot of empirical evidence for anybody saying my agent can be trusted to honor guardrails at all times.
• 49:44
  SPEAKER_03: Right.
• 49:45
  SPEAKER_03: Um I I don't think we're there yet as an industry, and most people aren't even thinking about these operating model impacts.
• 49:52
  SPEAKER_01: Yeah, yeah, there's so much work to be done.
• 49:54
  SPEAKER_01: Charles, my God.
• 49:56
  SPEAKER_01: I got to the first of my 12 questions.
• 49:58
  SPEAKER_01: No, this is a wonderful conversation.
• 50:00
  SPEAKER_01: I really appreciate it.
• 50:01
  SPEAKER_01: And I thank you so much for sharing your perspective.
• 50:04
  SPEAKER_01: It you know, it's clear that getting AI right entails so much, and there's so many capabilities, and it's getting better all the time.
• 50:11
  SPEAKER_01: It's very exciting, it's very scary because you know, organizations don't know what they don't know and they're trying to move forward with this stuff.
• 50:19
  SPEAKER_01: But I say, you know, a bias toward action is important, but the feedback loops are critical.
• 50:24
  SPEAKER_01: And um, for those listening, we'll we'll link to Charles uh work at Forrester uh and other resources in the show notes.
• 50:32
  SPEAKER_01: Um, if you enjoyed the conversation, please subscribe to Early AI Podcasts, wherever you get your podcasts.
• 50:38
  SPEAKER_01: And uh stay tuned for more insights from leaders of the AI and the enterprise.
• 50:43
  SPEAKER_01: Charles, thank you so much for your time today.
• 50:45
  SPEAKER_01: Thank you.
• 50:46
  SPEAKER_01: Take care.
• 50:48
  SPEAKER_00: Thank you for joining us on another deep dive into AI innovation.
• 50:52
  SPEAKER_00: Tune in next time when we introduce another industry expert and discuss how to maximize AI in your world.
• 50:58
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science and Vector.
• 51:04
  SPEAKER_00: To learn more, visit early.com.
• 51:06
  SPEAKER_00: That's E A R L E Y dot com.
• 51:09
  SPEAKER_00: Thank you for tuning in.