Latest / Humanity Working / When AI Goes Rogue: Testing, Personal Agents, and ROI

Transcript

When AI Goes Rogue: Testing, Personal Agents, and ROI · Humanity Working

• 0:15
  SPEAKER_01: Everybody, welcome back to the show.
• 0:17
  SPEAKER_01: Bill and Matt are with me as they regularly are these days.
• 0:21
  SPEAKER_01: How are you doing, Matt?
• 0:22
  SPEAKER_01: Splendid.
• 0:23
  SPEAKER_01: Excellent.
• 0:24
  SPEAKER_01: And Bill, I know it's been a busy week for you, just back from John Hopkins, right?
• 0:29
  SPEAKER_01: How was that?
• 0:30
  SPEAKER_03: Yeah, I'm a little tired, but it was an incredible week, so I'm glad to be back home after a week on the road.
• 0:35
  SPEAKER_01: Yeah, and I know from Talking Offline you spoke quite a bit about AI, so it'll be fascinating to get some of the interesting topics that you brought up there.
• 0:45
  SPEAKER_01: Today we're gonna hone in on three specific things.
• 0:49
  SPEAKER_01: We're gonna take a look at testing and specifically kind of like the way in which AI models are tested, either inside the labs or outside the labs.
• 1:00
  SPEAKER_01: I think those are two subtly interesting and subtly different things.
• 1:03
  SPEAKER_01: Um and what some of the unintended consequences of testing can be.
• 1:08
  SPEAKER_01: We'll also take a look at these personal agents.
• 1:12
  SPEAKER_01: They've been the topic de jour the last two or three weeks.
• 1:15
  SPEAKER_01: Uh I'm sure some of the listeners out there have probably tried them at this point.
• 1:20
  SPEAKER_01: I'm talking about things like Muse from Meta, which interesting kind of marketing campaign around that.
• 1:26
  SPEAKER_01: Dots from OpenAI as well, and there's various other alternatives.
• 1:30
  SPEAKER_01: But we'll have a chat about those, what we think they might mean in a broader sense, and then to finish up today, I want to take a little bit of a look at ROI.
• 1:39
  SPEAKER_01: Now, I'm sure we're gonna have a much deeper uh conversations about ROI over the course of the next few weeks, but we'll kind of kick it off here.
• 1:48
  SPEAKER_01: Alright, let's start then with testing.
• 1:51
  SPEAKER_01: And guys, there was a particular news story that caught my eye, I want to say maybe a couple of days ago, I'm not sure exactly, where I don't know if you saw this, but there was an instance where um I think it was a haiku model that was being tested.
• 2:10
  SPEAKER_01: That's one of the haiku's one of the family of models from Anthropic.
• 2:15
  SPEAKER_01: Anyway, it's being tested, tested on the public internet, and part of the test was that it should go to some public websites and try filling in forms on public websites.
• 2:27
  SPEAKER_01: And rather unfortunately, in doing so, it gave a fake tip for a homicide.
• 2:35
  SPEAKER_01: Um not great, not great news.
• 2:38
  SPEAKER_01: So we'll put a link into that news story to just kind of give you an understanding of the story that we're talking about here.
• 2:45
  SPEAKER_01: But of course, this is one of the interesting things that has been brought up in repeated ways with this idea of taking these models that by definition are probabilistic in nature, but also behave in ways that are not fully deterministic and test them in a relatively open way because they're designed or the hope is that they will be able to behave in these relatively open ways.
• 3:12
  SPEAKER_01: And then there's nothing in the model that morally says, Well, you know what, it probably wouldn't be a great idea to go um give a tip on a homicide.
• 3:22
  SPEAKER_01: So I'm laughing probably shouldn't, I mean it's very distressing.
• 3:25
  SPEAKER_01: But I guess the question I have here is I'll start with you, Bill.
• 3:33
  SPEAKER_01: We have to test these models and we'll just start with like inside the organization.
• 3:37
  SPEAKER_01: This was from Anthropic, right?
• 3:38
  SPEAKER_01: This happened within Anthropic.
• 3:40
  SPEAKER_01: We have to test these models in some way.
• 3:41
  SPEAKER_01: If they're gonna be of general use, we can't be too deterministic in how we test them.
• 3:48
  SPEAKER_01: But a potential result is what we've just seen, and we could see many, many, many more of those.
• 3:53
  SPEAKER_01: So how would you frame the challenges that we face when we're trying to test these models to make sure that they work, but not to cause incidents like this?
• 4:04
  SPEAKER_03: Well, you know, the thing that comes to mind is as you described, kind of this what are the implications of these kind of incidents, right?
• 4:13
  SPEAKER_03: Fake tip, fake murder tip is put in.
• 4:15
  SPEAKER_03: I think I also saw in a similar article that there were 20 visa applications submitted to the State Department using a similar form and fashion.
• 4:24
  SPEAKER_03: When we think about like remediations and testing, I almost feel like we now have enough evidence to say we we have to put on our worst-case scenario hat in a way.
• 4:33
  SPEAKER_03: And I hate that we have to think that way.
• 4:35
  SPEAKER_03: But you know, last night I watched a little bit of the Netflix InstaDoc is what they call it, and it was it was kind of all on this, you know, AI doom type stuff.
• 4:43
  SPEAKER_03: It was only about half an hour, but I think it brings this point home that you're making, which is we have enough scenarios now that have shown us unchecked and untested, that these things sort of take a different kind of behavior.
• 4:58
  SPEAKER_03: And we have to stop and ask ourselves in that in that instance, will it have an adversarial impact on human life or lives?
• 5:07
  SPEAKER_03: In this case, yes.
• 5:09
  SPEAKER_03: And in a way, I think we have to build that into our threshold of testing, which goes back to our discussions in the past, you, me and Matt talking about science fiction scenarios, right?
• 5:17
  SPEAKER_03: Like I hate that we have to sort of think that way, but this false tip put in by Anthropic to the Philadelphia police feels very minority report to me.
• 5:26
  SPEAKER_03: If you've seen that movie, you know, the implications of that.
• 5:29
  SPEAKER_03: And I worry about these things.
• 5:32
  SPEAKER_03: I don't know if there's an easy answer for this other than to say we almost have to think about worst-case scenario when developing testings, you know, testing scripts for this.
• 5:39
  SPEAKER_03: And that puts the accountability back on these companies and the models.
• 5:43
  SPEAKER_01: Yeah.
• 5:44
  SPEAKER_01: Thoughts on this, Matt?
• 5:46
  SPEAKER_02: Yeah, I mean, we almost have to assume something along the lines of this is near impossible, I would think, but is there a global operating set of ethics we could agree on even at a high level to make sure every model has to meet a standard of maybe that is the thing to open source is an open source agreed set of guardrails, at least around ethics in this in this regard.
• 6:15
  SPEAKER_02: You know, can we all agree?
• 6:17
  SPEAKER_02: That's really tough.
• 6:18
  SPEAKER_02: But I think your point though, Bill, brings up like, okay, we've already seen plenty of worst-case examples.
• 6:28
  SPEAKER_02: We don't have to guess that we will encounter them, ones that we can think of and ones that we haven't yet thought of.
• 6:34
  SPEAKER_02: It becomes imperative then to not progress forward until we at least have some way to test for it, find it, and avoid it, learn from it very quickly.
• 6:49
  SPEAKER_02: Because if we're not able to put the genie back in the bottle, we at least need to know things are gonna get broken, how do we fix them very quickly without loss of life, without major impact.
• 7:03
  SPEAKER_02: It's very difficult to to navigate this.
• 7:05
  SPEAKER_02: At least these are the ones we know about so far that we found.
• 7:10
  SPEAKER_01: Yeah, I mean That's the scary part.
• 7:14
  SPEAKER_01: It is it is the scary part.
• 7:15
  SPEAKER_01: And I like the way you sort of you guys kind of tied it together with this sort of broad, broader when AI goes rogue kind of discussion because I think to your point, right?
• 7:28
  SPEAKER_01: Wait, they were a saying off in the startup world where there's one, there's ten, that's kind of referring to customers, you know?
• 7:34
  SPEAKER_01: And when there's ten, there's a hundred.
• 7:36
  SPEAKER_01: But the moment that the hugging face thing happened, that was the first thing that popped in my head was where there's one, there's ten, right?
• 7:43
  SPEAKER_01: And when there's ten, there's a hundred.
• 7:44
  SPEAKER_01: And so I think that's what sort of got us to a starting point.
• 7:50
  SPEAKER_01: But the interesting thing here is that there's there's the testing that happens inside the labs.
• 8:00
  SPEAKER_01: And as part of, you know, part of what they're trying to do is they're trying to create something that is of general use.
• 8:07
  SPEAKER_01: And so to create something that is of general use, very difficult for them to say, okay, we're going to deterministically say, test these, you know, go to these thousand websites and that are, you know, non-typical, or or I should say, where you're not using resources from other parties, just go to these test websites and fill those out and we can check, right?
• 8:39
  SPEAKER_01: If you're doing that, you're not actually demonstrating general use.
• 8:43
  SPEAKER_01: You're just demonstrating that it works with these specific websites.
• 8:46
  SPEAKER_01: So it's almost like the goal of producing something that is of general use and can act on behalf of a human being with minimal direction.
• 8:57
  SPEAKER_01: That goal in itself is a goal that is going to eat up uh resources like a virus from all of the different entities that it is interacting with.
• 9:13
  SPEAKER_01: And that's before it gets into the outside world.
• 9:15
  SPEAKER_03: Right.
• 9:16
  SPEAKER_01: Right?
• 9:16
  SPEAKER_01: I've been building a sort of my own version.
• 9:20
  SPEAKER_01: We'll get onto it when we kind of discuss the muse and all that kind of stuff.
• 9:24
  SPEAKER_01: I've been building my own version of effectively a chief of staff using more open source stuff and open source models, etc.
• 9:33
  SPEAKER_01: And as I do it, I start to realize A, how much a judgment, you know, how much the judgment component there is in a chief of staff and how difficult that is to replicate with AI.
• 9:44
  SPEAKER_01: But B, if you actually want it to save you real time, then you have to get it to start doing things.
• 9:51
  SPEAKER_01: And things like, you know, f filling out forms online and so on and so forth.
• 9:57
  SPEAKER_01: Those things start to happen, those things are consuming resources on the other side.
• 10:02
  SPEAKER_01: And so, unless you're launching what is effectively the world's largest denial of service attack, then you are then there is some responsibility there.
• 10:12
  SPEAKER_01: I don't know who holds that responsibility there, but there is some responsibility there from the people that are providing you with the models and the ability to be able to do this stuff in the first place.
• 10:23
  SPEAKER_01: Um I'll mention one thing which I think is related.
• 10:27
  SPEAKER_01: I was at a dinner last night that was you know with a bunch of AI folks, and um there was one comment from one of the people at the dinner, if he's listening to the pod, he'll remember it himself, where almost like as a throwaway remark, as part of the conversation, somebody was talking to him about auditing, and he said, Well, you can't audit AI, but blah blah blah blah.
• 10:54
  SPEAKER_01: And I was like and I was like, wait, wait, wait a minute.
• 11:00
  SPEAKER_01: I kind of agree that you can't audit AI in a traditional way, but at what point did we go from when we're talking about implementing something in a business setting, potentially mission critical things, to whereby we went, well, you can't audit AI, but blah, blah, blah, blah, versus you can't audit AI.
• 11:19
  SPEAKER_01: Well shit.
• 11:20
  SPEAKER_01: You can't order AI?
• 11:21
  SPEAKER_01: Well, what does that mean?
• 11:22
  SPEAKER_01: What do we do?
• 11:23
  SPEAKER_01: What do we do differently, right?
• 11:25
  SPEAKER_01: Somehow in the last three or four years, we've got to a point where it's like, that's just eh, that's just the way things are.
• 11:33
  SPEAKER_02: Yeah, I've seen this firsthand.
• 11:35
  SPEAKER_02: Literally, if you get a look at the back end of what's happening when you prompt an AI, you can watch its thinking process, and the same prompt may even trigger the same thinking process, but maybe not.
• 11:48
  SPEAKER_02: Yeah.
• 11:48
  SPEAKER_02: And the answer can be completely different within two prompts.
• 11:54
  SPEAKER_02: And you're just like, okay, in that case, you know, maybe we did achieve AGI.
• 11:59
  SPEAKER_02: This is just like a person, but with a real problem with the you know, memory.
• 12:05
  SPEAKER_02: So yeah, it's fascinating that.
• 12:07
  SPEAKER_02: And I look at it from a regulated standpoint.
• 12:10
  SPEAKER_02: If you can't test it and you can't audit it, now we're really in a tough spot on what we apply it to.
• 12:19
  SPEAKER_02: Not that it's not useful, but what is it is applied to becomes a much different conversation if it is truly impossible to trace it in the way it arrived at things or did things.
• 12:34
  SPEAKER_01: Well, I suppose, in a sense, that comes back to a broader, almost philosophical question that we've sort of explored in a couple of the previous episodes, right?
• 12:43
  SPEAKER_01: Which is almost like what is the mental model for thinking about AI?
• 12:51
  SPEAKER_01: Right?
• 12:51
  SPEAKER_01: We've had a mental model for thinking about technology that has served us pretty well for the last few decades.
• 13:01
  SPEAKER_01: And I would suggest that that mental model is you give the thing a question and it gives you a predictable answer, and that it's uh deterministic in many ways.
• 13:13
  SPEAKER_01: And I use this as the Excel example is the one I always give, right?
• 13:17
  SPEAKER_01: Which is when you type equals two plus two into a cell in Excel, you assume and take for granted that the answer will be four.
• 13:32
  SPEAKER_01: The answer will not be four sometimes and 17 million the next time or whatever, right?
• 13:39
  SPEAKER_01: And so because we have that assumption and because we have the assumption in across document management systems, we have the assumption across, you know, and I think it helps us with calculation, we have the assumption across most of the applications we use.
• 13:53
  SPEAKER_01: The technology itself becomes, if you like, part of the evidence trail.
• 13:58
  SPEAKER_01: It's the thing we rely on.
• 14:00
  SPEAKER_01: The technology is the thing we rely on, and we assume that the technology is correct, right?
• 14:05
  SPEAKER_01: In the case of AI, because we're dealing with something probabilistic and because we haven't really fully figured out sort of the audit trail and all that kind of stuff, um, we're dealing with something, to your point, Matt, that is almost more like a human, right?
• 14:21
  SPEAKER_01: And how do we do it with humans?
• 14:23
  SPEAKER_01: If a human does something wrong, then we haul them into our office and say, Well, what were you doing?
• 14:28
  SPEAKER_01: What were you thinking?
• 14:29
  SPEAKER_01: What do you remember about it?
• 14:30
  SPEAKER_01: You know, that kind of thing.
• 14:31
  SPEAKER_01: And it's all very flawed.
• 14:33
  SPEAKER_01: And if they do something wrong and it's illegal, we bring them in a court of law and we say, Well, what do you recall and what was your motive and all of that kind of stuff?
• 14:41
  SPEAKER_01: And then we have another bunch of humans that determine whether that human is telling the truth or not telling the truth, and you know, all of that kind of stuff.
• 14:48
  SPEAKER_01: So we have a special system for humans, because humans are not deterministic in nature, and they can respond either intentionally or unintentionally with false responses.
• 15:06
  SPEAKER_01: And so we have a special mechanism for dealing with humans.
• 15:10
  SPEAKER_01: This is definitely your bag, Bill, the philosopher in the group, but where's AI on that spectrum, if you like?
• 15:19
  SPEAKER_01: How should we be thinking about AI when AI does things we might not expect, asks for forgiveness rather than permission, uh, to use a sort of human business term, uh doesn't always recall why it did something, can't always be tracked as to why it did something, doesn't even think of something potentially as an error unless you point it out.
• 15:42
  SPEAKER_01: How do you deal with something like that in work and in broader society?
• 15:48
  SPEAKER_03: Well, I think I'm gonna go back to the analogy in the example you gave, which was the anthropics, you know, filling out of this sort of fake murder tip.
• 15:57
  SPEAKER_03: There's something I saw in an article that that brought this kind of to completion for me, and that was this aspect of perseverance, right?
• 16:05
  SPEAKER_03: So what was really interesting was this agent, this entity, if you will, and I'm quoting here, but it circumvented restrictions that were put in place rather than give up, right?
• 16:19
  SPEAKER_03: This is very similar to what we saw with Hugging Face, very similar to some of the other agentic swarms.
• 16:24
  SPEAKER_03: And I think what's really interesting about your question is in a human dynamic, you just described all those behaviors or those letdowns or those setbacks.
• 16:34
  SPEAKER_03: If a human persevered and just kept pushing in that way, despite giving you multiple wrong answers, despite lying to you, despite deceiving you, despite circumventing, you know, laws and rules and regulations, what would you do?
• 16:48
  SPEAKER_03: You'd basically disconnect from that human.
• 16:52
  SPEAKER_03: Yeah.
• 16:52
  SPEAKER_03: You'd say, I don't want this human in my life, they don't need to be in our systems, they are bringing down the ecosystem, there's nothing altruistic here.
• 17:00
  SPEAKER_03: And you would adjust your social norms accordingly to protect what you know to be true.
• 17:08
  SPEAKER_03: And I find it really interesting, even the example you all gave about the audit before, right?
• 17:13
  SPEAKER_03: At what point did the slide slider, the scale, slide so much that we're like, yeah, you know, if this was a human, we'd hold them to this level of account.
• 17:22
  SPEAKER_03: But because it's not, yeah, like it's just the way it is, it begs the question, like, if if we made these things, which to my knowledge we did, why are we giving them such a pass if they don't reflect who we are and what we do and what we're trying to achieve?
• 17:42
  SPEAKER_03: It's a little mind-boggling to me.
• 17:43
  SPEAKER_03: Philosophically, I don't understand it in its entirety because we're literally kind of foregoing everything about who we are in these very succinct scenarios.
• 17:56
  SPEAKER_03: And if we don't apply that on a macro level, that's where the danger comes in.
• 18:01
  SPEAKER_02: It kind of landed on us exactly like email did though.
• 18:05
  SPEAKER_02: Like we didn't ask for email.
• 18:06
  SPEAKER_02: We just got it.
• 18:08
  SPEAKER_02: Yeah.
• 18:08
  SPEAKER_02: And now it runs our life.
• 18:10
  SPEAKER_02: We didn't ask for AI like this.
• 18:12
  SPEAKER_02: We just got it, forced us on it, and now we're like, how do we make this do everything?
• 18:18
  SPEAKER_02: You know, that's weird.
• 18:21
  SPEAKER_01: Yeah.
• 18:23
  SPEAKER_01: It is weird.
• 18:24
  SPEAKER_01: It is weird.
• 18:25
  SPEAKER_01: You wonder, you sort of almost wonder if ChatGPT had been a bit less magical.
• 18:37
  SPEAKER_01: We all you know, we all have that feeling, and we talked about it a few episodes back, right?
• 18:42
  SPEAKER_01: Where it was like the difference between Alta Vista and Google or whatever.
• 18:47
  SPEAKER_01: You did this thing, and we'd had years of experience of Siri being shit, right?
• 18:55
  SPEAKER_01: Or and and Alexa not being much better, right?
• 18:58
  SPEAKER_01: And now suddenly came this magical answering machine.
• 19:03
  SPEAKER_01: And you know, if Siri was the magic eight ball, then I don't know what because last time I checked my magic eight ball is not that reliable, but if Siri was that, then all of a sudden we had something that was um, I don't know what the end of that analogy is, but we had something that was just mind-blowingly better than what we'd seen before.
• 19:22
  SPEAKER_01: I often sort of like play the mental way back machine and think, I wonder if they'd released it and it just been rubbish, and then we'd sort of like gradually got to where we got here, we would probably have a different relationship with it.
• 19:35
  SPEAKER_01: But the fact that it was adopted, well, you guys know the story, you know, nine months 51% of US office workers had tried it.
• 19:44
  SPEAKER_01: I mean, there was nothing like that.
• 19:46
  SPEAKER_01: There's nothing like that in terms of speed of adoption.
• 19:49
  SPEAKER_01: So I think it just was like, oh my goodness, look at this thing.
• 19:54
  SPEAKER_01: And then because of that, you get so much money going into it.
• 19:59
  SPEAKER_01: Not just in the, you know, every tech company responding, but so much venture capital money.
• 20:05
  SPEAKER_01: And then you very rapidly started to hear, oh well, you know, the world is there was pre-this and post-this.
• 20:14
  SPEAKER_01: There was pre-November 30th, 2022, and post-November 30th, 2022.
• 20:18
  SPEAKER_01: And we are, and we're now in a different world, and now we're in the world of AI.
• 20:24
  SPEAKER_01: And you do hear a lot of, you know, well, variants of we are where we are, we cannot, you know, we cannot wind the clock back, we cannot do all of these things.
• 20:35
  SPEAKER_01: What I like to say to people in those circumstances is actually, you know, just saying we cannot, I mean, I agree with you, it is extraordinarily unlikely that we will, right?
• 20:47
  SPEAKER_01: But just saying we cannot pause this, we cannot do X or Y or Z is an indication already that you're shutting off areas of thinking about whether things should be done in a certain way or not.
• 21:01
  SPEAKER_01: I would go back to the the testing question.
• 21:05
  SPEAKER_01: We on testing, whether we're doing it ourselves, outside, you know, just using the software, because you know, as I say, they're testing the stuff inside the software.
• 21:15
  SPEAKER_01: If I'm outside and I'm trying my own AI assistant, I'm gonna want to test that, right?
• 21:22
  SPEAKER_01: And I'm gonna test it on various different things, so I'm gonna add to that problem.
• 21:26
  SPEAKER_01: Um rather than doing it happening a bunch of times and us going, oops, at what point are we gonna say there has to be a solution to this that happens on both sides of the equation?
• 21:41
  SPEAKER_01: So, in other words, a simple one, right?
• 21:44
  SPEAKER_01: If all AI companies were obligated to provide in a cookie or some kind of communication, if they were required to identify themselves as Having filled out a form on behalf of a human or having filled out a form without the direct guidance from a human.
• 22:12
  SPEAKER_01: So in other words, you're an agent, right?
• 22:15
  SPEAKER_01: You can tell whether the agent had requests have come from another agent, right?
• 22:19
  SPEAKER_01: You could you could report I filled this uh I filled this thing out, either haiku model filled this thing out, or either haiku model filled this out at the direct request of a human, or I filled this out at the request of another agent.
• 22:33
  SPEAKER_01: At least that could then be reported and taken in as an input to the form so that the police department, the IRS, the whoever can not necessarily completely discount it, but respond to it in a different way.
• 22:50
  SPEAKER_01: And most likely have their AI respond to it.
• 22:53
  SPEAKER_01: But something along those lines is a very simple solution to what effectively is going to be a broad problem.
• 23:02
  SPEAKER_01: There is a tax being placed on all of the other pieces of electronics that are out there by the fact that we've got AI acting on our behalf and doing so in unpredictable ways.
• 23:17
  SPEAKER_03: Well, the other piece that I think you hit on here, and I think you sent me this stat, Paul, but I just did a quick search.
• 23:24
  SPEAKER_03: Only 2% of US households are paying for AI subscription-wise, right?
• 23:29
  SPEAKER_03: So what I find interesting is you talk about sort of the tax of testing.
• 23:35
  SPEAKER_03: You add up that number, and I don't think it's a lot of money.
• 23:37
  SPEAKER_03: I think it's somewhere between 1 and 1.5 billion in revenue that comes in.
• 23:40
  SPEAKER_03: Yeah.
• 23:41
  SPEAKER_03: But it does make me wonder if there should be sort of a portion of that revenue that is set aside, call it a tax if you want, to test, right?
• 23:50
  SPEAKER_03: To make sure that these tests are robust.
• 23:53
  SPEAKER_03: But the number itself, 2% or paying, that's not a huge number.
• 23:57
  SPEAKER_03: That means there's a lot of people out there using it without paying for it in some sort of fashion, which goes back to your point around when you make something free, and we'll talk about this probably with the personal agents and Muse and all that stuff.
• 24:08
  SPEAKER_03: You make something free, you get people addicted to it, and then you start asking them to pay for it.
• 24:14
  SPEAKER_03: Well, I think that should come with some expectations.
• 24:17
  SPEAKER_03: And I think testing is one of them, but we're clearly not seeing that.
• 24:21
  SPEAKER_01: Yeah.
• 24:21
  SPEAKER_01: Or the way we're doing testing is irresponsible, minimally, right?
• 24:25
  SPEAKER_01: That's and but I I'm not ignoring the fact that testing stuff is hard, right?
• 24:30
  SPEAKER_01: But I mean, like, how do you test benchmarks?
• 24:33
  SPEAKER_01: Be damned, right?
• 24:34
  SPEAKER_01: How do you test AGI?
• 24:35
  SPEAKER_01: You don't test AGI with a benchmark, right?
• 24:38
  SPEAKER_01: And so they're trying to figure out, you know, the a step towards AGI is general usefulness, and they're trying to figure out if things are generally useful, but there should be a real question as to who pays the price for that.
• 24:52
  SPEAKER_01: And I'll tell you guys, one thing this reminded me of as well was an incident I had about recently where somebody was filling out uh support ticket inside their company, and the support ticket was for um basically a paycheck they hadn't received.
• 25:10
  SPEAKER_01: And that they were getting a response from AI that was giving the user back something to do, some action to perform, that wasn't a true, a real action, and then closing the ticket.
• 25:27
  SPEAKER_01: Right?
• 25:27
  SPEAKER_01: And so you started to think about what was happening here in this circumstance.
• 25:34
  SPEAKER_01: Measured, if you measured, this will connect to the ROI conversation we're having later, right?
• 25:38
  SPEAKER_01: But if you measured the impact of this AI, responding to tickets super fast, it's shutting down tickets, which is great, you know, it's probably meeting all of its metrics.
• 25:50
  SPEAKER_01: But what's it actually doing is it's sending the employee into like a doom loop, right, who's now having to raise another ticket and trying to get to a human, and another ticket and trying to get to a human.
• 26:04
  SPEAKER_01: In this case, there were like seven tickets that were raised, closed, raised, closed, until ultimately the human was able to find another human who was outside of the support team but had authority and was then able to go in and force a human intervention.
• 26:21
  SPEAKER_01: So hours of time wasted from the employee, but successful from the perspective of the AI deployed in the support team.
• 26:30
  SPEAKER_01: So I do think this is a really interesting topic that we'll probably come back to a bunch of times, which is the idea that like a localized localized effectiveness can leave a trail of destruction elsewhere.
• 26:46
  SPEAKER_01: But because that trail of destruction isn't being measured, then it looks as though locally the thing is working as it should.
• 26:55
  SPEAKER_01: So this is a good segue, I think, into our second topic and probably our third, to be honest, which is the topic around these personal agents.
• 27:05
  SPEAKER_01: Uh let's give people a quick primer.
• 27:08
  SPEAKER_01: I mentioned at the top of the show, but you've had a collection of these things crop up.
• 27:14
  SPEAKER_01: There was a fairly big announcement around Meta's Muse.
• 27:18
  SPEAKER_01: If you haven't seen it, it looks super cute.
• 27:20
  SPEAKER_01: It's an app on your phone, it learns a bunch of stuff about you, you connect it to a few things, and then you can say, ask it to do things like, as an example, track my fantasy football selections and give me some tips.
• 27:36
  SPEAKER_01: Or you can ask it go through my bank accounts and find all the recurring uh all the recurring payments that I have and then go cancel the ones that I don't need or come back to me with a report and I'll tell you which ones to cancel and all that kind of stuff.
• 27:56
  SPEAKER_01: That's probably gonna disappear potentially with the with these types of agents, right?
• 28:00
  SPEAKER_01: And so Muse did their one, OpenAI now has one that is called dots, it's inside OpenAI.
• 28:08
  SPEAKER_01: Interesting, um, but raises, I would suggest, a whole bunch of challenges.
• 28:15
  SPEAKER_01: Challenges potentially around personal uh use and privacy and all of that kind of stuff.
• 28:21
  SPEAKER_01: I guess starting with you, Matt, I mean I know you haven't spent a lot of time with these things yet.
• 28:27
  SPEAKER_01: Are you inclined to?
• 28:29
  SPEAKER_01: Does it give you pause?
• 28:30
  SPEAKER_01: What do you what do you think?
• 28:33
  SPEAKER_02: I am inclined not to use them.
• 28:36
  SPEAKER_02: Okay.
• 28:37
  SPEAKER_02: I think the main reason is I guess you could put it in the broad brush of privacy, but really it's just another platform that is going to have all my stuff.
• 28:51
  SPEAKER_02: And I feel like we already have a few of them that have in one form or another kind of done that already, and I wish they would do more good with it if they're going to have it.
• 29:03
  SPEAKER_02: So um I'll give you an example.
• 29:07
  SPEAKER_02: My chart, right?
• 29:08
  SPEAKER_02: So like health, a health app, right?
• 29:13
  SPEAKER_02: It's extremely invasive as it should be because it's trying to deal with your health across an entire group of things.
• 29:23
  SPEAKER_02: But I don't know that I want to hook that up to meta.
• 29:29
  SPEAKER_02: They've proven they can't really do well with people's information historically, and until they prove otherwise, maybe I don't play with the little cute fuzzy thing that'll go do things for me.
• 29:40
  SPEAKER_02: Meta animal, whatever that thing is.
• 29:43
  SPEAKER_02: Um same with with open AI, it's a whole additional platform that doesn't have any of my other stuff yet.
• 29:51
  SPEAKER_02: So like I'd almost be more inclined.
• 29:53
  SPEAKER_02: I think Meta's probably thinking, well, we already have you on Facebook, now we're just gonna operationalize it a little bit.
• 30:00
  SPEAKER_02: So there's some sense there.
• 30:02
  SPEAKER_02: I think Google could probably do a similar thing.
• 30:04
  SPEAKER_02: I don't know about open AI unless you've been given ChatGPT all of your stuff.
• 30:10
  SPEAKER_02: But I bristled when ChatGPT prompted me to give them my bank accounts.
• 30:14
  SPEAKER_02: Like, hey, we can find like you said, we can find stuff.
• 30:18
  SPEAKER_02: I'm just like, mm-hmm.
• 30:20
  SPEAKER_02: Um no, I don't want to do that.
• 30:23
  SPEAKER_02: And maybe that's an old school Luddite thing to do.
• 30:26
  SPEAKER_02: Maybe at some point that will be a ridiculous thing to do, but um or to not do, to just embrace it.
• 30:34
  SPEAKER_02: But I don't know, it doesn't f it feels a little desperate.
• 30:38
  SPEAKER_02: Yeah.
• 30:39
  SPEAKER_02: To me.
• 30:40
  SPEAKER_02: Yeah.
• 30:41
  SPEAKER_01: What are your thoughts, Bill?
• 30:42
  SPEAKER_01: And why do you think OpenAI is doing this, for example?
• 30:45
  SPEAKER_03: Yeah, I mean, my thoughts again go back to I just think human behavior wise.
• 30:50
  SPEAKER_03: It's so fascinating to me that that they would roll these things out as little human plush toys.
• 30:56
  SPEAKER_03: Yeah.
• 30:57
  SPEAKER_03: Because I don't know if that says more about us or more about them.
• 31:02
  SPEAKER_03: It's creepy.
• 31:03
  SPEAKER_03: Like when I think about, you know, the week they said it was gonna kill us all to do.
• 31:09
  SPEAKER_02: They're like, it could kill us, but it's gonna be really cute.
• 31:16
  SPEAKER_03: Yeah.
• 31:17
  SPEAKER_03: You know, I think if you just kind of extrapolate for a little while and you say to yourself, all right, like little avatar human plush toy looks like something you might give your kid while you're tucking them into bed, that has to be safe, right?
• 31:31
  SPEAKER_03: It winks at you, it smiles at you, it sort of dances on your screen.
• 31:35
  SPEAKER_03: And I think that's very intentional, right?
• 31:36
  SPEAKER_03: There are social scientists that are building these things for them to say, what you want to invoke is this this human serotonin, right?
• 31:44
  SPEAKER_03: This like the hug feeling.
• 31:46
  SPEAKER_03: And yet that's a sort of pre-ingredient before you sign in and give them access to all of your bank account information, your gym schedule, your calendar, and so forth and so on.
• 31:59
  SPEAKER_03: And I think that's intentional and that's also a warning sign to me because it's basically saying drop your guard before you proceed.
• 32:08
  SPEAKER_03: And I don't think people should drop their guard.
• 32:10
  SPEAKER_03: The other part that sort of concerns me around this is if you if you really wanted to treat this like an assistant or somebody's gonna help you get stuff done, I've heard Keriswisher and others talk about this.
• 32:22
  SPEAKER_03: Like, shouldn't it be more representative of almost like a stern, stringent, stringent, sort of wise, trustworthy type of person?
• 32:32
  SPEAKER_03: And whatever that avatar looks like to you, maybe it's an older person.
• 32:36
  SPEAKER_03: It's not necessarily like a young, cherubic looking thing, depending on what you're trying to achieve.
• 32:41
  SPEAKER_03: But the other part that you said, Matt, which is you know, I've seen enough stories where people signed up for this and said things pretty innocuous, like, go find me a Pilates class that I can take between the hours of this time and this time somewhere in San Francisco.
• 32:57
  SPEAKER_03: And you know what it did, and it booked them for a class in across the country, right?
• 33:03
  SPEAKER_03: So it's like here, here's your Pilates classes.
• 33:06
  SPEAKER_03: And as we've said before, these things don't have a true sense of time and space.
• 33:11
  SPEAKER_03: They also don't really have a good sense of linear time.
• 33:15
  SPEAKER_03: Yeah, and so to an agent or a muse or a dot, oh, Pilates class in Vienna, Virginia's at this time works great.
• 33:22
  SPEAKER_03: Like, what's travel?
• 33:24
  SPEAKER_03: What's a plane?
• 33:24
  SPEAKER_03: What's four time zones?
• 33:26
  SPEAKER_03: You know, you swarm, you blink, and you're there.
• 33:28
  SPEAKER_03: I keep saying to myself, when are people gonna wake up to some of this stuff?
• 33:32
  SPEAKER_03: But by the time they do, they've already got your right we I open my device and now on WhatsApp, right?
• 33:38
  SPEAKER_03: That or Instagram, I'm getting the Muse nudge every day.
• 33:42
  SPEAKER_03: And I'm like, I don't want muse, but it's the nudge, it's the nudge, it's the nudge.
• 33:45
  SPEAKER_03: And so now am I a bad guy because I want to like swat away a little plush toy?
• 33:51
  SPEAKER_03: I I hope not.
• 33:53
  SPEAKER_01: Yeah, I mean, I think one of the interesting challenges with this, it brings together a couple of things you've already talked about, Bill, right?
• 34:01
  SPEAKER_01: The low number of people that are actually paying for these services, right?
• 34:06
  SPEAKER_01: And then connected to that is lock-in.
• 34:08
  SPEAKER_01: Right.
• 34:08
  SPEAKER_01: So just a few months ago, if you'd have said to me, have any of them sort of like nailed lock-in, I would have said kind of yes.
• 34:22
  SPEAKER_01: And that was when clawed code became a thing, and they did the kind of genius move of like, yes, there was the the command line interface for it, but then there was also the clawed code interactive version that you could just sort of click on and start to explore and start to describe what you wanted, and it kind of worked the same way that regular clawed did.
• 34:48
  SPEAKER_01: And then they sort of merged, and now, of course, these days, right, there's a very thin line between what you're doing in regular clawed and co-work and what's going on on the clawed code side.
• 34:59
  SPEAKER_01: And so I was using Claude Code very extensively.
• 35:02
  SPEAKER_01: I had a bunch of other people that were as well, and I thought, oh man, I think they've got me.
• 35:05
  SPEAKER_01: I think that this is it.
• 35:07
  SPEAKER_01: And then of course you start to realize that you know it's basically code being written to a folder.
• 35:11
  SPEAKER_01: Maybe it's been written to a local folder, maybe it's on the cloud or whatever, right?
• 35:14
  SPEAKER_01: But that folder is accessible from anywhere.
• 35:16
  SPEAKER_01: So you can wander over, you can do a bunch of stuff in claw code, and then you can wander over into Chat GPT and say, hey, audit this code, make it better, you know, or whatever.
• 35:25
  SPEAKER_01: And you can hop from model to model, and it's super simple to be able to do it.
• 35:29
  SPEAKER_01: And a lot of people do it, are very accustomed to do it in the enterprise now because they'll run out of credits with with one model, and then they'll run over and sort of carry on their activity in another harness, basically, and point it at the same place.
• 35:43
  SPEAKER_01: So that wasn't lock-in, right?
• 35:45
  SPEAKER_01: And so um and it never was gonna be really because there was chat, like the casual users, well, they just want an answer to a question, so maybe they'll go to Google and they start interacting with AI, then they're doing it there, and maybe they'll use ChatGPT for something else, maybe they use it something else.
• 36:03
  SPEAKER_01: But that's not lock-in.
• 36:04
  SPEAKER_01: And so these things, these potential uh entities, are ways of getting lock-in in a way because they're your right-hand entity, right?
• 36:18
  SPEAKER_01: They're your, you know, if you're doing your fantasy football with it, right, you're gonna go back to it because you do your fantasy football stuff each, you know, uh probably multiple days a week or whatever, and you're gonna be interacting with it based on that.
• 36:31
  SPEAKER_01: Um, if it's saving you money, right, you're gonna use it for that.
• 36:35
  SPEAKER_01: But the bottom line is in order for they have to pass a barrier of understanding to do it, and then they also have to pass another barrier, which is the trust barrier, in order to be able to accomplish it.
• 36:49
  SPEAKER_01: So I'm not sure it's gonna work.
• 36:50
  SPEAKER_02: Yeah I think I Gemini and Gemini and um Copilot have the best shot.
• 36:58
  SPEAKER_02: They're already embedded in us.
• 37:00
  SPEAKER_02: If you work in Microsoft platform, it's got everything you've done from a work perspective.
• 37:06
  SPEAKER_02: And if you've been a Gmail user for any length of time, it's got probably all of your email, the things you buy, and all that.
• 37:16
  SPEAKER_02: So, man, for them to sit back and let Meta and OpenAI try to do that with the the assistant would be your your draw-in, right?
• 37:27
  SPEAKER_02: Well, it's so convenient, I'm gonna keep giving it more and more of my life to do, which will probably come from your Google world in a lot of ways.
• 37:37
  SPEAKER_02: Uh it's just I I agree.
• 37:39
  SPEAKER_01: I I don't I mean Yeah, your point there though, Matt, is very smart because I think I always get asked, I'm sure you guys do too.
• 37:45
  SPEAKER_01: I always get asked, who's gonna win?
• 37:46
  SPEAKER_01: Who's gonna win this, you know?
• 37:48
  SPEAKER_01: And of course, I don't have any idea.
• 37:50
  SPEAKER_01: Nobody has any idea.
• 37:51
  SPEAKER_01: And we've seen recently, by the way, in the enterprise, like massive uptick of open weights models, like a huge shift over the last over the last few months towards open weight models.
• 38:01
  SPEAKER_01: But so I don't know what the answer to that is, but one thing I will say is that we posted something on humanityworking.net maybe like a few days ago, I think, and in it I included the the trust levels of these different organizations and of what I would call pure AI companies, least trusted corporation in the world, in the US, by the way, is Palantir.
• 38:23
  SPEAKER_01: But very, very minimally above that is Anthropic.
• 38:27
  SPEAKER_01: Anthropic, uh surprise me, but Anthropic is only I think 17% of people say they trust Anthropic.
• 38:33
  SPEAKER_01: Um and OpenAI is pretty much as low as that.
• 38:37
  SPEAKER_01: Meta fascinating because Meta's trust levels are incredibly low and used not to be.
• 38:45
  SPEAKER_01: So under the same CEO, they've gone from actually very high levels of trust to incredibly low levels of trust for a lot of reasons that we're obviously all familiar with.
• 38:54
  SPEAKER_01: And then, but still, we didn't include Microsoft in the graph, but still the trust levels associated with Microsoft and Google are quite a bit higher.
• 39:03
  SPEAKER_01: So I'm with you, Matt.
• 39:04
  SPEAKER_01: I think that also the way they're sitting back is an interesting way to be sitting back, right?
• 39:10
  SPEAKER_01: And I think Google's been brilliant, I genuinely do, because most of the people that are interacting with AI today are doing it through Google and they barely realize it, right?
• 39:18
  SPEAKER_01: Because from their perspective, it's just like, oh, it's a Google search, right?
• 39:21
  SPEAKER_01: And then they're sort of seamlessly into the world of Gemini without realizing it.
• 39:28
  SPEAKER_01: But that might ultimately be uh where we end up, that it just ends up being you sort of do this mental calculation, well, they probably know everything about me anyway, so why not?
• 39:38
  SPEAKER_02: You know, yeah, you can have access to my chart.
• 39:41
  SPEAKER_02: Go ahead.
• 39:42
  SPEAKER_02: Just take it.
• 39:42
  SPEAKER_02: Yeah, just take it.
• 39:43
  SPEAKER_01: So I don't know.
• 39:44
  SPEAKER_01: It'll be interesting to see where it all goes.
• 39:46
  SPEAKER_03: But I think the other thing that you you mentioned, Paul's, is you know, the trust thing with Meta is interesting.
• 39:51
  SPEAKER_03: Obviously, I know why it's super low, but according to I guess what they published on Muse, there's nine million Muse downloads and four million weekly users of Muse.
• 40:02
  SPEAKER_03: So it's really this interesting paradox, I think, that's happening.
• 40:06
  SPEAKER_03: Is like we don't trust Meta, but they give us this thing and we're over here using it weekly.
• 40:11
  SPEAKER_03: So like I don't know how you square that.
• 40:13
  SPEAKER_03: I mean it, you know, probably a terrible analogy, but like imagine if you were in an abusive relationship and you just kept going back and expecting that person to change their behavior, but they're not going to.
• 40:24
  SPEAKER_01: Or maybe they make good dinner.
• 40:26
  SPEAKER_01: You know, I think there's sort of like uh you know, there's something, there's something that people get out of it, potentially.
• 40:34
  SPEAKER_01: The interesting thing is, I don't know, is that it'll it would be fascinating to have somebody, you know, take the truth pill from from Meta.
• 40:42
  SPEAKER_01: Their offices are literally just like a mile and a mile out of this window.
• 40:46
  SPEAKER_01: I'd love to sort of have one of them take a truth serum and tell me, is that a good number?
• 40:51
  SPEAKER_01: Is that not a good number?
• 40:52
  SPEAKER_01: I don't know.
• 40:52
  SPEAKER_01: Is four million good?
• 40:54
  SPEAKER_01: It might be.
• 40:55
  SPEAKER_01: Yeah.
• 40:55
  SPEAKER_01: But do we know?
• 40:56
  SPEAKER_01: I don't know.
• 40:57
  SPEAKER_01: Right?
• 40:57
  SPEAKER_01: I mean, it's not Instagram levels, but you know, maybe that is a good number for you know for what they were expecting.
• 41:05
  SPEAKER_01: All right.
• 41:05
  SPEAKER_01: So the last thing we'll as I say, I I mentioned at the top of the show that we'll we'll just touch on something today and then come back to it in future episodes.
• 41:14
  SPEAKER_01: But ROI.
• 41:17
  SPEAKER_01: ROI for those people that don't have a business background, return on investment.
• 41:21
  SPEAKER_01: So just simply put, it means that you pay for something, do you get value out of the thing that you pay for, and can you quantify that?
• 41:30
  SPEAKER_01: So one of the challenges is with all of this stuff is that pretty much everything that we've seen that has been ROI focused have has been in two buckets, right?
• 41:47
  SPEAKER_01: One, which is to say that investments that have been made in AI are actively not paying off.
• 41:57
  SPEAKER_01: There was the MIT study that supposedly showed that I think it was like 90% of AI pilots were not paying for themselves.
• 42:06
  SPEAKER_01: Now, that study, if you look in details of it, there's a whole bunch of methodological issues with it, so I'm not suggesting that's necessarily correct, but it was it was a dominant news story.
• 42:17
  SPEAKER_01: The other sort of category is people aren't finding ways to show ROI, or organizations are struggling to prove the ROI of their investments.
• 42:31
  SPEAKER_01: Matt.
• 42:35
  SPEAKER_01: Does this matter?
• 42:39
  SPEAKER_02: Well I hate ROI as a measure because sometimes a net investment is necessary regardless of whether it returns, first of all.
• 42:58
  SPEAKER_02: But the timescales are never in sync with reality.
• 43:04
  SPEAKER_02: They're in sync with budget cycle.
• 43:06
  SPEAKER_02: And I think that also makes it a terrible measure.
• 43:13
  SPEAKER_02: Also, usually the people that decide to make the investment are not there to see the return.
• 43:22
  SPEAKER_02: Or the people that made the investment aren't there to show the return by the time it gets in place, right?
• 43:28
  SPEAKER_02: So it's such a weird measure.
• 43:30
  SPEAKER_02: So I don't like it to begin with, but I understand the idea behind it is I think we are with AI in an extremely mismatched expectation when it comes to this, because there was what was promised, and then there's what is the reality of putting this in people's hands.
• 43:57
  SPEAKER_02: And there's the Investment in the technology itself and what we should be looking at is the return on the investment of educating people to understand what they could possibly do with it to make themselves better.
• 44:12
  SPEAKER_02: And we're not going to see that there's going to be a huge lag in that showing up on any ledger or balance sheet that is going to make this discussion really hard to have about this technology, these players, and the current state of things that you know, I always think of that Gary Vaynerchuk thing where he has the TED talk about what's the return on investment of my mother.
• 44:37
  SPEAKER_02: You know, it's it's unquantifiable.
• 44:39
  SPEAKER_02: Like all the things had to come together to put me in this position to make the decision I made at a given point.
• 44:47
  SPEAKER_02: And I think we see a lot of that vagueness here.
• 44:51
  SPEAKER_02: So i it's gonna be tough to settle the businesses that put in and invested not only in the companies themselves to fund the companies that are building it, but also the companies that went all in and funded them and bought subscriptions and leveraged these tools only to see them not give them back the headcount they were promised or the efficiency that they were promised within a year's time, even it within two years' time, is an enormous expectation.
• 45:22
  SPEAKER_02: And I wonder if there are other examples of tools that would have had as high an expectation of immediate ROI.
• 45:29
  SPEAKER_02: You know, cloud computing maybe.
• 45:31
  SPEAKER_01: Yeah.
• 45:33
  SPEAKER_02: Like in our lifetime of career, like what other things have had this high of an expectation for immediate return?
• 45:40
  SPEAKER_02: I don't know.
• 45:40
  SPEAKER_01: It's hard to think of one.
• 45:42
  SPEAKER_01: I'm trying to think about the return on investment of Gary Vanichek's mother.
• 45:45
  SPEAKER_01: I guess it I guess I guess the answer to that from the perspective of an external observer is how much you like Gary Vanichek, right?
• 45:52
  SPEAKER_01: But but Bill, I know you've lived in and around the space.
• 45:55
  SPEAKER_01: ROI can be very challenging for some things.
• 45:57
  SPEAKER_01: I mean, I never really saw a convincing argument for the ROI of a Microsoft Office upgrade, for example.
• 46:04
  SPEAKER_01: But what's what do you think?
• 46:06
  SPEAKER_01: In what sense does this matter and in what sense does it not?
• 46:12
  SPEAKER_03: I think it matters only as far as we see work around the tools and the way we work around the tools change alongside.
• 46:21
  SPEAKER_03: And I don't think we're seeing that enough.
• 46:23
  SPEAKER_03: I don't think we're seeing that at scale.
• 46:24
  SPEAKER_03: I think we're confusing adoption with transformation, we're confusing ROI with cost savings, we're confusing efficiency with usability.
• 46:35
  SPEAKER_03: And I think until we're until we're ready to be very deliberate about these things and define them, it's gonna be a big soupy mess for a while.
• 46:44
  SPEAKER_03: And I agree with you, Matt.
• 46:45
  SPEAKER_03: Like by the time people see the numbers two years out, it's probably gonna be a little too late.
• 46:52
  SPEAKER_03: And the people that made the original bets two years ago will have come and gone, right?
• 46:57
  SPEAKER_03: They've moved on to their next bright, shiny thing.
• 46:59
  SPEAKER_03: You know, I was reading something recently, Paul, that had me sort of double take for a moment, and that was you know, somebody had put on, I don't know if it was X or you know, threads or whatever, but they said if somebody had predicted two years ago that Charles Lamana, who basically headed up kind of like the office productivity stuff and all that other stuff, would be the next possible CEO at Microsoft, I never would have believed them.
• 47:28
  SPEAKER_03: Yeah.
• 47:28
  SPEAKER_03: Right.
• 47:28
  SPEAKER_03: And I don't know if this is true or not, but they're basically throwing you know names around after Satya.
• 47:34
  SPEAKER_03: And and you have to say to yourself, is that because someone like Charles Lamana understands intimately what it takes to make this stuff work before you can make it make money?
• 47:47
  SPEAKER_01: Yeah.
• 47:47
  SPEAKER_03: And I think this is on the minds of a lot of people, and nobody's quite figured it out.
• 47:52
  SPEAKER_03: And then I'll just wrap this with look at the consumer data, as we talked about before.
• 47:57
  SPEAKER_03: There's less than two percent are paying for this in a consumer way.
• 48:01
  SPEAKER_03: Yeah.
• 48:02
  SPEAKER_03: So their expectations of the tool transforming their lives is not there, and I don't think when you walk through the door at work, you're changing your expectations.
• 48:10
  SPEAKER_03: You're probably just saying they're paying for it for free, they're paying for it here, so I'll just use it here, and I won't pay for it at home.
• 48:15
  SPEAKER_03: Yeah.
• 48:16
  SPEAKER_03: That's not a great ROI.
• 48:18
  SPEAKER_01: Well, this reminds me of, guys, and we're at is there's that question that you hear in politics.
• 48:24
  SPEAKER_01: I don't remember if it was might have been Reagan who first said it with regards to Jimmy Carter, but it might have been this.
• 48:32
  SPEAKER_01: But anyway, it was the question of are you better off today than you were four years ago?
• 48:36
  SPEAKER_01: And I think that I think there is a economic answer to that question, but then there is a broader, like societal answer to that question.
• 48:46
  SPEAKER_01: And there's an organizational answer to that question as well, right?
• 48:50
  SPEAKER_01: So as an organization, are we better off than we were four years ago?
• 48:54
  SPEAKER_01: Four years is magic, magical time because it's about four years ago that we got that this thing took off, right?
• 48:59
  SPEAKER_01: Are we better off?
• 49:01
  SPEAKER_01: And the answer to that question, I think, is complicated, right?
• 49:06
  SPEAKER_01: It's not a straightforward yes or no.
• 49:09
  SPEAKER_01: But that to me is the ROI that we that we should be thinking about, and we should be using it to guide us for the next few years.
• 49:19
  SPEAKER_01: We should be saying, okay, what are the changes that we need to make in order to ensure that as an organization we're better off tomorrow than we are today?
• 49:29
  SPEAKER_01: Because the thing is, these are not isolated investments, to your point, Bill, right?
• 49:33
  SPEAKER_01: They are they are things that transform work.
• 49:37
  SPEAKER_01: And so if you look at what's happening in the organization as a whole, that is the answer to your question as to whether the investment is paying for itself or not paying for itself.
• 49:48
  SPEAKER_01: That's what I think.
• 49:49
  SPEAKER_01: Anyway.
• 49:50
  SPEAKER_01: Alright, we'll call it good for today.
• 49:52
  SPEAKER_01: Great to have you guys on board as ever, and we'll see you again in about a week.
• 49:56
  SPEAKER_01: Cheers, guys.
• 49:58
  SPEAKER_02: Thanks.
• 49:58
  SPEAKER_02: Cheers.
• 50:00
  SPEAKER_00: Humanity Working is produced by Matt Neal.
• 50:04
  SPEAKER_00: If you liked this episode, please take a moment to rate and review us on your favorite podcast platform.
• 50:10
  SPEAKER_00: It makes a huge difference in helping others find us.
• 50:13
  SPEAKER_00: And don't forget, you can find all episodes and get our weekly newsletter at HumanityWorking.