Latest / Sidecar Sync / Fargo vs. Klarna & The Rise of Reasoning Models | 81

Transcript

Fargo vs. Klarna & The Rise of Reasoning Models | 81 · Sidecar Sync

• 0:00
  Speaker 1: The most important thing for all of you associations to note is that you have options.
• 0:04
  Speaker 1: You have ways of doing secure, private AI inference.
• 0:08
  Speaker 1: There's a number of ways to do this and you can even do it locally, on-device.
• 0:14
  Speaker 2: Welcome to Sidecar Sync, your weekly dose of innovation.
• 0:17
  Speaker 2: If you're looking for the latest news, insights and developments in the association world, especially those driven by artificial intelligence, you're in the right place.
• 0:29
  Speaker 2: We cut through the noise to bring you the most relevant updates, with a keen focus on how AI and other emerging technologies are shaping the future.
• 0:34
  Speaker 2: No fluff, just facts and informed discussions.
• 0:37
  Speaker 2: I'm Amit Nagarajan, Chairman of Blue Cypress, and I'm your host.
• 0:43
  Speaker 1: Greetings and welcome to to the Sidecar Stink, your source for content at the intersection of all things artificial intelligence and the world of associations.
• 0:52
  Speaker 1: My name is Dmitry Nagarajan.
• 0:55
  Speaker 3: And my name is Mallory Mejiaz.
• 0:57
  Speaker 1: And we are your hosts and, as always, we've prepared an awesome episode for you guys to get some really interesting topics at the forefront of AI and we're going to talk all about how they apply to you in the world of associations.
• 1:10
  Speaker 1: So excited to get into that.
• 1:12
  Speaker 1: But first of all, Mallory, how are you doing today?
• 1:15
  Speaker 3: I'm doing pretty well myself, amit.
• 1:17
  Speaker 3: It's a nice chilly day in Atlanta, so I'm enjoying that.
• 1:21
  Speaker 3: Been getting outside a lot recently, since the weather's been mostly warm and, yeah, I've had some fun auditions come through on the acting front, so it's been a good productive weekend for me.
• 1:34
  Speaker 1: What about you Fantastic?
• 1:35
  Speaker 1: Well, you know I joke around a lot of times when I'm in New Orleans, which is home base for me, that that's the center of the universe for associations.
• 1:41
  Speaker 1: Of course it really isn't no-transcript.
• 2:07
  Speaker 1: Yeah, I had a breakfast chat with somebody and have a few more meetings lined up meeting with some of our team members across our company, so it's always a productive time in DC.
• 2:17
  Speaker 1: It's pretty much nonstop from early morning till late in the evening when I get into town.
• 2:22
  Speaker 3: Yeah, I was saying to Amit before we started recording.
• 2:25
  Speaker 3: I didn't know how he was possibly going to squeeze in this podcast with his schedule today of all these meetings, but you showed up, Amit, I'm really happy we're here.
• 2:35
  Speaker 1: Well, episode 81,.
• 2:36
  Speaker 1: We got to keep the streak going and this is so much fun to record and I'm always interested in making time for it.
• 2:42
  Speaker 1: My audio quality may not be as good as normal, unfortunately for this episode, so apologies in advance if that is the case and that is your experience.
• 2:50
  Speaker 1: But I'll be back to a normal recording session shortly, but for now I am on the road and doing my best.
• 2:57
  Speaker 3: We take the sidecar sink all over.
• 2:59
  Speaker 3: I don't know if we've ever done it internationally yet.
• 3:02
  Speaker 3: I'm trying to think on my end.
• 3:10
  Speaker 1: I don't think I've ever recorded in another country.
• 3:11
  Speaker 1: What about you, Amit?
• 3:11
  Speaker 1: I don't believe so, but that sounds like a challenge.
• 3:12
  Speaker 3: I think I need to book a flight somewhere.
• 3:13
  Speaker 3: Yeah, hey, let's do it.
• 3:14
  Speaker 3: We'll do like maybe a Mexico version of the sidecar sink.
• 3:17
  Speaker 3: That'd be fun.
• 3:18
  Speaker 3: Well, today, as Amit mentioned, we have some exciting topics lined up for you.
• 3:23
  Speaker 3: We're going to first be exploring this Wells Fargo AI assistant and then doing a little bit of a reflection on an episode we did it was actually episode 21 where we talked about Klarna's AI assistant, just to do a little compare and contrast.
• 3:36
  Speaker 3: And then we will be talking about the latest Microsoft's PHY4 family of models with some great naming conventions, as we always chat about on the Sidecar Sync podcast.
• 3:47
  Speaker 3: So, first and foremost, the Wells Fargo AI Assistant is called Fargo.
• 3:51
  Speaker 3: It's an advanced virtual assistant integrated into the Wells Fargo mobile app that helps customers with a wide variety of banking tasks through both voice and text interactions, from checking balances to processing payments and handling refunds, and also providing personalized financial guidance.
• 4:08
  Speaker 3: Fargo serves as a 24-7 banking assistant for Wells Fargo customers.
• 4:13
  Speaker 3: The assistant uses a model agnostic architecture and it employs different specialized LLMs for various tasks.
• 4:21
  Speaker 3: So that's that multi-agent framework that we talk about often on this podcast.
• 4:24
  Speaker 3: It has a privacy-first design, so no personally identifiable information is exposed to external language models and sensitive data is processed locally before any cloud interaction.
• 4:36
  Speaker 3: They're seeing some impressive results so far.
• 4:40
  Speaker 3: So there have been 245.4 million interactions with the assistant in 2024, which is actually double what they projected and these are interactions entirely without human intervention.
• 4:53
  Speaker 3: So around 250 million interactions without human intervention.
• 4:57
  Speaker 3: They're seeing deep engagement with their AI assistant, so 2.7 interactions per session on average with their AI assistants.
• 5:07
  Speaker 3: So 2.7 interactions per session on average, and across the board with their AI initiatives they're seeing a 3 to 10x increase in customer engagement and something that's also interesting to note.
• 5:13
  Speaker 3: We've talked about bias that's built into AI models because of the material that it's trained on.
• 5:18
  Speaker 3: We've also talked about bias with humans right, because when we're making decisions, we're pulling on all of our previous experience as well.
• 5:25
  Speaker 3: Something that's been interesting with their AI initiatives at Wells Fargo is they're seeing some bias reduction in certain areas, so the AI has led to fairer lending decisions when it comes to loans, which I think is quite interesting to note.
• 5:39
  Speaker 3: Behind the scenes, pega is the company particularly their customer decision hub behind all these AI initiatives at Wells Fargo, and it helps them analyze billions of interactions to determine the next best conversation for each customer, making Fargo's responses highly personalized and relevant across channels and, as I mentioned in episode, so it was episode 21.
• 6:03
  Speaker 3: Right now we're recording episode 81.
• 6:05
  Speaker 3: And, as I mentioned in episode, so it was episode 21.
• 6:06
  Speaker 3: Right now we're recording episode 81.
• 6:08
  Speaker 3: So a long time ago, 60 weeks ago, we talked about the Klarna AI Assistant and I wanted to do a little bit of a reflection.
• 6:12
  Speaker 3: There aren't a ton of stark differences, but there are a few.
• 6:17
  Speaker 3: So, despite serving fewer customers, wells Fargo has about 70 million customers.
• 6:21
  Speaker 3: Klarna has 150 million.
• 6:23
  Speaker 3: So substantially different has about 70 million customers.
• 6:26
  Speaker 3: Klarna has 150 million, so substantially different.
• 6:30
  Speaker 3: Wells Fargo handled that 250 million interactions that I mentioned in the whole year of 2024, compared to Klarna's 2 million-ish in its first month.
• 6:34
  Speaker 3: They haven't published their full number for 2024.
• 6:37
  Speaker 3: But even comparatively 2 million in one month if it continued on that trend or even considerably increased each month in one month, if it continued on that trend or even considerably increased each month 250 million interactions at Wells Fargo is pretty impressive for their 70 million customer base.
• 6:50
  Speaker 3: Klarna also publicly stated that its AI assistant was doing the work of about 700 full-time customer service agents, handling two-thirds of all customer service chats.
• 7:02
  Speaker 3: Wells Fargo has not published a specific equivalent number of agents replaced.
• 7:07
  Speaker 3: But given Fargo's scale as far as exceeding Klarna in both total interactions and per customer engagement, I would say it's reasonable to infer that Fargo automates work that would require potentially thousands of agents.
• 7:19
  Speaker 3: And something also worth noting is the feature evolution in both.
• 7:24
  Speaker 3: So the Klarna assistant has expanded from customer service to shopping recommendations, personalized shopping feed, multilingual support and chat GPT integration for shopping advice.
• 7:34
  Speaker 3: The Wells Fargo assistant has added AI driven spending insights, actionable financial tips, improved money movement and financial insight summaries.
• 7:43
  Speaker 3: So going beyond this basic customer service, routine interaction and really providing further value to the consumer, which I think is quite interesting.
• 7:52
  Speaker 3: So, amit, you've been talking about virtual assistants, ai agents, really from the beginning, from the beginning of this podcast, for sure.
• 8:00
  Speaker 3: How have you seen that conversation evolve, particularly over the last year?
• 8:06
  Speaker 1: You know it's interesting, you mentioned episode 21 versus 81.
• 8:10
  Speaker 1: So it's exactly 60 episodes, or roughly 60 weeks ago.
• 8:13
  Speaker 1: When we talked about Klarna, I think back then both of us were really excited and impressed by what Klarna had achieved with, at the time, a fairly early model I mean it was GPT-4, if I recall correctly but it was, compared to what we have now, a very rudimentary model, and what they achieved was pretty remarkable.
• 8:30
  Speaker 1: And so now it is good to have that perspective because in 60 weeks we've had roughly a little bit over two AI doublings in power, so a lot of fascinating things to unpack here.
• 8:40
  Speaker 1: So to your question of what I've seen evolve feature set increase is definitely something I think that makes sense, because if you can engage people in a way that they find pleasing, that they find useful, they'll come back more.
• 8:54
  Speaker 1: And if you have more functionality to offer, then you can go deeper.
• 8:57
  Speaker 1: So you know, if Wells Fargo is able to, for example, provide spending insights directly in their platform, that could be really useful for a lot of people, especially if you have a credit card and a bank account with Wells Fargo or maybe some other things.
• 9:11
  Speaker 1: That broader set of insights that you could get from your bank could be pretty powerful and pretty helpful.
• 9:17
  Speaker 1: It could help you make better spending decisions.
• 9:19
  Speaker 1: It could help you make better decisions with respect to investing, even and those are things that third-party apps have been doing for a while products like Mint or a number of others Rocket Money is another one that have some AI features, but this is an opportunity for a platform like a bank to bring some of that engagement back to the bank as the core platform for most people's primary financial interactions.
• 9:43
  Speaker 1: So I think that's interesting.
• 9:50
  Speaker 1: In my experience, the association community has been moving a little bit slower than I'd like in terms of member service agents.
• 9:53
  Speaker 1: Overall, people have been doing bits and pieces.
• 9:55
  Speaker 1: We have, in our own family of companies, a number of groups that are working on things that are in this space, one of which is obviously Betty, which has about 100 associations working now and growing quickly, and Betty is definitely in this realm as the knowledge agent, the expert agent in terms of all things association knowledge.
• 10:14
  Speaker 1: We've mentioned previously on this podcast.
• 10:16
  Speaker 1: We're launching something specifically for member service that deals with routing incoming asynchronous messages like emails and SMS and so forth, but you know, I'd say that we're still in super, super early innings.
• 10:29
  Speaker 1: So if you're an association that's thinking about this, saying, hey, we'd love to have something like the Wells Fargo assistant or like the Klarna assistant, you've still got plenty of time ahead of you.
• 10:40
  Speaker 1: But I wouldn't, you know, spend all year thinking about it.
• 10:42
  Speaker 1: I'd run an experiment.
• 10:43
  Speaker 1: To me, this is.
• 10:46
  Speaker 1: What's so powerful about this particular use case is it's both sides of the value equation.
• 10:50
  Speaker 1: The one side is cost reduction or efficiency improvements, but the other side is improving the value to the customer, which is the biggest thing.
• 10:58
  Speaker 1: When you see people using a service more and more, that should light up a light bulb for you.
• 11:03
  Speaker 1: It says, hey, there's something good here.
• 11:05
  Speaker 1: When we see, for example, engagement in a web-based search tool compared to a web-based knowledge agent, where the knowledge agent has literally 50x longer session times than a search tool, that should tell you something about the value you're creating.
• 11:19
  Speaker 1: It's not that it takes 50 times longer to get the information.
• 11:21
  Speaker 1: It's quite the opposite, in fact, the knowledge agent is much, much faster at getting people the information they want.
• 11:31
  Speaker 1: But rather, because people found value and it's low friction, low time to value for the customer, they come back more.
• 11:35
  Speaker 1: So if they come back more, there's more opportunities to engage more opportunities to create value and have a reinforcement cycle.
• 11:41
  Speaker 1: So I find it to be a really, really exciting area for associations to jump into, but, as I said, I think it's still super early.
• 11:49
  Speaker 3: Okay, I was going to say I'm sure we have some listeners thinking, well great, wells Fargo did it and Klarna with their 70 million and 150 million customers respectively, that's feasible for them.
• 11:59
  Speaker 3: You said it's still early stages for associations.
• 12:02
  Speaker 3: Can you contextualize what you mean by that?
• 12:04
  Speaker 3: So what would you say is currently feasible right now for a pilot project with a member service agent?
• 12:11
  Speaker 1: I think you could stand up a member service agent over the next three to six months in your association a number of different ways.
• 12:18
  Speaker 1: There are a number of tools you could use for that, using either off-the-shelf tools and just string them together with different kinds of agent frameworks.
• 12:28
  Speaker 1: You could certainly partner with companies that specialize in this, either in the association market, like our companies, or companies that are outside of the association market who do this kind of work.
• 12:39
  Speaker 1: There's companies that are focused on kind of large enterprise, like the one that you mentioned.
• 12:43
  Speaker 1: There's also a company called Decagon, and Spiera is another one, that do customer service agents kind of at the very high end of the market, and people in the association market, I think, are gonna have association specific solutions more and more.
• 12:55
  Speaker 1: Obviously what we're focused on is that, but you're gonna see more and more choice there.
• 12:59
  Speaker 1: So I think there's off the shelf stuff you deploy and you can also build something in this space.
• 13:05
  Speaker 1: I think this is a great opportunity for an experimentation round where you could do something really, really small.
• 13:11
  Speaker 1: Don't try to boil the ocean and solve all customer service or member service inquiries.
• 13:15
  Speaker 1: Focus on a pain point.
• 13:17
  Speaker 1: For example, many associations have a highly seasonal volume of activity that comes in around their annual conference.
• 13:26
  Speaker 1: So prior to the annual conference they might have a fairly reasonable inflow of inquiries, but right before and during and after the conference they might have, let's say, a 30 or 60 day window of time on the calendar where it's just completely crazy.
• 13:41
  Speaker 1: Well, what if we could put in place a great member slash event service AI that could help field 50, 60, 70% of those questions that are fairly repetitive?
• 13:52
  Speaker 1: That's a super achievable thing, and within the narrower context of events, the domain of questions usually are far narrower.
• 13:59
  Speaker 1: So I think that's an easy thing to go experiment with.
• 14:03
  Speaker 1: Overall, what I'd say is, to me, the thing that you have to remember is yes, you're an association, you're not Wells Fargo.
• 14:09
  Speaker 1: Yes, you're an association, you're not Amazon.
• 14:12
  Speaker 1: But the technologies have come down so much in cost, they're so much more accessible and they're so much more powerful that not only can you do this as an association, but you're going to be expected to.
• 14:23
  Speaker 1: Your members don't care that you're not Wells Fargo or Amazon or Netflix or Karnak.
• 14:28
  Speaker 1: They just expect the same quality of experience from you that they expect from their largest consumer experiences.
• 14:34
  Speaker 1: And it may not be fair, but fairness doesn't really matter.
• 14:37
  Speaker 1: If the eye of the consumer, the expectation and the bar has been set at this level, they're going to expect it soon enough from you, so might as well get ahead of that and provide them something slightly before they might expect it from the association.
• 14:50
  Speaker 3: And then you can provide that additional value, those insights, things that really, really help your members in their profession or industry to further create that value-based relationship.
• 15:00
  Speaker 1: You know, I'd say this is also a great time to reinforce a concept we've talked about on the pod Mallory a number of times, which is how do you prioritize your energy?
• 15:09
  Speaker 1: Your energy might be classified as human labor, like your team's time, your volunteers' time, also your dollars.
• 15:16
  Speaker 1: That's part of the energy flow, right?
• 15:18
  Speaker 1: Where do you invest?
• 15:19
  Speaker 1: And a lot of people are saying, well, our infrastructure is so terrible.
• 15:24
  Speaker 1: We've got ancient systems.
• 15:36
  Speaker 1: You know, we've got a really old MS and we've got to replace that thing or an old LMS decent chance they'll still work this year and next year.
• 15:43
  Speaker 1: And the question is instead of replacing a major system like that, which is, you know, significant effort, sometimes takes 18 to 24 months to fully go through a process like that, sometimes longer.
• 15:53
  Speaker 1: What if you didn't do that?
• 15:55
  Speaker 1: Right, and you said, hey, we're going to deprioritize some of those classical association IT things and instead invest a few dollars and some time, time being the most important ingredient to experiment with this use case.
• 16:07
  Speaker 1: Right, go figure out how to make a member services agent work for you as your priority.
• 16:11
  Speaker 1: Let's say you did that for the next six months and you hit the pause button on a pending AMS selection or AMS implementation.
• 16:17
  Speaker 1: The amount of value you create for members from this technology is so much higher.
• 16:23
  Speaker 1: It's dramatically different than what an internal system replacement might yield.
• 16:27
  Speaker 1: So again, I'm not suggesting that you work with an unstable, shaky foundation, with ancient technology forever.
• 16:35
  Speaker 1: But if you have to choose between something like this and infrastructure improvements that, frankly, nobody's going to really notice on the external side, I'd focus on this and maybe you don't have to choose between the two in your group and your association, but most people do have to choose between those kinds of priorities.
• 16:50
  Speaker 1: So I decided it'd be a good time to remind people that you can attack these things if you're willing to say no to stuff.
• 16:55
  Speaker 1: You just have to draw a line in the sand and say you know what.
• 16:57
  Speaker 1: We're going to put a pause on all these old, classical types of systems and projects and we're going to keep them running, obviously, but we're not going to invest big dollars and big energy in these older technologies.
• 17:08
  Speaker 1: Instead, we're going to focus on making these new AI things work.
• 17:11
  Speaker 1: Last thing I'll say about that is once you do these kinds of new projects, you'll actually reframe what you think you need.
• 17:18
  Speaker 1: When it comes time to replace some of that infrastructure, you might think you know what you want in that next generation AMS, but frankly you probably don't.
• 17:25
  Speaker 1: When you build an AI technology or two and deploy it into production, you will get so much better of a sense of where your members want you to go, and that might change the requirements for what that new AMS is going to do.
• 19:28
  Speaker 1: And then the last thing related to that, by the way, is the AMS vendors are also figuring that out.
• 19:33
  Speaker 1: Whether you're talking about a traditional AMS vendor in the space or some other type of solution, everybody in these types of database applications is working really hard right now to figure out how to AI enable their systems, so I'd give them a little bit of time too.
• 19:47
  Speaker 1: I think you'll have better choice and you'll have better visibility into what you're actually going to get.
• 19:53
  Speaker 3: Mm.
• 19:53
  Speaker 3: Hmm, hearing you talk about that.
• 19:55
  Speaker 3: We've just hit the one year mark of moving to Atlanta and it made me think of our experience last year of moving into an apartment we had never seen and trying to furnish it before we were there and realizing sometimes you just need to be there physically in the space before you realize, oh OK, we need this size couch, we need a TV right here.
• 20:14
  Speaker 3: It makes me think of you or of associations specifically trying to replace their AMS and then perhaps getting to that point and thinking, oh gosh, now with AI, we realize we need all these other features and all this other infrastructure.
• 20:26
  Speaker 3: So I think it's a really valid point, amit, and I want to talk a little bit about this pilot project that you mentioned.
• 20:32
  Speaker 3: I can definitely resonate, having been the primary point person at Sidecar who would take in a lot of inquiries approaching digital.
• 20:40
  Speaker 3: Now the conference, before the event and after the event.
• 20:43
  Speaker 3: However, I would think if you came to me and said we're going to, you know, roll out this AI agent and there's going to be potentially no human intervention, right, we're just going to roll it out, I would be intimidated by that and, to be honest, scared that it wouldn't work.
• 20:58
  Speaker 3: So and I'm sure our association listeners feel the same way what is your thought on the pilot project of trying to roll out an agent that has no human intervention versus trying to roll out an agent that does the routing, like you kind of briefly mentioned earlier?
• 21:14
  Speaker 3: Is no human intervention the goal?
• 21:16
  Speaker 3: Talk me through that.
• 21:18
  Speaker 1: I don't think that's the goal at all in almost all cases.
• 21:21
  Speaker 1: I don't think that's the case either for Klarna or Wells Fargo, as I understand their models.
• 21:24
  Speaker 1: It's more about making available instant and high-quality responses for most things but at the same time being able to interact with a human agent when appropriate.
• 21:34
  Speaker 1: And this might sound like we're trying to find a silver lighting in terms of the employment side of the equation here in saying that the humans can focus on higher value activities.
• 21:45
  Speaker 1: That's oftentimes consultants speak for, saying they're going to be laid off.
• 21:49
  Speaker 1: In reality there's some of that that might happen in the association market, probably not so much, but in the broader market, if you have 10,000 people in a call center, maybe you don't need 10,000, maybe you need 2000,.
• 22:01
  Speaker 1: But you need your best 2000 people.
• 22:03
  Speaker 1: So there's some issues there for sure, when you think about that across an entire sector.
• 22:07
  Speaker 1: But for the association world, I think of it this way that you know your member services folks, your event services folks.
• 22:13
  Speaker 1: They have a lot more to offer than just answering rote inquiries, asking, like people say hey, when do I need to register?
• 22:23
  Speaker 1: Where can I check in?
• 22:23
  Speaker 1: Can I bring my spouse to this particular function?
• 22:25
  Speaker 1: What's the guest registration fee.
• 22:26
  Speaker 1: You know where can I find this particular article, All these kinds of basic help, desk questions, and AI can nail all those things and those people who are asking those questions are going to be happier with a better answer.
• 22:36
  Speaker 1: That's nearly instant.
• 22:37
  Speaker 1: But those member services reps, those event folks can have conversations with people, can learn more from those members, can take time to actually have live synchronous phone calls and video calls, to really be the concierge, to help provide an experience, so that it feels like you're checking into the four seasons when we come to the Euro event rather than checking into the Red Roof Inn.
• 23:00
  Speaker 1: So you know, the whole idea is that you want to level up the caliber of service and the quality of service that you provide, and you can do that.
• 23:08
  Speaker 1: You can, you know, punch way above your weight class by using AI to take care of the rote stuff.
• 23:14
  Speaker 1: Coming back to your point, that's where this concept in agentic systems, called human in the loop, is so critical.
• 23:20
  Speaker 1: That's for key decision-making, but it's also for escalation, where the AI should be trained, and can easily be trained, to be smart enough to not try to take on everything right, when you can tell the AI hey, for these three or four different kinds of inquiries.
• 23:33
  Speaker 1: We can answer it in these different ways.
• 23:36
  Speaker 1: These are the ways.
• 23:41
  Speaker 1: These are the tools that are available.
• 23:42
  Speaker 1: You might have knowledge agency.
• 23:43
  Speaker 1: You might have capabilities around database lookups.
• 23:44
  Speaker 1: There might be two or three different things that the agent is really good at, but we can tell the agent to err on the side of getting a human involved.
• 23:50
  Speaker 1: If there's any question as to the value of the quality or independent of the objective purpose of the call or the inquiry, let's just say that the AI detects a tone of frustration.
• 24:04
  Speaker 1: Let's say that there's two or three iterations of emails and the AI detects that the person's just not particularly happy.
• 24:10
  Speaker 1: You know, AI is really really good at reading into the emotion from just plain text, and that's even more true with audio.
• 24:18
  Speaker 1: If you were to do this with audio capabilities and then to be able to detect that and say, hey, you know what I think?
• 24:24
  Speaker 1: Mallory is not super happy with me right now, it's the AI.
• 24:27
  Speaker 1: I'm going to forward this message to somebody else, for a human in that case, right To help Mallory out.
• 24:37
  Speaker 3: Yep, as you said, ai is pretty good at detecting sentiment.
• 24:40
  Speaker 3: It's not something you think it would be good at, but, like word choice and especially if it has more information through audio video, it does a pretty decent job at it.
• 24:49
  Speaker 1: Well, you can also tell, like, how someone, if someone's coming back two or three times and they feel like they're asking the same thing repetitively and they they like, or even like using a simple like phrase, like as I said.
• 25:00
  Speaker 1: Right, when I say as I said, I feel like I'm repeating myself, and I find myself doing that with customer service reps in that, you know, kind of ongoing, infinite loop of emailing people who really don't have a great idea of what I'm after but are there to kind of, you know, address my issue in some way.
• 25:18
  Speaker 1: So I think there's a lot of opportunity here.
• 25:19
  Speaker 1: But yes to your point, mallory, you make a really important one.
• 25:22
  Speaker 1: I wouldn't try to like just hand this over to the robot and say good luck and hope to see in the future, because I don't think that's a complete solution.
• 25:29
  Speaker 1: I think you have to level up what the humans do in this equation.
• 25:39
  Speaker 3: My last question here, amit, is that the ability to process sensitive data locally before any cloud interaction is a major privacy advancement and is, of course, essential for things like banking or buy now, pay later.
• 25:48
  Speaker 3: When you're dealing with people's payment information, do you think that this is a necessity for associations?
• 25:55
  Speaker 1: I think it's an important concept that associations should be aware.
• 25:59
  Speaker 1: A lot of people make the assumption they'll say to me, for example, oh, I love the idea of whatever the application is, but they'll say, I have all this sensitive data, or it might not be sensitive like patient data or banking data or something like that, but it might be just.
• 26:13
  Speaker 1: We have a lot of content in our private knowledge repository.
• 26:17
  Speaker 1: We don't want to send that to chat, gpt or to cloud.
• 26:19
  Speaker 1: We just don't trust them and that's a reasonable concern.
• 26:22
  Speaker 1: But people make the assumption that that's a dead end.
• 26:24
  Speaker 1: Right, that that's the end of the conversation.
• 26:27
  Speaker 1: Whereas there's both ways of doing private deployment in the cloud of your own models where you could say, hey, I'm going to run Lama or a number of other models in a private cloud deployment and to what you specifically brought up, these models are shrinking, their capabilities are growing and they're shrinking in size.
• 26:43
  Speaker 1: You can actually run them locally on a phone, in a web browser and in ways that also provide additional privacy.
• 26:52
  Speaker 1: So I don't know exactly what Wells Fargo is doing, but Apple's strategy around this sounds similar.
• 26:56
  Speaker 1: I don't know exactly what Wells Fargo is doing, but Apple's strategy around this sounds similar.
• 26:59
  Speaker 1: What they'll do is on the phone itself, the LLM that's running locally a very, very small LLM will try to get the essence of what you've asked and then determine if it can answer the question locally or if it will need to promote a portion of that information.
• 27:13
  Speaker 1: Abstracting out anything personally you may have shared with, just the general concept get higher order knowledge from a remote LLM, also operating in a secure manner, and then pull that back to the local LLM to synthesize a response that then reintroduces your personal information.
• 27:28
  Speaker 1: But the personal information never really left the local environment.
• 27:31
  Speaker 1: The most important thing for all of you associations to note is that you have options.
• 27:37
  Speaker 1: You have ways of doing secure, private AI inference.
• 27:41
  Speaker 1: There's a number of ways to do this and you can even do it locally on-device, and that's going to continue to be the case.
• 27:45
  Speaker 1: There's all this growing collective body of language models that you can run that are smaller and smaller, that run extremely efficiently on desktop computers and laptops and even on phones.
• 27:59
  Speaker 3: Well, you really set me up perfectly there, amit.
• 28:01
  Speaker 3: To go to topic two, which is Microsoft's 5.4 models small language models with big reasoning power.
• 28:08
  Speaker 3: So Microsoft just released a new family of 5.4 models, including these great names 5.4 Reasoning, 5.4 Reasoning Plus and 5-4 Mini Reasoning.
• 28:19
  Speaker 3: Those aren't too bad.
• 28:20
  Speaker 3: I've seen worse, I would say, come out of OpenAI.
• 28:22
  Speaker 3: The 5-4 Reasoning models are very much a part of the broader trend toward reasoning or thinking models, called either-or, that can perform advanced reasoning, an ability to analyze complex scenarios, apply structured logic and solve problems in a way that resemble human thinking.
• 28:39
  Speaker 3: So, to break down that 5-4 family, we've got 5-4 reasoning, which is a 14 billion parameter open weight model, fine-tuned for complex reasoning, math, science and coding tasks.
• 28:51
  Speaker 3: It uses supervised fine-tuning with high quality, curated data enabling it to generate detailed reasoning chains and match or surpass much larger models on benchmarks.
• 29:01
  Speaker 3: Then we've got 5.4 Reasoning Plus, which builds on that 5.4 Reasoning model that I just mentioned, further trained with reinforcement learning and able to use 1.5x more tokens for even higher accuracy.
• 29:15
  Speaker 3: It matches or exceeds the performance of much larger models like DeepSeek R1, which we've covered on the podcast, which, as a note, has six hundred and seventy one billion parameters compared to the 14 billion parameters of this model and OpenAI's O3 Mini on several key benchmarks.
• 29:33
  Speaker 3: And then we've got 5.4 Mini Reasoning, a compact three point8 billion parameter model optimized for mathematical reasoning and educational use, suitable for deployment on resource-limited devices like mobile phones and edge hardware.
• 29:47
  Speaker 3: So Amit was already kind of gearing up to mention a lot of the practical benefits of smaller models.
• 29:52
  Speaker 3: They can run locally on PCs, mobile devices and edge hardware.
• 29:56
  Speaker 3: They're also designed for offline use in co-pilot and PCs and, of course, there are lower computational requirements that make them more accessible and cost effective.
• 30:05
  Speaker 3: All three of these models are openly available under permissive licenses and they can be accessed through Azure AI Foundry and Hugging Face.
• 30:14
  Speaker 3: So, amit, what are your initial thoughts on the 5-4 family of models?
• 30:21
  Speaker 1: Well, first of all, let's spell this out for folks, because 5-4 might be pronounced or spelled differently.
• 30:29
  Speaker 1: It's P-H-I dash number four, and that's, I think, part of what makes it so hard to pronounce is there was five, three, and then it's almost like you're saying five, but yeah, five, four you're right but yeah, that's what I always think.
• 30:43
  Speaker 1: When I first started hearing this but it's p-h-i-p-h-i-dash-four my thoughts are, wow, this is really exciting.
• 30:51
  Speaker 1: So what you said that's part of many really interesting comments is that, not across all benchmarks, but across several important benchmarks.
• 30:59
  Speaker 1: 5.4 Reasoning Plus, which I'll talk about in more detail in a second, matches or exceeds DeepSeek R1, which, if you recall, r1 shook the world back in January February timeframe.
• 31:11
  Speaker 1: People freaked out because it was as performant as OpenAI's then most powerful AI reasoning model, the O1 and O3 mini capabilities.
• 31:22
  Speaker 1: So here's the deal.
• 31:23
  Speaker 1: Here's the way I think about this is that this is a tiny model 14 billion parameters and today's model size is really small, capable of being run, probably on some phones, but definitely on a PC or a Mac on some phones, but definitely on a PC or a Mac.
• 31:39
  Speaker 1: And one of the ways they're able to make it perform, as well as by giving it more time to think when you ask the question.
• 31:42
  Speaker 1: So, reasoning slash thinking models it sounds like some new category of model.
• 31:47
  Speaker 1: It's this really cool, complex thing.
• 31:49
  Speaker 1: In reality, actually, it's not all that different from the models we've had in the past.
• 31:54
  Speaker 1: It's essentially saying hey, model, I want you to spend time to think about this problem, to be able to spend time just like going deeper and thinking through the problem, breaking it down step by step into small chunks and then compiling the results of each of those sub steps into an answer.
• 32:12
  Speaker 1: Another way to think about it is that the model is able to revise something that it thought about previously.
• 32:19
  Speaker 1: When we interviewed Ian Andrews from Grok Grok with a Q, he used an analogy that I love and I've repeated this a number of times which is that it's like giving the model a backspace key where the model can edit its prior response as opposed to simply writing as fast as it can.
• 32:32
  Speaker 1: So that's what these reasoning and thinking models do, and the way to think about it for you is that you know you have access in a 14 billion parameter model is something that previously, literally two months ago, required a 671 billion parameter model, which makes it possible to run all sorts of workloads on smaller and smaller devices.
• 32:52
  Speaker 1: So, and then the mini model is a much smaller model.
• 32:56
  Speaker 1: It's, you know, a quarter the size of the main 5.4 model, but it also is trained to use more compute resources when you ask questions, so it can reason through problems, and that model is suitable for running on edge hardware, which would include phones and other devices that have much smaller memory and computational ability.
• 33:16
  Speaker 1: So I find all of this to be super exciting.
• 33:18
  Speaker 1: It just reinforces the trend line of what we talked about.
• 33:21
  Speaker 1: I've been saying this for a few years now that I'm actually more excited about the compact, small, super efficient, lightweight models becoming smarter than I am about frontier models like Cloud37, sonnet, gemini, p5 Pro.
• 33:33
  Speaker 1: Those are awesome.
• 33:35
  Speaker 1: The fact that these super powered models that run only in the cloud are getting smarter is, of course, exciting, but the fact that these small, really efficient models can do so much more is just stunning.
• 33:46
  Speaker 1: I mean, what you have in Phi for Reasoning Plus is better than what you had six months ago in the very best models in the world, and you now can run that on your computer for free.
• 33:57
  Speaker 1: That's a pretty stunning advancement in a very short number of months.
• 34:01
  Speaker 3: Mm, hmm, I know you, you and I we like to geek out about all the minute details of all these models, because that's part of our job and I think we just enjoy learning about it.
• 34:11
  Speaker 3: But you mentioned the trend lines and I always think it's important with these model conversations to zoom out a little bit and look at the bigger picture.
• 34:17
  Speaker 3: So what you just said is really profound.
• 34:19
  Speaker 3: But what do you think this trend line means with the smaller, more powerful models, specifically for associations?
• 34:25
  Speaker 1: think this trend line means with the smaller, more powerful models, specifically for associations.
• 34:29
  Speaker 1: Well, I think you know.
• 34:32
  Speaker 1: Going back to the last conversation, if there are certain types of data that you have in your organization that you're not comfortable sharing with any of the other any of the model providers Anthropic or Google or OpenAI you can take this FI model.
• 34:42
  Speaker 1: You can run it, even on your own physical hardware if you want to, or you can run it in a virtual private cloud environment one of the major cloud providers, where it's completely contained and as secure as anything else any other computer program that you run.
• 34:57
  Speaker 1: Most people have gotten pretty comfortable about secure private cloud deployment where, in a cloud like Google or AWS or Azure, you can set up resources that are 100% secured and private and, by most measures, far more secure than computers.
• 35:13
  Speaker 1: You run physically, like on your own physical hardware and run whatever programs you want.
• 35:19
  Speaker 1: Right, traditional computer programs and an AI model is just a computer program.
• 35:23
  Speaker 1: It works differently than a traditional computer program but it is a computer program.
• 35:25
  Speaker 1: It works differently than a traditional computer program but it is a computer program and you can run it on hardware.
• 35:28
  Speaker 1: You have absolute control over right.
• 35:31
  Speaker 1: So if you have that ability.
• 35:33
  Speaker 1: It opens up a class of applications that associations have often told me that they are uncomfortable with, which is things related to clinical data that they might've access to if they're a healthcare association, if they're a financial association association, maybe there's benchmarking data that they receive from some of their members that they don't feel comfortable passing to open ai, or anybody else for that matter.
• 35:53
  Speaker 1: These kinds of applications now can be brought into a totally secure environment and run with incredible accuracy.
• 36:01
  Speaker 1: So it opens up a ton of doors.
• 36:03
  Speaker 1: If you're worried about passing your content to an AI system because you worry that they'll somehow subsume your content into their corpus of training data that they'll use for future models which, by the way, little bit skeptical, even if the legal agreement says that it can't be used in certain ways, you might say, okay, well, I'd rather just be totally sure and I'm just going to run this type of model on my own.
• 36:32
  Speaker 1: So it opens up a lot of doors.
• 36:35
  Speaker 1: The other thing to think about is independent of the privacy security conversation.
• 36:39
  Speaker 1: Smaller models run faster with less energy, less resources, and are cheaper to run.
• 36:46
  Speaker 1: So if you have a little model like this, that's as smart as what was previously requiring a giant model and you can now run it with a really cost-effective small model.
• 36:56
  Speaker 1: You can do more, right?
• 36:57
  Speaker 1: You might have a hundred million documents that go back to the beginning of your association's formation and you might like to analyze them in all sorts of new ways that you previously would have thought to be totally unattainable.
• 37:09
  Speaker 1: Because you might have said well, we have this idea in mind, where you know, we have a million documents of every paper we've ever published and every opinion that's ever been written on every paper, and we would like to ask certain questions of every one of those papers.
• 37:22
  Speaker 1: Right, have a detailed analysis done of each of those papers in order to capture, like some metadata or some structured insight from all those papers.
• 37:34
  Speaker 1: And let's just say, a year ago you thought about this idea.
• 37:35
  Speaker 1: It was a cool idea.
• 37:36
  Speaker 1: But then you're like, yeah, it would cost like between two and $3 to do that per paper and we have a couple million pieces of content.
• 37:40
  Speaker 1: That's just not going to scale.
• 37:42
  Speaker 1: But now if you have a 97, 98% cost reduction, which is basically what you get here, that might cost you a few thousand dollars, right?
• 37:49
  Speaker 1: Or maybe $10,000.
• 37:50
  Speaker 1: You might say you know what?
• 37:51
  Speaker 1: That's actually pretty reasonable.
• 37:52
  Speaker 1: And if you wait six more months, it might be basically zero.
• 37:56
  Speaker 1: So the cost curve compression is really compelling, as well as the privacy it opens up the door to just use way, way more of this inference that we keep talking about opens up the door to just use way, way more of this inference that we keep talking about.
• 38:12
  Speaker 3: Amit, this is just not something I'm up to speed on, so I'll ask, in case we have some listeners as well that have the same question.
• 38:14
  Speaker 3: But when you talked about running models privately in your own cloud environment versus running them locally, is one as secure as the other, or is one more secure than the other?
• 38:23
  Speaker 1: You know there's pros and cons to each approach.
• 38:25
  Speaker 1: So let's say that I have the old school way of doing it, that in my office I have a computer server and I run that physical server.
• 38:34
  Speaker 1: I am responsible for site security to make sure no one physically enters that location.
• 38:39
  Speaker 1: I'm responsible for network security.
• 38:41
  Speaker 1: I'm responsible for the whole thing.
• 38:43
  Speaker 1: Right, and so traditionally IT departments and associations did that.
• 38:46
  Speaker 1: They would run, you know, they'd have server rooms where they'd have you know, racks of these servers and they'd run them and they were responsible for all of that.
• 38:53
  Speaker 1: You know the site security and the digital security and I would argue that, generally speaking, that is going to be less secure than a modern cloud provider that have rigorous, tight, military-grade physical security around their sites way, way more than any association is ever going to have.
• 39:11
  Speaker 1: And from a digital security perspective, implementing your own approach to cyber security is really important for your own resources.
• 39:19
  Speaker 1: But cloud service providers tend to have really really good built-in security architectures that are a good starting point.
• 39:26
  Speaker 1: So I generally am a skeptic of anyone who tells me that they can run a more secure local environment than a well-implemented cloud environment.
• 39:35
  Speaker 1: I think most security experts would tend to agree with that.
• 39:38
  Speaker 1: Certainly for smbs, like small to medium-sized businesses, which associations fit into, there are exceptions to every statement.
• 39:44
  Speaker 1: Obviously, there are some organizations who would argue you know, we have even stronger site security and visual security than any cloud provider.
• 39:51
  Speaker 1: Certain information we have justifies this, and sure, there's always exceptions, but for the vast, vast majority of our listeners in this market, cloud-based deployment is going to work really, really well.
• 40:03
  Speaker 1: It just has to be well thought out.
• 40:05
  Speaker 1: You could create a cloud-based resource that you leave a wide open backdoor to without thinking about where you're just like oh, I'm just going to post a password to my website on Reddit and let anyone log in.
• 40:16
  Speaker 1: I mean, that sounds totally stupid, but the reality is there's all sorts of human factors that go into compromising security all the time.
• 40:23
  Speaker 1: That can affect you either way.
• 40:26
  Speaker 1: Local inference on a device that an end user uses, though, is actually really interesting as a complement to that, because, let's say again in the Wells Fargo case, I'm talking to my banking assistant on my phone and I have to get all this information about and I'm talking about, like, my salary or my investment strategy and my net worth and all this other information.
• 40:45
  Speaker 1: Maybe that information isn't really what that local AI about, like my salary or my investment strategy and my net worth and also their information.
• 40:46
  Speaker 1: Maybe that information isn't really what that local ai needs help with.
• 40:50
  Speaker 1: Maybe it needs to help.
• 40:51
  Speaker 1: It needs to help reasoning through some general ideas.
• 40:54
  Speaker 1: So, then, guide what the local llm does.
• 40:56
  Speaker 1: So, instead of sharing my salary and my net worth from the local conversation, let's say with the remote ai, what it does is, hey, I'm with a consumer, they're working through these kinds of problems, Can you give me some general guidance on A, B and C?
• 41:10
  Speaker 1: And then the remote LLM throws way more compute at it and comes up with a stronger answer, feeds it back to the local LLM and says, hey, here's the direction you should go with.
• 41:19
  Speaker 1: And then the local LLM then takes that private data, infuses it back into the answer from the remote LLM and gives me an experience that's really really high quality in my phone right and my personal data never left my phone.
• 41:32
  Speaker 1: And the same thing can be done for healthcare, and that could be a compliment that associations can take advantage of.
• 41:38
  Speaker 1: Let's say that you're a medical association and you want to provide capabilities for your members to have chats with you that are specific, down to the case level on a particular patient they're working with.
• 41:50
  Speaker 1: You probably don't want any of that.
• 41:52
  Speaker 1: You know healthcare data to ever come back to you, right?
• 41:55
  Speaker 1: So what if you have local LLM that did part of the processing and, just like I said, abstracted out the problem removing patient-specific data, then got a knowledge agent that has a tremendous amount of content and compute capability to formulate an answer and then re-infuse that back with the local data?
• 42:12
  Speaker 1: There's ways to do that as well, and there's applications for associations for sure.
• 42:18
  Speaker 3: Yep, I've mentioned my husband's in healthcare and he is just waiting for the day that he has exactly what you mentioned, where he could drop in some patient info and get that resolved with better accuracy perhaps than he could have found doing some searches online.
• 42:32
  Speaker 1: I was just going to say one other thing, mallory, that I think our listeners might find interesting.
• 42:35
  Speaker 1: For those of you that have heard me talk about the Acquired podcast before I heard Mallory mention it, we're big fans of the work that those guys do.
• 42:42
  Speaker 1: That's a long form like business history style podcast.
• 42:46
  Speaker 1: There's a new episode they just dropped as of the recording of this podcast in spring of 25.
• 42:52
  Speaker 1: They dropped an episode on Epic.
• 42:54
  Speaker 1: Epic is a software company in the healthcare space and what's super interesting?
• 43:00
  Speaker 1: There's a lot of interesting things about that particular episode, but there's a lot of talk about AI and a company like Epic and what they're going to do in the healthcare field, and they're by far the dominant player in providing tools like MyChart that patients use and the EMRs, ehrs and billing systems and so forth that hospitals and health systems use, and certainly those kinds of tools will likely soon feature AI capabilities for doctors to use.
• 43:24
  Speaker 1: So you might ask the question well, what is the role of the association?
• 43:27
  Speaker 1: How do we provide value when the hospital might have an AI bot built into their secure EHR EMR system?
• 43:35
  Speaker 1: And the answer in my mind is to complement that where you have certain things that nobody else has, particularly your content, and over time you can capture other forms of experiential data that would be unique to you, that could be complementary to what people get out of an EHR EMR.
• 43:52
  Speaker 1: So I think there's actually a very bright story.
• 43:54
  Speaker 1: Whether those things can interoperate and integrate with the experience that a doctor or a medical practitioner member may have is a big question, because companies like Epic, specifically, are famously very guarded about integrations.
• 44:07
  Speaker 1: But I think there's an opportunity here for many associations in a similar capacity outside of healthcare as well, to do things where you can complement a line of business system that your members use every day.
• 44:19
  Speaker 3: I'm going to have to tell Bailey about that episode.
• 44:21
  Speaker 3: I've actually gotten him on the Acquired podcast.
• 44:23
  Speaker 3: He listened to the Costco episode as well and really enjoyed it, so he will certainly enjoy the Epic one, amit.
• 44:28
  Speaker 3: My last question was about this trend line.
• 44:31
  Speaker 3: Again, it seems like when you zoom out, we're seeing smaller and smaller models become more and more powerful In your mind.
• 44:40
  Speaker 3: In the next five years, if you could zoom out, do you think we'll be looking at tons of models that are, you know, millions of parameters and like more powerful than we could possibly imagine, I guess.
• 44:51
  Speaker 3: Are we trending toward creating as small and small models as we can, or is there a place for the giant ones and the small ones as well?
• 45:01
  Speaker 1: I think it's both.
• 45:01
  Speaker 1: I think that you know if you can further compact these models down to the point where, let's say, we could come up with 12 months from now.
• 45:08
  Speaker 1: You said five years.
• 45:09
  Speaker 1: I don't know that I can think that far out.
• 45:11
  Speaker 1: I think that's next year.
• 45:12
  Speaker 1: You're right, that was a hard question say that an equivalent model to the 5.4 reasoning model is available in 100 million parameter or 200 million parameter model.
• 45:31
  Speaker 1: Right, like you know, 10x or even 50 or 100x smaller than the current 5.4 model.
• 45:35
  Speaker 1: That could run in a web browser, that could run on really, really lightweight phones not even like an iPhone 16, but something much smaller than that.
• 45:43
  Speaker 1: And so if that's the case, then now you have really high end reasoning capability in a super compact form.
• 45:49
  Speaker 1: You know you could have it running pretty much everywhere.
• 45:51
  Speaker 1: You can have that capability in your earbuds, you know.
• 45:53
  Speaker 1: So those capabilities becoming smaller and smaller is good.
• 45:58
  Speaker 1: I think what you're also going to see is that the state of the art frontier models will keep getting smarter and smarter.
• 46:05
  Speaker 1: You know, one of the stats that I think has been missed by a lot of folks is I think it was the most recent O3 release by OpenAI, and Gemini 2.5 Pro and Cloud 3.7 in extended thinking motor are kind of similar in terms of where they're at, but this benchmark shows that O3 is approximately on par with about the 80th percentile of performance of PhDs across all disciplines.
• 46:27
  Speaker 1: So let's unpack that for just a second 70, 80 percentile of PhDs.
• 46:31
  Speaker 1: That means that if you put the average PhD, which is nose-louch, typically right in the middle, that's the 50 percentile.
• 46:38
  Speaker 1: So O3 is at the 70 to 80 percentile of performance of PhDs, and not just in one field but across a number of different disciplines, ranging from history to philosophy to various forms of science and engineering.
• 46:50
  Speaker 1: So it's pretty stunning what you have, and that's an O3, which is a big, heavy, expensive reasoning model.
• 46:56
  Speaker 1: But if you can have that capability distilled down into smaller and smaller and smaller models, even if these models didn't get any smarter right, that's pretty darn smart.
• 47:06
  Speaker 1: And if you make it super, super fast, small, cost-effective, energy efficient, the doors that open up are really compelling.
• 47:16
  Speaker 3: That is a great place to wrap up this episode.
• 47:18
  Speaker 3: What would you do if you had all those PhDs at your fingertips running on your phone and your earbuds?
• 47:24
  Speaker 3: I don't know, we might be there pretty soon.
• 47:26
  Speaker 3: Everybody, thank you for tuning in to today's episode and we'll see you all next week.
• 47:33
  Speaker 2: Thanks for tuning in to Sidecar Sync this week.
• 47:35
  Speaker 2: Looking to dive deeper?
• 47:37
  Speaker 2: Download your free copy of our new book Ascend Unlocking the Power of AI for Associations at ascendbookorg.
• 47:45
  Speaker 2: It's packed with insights to power your association's journey with AI.
• 47:49
  Speaker 2: And remember Sidecar is here with more resources, from webinars to boot camps, to help you stay ahead in the association world.
• 47:56
  Speaker 2: We'll catch you in the next episode.
• 47:58
  Speaker 2: Until then, keep learning, keep growing and keep disrupting.