Latest / Sidecar Sync / Generating Instant Avatars with HeyGen & Anthropic’s New ‘Computer Use’ Feature | 53

Transcript

Generating Instant Avatars with HeyGen & Anthropic’s New ‘Computer Use’ Feature | 53 · Sidecar Sync

• 0:00
  Speaker 1: Now we're shifting to the point where the computers can understand us.
• 0:03
  Speaker 1: As part of understanding us and gaining these new capabilities, they understand our world too.
• 0:08
  Speaker 1: Welcome to Sidecar Sync, your weekly dose of innovation.
• 0:14
  Speaker 1: If you're looking for the latest news, insights and developments in the association world, especially those driven by artificial intelligence, you're in the right place.
• 0:23
  Speaker 1: We cut through the noise to bring you the most relevant updates, with a keen focus on how AI and other emerging technologies are shaping the future.
• 0:31
  Speaker 1: No fluff, just facts and informed discussions.
• 0:34
  Speaker 1: I'm Amit Nagarajan, chairman of Blue Cypress, and I'm your host.
• 0:38
  Speaker 1: Hey everybody, and welcome to the Sidecar Sync, your source of all things AI for the association community.
• 0:47
  Speaker 1: We're excited to be here again.
• 0:48
  Speaker 1: My name is Amit Nagarajan.
• 0:51
  Speaker 2: And my name is Mallory Mejiaz.
• 0:53
  Speaker 1: And we are your hosts.
• 0:54
  Speaker 1: We can't wait to get into our topics for today, which, as usual, are interesting things happening in the world of AI, but how they apply specifically to the world of associations and nonprofits.
• 1:04
  Speaker 1: Before we dive in, let's take a moment to hear a quick word from our sponsor.
• 1:09
  Speaker 2: Digital Now is your chance to reimagine the future of your association.
• 1:13
  Speaker 2: Join us in the nation's capital, washington DC, from October 27th through the 30th, for Digital Now, hosted at the beautiful Omni Shoreham Hotel.
• 1:22
  Speaker 2: Over two and a half days, we'll host sessions focused on driving digital transformation, strategically embracing AI and empowering top association leaders with Silicon Valley-level insights.
• 1:33
  Speaker 2: Together, we'll rethink association business models through the lens of AI, ensuring your organization not only survives but thrives in the future.
• 1:41
  Speaker 2: Enjoy keynotes from world-class speakers joining us, from organizations like Google, the US Department of State and the US Chamber of Commerce.
• 1:49
  Speaker 2: This is your chance to network with key association leaders, learn from the experts and future-proof your association.
• 1:57
  Speaker 2: Don't just adapt to the future, create it at Digital Now.
• 2:01
  Speaker 2: As you all heard from our sponsor, digital Now is upon us next week.
• 2:09
  Speaker 2: October 27th is the official kickoff date.
• 2:11
  Speaker 2: Amit, how are you feeling about that?
• 2:14
  Speaker 1: I'm feeling fantastic.
• 2:15
  Speaker 1: It's so exciting.
• 2:16
  Speaker 1: We've got record-breaking registrations, we've got an awesome venue, amazing speakers lined up.
• 2:22
  Speaker 1: It's going to be a lot of fun.
• 2:24
  Speaker 2: It's been so neat to see it all come together.
• 2:27
  Speaker 2: I'm sure a lot of our listeners can relate.
• 2:29
  Speaker 2: Events can be very time intensive to plan.
• 2:32
  Speaker 2: At the beginning you don't see how it's all going to come together in the end.
• 2:36
  Speaker 2: But, being a few days out, I'm really excited to see our audience, to see all the relationships we've built throughout the last few years and to celebrate the biggest Digital Now we've ever had.
• 2:47
  Speaker 1: It's going to be awesome, yeah, and when you run an event for association people and knowing that association people are professionals in event planning, it raises the bar a little bit in terms of expectations.
• 2:58
  Speaker 1: But it's also awesome because people who attend the event tend to know how challenging it can be to pull together an event even at the scale of something like Digital Now, compared to many of the associations who listen into this podcast or watch us on YouTube have events that are 10 or 50 times the size of Digital Now.
• 3:15
  Speaker 1: But you really want to have every detail nailed down and AI can help with some of that.
• 3:20
  Speaker 1: But at the end of the day, it's people helping people at an in-person conference, which you know.
• 3:26
  Speaker 1: Just as an aside, I'm really bullish on in-person conferences for associations.
• 3:31
  Speaker 1: I think it's one of the things associations do an incredible job with and no matter what happens with technology like biologically, we're wired to want community and connection and having that in person, you know, is something that's kind of roared back to life after COVID and I think is going to continue to be a strong suit.
• 3:47
  Speaker 1: That doesn't mean that associations don't have to adapt their conferences to take into account AI augmentation in various ways.
• 3:54
  Speaker 1: Of course they should and they will, but I'm really pumped about it.
• 3:57
  Speaker 1: I think conference season is just a fantastic time of the year.
• 4:02
  Speaker 2: I agree and I know this week, Amit, you're doing a lot of thinking and reflection about your own keynote.
• 4:07
  Speaker 2: Are there any details or sneak peeks you'd want to share with our listeners?
• 4:12
  Speaker 1: Well, I have to figure out what I'm talking about first.
• 4:13
  Speaker 1: You know it's you know as you know, because you're helping me put it together.
• 4:20
  Speaker 1: It's like I don't like to prepare a keynote too far in advance of when I'm going to be on stage somewhere, because it just ultimately things are changing so fast and some of the things we're going to be talking about next week during my keynote to open up the event are things that have happened in the last week or two.
• 4:37
  Speaker 1: So you know you can definitely count on it being focused on the latest and greatest in AI generally, but what I plan to do is talk specifically about agents.
• 4:46
  Speaker 1: So most of my keynote is about how exponential change is driving the rise of agents and how agents can dramatically affect customer service, both from the perspective of the association in terms of the workload, but also from the viewpoint of the member receiving a far superior experience to what they've ever envisioned possible.
• 5:07
  Speaker 1: So that's part of what I'm talking about and hopefully I'll be setting the stage for keynotes and other speakers to follow me that will be able to weave in other specialized topics, but I think that the world of agents is an area that we've just got to pay more attention to.
• 5:22
  Speaker 1: So much focus goes towards hey, there's a new model released and this model is really cool.
• 5:26
  Speaker 1: Like yesterday, cloud 3.5 Sonnet update came out, including computer use, and that's really cool.
• 5:33
  Speaker 1: But that's just a model, and so I say just a model.
• 5:37
  Speaker 1: It's incredibly powerful and very exciting, but with Agents you can build all this other software around these models and do things today, even with last year's models that are just stunning that people don't realize.
• 5:47
  Speaker 1: So we're trying to cast a bigger spotlight on agents and that's what my keynote aims to do.
• 5:52
  Speaker 1: So if you have not yet registered for Digital Now, you're a slacker, but you can still come, so sign up today.
• 6:02
  Speaker 2: You are a slacker, but you're not the only one.
• 6:02
  Speaker 2: We've seen, as we've mentioned, a lot of registrations come in in the past few weeks.
• 6:05
  Speaker 2: We still have some room.
• 6:07
  Speaker 2: I mean, we're not capping digital now this year.
• 6:09
  Speaker 2: So if that sounds of interest to you, please join us at the Omni Shore Room Hotel October 27th.
• 6:14
  Speaker 2: And, as a reminder, we will be recording the keynote sessions as well and putting those into our AI Learning Hub if you can't make it or are joining from a faraway place.
• 6:23
  Speaker 2: If you can't make it or are joining from a faraway place, today we are talking about two topics.
• 6:27
  Speaker 2: First and foremost, hey Jen's Instant Avatars.
• 6:34
  Speaker 2: I think that'll be a fun conversation.
• 6:35
  Speaker 2: And then we're also talking about exactly what Amit just mentioned Claude's computer use feature that they just rolled out.
• 6:38
  Speaker 2: So hey Jen Instant Avatars.
• 6:41
  Speaker 2: To remind you, haijin is an AI-powered video creation platform that specializes in generating realistic video avatars.
• 6:49
  Speaker 2: If you have joined us for our Intro to AI webinar, you'll know that I typically show a video from Haijin that actually Amit created, translating his video and his voice into different languages.
• 6:59
  Speaker 2: Their interactive avatars create AI-powered digital versions of individuals that are capable of joining Zoom meetings and other live interactions, mimicking the user's appearance, voice and decision-making style.
• 7:13
  Speaker 2: More on that later.
• 7:14
  Speaker 2: The avatars are designed to look and sound like the user, with the ability to think and respond in a way that reflects the user's personality and preferences.
• 7:23
  Speaker 2: Users can input specific information, like company data or brand guidelines, to ensure the avatar accurately represents them or their organization.
• 7:32
  Speaker 2: These avatars can be used for various applications like customer support, online coaching, sales calls, job interviews, language learning and even therapy sessions.
• 7:42
  Speaker 2: This comes from Hajin, just so you know.
• 7:44
  Speaker 2: The avatars integrate with Google Calendar for scheduling and can connect to large language models via API for enhanced interaction capabilities.
• 7:53
  Speaker 2: So I'll let you all in on a secret.
• 7:56
  Speaker 2: Our plan for today's episode was actually to have one of these interactive avatars join us on this Zoom call.
• 8:02
  Speaker 2: I tested it out and I'll let you all know.
• 8:05
  Speaker 2: We were not impressed enough to actually include that on the pod, so I didn't create an avatar of myself or Amit.
• 8:12
  Speaker 2: I thought that would be a little too weird.
• 8:13
  Speaker 2: I used one of their avatar templates and I trained it up on the Sidecar website and I provided it a prompt that I came up with using Claude.
• 8:22
  Speaker 2: Actually, that said, you're a seasoned technology strategist specializing in AI implementation for professional and trade associations, so on and so forth with different areas of expertise.
• 8:33
  Speaker 2: I then had the avatar join a Zoom call with me and then recorded and shared that with Amit, but overall not super impressed.
• 8:41
  Speaker 2: I was able to interact with the avatar and say, you know, welcome to the Sidecar Sync podcast, and it said, oh, I'm glad to be on this podcast.
• 8:48
  Speaker 2: And then it was able to talk about AI for associations, able to discuss some of the sidecar offerings around AI education.
• 8:56
  Speaker 2: I can surely say this is not going to replace me or any of you at this moment on a Zoom call, on a customer service call, on a sales call or a coaching call.
• 9:07
  Speaker 2: That's not to say that won't be the case in a few months, maybe not weeks, but maybe a few months.
• 9:13
  Speaker 2: So, amit, I'm curious, what do you have to say about Hajin's interactive avatars?
• 9:20
  Speaker 1: Well, my point of view is that the I mean what you shared the other day is that little video of you having a chat with the avatar through Zoom was really impressive, on the one hand because it was near real time video generation responding to you.
• 9:36
  Speaker 1: I thought the quality of the content in terms of what it was saying was reasonable.
• 9:41
  Speaker 1: They're obviously using some strong LLM underneath the hood to do that.
• 9:45
  Speaker 1: I thought the quality of the synthesized audio was pretty poor and it also is not really optimized for giving shorter and sweeter responses, as opposed to more of like a text-based response where they ask a question like hey, what's the best way for my association to start using AI?
• 10:01
  Speaker 1: And the avatar just kind of goes on and on and on.
• 10:05
  Speaker 1: So I think that that part of it was less conversational and more speechy and the quality of the audio wasn't as great as like what I've seen from 11 Labs or ChatGPT's Advanced Voice Mode, which is also available through their real-time API.
• 10:20
  Speaker 1: Now, that being said, hey, jen's video is head and shoulders above everybody else's in terms of real-time time avatar like stuff.
• 10:28
  Speaker 1: It's not trying to compete with video models like Runway or things like that or Sora.
• 10:33
  Speaker 1: It's not in that realm, even though it's video.
• 10:35
  Speaker 1: It's more about, like you know, business use cases where we're saying, hey, we're going to create an avatar or an avatar based on a video we provide.
• 10:43
  Speaker 1: Right, create an avatar or an avatar based on a video we provide.
• 10:45
  Speaker 1: So it's really really good at that the fact that it's real-time now versus previously.
• 10:52
  Speaker 1: If I wanted to get my AI avatar to say something, I'd give it a text script and then I'd wait somewhere between 10 to 30 minutes and then HeyGen would say, hey, it's ready.
• 10:58
  Speaker 1: And then you'd download it and you'd view it, and it was usually fine A little bit glitchy here and there, but it was usually fine.
• 11:04
  Speaker 1: Now, a year later, you know you can do near real time conversations.
• 11:08
  Speaker 1: That what it really tells you is the pace of progress is crazy.
• 11:11
  Speaker 1: To your earlier point, mallory, I anticipate in the next three to six months that the quality of all the stuff that I just criticized will be radically improved.
• 11:21
  Speaker 1: You know, when you have something like 11 Labs, for example, I'm a big fan.
• 11:25
  Speaker 1: I use their tools all the time when I want to either do text to speech generation or speech to speech dubbing, where, for example, I record something but, for whatever reason, I don't want my voice to be the voice in a presentation or a video or something like that.
• 11:39
  Speaker 1: Well, I can take just the audio clip, upload it to 11 Labs and get an awesome dubbing in a synthesized voice, and the same thing with text-to-speech.
• 11:48
  Speaker 1: It's really, really good.
• 11:49
  Speaker 1: The advantage, by the way, of doing audio-to-audio is the timing, the intonation, all that stuff is coming from the information in your voice and then it's just translated into somebody else's voice essentially, or an AI's voice, else's voice essentially, or an AI's voice, whereas with text to speech, you can't really give it that much information to generate the right timing and the right sequence and the right intonations and all that.
• 14:00
  Speaker 1: But in any event, the point about it is 11 Labs is a super specialized audio platform and it's awesome.
• 14:06
  Speaker 1: It's way better than HeyGen's audio.
• 14:08
  Speaker 1: But it's kind of like saying listen, do we have specialization, so much so that we need to use these different things and stitch them together?
• 14:17
  Speaker 1: I suspect that because everything is improving so fast, heygen, being a video platform, will get very good audio.
• 14:24
  Speaker 1: It's kind of like saying this like you know, if you were to say OK, well, boeing has an airplane, that's a jetliner that cruises at 550 miles per hour, a jetliner that cruises at 550 miles per hour, but Airbus has a hypersonic plane that can get you from Paris to New York City in two hours.
• 14:43
  Speaker 1: You know there's a marked difference there between the two.
• 14:45
  Speaker 1: It makes the Boeing jetliner seem like it's super antiquated, even though it's, you know, at the moment state of the art right.
• 14:52
  Speaker 1: But the point would be that, you know, similarly, the audio in HeyGen's product seems clunky compared to 11 Labs.
• 15:00
  Speaker 1: But you know, if all the airliners are kind of cruising at Mach 5 or whatever, then it's going to be very different and like, the incremental difference is so small you wouldn't notice.
• 15:09
  Speaker 1: So my point is, heygen's audio is going to get a hell of a lot better really fast.
• 15:13
  Speaker 1: I would say that for now I wouldn't probably invite a hey Jen avatar into my conversations people.
• 15:19
  Speaker 1: But that's exactly where this stuff is going to go Within the next few years.
• 15:23
  Speaker 1: Certainly you're going to have all sorts of AI to avatars joining meetings.
• 15:26
  Speaker 1: You might even have an avatar of yourself joining a meeting on your behalf, which is maybe scary, but that's all going to be happening.
• 15:36
  Speaker 2: Certainly exciting stuff on the Hagen front.
• 15:38
  Speaker 2: In terms of creating the avatar, it was a bit like creating a custom GPT up.
• 15:42
  Speaker 2: That said, if you assign your avatar a name so let's say Susan, for example, every time your avatar responds it might say my name is Susan and here's everything about AI for association.
• 16:11
  Speaker 2: So it just seemed a bit clunky all the way around.
• 16:14
  Speaker 2: Our avatar was named Wayne.
• 16:15
  Speaker 2: I did not change the name, so we didn't encounter that issue.
• 16:18
  Speaker 2: But across the board, I think the only thing that was surprising to me and I don't know what your thoughts are on this, amit were just how all of the marketing around it was so intense.
• 16:27
  Speaker 2: Like you're going to send this avatar to a Zoom meeting on your behalf.
• 16:30
  Speaker 2: It's going to make decisions like in your decision-making style.
• 16:33
  Speaker 2: I think we're a ways off from that.
• 16:42
  Speaker 1: Yeah, I agree, we're, you know, two or three kind of leaps away from that which you know might be two or three kind of AI cycles, so maybe 18 months, 24 months, something like that.
• 16:49
  Speaker 1: And then there's even if the technology was there, if you just kind of like, let's just say that we presuppose that in 24 months time we have an AI avatar version that is as good as you could possibly want, right In terms of quality and just accuracy of answers, all that stuff.
• 17:06
  Speaker 1: The question then is do you want that or not?
• 17:10
  Speaker 1: Do you actually want that?
• 17:11
  Speaker 1: Do you want to use it as an additional participant in the meeting?
• 17:14
  Speaker 1: Do you want it to attend a meeting on your behalf?
• 17:16
  Speaker 1: If you can't make it, I don't know.
• 17:18
  Speaker 1: I think those are really interesting questions to ask.
• 17:22
  Speaker 2: Well, that was actually my next question for you, amit was in this ideal world, latency is reduced.
• 17:26
  Speaker 2: Quality of the avatar shoots up.
• 17:28
  Speaker 2: Do you right in this moment, do you see that as a path that business leaders will go on?
• 17:33
  Speaker 2: It seems kind of hard to grasp that I would send an avatar to a meeting in my place.
• 17:38
  Speaker 1: I could see myself having an avatar for an AI assistant, where there's some kind of visual representation of my AI assistant that knows me really well, but it's clearly a different entity than me.
• 17:51
  Speaker 1: I don't personally think I'll ever feel comfortable with the idea of a version of me interacting with people, Maybe if it's known to be like, hey, this avatar can answer questions or whatever.
• 18:02
  Speaker 1: But it's clearly just an AI that's trained on content.
• 18:04
  Speaker 1: But you know, people very quickly get over, lose track of the fact that they're dealing with AI and you know if they see you and they hear you, they're going to quickly think it's you, and so I'm not comfortable with that.
• 18:16
  Speaker 1: No matter how good the AI is, I'd rather be in the meeting, and so I'm not comfortable with that.
• 18:20
  Speaker 1: No matter how good the AI is, I'd rather be in the meeting.
• 18:22
  Speaker 1: But I do think that what's going to happen is AI avatars of your assistants.
• 18:24
  Speaker 1: You'll be able to kind of shape and mold them to whatever you want.
• 18:27
  Speaker 1: They could be some alien looking creature or they could be a person or an animation or whatever, and you could include those in your meetings.
• 18:35
  Speaker 1: And one of the things that may have to happen is if you want to sell me something.
• 18:39
  Speaker 1: Instead of getting a hold of me, you'll always have to go through my gatekeeper, which is my AI assistant that knows me really well and the goal of that assistant isn't to block all potential opportunities, but to really intelligently filter things out and knows me well enough to know what I actually want.
• 18:54
  Speaker 1: To take a look at.
• 18:55
  Speaker 2: I agree with that.
• 18:58
  Speaker 2: I think if I ever were to meet with a Meath avatar at this point in time I wouldn't take anything it said seriously because I'd be like, well, what does a Meath really think about what a Meath avatar has to say?
• 19:09
  Speaker 2: So I agree with you.
• 19:10
  Speaker 2: Or perhaps an interesting use case in the future would be some sort of like an avatar that takes the counterpoint to whatever you're talking about in your meetings, just to help you kind of brainstorm and flesh out ideas.
• 19:22
  Speaker 2: That could be interesting.
• 19:23
  Speaker 1: Totally, and then an avatar for, like, a really great facilitator who can ensure that everyone in the conversation has had an active opportunity to participate, using things like personality profiles, for example, to employ techniques like saying, hey, the person that has, you know, usually it's the people that have kind of the you know, high extroverted kind of character traits that tend to be the ones that talk the most in a meeting.
• 19:46
  Speaker 1: That doesn't mean they have the best ideas, but a lot of times they just consume all the oxygen in the room just as soon as they walk in, and so potentially, there's help on the way for that.
• 19:55
  Speaker 1: If you have an AI avatar, that's job is to facilitate the meeting and try to pull ideas out of everyone.
• 20:01
  Speaker 1: That could be super powerful and valuable too, just to remind everyone that there's more people in the room that may, you know, jump into the speaking ring right away.
• 20:08
  Speaker 1: I think it's an interesting technology also in the world of associations, in the context of learning.
• 20:15
  Speaker 1: So you think about, like Sidecar's AI learning hub and think about all the different courses that are there, and then think about, like, if you had an avatar available throughout that experience, that you could just have a voice video to video, voice to voice conversation with about the content, about any topic, and it was an avatar of the instructor for that particular course.
• 20:34
  Speaker 1: That could be very interesting.
• 20:37
  Speaker 1: Now, it may be better to choose an avatar that's not the instructor, but some animated cartoony thing that's like hey, I'm the AI assistant to this instructor and I'm here, I know all the material, I'm here to help you, and that way it's clearly not that person.
• 20:49
  Speaker 1: That's probably more comfortable, but still very useful.
• 20:52
  Speaker 1: So you know, I think that's a way of adding a real time element to asynchronous learning.
• 20:57
  Speaker 1: That could be very powerful.
• 20:58
  Speaker 1: It could also be something where it's like hey, I'm having a hard time with this particular concept.
• 21:03
  Speaker 1: I don't really understand how vectors work.
• 21:05
  Speaker 1: Can you walk me through it?
• 21:06
  Speaker 1: And then that avatar is in the context of your learning experience, knows where you're at in your learning journey based on your LMS data, knows all the curriculum in the LMS and can give you really guided help.
• 21:18
  Speaker 1: Knows all the curriculum in the LMS and can give you really guided help.
• 21:19
  Speaker 1: And we talked about similar ideas in the past in the world of AI education or AI for education.
• 21:29
  Speaker 1: I should say where you think about what the Khan Academy has done with Conmigo.
• 21:31
  Speaker 1: It's actually somewhat similar.
• 21:32
  Speaker 1: That's just a text thing versus video to video.
• 21:36
  Speaker 2: Oh, that would be an excellent use case for us to roll out an avatar for each course.
• 21:40
  Speaker 2: I'm going to add that to the docket.
• 21:42
  Speaker 1: I bet most LMSs that are investing in R&D anyway which is not every LMS, but most LMSs that are looking ahead are going to add features like this.
• 21:50
  Speaker 2: For all of our listeners and viewers.
• 21:52
  Speaker 2: I would say at this point, unless you just want to try it out for fun, heygen Interactive Avatars is probably not something you're going to be using in your meetings regularly, but certainly something to keep an eye on in the next few months.
• 22:04
  Speaker 1: Yeah, I agree with that.
• 22:05
  Speaker 1: I would just say I do think checking out HeyGen and there's a few other tools like it, but HeyGen is the one that seems to be the market leader for video translation is definitely a use case.
• 22:14
  Speaker 1: I'd encourage people to go try.
• 22:15
  Speaker 1: If you haven't yet translated a video of yourself speaking in whatever your native language is into another language, go try that.
• 22:22
  Speaker 1: Just do a 30-second video on your phone, upload it to HeyGen, ask it to translate it into some other language and see what happens.
• 22:28
  Speaker 1: It should blow your mind.
• 22:29
  Speaker 1: If you haven't gone through that yourself, it just opens up your brain to the idea that these tools exist.
• 22:35
  Speaker 1: The AI avatar feature is their newest feature at HeyGen, and that one is what Mallory and I are saying is perhaps not quite ready for primetime from our viewpoint, but you know, if you go check it out and you have a different point of view, shoot us a note.
• 22:46
  Speaker 1: We'd love to talk to you about it.
• 22:48
  Speaker 2: Yep, we might bring you on a future pod episode.
• 22:50
  Speaker 2: Just a warning, all right.
• 22:53
  Speaker 2: Next up we're talking about cloud computer use.
• 22:56
  Speaker 2: Anthropic introduced a new feature called computer use for its AI model, cloud, which allows the AI to control a computer in a manner similar to human users.
• 23:05
  Speaker 2: This feature is still in the public beta phase and is available through Anthropic's API, and it enables cloud to perform various tasks by interacting with computer interfaces, such as moving a cursor, typing and executing commands based on visual inputs from the screen.
• 23:20
  Speaker 2: Claude can perceive the computer screen, identify elements and interact with them by moving the cursor and clicking based on pixel positions, which allows the model to use everyday software and tools without requiring task-specific programming.
• 23:34
  Speaker 2: The AI can perform complex tasks autonomously, like filling out forms by gathering data from spreadsheets and CRM systems, or navigating web browsers to complete coding tasks.
• 23:46
  Speaker 2: Despite its capabilities, claude's computer use feature is still experimental and can be error prone, and its ability to use computers is not yet on par with human proficiency.
• 23:56
  Speaker 2: On evaluations like Osworld, its scores a 14.9%, whereas typical human scores range from 70 to 75%, but it's nearly double the score of the next best AI model in the same category, which scored 7.7%.
• 24:11
  Speaker 2: So, amit, you shared this with me.
• 24:13
  Speaker 2: I think it just dropped yesterday, so this is like hot off the press for our association listeners.
• 24:18
  Speaker 2: What do you see as some near-term use cases for computer use?
• 24:24
  Speaker 1: Well, you're right that at the moment, these models aren't as good as a human in terms of their use of a computer, but they are really good in certain narrower contexts and they're also good at learning over time, as you tell them.
• 24:35
  Speaker 1: Well, no, not that I meant this.
• 24:36
  Speaker 1: One of the things I think that's going to be really powerful for this is improving quality assurance.
• 24:42
  Speaker 1: So think about your association's website and the number of times people tell you that there's a broken link on it or that a particular feature doesn't work.
• 24:51
  Speaker 1: You do QA.
• 24:52
  Speaker 1: Typically, when you have a new system that you're putting in, like a new e-commerce system or a new LMS, you'll do a bunch of QA on it.
• 24:58
  Speaker 1: And then hopefully, hopefully you're doing a bunch of QA on it before you go live with it.
• 25:02
  Speaker 1: And then, once you go live, that kind of goes away and you just start incrementally changing it, and so then you add this button and that button and this image and that image and new pages, and all this stuff is happening and over time it's like a death by a thousand paper cuts scenario and then your website no longer works well, it's slow.
• 25:20
  Speaker 1: So there have been automated quality assurance tools for years and there's companies and the whole industry around this.
• 25:26
  Speaker 1: But those tools have been the domain of QA engineers.
• 25:29
  Speaker 1: These are very technical tools.
• 25:30
  Speaker 1: They're super expensive Like a good QA engineering kind of tool set can cost into the many tens of thousands of dollars per person but they have been able to create automation scripts and this is for years now that are very, very valuable and that's why companies pay for those.
• 25:48
  Speaker 1: But with this kind of advancement in AI, both the capabilities are better, but it's also available to anyone.
• 25:53
  Speaker 1: So you could, mallory, go to Claude's new computer use and say hey, this is our Sidecar AI Learning Hub LMS site.
• 26:02
  Speaker 1: I want you to log in and I want you to test all the courses I want you to go through and make sure that the course links work, that the videos size correctly.
• 26:11
  Speaker 1: I want you to do this on desktop and I also want you to simulate mobile use using this tool and you can give Claude the tools Now.
• 26:18
  Speaker 1: If you do that right now, I suspect the the current version of cloud would probably not be able to do the whole thing, but very close, and it's going to keep improving at this pace where you know you could put in place that type of an assistant that just every day is running that qa on a continuous basis and sending you feedback saying hey, mallory, there's this broken link on this page.
• 26:39
  Speaker 1: So there's stuff like that.
• 26:42
  Speaker 1: The broader way to think about this is for a long time we have had to adapt to the interface of the technology.
• 26:50
  Speaker 1: So think about going way back in time and say well, originally, mainframe type computers, you actually didn't even interact with the keyboard.
• 26:59
  Speaker 1: You would load your program with punch cards and these were literally perforated.
• 27:05
  Speaker 1: You know thicker pieces of paper cards that you know that's how you would encode a program using special machines.
• 27:12
  Speaker 1: I never worked with punch cards I'm not that old but my dad told me a lot about it, and so he actually did that in his college work and loaded the punch cards for his PhD thesis and ran them on a computer.
• 27:21
  Speaker 1: And that is a great example of an incredibly inefficient way of communicating with a machine Right.
• 27:29
  Speaker 1: And then from there we went to you know little green screens and keyboards, and then from there we got this thing called a mouse, which was a pointing device where we could tell the computer what part of the screen we wanted to interact with, and graphical user interfaces blew up from there.
• 27:42
  Speaker 1: And then, with mobile, we had touch capabilities, multi-touch capabilities, and now we're getting into audio and video.
• 27:49
  Speaker 1: So our interfaces with technology keep getting better, and that's what made ChatGPT in 2022 such a revelatory experience for people, because for the first time for most people, they could interact with a computer in their choice of language, as opposed to having to use the computer's preferred way of interacting, whether that's punch cards, keyboards or, if you want to make it do something more complex, programming code.
• 28:12
  Speaker 1: Now we're shifting to the point where the computers can understand us, and that's kind of cool.
• 28:17
  Speaker 1: But the other thing that's interesting and this comes back to this topic is that, as part of understanding us and gaining these new capabilities, they understand our world too, including all of the devices we interact with, and the computer screen is one of the great examples is if the computer can see the computer, and then this AI can control the computer.
• 28:38
  Speaker 1: It has access to every piece of software that's ever been written.
• 28:41
  Speaker 1: It has access to everything right, and it already has the world's knowledge, and so it's a very powerful idea, because not all softwares are going to be available in a technical way, like via API, where the AI can interact with it, but there's lots of software that is used across so many businesses Like.
• 28:59
  Speaker 1: Imagine if your association had a legacy AMS and that AMS has been in place for God knows how long, but it basically works.
• 29:07
  Speaker 1: It might be really kludgy, but it works, and you don't want to spend millions of dollars in two years or longer to replace it.
• 29:13
  Speaker 1: But what if you could use AI, using computer use, to automate processes?
• 29:19
  Speaker 1: Right, and then you use the AI essentially as a way of extending the life of a legacy tool.
• 29:24
  Speaker 1: There's a lot of examples like that that come to mind.
• 29:27
  Speaker 1: The broader concept again, though, is computers using our interfaces to connect with and extend use cases.
• 29:33
  Speaker 1: It's another reason why humanoid robots are such a big thing.
• 29:36
  Speaker 1: The whole world.
• 29:37
  Speaker 1: We've crafted it in a way that basically interacts with us, so we have two hands and two feet and legs and all this other stuff.
• 29:45
  Speaker 1: So if we can make robotics that are similar to us, they just plug right in to everything else.
• 29:51
  Speaker 1: So that's the idea behind this in my mind is it's an accelerant.
• 29:56
  Speaker 1: It's basically opening up a whole bunch of new doors for the AI to walk through.
• 30:02
  Speaker 2: If you check out the demo video, which we will include some of those in the show notes, you can see that Claude is actually taking screenshots of your screen over and over again and that's how it's figuring out, kind of what to do next and what to click and how to fill out these forms, which seems a little bit clunky.
• 30:16
  Speaker 2: So to your point, amit, interfaces have gotten better for us throughout history, but then I'm wondering do we now go backward with AI or not, or backwards for us in the sense that it doesn't seem like an AI needs a computer right to interact with all these things.
• 30:32
  Speaker 2: You mentioned APIs being an option, but do you see, do you think that AI will eventually go away from using our systems and kind of create more of a direct path?
• 30:42
  Speaker 1: I think there's definitely a possibility.
• 30:44
  Speaker 1: I mean, AI systems left to their own devices will actually create their own languages, you know, because English is pretty inefficient.
• 30:49
  Speaker 1: But I don't know if we want that or not.
• 30:51
  Speaker 1: I think there may be value in always forcing the AIs to go down a path that we can interpret and understand Well an API calling an API.
• 31:03
  Speaker 1: For example, let's say I have a more modern AMS and I want to create a new member in the database.
• 31:08
  Speaker 1: If I have an API to do that, that's a structured way of connecting to the AMS programmatically.
• 31:13
  Speaker 1: I could connect that to Zapier, I could connect that to a CDP, I can connect that to whatever, and AI can, of course, connect to APIs as well.
• 31:22
  Speaker 1: That's going to be a higher resolution way of doing it, if you will or a higher probability of success kind of way of doing it, but there's a lot of things in the world that don't have those kinds of capabilities.
• 31:33
  Speaker 1: Or you have systems that may have APIs, but the API is very limited and the user interface can do a whole lot more.
• 31:39
  Speaker 1: So there's a ton of really interesting business use cases around this.
• 31:44
  Speaker 1: I also think that, independent of the computer doing the driving, think of it this way too.
• 31:49
  Speaker 1: If you enable a tool like this to watch your screen and watch you work while you're perhaps talking to it continuously speech-wise right, Forget about video, but just audio.
• 31:59
  Speaker 1: And I'm talking to Claude like I'm talking to you, I'm saying hey, Claude, listen, I'm about to work on this Excel spreadsheet.
• 32:05
  Speaker 1: I'm going to do this and I'm stuck here.
• 32:07
  Speaker 1: What do you think's wrong?
• 32:08
  Speaker 1: And then Claude says oh well, that cell it should be this, Let me change it for you.
• 32:11
  Speaker 1: And it changes, so you're kind of like working in parallel.
• 32:14
  Speaker 1: It's almost as if you imagine hey, I opened up a Google Sheet and you and I are on an really interesting.
• 32:19
  Speaker 1: There are limitations right now, partly the model's intelligence.
• 32:21
  Speaker 1: Part of it, though, is the latency, with the way it's doing what you described, taking this sequence of screenshots, which becomes kind of like a children's flipbook of what's happening.
• 32:37
  Speaker 1: Animation-wise, it's very low res, but that's because there's some major limitations with vision models right now.
• 32:42
  Speaker 1: All that's changing.
• 32:44
  Speaker 1: We're on this accelerating exponential curve, and so vision models today, the way you can efficiently interact with them is to pass them smaller images that they can process, make sense of and then drive the next thing that the model does.
• 32:59
  Speaker 1: Over a period of time, you're going to have a real-time, high-resolution video feed going into the model.
• 33:03
  Speaker 1: It's going to have complete access to that on a continuous basis and we're not going to even think about it, Just like we're using Zoom today and we have HD video in both directions and we're not thinking about that as a remarkable thing, even though it really is.
• 33:16
  Speaker 1: So we will soon have that as well, where the AI will be with us with continuous video.
• 33:21
  Speaker 1: It's like 30 frames per second or higher, as opposed to like one frame every three seconds, which I think is what this thing's doing right now.
• 33:28
  Speaker 2: And think about how powerful for onboarding new team members to just have the AI kind of follow you throughout your work for maybe a few weeks you would have the ultimate onboarding avatar, even potentially that you could use through Haygen.
• 33:40
  Speaker 1: Well, and now imagine a version of this type of technology that is your association's AI avatar.
• 33:47
  Speaker 1: That is perhaps the video part, perhaps not, but is capable of doing things like computer use in combination with knowing everything about your association.
• 33:58
  Speaker 1: So let's say, for example, you are a CPA society and your members are accountants, what if your organization had an AI assistant that was capable, if the individual authorized it, to be able to kind of be part of their computer session, and your AI agent, of course, knew everything that the CPA society has training on, and which is a vast amount of content, and so that becomes a more powerful agent to potentially help.
• 34:23
  Speaker 1: And so I think it's almost like you think of the AIs as different collaborators.
• 34:27
  Speaker 1: I might say, hey, I'm going to invite this CPA Society's AI agent to hang out with me right now because I'm working on this report and I want that level of expertise.
• 34:37
  Speaker 1: I just don't want the generic clod hanging out with me because that's not quite at that level.
• 34:41
  Speaker 1: Or maybe it's highly specialized software used for taxes or something.
• 34:45
  Speaker 1: I'm kind of making this up.
• 34:46
  Speaker 1: But every domain has uniquenesses to it and associations living in those veins where they have that deep domain expertise.
• 34:52
  Speaker 1: But if they can essentially activate that expertise through new modalities and new channels like this.
• 34:58
  Speaker 1: It gets more and more exciting, so I see this being super relevant to associations in terms of internal use, where you can think about things that you do repetitively or you should be doing more like the QA stuff I mentioned earlier and use this type of technology, play with it to try to automate some of that to save yourself time.
• 35:15
  Speaker 1: And then the other side of it is think about how this affects your members and think about ways you could potentially help those members use this technology.
• 35:23
  Speaker 1: In some cases, it might just be training them on it, providing them an ability to learn this stuff through focused training that's relevant to them in their context.
• 35:33
  Speaker 1: In some cases, it might be custom tools.
• 35:37
  Speaker 2: I think I'm in AI agent mode because of your keynote session, amit.
• 35:40
  Speaker 2: But I'm curious here we have a model FOD that can take action through a feature called computer use.
• 35:47
  Speaker 2: Are we talking about an agent here, or would you still call that something separate from an AI agent?
• 35:53
  Speaker 1: I think it's kind of semantics at some point where there are things that computer use 100% is agentic in the sense that it has access to tools, it's able to interact with your computer, it's able to kind of semi-autonomously do things with your computer.
• 36:08
  Speaker 1: So, but it's packaged within the model.
• 36:11
  Speaker 1: Now, is cloud 3.5 Sonnet capable of doing that as a raw LLM?
• 36:14
  Speaker 1: No, and LLM, lvm, you know, basically a model.
• 36:18
  Speaker 1: It's.
• 36:18
  Speaker 1: There's software engineering on top of that that's allowed cloud to see your screen, take the screenshots, pass them in.
• 36:24
  Speaker 1: So there's definitely an agentic layer, even though it's being kind of packaged into the cloud offering.
• 36:30
  Speaker 1: But the reason I say it's semantics and you're just kind of I don't know that we ultimately are going to care too much, is that these are just going to be called systems.
• 36:36
  Speaker 1: They're AI systems.
• 36:37
  Speaker 1: There are one or more models involved and kind of the agentic sauce that surrounds those models will be a lot of times the things that actually drive the real world value.
• 36:48
  Speaker 1: But I think the main differentiation between the two is that models are passive and waiting for you to tell them that you want some specific thing and that agents can be active and take action on your behalf either autonomously or semi-autonomously, and that's the definitional break I think that most people use.
• 37:08
  Speaker 1: It's certainly how I think about it, but I think ultimately it's not going to matter.
• 37:11
  Speaker 1: It's just going to be AI solutions or AI systems and you're going to have bits and pieces coming together.
• 37:15
  Speaker 1: It's just like you know.
• 37:17
  Speaker 1: Think about a system you're more familiar with, like a CRM software.
• 37:19
  Speaker 1: Within the CRM software, you have databases, you have web servers, you have user interfaces.
• 37:25
  Speaker 1: You have all these.
• 37:26
  Speaker 1: There's network elements, there's all these different components that you pull together.
• 37:32
  Speaker 2: The same thing is true in this world too.
• 37:33
  Speaker 2: That makes sense.
• 37:34
  Speaker 2: So essentially, as we've talked about with Microsoft Copilot, wave 2, with Claude's computer use, we really have the ability to create agents or agent-like systems at our fingertips.
• 37:46
  Speaker 2: So I think really the big question here is what are people going to do with it?
• 37:51
  Speaker 2: What are associations going to do with that?
• 37:52
  Speaker 1: But even our individual listeners, yeah, what I'd love to be able to do with the earlier example is say hey, Claude, hang out with me, we're going to do a QA testing session on this website.
• 38:02
  Speaker 1: And now we know, okay, the Sidecar AI Learning Hub is awesome, we tested it.
• 38:07
  Speaker 1: Now, hey, Claude, listen, what I want you to do now is to create a version of yourself that can rerun that exact set of tests as frequently as I'd like you to run it.
• 38:17
  Speaker 1: And then Claude says no problem, Claude probably has to go write some code to do that, which Claude can do.
• 38:22
  Speaker 1: Right, All these models can do that.
• 38:23
  Speaker 1: Well.
• 38:23
  Speaker 1: And then test it and say does that test replicate what our session had?
• 38:32
  Speaker 1: And if so, then it's like, yeah, it's ready to go.
• 38:33
  Speaker 1: And so we'll say that's QA Cloud.
• 38:33
  Speaker 1: And so then I'll say to QA Cloud hey, I want you to run QA Cloud every day and send me a result, a report result, and tell me if there's any issues that were found and classify them as high priority, medium priority, low priority.
• 38:44
  Speaker 1: And then, if you have these issues and you know what the problem is, recommend what the remediation is.
• 38:50
  Speaker 1: And then the next step beyond that is to say, well, in certain categories where it's like saying oh, there's a broken link.
• 38:55
  Speaker 1: Well, if we so, authorize Claude or mini Claude or QA Claude or what we want to call them, we can say, well, here's our admin access to this particular account.
• 39:05
  Speaker 1: You can go in there and change that if you find a broken link.
• 39:08
  Speaker 1: And of course every level of this, you have to trust the technology more and there's varying levels of comfort for that.
• 39:14
  Speaker 1: But you can envision a thing where it's kind of like self-healing, where it goes from QA to actually making the repair.
• 39:20
  Speaker 1: So maybe you still get notification that something was corrected but maybe you don't.
• 39:25
  Speaker 1: You know, modern computer systems constantly have all sorts of issues and there are many self-correcting mechanisms in places and databases on networks.
• 39:34
  Speaker 1: I mean, the entire design of the internet is a continuous self-healing, self-correcting architecture and so that's how a lot of modern systems work.
• 39:41
  Speaker 1: So I could see that being applied to association business processes.
• 39:46
  Speaker 2: I don't know if there are all that many people out there who like QA I know that I do not.
• 39:49
  Speaker 2: It's kind of tedious, but I'm just thinking how exciting it is to live in a time where all these things that are pain points, tasks we don't enjoy, doing tasks that are really tedious AI is going to be able to do for us, and thinking about all that time it's going to free up for us to do the things we enjoy and the things that are more human yeah, and another element of qa is get the ai to be like a simulated student to take your course end to end and actually consume the content, give you feedback, attempt to attempt to answer the questions.
• 40:18
  Speaker 1: you'd have to do a lot of work to make sure the ai doesn't use its prior training, uh, and knowledge, to answer questions in an exam, for example, but or, or what?
• 40:26
  Speaker 1: Do you cap it off and say, hey, you're capped off at a 12th grade education or whatever.
• 40:30
  Speaker 1: There's all sorts of things you could theorize around that, but there's just an enormous number of things you could do with this technology and I think the feedback opportunity is tremendous.
• 40:41
  Speaker 2: All right, everyone.
• 40:42
  Speaker 2: Thanks for tuning in to episode 53 of the Sidecar Sync podcast.
• 40:46
  Speaker 2: We will see you all next week.
• 40:48
  Speaker 2: Host Digital Now.
• 40:52
  Speaker 1: Thanks for tuning in to Sidecar Sync this week.
• 40:54
  Speaker 1: Looking to dive deeper?
• 40:55
  Speaker 1: Download your free copy of our new book Ascend Unlocking the Power of AI for Associations at ascendbookorg.
• 41:04
  Speaker 1: It's packed with insights to power your association's journey with AI.
• 41:08
  Speaker 1: And remember Sidecar is here with more resources, from webinars to boot camps, to help you stay ahead in the association world.
• 41:15
  Speaker 1: We'll catch you in the next episode.
• 41:17
  Speaker 1: Until then, keep learning, keep growing and keep disrupting.